Impact of Flawed Multiple-Choice Questions on Tests and Student Achievements in Medical Education

Abstract

Objectives: The objectives were to determine the frequency of flawed items in MCQ based tests in Azad Jammu and Kashmir Medical College (AJK-MC) Muzaffarabad, to compare the Quality of examination before and after removing the flawed MCQ items from the tests, to compare student’s achievements on standard (without flawed items) and flawed tests and to find out the student group/ groups whom result is maximally affected by the presence of flawed MCQs items in examinations.

Methods: 10 summative examinations in AJK-MC Muzaffarabad were included in the study. The frequencies of flawed MCQ items with different types of item writing flaws in tests were identified. The student scores in each test were compared before and after removal of flawed items and its effects were evaluated in high, moderate and low achieving groups of students. The Mann‐Whitney U test was used for comparison of student’s scores on standard and flawed tests. The internal consistency reliability of tests was measured by Kuder-Richardson (KR- 20) formula. Chi Square and Fischer Exact tests were used to determine the association between different categorical variables in the study.

Results: There was high frequency of flawed items ranging from 7 to 52 percent in different tests. There were seven most common item writing flaws. More students failed in flawed as compared with standard tests in most of examinations.

Conclusion: Item writing flaws are common in in-house developed MCQ tests and there are unintended negative consequences of using flawed items on test scores and student achievements.