Quantifying the Effects of Guessing, Position Bias and Prior Knowledge in Multiple Choice Exams

Abstract

“Can an examinee pass a multiple-choice (MC) exam by chance?” Many studies have tried to address this question. However, these studies ignore different types of position bias associated with picking an item from a list. Despite the presence of considerable evidence on the existence of position bias in guessing answers in MC exams, these studies assume an examinee chooses answers with equal probability. This paper seeks to fill this gap by quantifying the chance of success in MC exams due to guessing when different types of position bias and prior knowledge are taken into consideration. The paper proposes a probabilistic model for position bias and uses it to conduct a series of computer simulations for quantifying the chance of passing an MC exam. Results show that the chance of passing an MC exam by pure random guessing is generally negligible even for a poorly set MC exam. Furthermore, results show that chances for an examinee with a fair amount of prior knowledge passing an MC exam of acceptable standard are between very high and perfect. Since a typical examinee is expected to possess some amount of prior knowledge, these results imply that despite their popularity, MC exams should be avoided particularly in high-stake exams where they can lead to false positives.

Keywords: Multiple-choice exams; multiple-choice questions; position bias; computer simulation.