An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis

<p>Abstract</p> <p>Background</p> <p>Four- or five-option multiple choice questions (MCQs) are the standard in health-science disciplines, both on certification-level examinations and on in-house developed tests. Previous research has shown, however, that few MCQs have three or four functioning distractors. The purpose of this study was to investigate non-functioning distractors in teacher-developed tests in one nursing program in an English-language university in Hong Kong.</p> <p>Methods</p> <p>Using item-analysis data, we assessed the proportion of non-functioning distractors on a sample of seven test papers administered to undergraduate nursing students. A total of 514 items were reviewed, including 2056 options (1542 distractors and 514 correct responses). Non-functioning options were defined as ones that were chosen by fewer than 5% of examinees and those with a positive option discrimination statistic.</p> <p>Results</p> <p>The proportion of items containing 0, 1, 2, and 3 functioning distractors was 12.3%, 34.8%, 39.1%, and 13.8% respectively. Overall, items contained an average of 1.54 (SD = 0.88) functioning distractors. Only 52.2% (n = 805) of all distractors were functioning effectively and 10.2% (n = 158) had a choice frequency of 0. Items with more functioning distractors were more difficult and more discriminating.</p> <p>Conclusion</p> <p>The low frequency of items with three functioning distractors in the four-option items in this study suggests that teachers have difficulty developing plausible distractors for most MCQs. Test items should consist of as many options as is feasible given the item content and the number of plausible distractors; in most cases this would be three. Item analysis results can be used to identify and remove non-functioning distractors from MCQs that have been used in previous tests.</p

The conceptualisation and measurement of DSM-5 Internet Gaming Disorder: the development of the IGD-20 Test

Background: Over the last decade, there has been growing concern about ‘gaming addiction’ and its widely documented detrimental impacts on a minority of individuals that play excessively. The latest (fifth) edition of the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders (DSM-5) included nine criteria for the potential diagnosis of Internet Gaming Disorder (IGD) and noted that it was a condition that warranted further empirical study. Aim: The main aim of this study was to develop a valid and reliable standardised psychometrically robust tool in addition to providing empirically supported cut-off points. Methods: A sample of 1003 gamers (85.2% males; mean age 26 years) from 57 different countries were recruited via online gaming forums. Validity was assessed by confirmatory factor analysis (CFA), criterion-related validity, and concurrent validity. Latent profile analysis was also carried to distinguish disordered gamers from non-disordered gamers. Sensitivity and specificity analyses were performed to determine an empirical cut-off for the test. Results: The CFA confirmed the viability of IGD-20 Test with a six-factor structure (salience, mood modification, tolerance, withdrawal, conflict and relapse) for the assessment of IGD according to the nine criteria from DSM-5. The IGD-20 Test proved to be valid and reliable. According to the latent profile analysis, 5.3% of the total participants were classed as disordered gamers. Additionally, an optimal empirical cut-off of 71 points (out of 100) seemed to be adequate according to the sensitivity and specificity analyses carried

Online Assessment of Applied Anatomy Knowledge: The Effect of Images on Medical Students' Performance

Anatomical examinations have been designed to assess topographical and/or applied knowledge of anatomy with or without the inclusion of visual resources such as cadaveric specimens or images, radiological images, and/or clinical photographs. Multimedia learning theories have advanced the understanding of how words and images are processed during learning. However, the evidence of the impact of including anatomical and radiological images within written assessments is sparse. This study investigates the impact of including images within clinically oriented single-best-answer questions on students' scores in a tailored online tool. Second-year medical students (n = 174) from six schools in the United Kingdom participated voluntarily in the examination, and 55 students provided free-text comments which were thematically analyzed. All questions were categorized as to whether their stimulus format was purely textual or included an associated image. The type (anatomical and radiological image) and deep structure of images (question referring to a bone or soft tissue on the image) were taken into consideration. Students scored significantly better on questions with images compared to questions without images (P

Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper

Background: Reliable and valid written tests of higher cognitive function are difficult to produce, particularly for the assessment of clinical problem solving. Modified Essay Questions (MEQs) are often used to assess these higher order abilities in preference to other forms of assessment, including multiple-choice questions (MCQs). MEQs often form a vital component of end-of-course assessments in higher education. It is not clear how effectively these questions assess higher order cognitive skills. This study was designed to assess the effectiveness of the MEQ to measure higher-order cognitive skills in an undergraduate institution. Methods: An analysis of multiple-choice questions and modified essay questions (MEQs) used for summative assessment in a clinical undergraduate curriculum was undertaken. A total of 50 MCQs and 139 stages of MEQs were examined, which came from three exams run over two years. The effectiveness of the questions was determined by two assessors and was defined by the questions ability to measure higher cognitive skills, as determined by a modification of Bloom's taxonomy, and its quality as determined by the presence of item writing flaws. Results: Over 50% of all of the MEQs tested factual recall. This was similar to the percentage of MCQs testing factual recall. The modified essay question failed in its role of consistently assessing higher cognitive skills whereas the MCQ frequently tested more than mere recall of knowledge. Conclusion: Construction of MEQs, which will assess higher order cognitive skills cannot be assumed to be a simple task. Well-constructed MCQs should be considered a satisfactory replacement for MEQs if the MEQs cannot be designed to adequately test higher order skills. Such MCQs are capable of withstanding the intellectual and statistical scrutiny imposed by a high stakes exit examination.Edward J Palmer, Peter G Devit

The development of a knowledge test of depression and its treatment for patients suffering from non-psychotic depression: a psychometric assessment

<p>Abstract</p> <p>Background</p> <p>To develop and psychometrically assess a multiple choice question (MCQ) instrument to test knowledge of depression and its treatments in patients suffering from depression.</p> <p>Methods</p> <p>A total of 63 depressed patients and twelve psychiatric experts participated. Based on empirical evidence from an extensive review, theoretical knowledge and in consultations with experts, 27-item MCQ knowledge of depression and its treatment test was constructed. Data collected from the psychiatry experts were used to assess evidence of content validity for the instrument.</p> <p>Results</p> <p>Cronbach's alpha of the instrument was 0.68, and there was an overall 87.8% agreement (items are highly relevant) between experts about the relevance of the MCQs to test patient knowledge on depression and its treatments. There was an overall satisfactory patients' performance on the MCQs with 78.7% correct answers. Results of an item analysis indicated that most items had adequate difficulties and discriminations.</p> <p>Conclusion</p> <p>There was adequate reliability and evidence for content and convergent validity for the instrument. Future research should employ a lager and more heterogeneous sample from both psychiatrist and community samples, than did the present study. Meanwhile, the present study has resulted in psychometrically tested instruments for measuring knowledge of depression and its treatment of depressed patients.</p

Direct and Constructivist Instructional Design: A Comparison of Efficiency Using Mental Workload and Task Performance

This paper investigates the efficiency of two instructional design conditions: a traditional design based on the direct instruction approach to learning and its extension with a collaborative activity based upon the community of inquiry approach to learning. This activity was built upon a set of textual trigger questions to elicit cognitive abilities and support knowledge formation. A total of 115 students participated in the experiments and a number of third-level computer science classes where divided in two groups. A control group of learners received the former instructional design while an experimental group also received the latter design. Subsequently, learners of each group individually answered a multiple-choice questionnaire, from which a performance measure was extracted for the evaluation of the acquired factual, conceptual and procedural knowledge. Two measures of mental workload were acquired through self-reporting questionnaires: one unidimensional and one multidimensional. These, in conjunction with the performance measure, contributed to the definition of a measure of efficiency. Evidence showed the positive impact of the added collaborative activity on efficiency

Simulated consultations: a sociolinguistic perspective

Background: Assessment of consulting skills using simulated patients is widespread in medical education. Most research into such assessment is sited in a statistical paradigm that focuses on psychometric properties or replicability of such tests. Equally important, but less researched, is the question of how far consultations with simulated patients reflect real clinical encounters – for which sociolinguistics, defined as the study of language in its socio-cultural context, provides a helpful analytic lens. Discussion: In this debate article, we draw on a detailed empirical study of assessed role-plays, involving sociolinguistic analysis of talk in OSCE interactions. We consider critically the evidence for the simulated consultation (a) as a proxy for the real; (b) as performance; (c) as a context for assessing talk; and (d) as potentially disadvantaging candidates trained overseas. Talk is always a performance in context, especially in professional situations (such as the consultation) and institutional ones (the assessment of professional skills and competence). Candidates who can handle the social and linguistic complexities of the artificial context of assessed role-plays score highly – yet what is being assessed is not real professional communication, but the ability to voice a credible appearance of such communication. Summary: Fidelity may not be the primary objective of simulation for medical training, where it enables the practising of skills. However the linguistic problems and differences that arise from interacting in artificial settings are of considerable importance in assessment, where we must be sure that the exam construct adequately embodies the skills expected for real-life practice. The reproducibility of assessed simulations should not be confused with their validity. Sociolinguistic analysis of simulations in various professional contexts has identified evidence for the gap between real interactions and assessed role-plays. The contextual conditions of the simulated consultation both expect and reward a particular interactional style. Whilst simulation undoubtedly has a place in formative learning for professional communication, the simulated consultation may distort assessment of professional communication These sociolinguistic findings contribute to the on-going critique of simulations in high-stakes assessments and indicate that further research, which steps outside psychometric approaches, is necessary