An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis

<p>Abstract</p> <p>Background</p> <p>Four- or five-option multiple choice questions (MCQs) are the standard in health-science disciplines, both on certification-level examinations and on in-house developed tests. Previous research has shown, however, that few MCQs have three or four functioning distractors. The purpose of this study was to investigate non-functioning distractors in teacher-developed tests in one nursing program in an English-language university in Hong Kong.</p> <p>Methods</p> <p>Using item-analysis data, we assessed the proportion of non-functioning distractors on a sample of seven test papers administered to undergraduate nursing students. A total of 514 items were reviewed, including 2056 options (1542 distractors and 514 correct responses). Non-functioning options were defined as ones that were chosen by fewer than 5% of examinees and those with a positive option discrimination statistic.</p> <p>Results</p> <p>The proportion of items containing 0, 1, 2, and 3 functioning distractors was 12.3%, 34.8%, 39.1%, and 13.8% respectively. Overall, items contained an average of 1.54 (SD = 0.88) functioning distractors. Only 52.2% (n = 805) of all distractors were functioning effectively and 10.2% (n = 158) had a choice frequency of 0. Items with more functioning distractors were more difficult and more discriminating.</p> <p>Conclusion</p> <p>The low frequency of items with three functioning distractors in the four-option items in this study suggests that teachers have difficulty developing plausible distractors for most MCQs. Test items should consist of as many options as is feasible given the item content and the number of plausible distractors; in most cases this would be three. Item analysis results can be used to identify and remove non-functioning distractors from MCQs that have been used in previous tests.</p

Portuguese validation of the Internet Addiction Test: An empirical study

Abstract Background and aims Research into Internet addiction (IA) has increased greatly over the last decade. Despite its various definitions and general lack of consensus regarding its conceptualisation amongst researchers, instruments for measuring this phenomenon have proliferated in a number of countries. There has been little research on IA in Portugal and this may be partly due to the absence of standardised measurement tools for assessing IA. Methods This study attempted to address this issue by adapting a Portuguese version of the Internet Addiction Test (IAT) via a translation-back translation process and Confirmatory Factor Analysis in a sample of 593 Portuguese students that completed a Portuguese version of the IAT along with questions related to socio-demographic variables. Results The findings suggested that the IAT appears to be a valid and reliable instrument for measuring IA among Portuguese young adults as demonstrated by its satisfactory psychometric properties. However, the present findings also suggest the need to reword and update some of the IAT's items. Prevalence of IA found in the sample was 1.2% and is discussed alongside findings relating to socio-demographic correlates. Limitations and implications of the present study are also discussed. Conclusions The present study calls for a reflection of the IAT while also contributing to a better understanding of the basic aspects of IA in the Portuguese community since many health practitioners are starting to realise that Internet use may pose a risk for some individuals

Online Assessment of Applied Anatomy Knowledge: The Effect of Images on Medical Students' Performance

Anatomical examinations have been designed to assess topographical and/or applied knowledge of anatomy with or without the inclusion of visual resources such as cadaveric specimens or images, radiological images, and/or clinical photographs. Multimedia learning theories have advanced the understanding of how words and images are processed during learning. However, the evidence of the impact of including anatomical and radiological images within written assessments is sparse. This study investigates the impact of including images within clinically oriented single-best-answer questions on students' scores in a tailored online tool. Second-year medical students (n = 174) from six schools in the United Kingdom participated voluntarily in the examination, and 55 students provided free-text comments which were thematically analyzed. All questions were categorized as to whether their stimulus format was purely textual or included an associated image. The type (anatomical and radiological image) and deep structure of images (question referring to a bone or soft tissue on the image) were taken into consideration. Students scored significantly better on questions with images compared to questions without images (P

Direct and Constructivist Instructional Design: A Comparison of Efficiency Using Mental Workload and Task Performance

This paper investigates the efficiency of two instructional design conditions: a traditional design based on the direct instruction approach to learning and its extension with a collaborative activity based upon the community of inquiry approach to learning. This activity was built upon a set of textual trigger questions to elicit cognitive abilities and support knowledge formation. A total of 115 students participated in the experiments and a number of third-level computer science classes where divided in two groups. A control group of learners received the former instructional design while an experimental group also received the latter design. Subsequently, learners of each group individually answered a multiple-choice questionnaire, from which a performance measure was extracted for the evaluation of the acquired factual, conceptual and procedural knowledge. Two measures of mental workload were acquired through self-reporting questionnaires: one unidimensional and one multidimensional. These, in conjunction with the performance measure, contributed to the definition of a measure of efficiency. Evidence showed the positive impact of the added collaborative activity on efficiency

Evidence-based practice in neonatal health: knowledge among primary health care staff in northern Viet Nam

<p>Abstract</p> <p>Background</p> <p>An estimated four million deaths occur each year among children in the neonatal period. Current evidence-based interventions could prevent a large proportion of these deaths. However, health care workers involved in neonatal care need to have knowledge regarding such practices before being able to put them into action.</p> <p>The aim of this survey was to assess the knowledge of primary health care practitioners regarding basic, evidence-based procedures in neonatal care in a Vietnamese province. A further aim was to investigate whether differences in level of knowledge were linked to certain characteristics of community health centres, such as access to national guidelines in reproductive health care, number of assisted deliveries and geographical location.</p> <p>Methods</p> <p>This cross-sectional survey was completed within a baseline study preparing for an intervention study on knowledge translation (Implementing knowledge into practice for improved neonatal survival: a community-based trial in Quang Ninh province, Viet Nam, the NeoKIP project, ISRCTN44599712). Sixteen multiple-choice questions from five basic areas of evidence-based practice in neonatal care were distributed to 155 community health centres in 12 districts in a Vietnamese province, reaching 412 primary health care workers.</p> <p>Results</p> <p>All health care workers approached for the survey responded. Overall, they achieved 60% of the maximum score of the questionnaire. Staff level of knowledge on evidence-based practice was linked to the geographical location of the CHC, but not to access to the national guidelines or the number of deliveries at the community level. Two separated geographical areas were identified with differences in staff level of knowledge and concurrent differences in neonatal survival, antenatal care and postnatal home visits.</p> <p>Conclusion</p> <p>We have identified a complex pattern of associations between knowledge, geography, demographic factors and neonatal outcomes. Primary health care staff knowledge regarding neonatal health is scarce. This is a factor that is possible to influence and should be considered in future efforts for improving the neonatal health situation in Viet Nam.</p

Simulated consultations: a sociolinguistic perspective

Background: Assessment of consulting skills using simulated patients is widespread in medical education. Most research into such assessment is sited in a statistical paradigm that focuses on psychometric properties or replicability of such tests. Equally important, but less researched, is the question of how far consultations with simulated patients reflect real clinical encounters – for which sociolinguistics, defined as the study of language in its socio-cultural context, provides a helpful analytic lens. Discussion: In this debate article, we draw on a detailed empirical study of assessed role-plays, involving sociolinguistic analysis of talk in OSCE interactions. We consider critically the evidence for the simulated consultation (a) as a proxy for the real; (b) as performance; (c) as a context for assessing talk; and (d) as potentially disadvantaging candidates trained overseas. Talk is always a performance in context, especially in professional situations (such as the consultation) and institutional ones (the assessment of professional skills and competence). Candidates who can handle the social and linguistic complexities of the artificial context of assessed role-plays score highly – yet what is being assessed is not real professional communication, but the ability to voice a credible appearance of such communication. Summary: Fidelity may not be the primary objective of simulation for medical training, where it enables the practising of skills. However the linguistic problems and differences that arise from interacting in artificial settings are of considerable importance in assessment, where we must be sure that the exam construct adequately embodies the skills expected for real-life practice. The reproducibility of assessed simulations should not be confused with their validity. Sociolinguistic analysis of simulations in various professional contexts has identified evidence for the gap between real interactions and assessed role-plays. The contextual conditions of the simulated consultation both expect and reward a particular interactional style. Whilst simulation undoubtedly has a place in formative learning for professional communication, the simulated consultation may distort assessment of professional communication These sociolinguistic findings contribute to the on-going critique of simulations in high-stakes assessments and indicate that further research, which steps outside psychometric approaches, is necessary

Distractor Functioning in Modified Items for Test Accessibility

Effective distractors in multiple-choice items should attract lower ability students, those with misconceptions or limited knowledge and skills, because they are based on common misconceptions or errors in logic. A large, multi-state data set collected for a quasi-experimental study on test modifications was analyzed to measure the impact on distractor functioning. The key modification of interest was the removal of the weakest of three distractors, from 39 items in reading and 39 items in mathematics. Distractor functioning was neither systematically improved nor systematically weakened through the modification process. However, more than 70% of the distractors became more discriminating. A moderate correlation between distractor selection rate and distractor discrimination, in mathematics, may have indicated that the modified items were being missed by the appropriate students. Implications of these findings for test developers are discussed