Script concordance test: an approach to the evaluation of clinical reasoning in uncertain contexts

Little research has been done in Brazilian medical education on the evaluation of clinical reasoning in situations of uncertainty. The most common tests are still multiple-choice, which are capable of evaluating skills when dealing with well-defined problems. However, in practice the majority of situations involve uncertainties. A method for the evaluation of clinical reasoning in contexts of uncertainty was developed on the basis of the cognitive script theory in relation to professional reasoning. The objectives of the research were to develop, apply, and analyze this methodology in a Brazilian educational setting, based on clinical situations in Geriatrics that involved diagnostic, therapeutic, or ethical dilemmas. A group of specialists in this area and a group of undergraduate students that were completing their training in the Geriatrics internship took the test. Comparison of the results led to evidence of the instrument's validity, capable of distinguishing clinical reasoning according to the participants' level of experience. The mean score for the specialists (80,41) was higher than that of students (70,71) (p < 0,001). In addition, analyses of the internal consistency and a G study design furnished results that are consistent with a scoring system that seeks to evaluate a professional skill. In conclusion, a proposal for a script concordance test in the Portuguese language, applied in a Brazilian teaching institution, may be a viable alternative for evaluating clinical reasoning in contexts of uncertainty.A avaliação do raciocínio clínico em situações de incerteza é pouco pesquisada na educação médica. Os testes escritos mais aplicados são de múltipla escolha, capazes de avaliar como se lida com problemas bem definidos. Porém, a maioria das situações contém incertezas. Um método de avaliação do raciocínio clínico em contextos de incerteza foi desenvolvido a partir da teoria de scripts, com situações em geriatria. Um grupo de especialistas e um grupo de estudantes de graduação resolveram o teste. Acomparação entre os resultados trouxe indícios da validade do instrumento, capaz de diferenciar o raciocínio relacionado ao nível de experiência profissional. A média dos escores dos especialistas (80,41) foi superior à dos estudantes (70,71), p < 0,001. As análises de consistência interna e um estudo G forneceram resultados que estão de acordo com metodologias que buscam avaliar uma competência profissional. Concluiu-se que uma proposta de teste de concordância de scripts em língua portuguesa aplicado em uma instituição de ensino brasileira pode ser uma alternativa para a avaliação do raciocínio clínico em contextos de incerteza.Universidade Federal de São Paulo (UNIFESP)UNIFESP, EPM, São Paulo, BrazilSciEL

Health-related quality of life after treatment for bladder cancer in England

Background Little is known about quality of life after bladder cancer treatment. This common cancer is managed using treatments that can affect urinary, sexual and bowel function. Methods To understand quality of life and inform future care, the Department of Health (England) surveyed adults surviving bladder cancer 1–5 years after diagnosis. Questions related to disease status, co-existing conditions, generic health (EQ-5D), cancer-generic (Social Difficulties Inventory) and cancer-specific outcomes (Functional Assessment of Cancer Therapy—Bladder). Results In total, 673 (54%) patients responded; including 500 (74%) men and 539 (80%) with co-existing conditions. Most respondents received endoscopic treatment (60%), while 92 (14%) and 99 (15%) received radical cystectomy or radiotherapy, respectively. Questionnaire completion rates varied (51–97%). Treatment groups reported ≥1 problem using EQ-5D generic domains (59–74%). Usual activities was the most common concern. Urinary frequency was common after endoscopy (34–37%) and radiotherapy (44–50%). Certain populations were more likely to report generic, cancer-generic and cancer-specific problems; notably those with co-existing long-term conditions and those treated with radiotherapy. Conclusion The study demonstrates the importance of assessing patient-reported outcomes in this population. There is a need for larger, more in-depth studies to fully understand the challenges patients with bladder cancer face

The past, present and future of Scientific discourse.

Published versio

Using Differential Item Functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment

BACKGROUND: Fairness is a critical component of defensible assessment. Candidates should perform according to ability without influence from background characteristics such as ethnicity or sex. However, performance differs by candidate background in many assessment environments. Many potential causes of such differences exist, and examinations must be routinely analysed to ensure they do not present inappropriate progression barriers for any candidate group. By analysing the individual questions of an examination through techniques such as Differential Item Functioning (DIF), we can test whether a subset of unfair questions explains group-level differences. Such items can then be revised or removed. METHODS: We used DIF to investigate fairness for 13,694 candidates sitting a major international summative postgraduate examination in internal medicine. We compared (a) ethnically white UK graduates against ethnically non-white UK graduates and (b) male UK graduates against female UK graduates. DIF was used to test 2773 questions across 14 sittings. RESULTS: Across 2773 questions eight (0.29%) showed notable DIF after correcting for multiple comparisons: seven medium effects and one large effect. Blinded analysis of these questions by a panel of clinician assessors identified no plausible explanations for the differences. These questions were removed from the question bank and we present them here to share knowledge of questions with DIF. These questions did not significantly impact the overall performance of the cohort. Group-level differences in performance between the groups we studied in this examination cannot be explained by a subset of unfair questions. CONCLUSIONS: DIF helps explore fairness in assessment at the question level. This is especially important in high-stakes assessment where a small number of unfair questions may adversely impact the passing rates of some groups. However, very few questions exhibited notable DIF so differences in passing rates for the groups we studied cannot be explained by unfairness at the question level

Am I getting an accurate picture: a tool to assess clinical handover in remote settings?

BACKGROUND: Good clinical handover is critical to safe medical care. Little research has investigated handover in rural settings. In a remote setting where nurses and medical students give telephone handover to an aeromedical retrieval service, we developed a tool by which the receiving clinician might assess the handover; and investigated factors impacting on the reliability and validity of that assessment. METHODS: Researchers consulted with clinicians to develop an assessment tool, based on the ISBAR handover framework, combining validity evidence and the existing literature. The tool was applied 'live' by receiving clinicians and from recorded handovers by academic assessors. The tool's performance was analysed using generalisability theory. Receiving clinicians and assessors provided feedback. RESULTS: Reliability for assessing a call was good (G = 0.73 with 4 assessments). The scale had a single factor structure with good internal consistency (Cronbach's alpha = 0.8). The group mean for the global score for nurses and students was 2.30 (SD 0.85) out of a maximum 3.0, with no difference between these sub-groups. CONCLUSIONS: We have developed and evaluated a tool to assess high-stakes handover in a remote setting. It showed good reliability and was easy for working clinicians to use. Further investigation and use is warranted beyond this setting

Direct observation of clinical skills feedback scale:development and validity evidence

Construct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace.Background: Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use.Approach: Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS. Four rater types (residents n = 21, medical students n = 8, faculty n = 12, and educators n = 12) used the DOCS-FBS to rate videotaped feedback encounters of variable quality. The psychometric properties of the scale were determined using a generalizability analysis. Participants also completed a survey to gather data on a 5-point Likert scale to inform the ease of use, clarity, knowledge acquisition, and acceptability of the scale.Results: Mean video ratings ranged from 1.38 to 2.96 out of 3 and followed the intended pattern suggesting that the tool allowed raters to distinguish between examples of higher and lower quality feedback. There were no significant differences between rater type (range = 2.36-2.49), suggesting that all groups of raters used the tool in the same way. The generalizability coefficients for the scale ranged from 0.97 to 0.99. Item-total correlations were all above 0.80, suggesting some redundancy in items. Participants found the scale easy to use (M = 4.31/5) and clear (M = 4.23/5), and most would recommend its use (M = 4.15/5). Use of DOCS-FBS was acceptable to both trainees (M = 4.34/5) and supervisors (M = 4.22/5).Conclusions: The DOCS-FBS can reliably differentiate between feedback encounters of higher and lower quality. The scale has been shown to have excellent internal consistency. We foresee the DOCS-FBS being used as a means to provide objective evidence that faculty development efforts aimed at improving feedback skills can yield results through formal assessment of feedback quality.</p

Scalable production of large quantities of defect-free few-layer graphene by shear exfoliation in liquids

To progress from the laboratory to commercial applications, it will be necessary to develop industrially scalable methods to produce large quantities of defect-free graphene. Here we show that high-shear mixing of graphite in suitable stabilizing liquids results in large-scale exfoliation to give dispersions of graphene nanosheets. X-ray photoelectron spectroscopy and Raman spectroscopy show the exfoliated flakes to be unoxidized and free of basal-plane defects. We have developed a simple model that shows exfoliation to occur once the local shear rate exceeds 10(4) s(-1). By fully characterizing the scaling behaviour of the graphene production rate, we show that exfoliation can be achieved in liquid volumes from hundreds of millilitres up to hundreds of litres and beyond. The graphene produced by this method performs well in applications from composites to conductive coatings. This method can be applied to exfoliate BN, MoS2 and a range of other layered crystals

The impact of large scale licensing examinations in highly developed countries: a systematic review

BACKGROUND: To investigate the existing evidence base for the validity of large-scale licensing examinations including their impact. METHODS: Systematic review against a validity framework exploring: Embase (Ovid Medline); Medline (EBSCO); PubMed; Wiley Online; ScienceDirect; and PsychINFO from 2005 to April 2015. All papers were included when they discussed national or large regional (State level) examinations for clinical professionals, linked to examinations in early careers or near the point of graduation, and where success was required to subsequently be able to practice. Using a standardized data extraction form, two independent reviewers extracted study characteristics, with the rest of the team resolving any disagreement. A validity framework was used as developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education to evaluate each paper’s evidence to support or refute the validity of national licensing examinations. RESULTS: 24 published articles provided evidence of validity across the five domains of the validity framework. Most papers (n = 22) provided evidence of national licensing examinations relationships to other variables and their consequential validity. Overall there was evidence that those who do well on earlier or on subsequent examinations also do well on national testing. There is a correlation between NLE performance and some patient outcomes and rates of complaints, but no causal evidence has been established. CONCLUSIONS: The debate around licensure examinations is strong on opinion but weak on validity evidence. This is especially true of the wider claims that licensure examinations improve patient safety and practitioner competence

The incidence of unpleasant dreams after sub-anaesthetic ketamine

Ketamine is an N-methyl-D-aspartate (NMDA)receptor antagonist with psychotogenic effects and for whichthere are diverse reports of whether pleasant or unpleasantdreams result during anaesthesia, post-operatively or aftersub-anaesthetic use. The aim was to assess in healthy volunteers the incidence ofunpleasant dreams over the three nights after receiving asub-anaesthetic dose of ketamine, in comparison to placebo,and with retrospective home nightmare frequency as acovariate.Thirty healthy volunteers completed questionnairesabout retrospective home dream recall and were then giveneither ketamine or placebo. Ketamine resulted in significantly more meandream unpleasantness relative to placebo and caused athreefold increase in the odds ratio for the incidence of anunpleasant dream. The number of dreams reported over thethree nights did not differ between the groups. Theincidence of unpleasant dreams after ketamine use waspredicted by retrospectively assessed nightmare frequencyat home.Ketamine causes unpleasant dreams over thethree post-administration nights. This may be evidence of aresidual psychotogenic effect that is not found on standardself-report symptomatology measures or a result of disturbedsleep electrophysiology. The results have theoretical implications for the relationship between nightmares and schizotypy

Lack of privileged access to awareness for rewarding social scenes in Autism Spectrum Disorder

Reduced social motivation is hypothesised to underlie social behavioural symptoms of Autism Spectrum Disorder (ASD). The extent to which rewarding social stimuli are granted privileged access to awareness in ASD is currently unknown. We use continuous flash suppression to investigate whether individuals with and without ASD show privileged access to awareness for social over nonsocial rewarding scenes that are closely matched for stimulus features. Strong evidence for a privileged access to awareness for rewarding social over nonsocial scenes was observed in neurotypical adults. No such privileged access was seen in ASD individuals, and moderate support for the null model was noted. These results suggest that the purported deficits in social motivation in ASD may extend to early processing mechanisms