Assessing fitness-to-practice of overseas-trained health practitioners by Australian registration & accreditation bodies

Assessment of fitness-to-practice of health professionals trained overseas and who wish to practice in Australia is undertaken by a range of organisations. These organisations conduct assessments using a range of methods. However there is very little published about how these organisations conduct their assessments. The purpose of the current paper is to investigate the methods of assessment used by these organisations and the issues associated with conducting these assessments

Development of a portfolio of learning for postgraduate family medicine training in South Africa: a Delphi study

<p>Abstract</p> <p>Background</p> <p>Within the 52 health districts in South Africa, the family physician is seen as the clinical leader within a multi-professional district health team. Family physicians must be competent to meet 90% of the health needs of the communities in their districts. The eight university departments of Family Medicine have identified five unit standards, broken down into 85 training outcomes, for postgraduate training. The family medicine registrar must prove at the end of training that all the required training outcomes have been attained. District health managers must be assured that the family physician is competent to deliver the expected service. The Colleges of Medicine of South Africa (CMSA) require a portfolio to be submitted as part of the uniform assessment of all registrars applying to write the national fellowship examinations. This study aimed to achieve a consensus on the contents and principles of the first national portfolio for use in family medicine training in South Africa.</p> <p>Methods</p> <p>A workshop held at the WONCA Africa Regional Conference in 2009 explored the purpose and broad contents of the portfolio. The 85 training outcomes, ideas from the WONCA workshop, the literature, and existing portfolios in the various universities were used to develop a questionnaire that was tested for content validity by a panel of 31 experts in family medicine in South Africa, via the Delphi technique in four rounds. Eighty five content items (national learning outcomes) and 27 principles were tested. Consensus was defined as 70% agreement. For those items that the panel thought should be included, they were also asked how to provide evidence for the specific item in the portfolio, and how to assess that evidence.</p> <p>Results</p> <p>Consensus was reached on 61 of the 85 national learning outcomes. The panel recommended that 50 be assessed by the portfolio and 11 should not be. No consensus could be reached on the remaining 24 outcomes and these were also omitted from the portfolio. The panel recommended that various types of evidence be included in the portfolio. The panel supported 26 of the 27 principles, but could not reach consensus on whether the portfolio should reflect on the relationship between the supervisor and registrar.</p> <p>Conclusion</p> <p>A portfolio was developed and distributed to the eight departments of Family Medicine in South Africa, and the CMSA, to be further tested in implementation.</p

The reliability of in-training assessment when performance improvement is taken into account

During in-training assessment students are frequently assessed over a longer period of time and therefore it can be expected that their performance will improve. We studied whether there really is a measurable performance improvement when students are assessed over an extended period of time and how this improvement affects the reliability of the overall judgement. In-training assessment results were obtained from 104 students on rotation at our university hospital or at one of the six affiliated hospitals. Generalisability theory was used in combination with multilevel analysis to obtain reliability coefficients and to estimate the number of assessments needed for reliable overall judgement, both including and excluding performance improvement. Students’ clinical performance ratings improved significantly from a mean of 7.6 at the start to a mean of 7.8 at the end of their clerkship. When taking performance improvement into account, reliability coefficients were higher. The number of assessments needed to achieve a reliability of 0.80 or higher decreased from 17 to 11. Therefore, when studying reliability of in-training assessment, performance improvement should be considered

The national portfolio for postgraduate family medicine training in South Africa : a descriptive study of acceptability, educational impact, and usefulness for assessment

Background: Since 2007 a portfolio of learning has become a requirement for assessment of postgraduate family medicine training by the Colleges of Medicine of South Africa. A uniform portfolio of learning has been developed and content validity established among the eight postgraduate programmes. The aim of this study was to investigate the portfolio's acceptability, educational impact, and perceived usefulness for assessment of competence. Methods: Two structured questionnaires of 35 closed and open-ended questions were delivered to 53 family physician supervisors and 48 registrars who had used the portfolio. Categorical and nominal/ordinal data were analysed using simple descriptive statistics. The open-ended questions were analysed with ATLAS.ti software. Results: Half of registrars did not find the portfolio clear, practical or feasible. Workshops on portfolio use, learning, and supervision were supported, and brief dedicated time daily for reflection and writing. Most supervisors felt the portfolio reflected an accurate picture of learning, but just over half of registrars agreed. While the portfolio helped with reflection on learning, participants were less convinced about how it helped them plan further learning. Supervisors graded most rotations, suggesting understanding the summative aspect, while only 61% of registrars reflected on rotations, suggesting the formative aspects are not yet optimally utilised. Poor feedback, the need for protected academic time, and pressure of service delivery impacting negatively on learning. Conclusion: This first introduction of a national portfolio for postgraduate training in family medicine in South Africa faces challenges similar to those in other countries. Acceptability of the portfolio relates to a clear purpose and guide, flexible format with tools available in the workplace, and appreciating the changing educational environment from university-based to national assessments. The role of the supervisor in direct observations of the registrar and dedicated educational meetings, giving feedback and support, cannot be overemphasized

Context factors in general practitioner-patient encounters and their impact on assessing communication skills--an exploratory study

Contains fulltext : 118212.pdf (publisher's version ) (Open Access)BACKGROUND: Assessment of medical communication performance usually focuses on rating generically applicable, well-defined communication skills. However, in daily practice, communication is determined by (specific) context factors, such as acquaintance with the patient, or the presented problem. Merely valuing the presence of generic skills may not do justice to the doctor's proficiency.Our aim was to perform an exploratory study on how assessment of general practitioner (GP) communication performance changes if context factors are explicitly taken into account. METHODS: We used a mixed method design to explore how ratings would change. A random sample of 40 everyday GP consultations was used to see if previously identified context factors could be observed again. The sample was rated twice using a widely used assessment instrument (the MAAS-Global), first in the standard way and secondly after context factors were explicitly taken into account, by using a context-specific rating protocol to assess communication performance in the workplace. In between first and second rating, the presence of context factors was established. Item score differences were calculated using paired sample t-tests. RESULTS: In 38 out of 40 consultations, context factors prompted application of the context-specific rating protocol. Mean overall score on the 7-point MAAS-Global scale increased from 2.98 in standard to 3.66 in the context-specific rating (p<0.00); the effect size for the total mean score was 0.84. In earlier research the minimum standard score for adequate communication was set at 3.17. CONCLUSIONS: Applying the protocol, the mean overall score rose above the level set in an earlier study for the MAAS-Global scores to represent 'adequate GP communication behaviour'. Our findings indicate that incorporating context factors in communication assessment thus makes a meaningful difference and shows that context factors should be considered as 'signal' instead of 'noise' in GP communication assessment. Explicating context factors leads to a more deliberate and transparent rating of GP communication performance