The posts for the articles in the July issue of Medical Education are now up and ready for your comments!
You can access the issue on the journal website by clicking here.
Posts for articles in July issue of Medical Education
Reply
Comparative effects of test-enhanced learning and self-explanation on long-term retention
Online abstract click here
Douglas P Larsen, Andrew C Butler and Henry L Roediger III
Context
Educators often encourage students to engage in active learning by generating explanations for the material being learned, a method called self-explanation. Studies have also demonstrated that repeated testing improves retention. However, no studies have directly compared the two learning methods.
Methods
Forty-seven Year 1 medical students completed the study. All students participated in a teaching session that covered four clinical topics and was followed by four weekly learning sessions. In the learning sessions, students were randomised to perform one of four learning activities for each topic: testing with self-generated explanations (TE); testing without explanations (T); studying a review sheet with self-generated explanations (SE), and studying a review sheet without explanations (S). Students repeated the same activity for each topic in all four sessions. Six months later, they took a free-recall clinical application test on all four topics.
Results
Repeated testing led to better long-term retention and application than repeatedly studying the material (p < 0.0001, η2 = 0.33). Repeated generation of self-explanations also improved long-term retention and application, but the effect was smaller (p < 0.0001, η2 = 0.08). When data were collapsed across topics, both testing conditions produced better final test performance than studying with self-explanation (TE = 40% > SE = 29% [p = 0.001, d = 0.70]; T = 36% > SE = 29% [p = 0.02, d = 0.48]). Studying with self-explanation led to better retention and application than studying without self-explanation (SE = 29% > S = 20%; p = 0.001, d = 0.68). Our analyses showed significant interaction by topic (p = 0.001, η2 = 0.06), indicating some variation in the effectiveness of the interventions among topics.
Conclusions
Testing and generating self-explanations are both learning activities that can be used to produce superior long-term retention and application of knowledge, but testing is generally more effective than self-explanation alone.
Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review
Online abstract click here
J Eric Jelovsek, Nathan Kow and Gouri B Diwadkar
Context
The Accreditation Council for Graduate Medical Education (ACGME) Milestone Project mandates programmes to assess the attainment of training outcomes, including the psychomotor (surgical or procedural) skills of medical trainees. The objectives of this study were to determine which tools exist to directly assess psychomotor skills in medical trainees on live patients and to identify the data indicating their psychometric and edumetric properties.
Methods
An electronic search was conducted for papers published from January 1948 to May 2011 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Science electronic databases and the review of references in article bibliographies. A study was included if it described a tool or instrument designed for the direct observation of psychomotor skills in patient care settings by supervisors. Studies were excluded if they referred to tools that assessed only clinical or non-technical skills, involved non-medical health professionals, or assessed skills performed on a simulator. Overall, 4114 citations were screened, 168 (4.1%) articles were reviewed for eligibility and 51 (1.2%) manuscripts were identified as meeting the study inclusion criteria. Three authors abstracted and reviewed studies using a standardised form for the presence of key psychometric and edumetric elements as per ACGME and American Psychological Association (APA) recommendations, and also assigned an overall grade based on the ACGME Committee on Educational Outcome Assessment grading system.
Results
A total of 30 tools were identified. Construct validity based on associations between scores and training level was identified in 24 tools, internal consistency in 14, test–retest reliability in five and inter-rater reliability in 20. The modification of attitudes, knowledge or skills was reported using five tools. The seven-item Global Rating Scale and the Procedure-Based Assessment received an overall Class 1 ACGME grade and are recommended based on Level A ACGME evidence.
Conclusions
Numerous tools are available for the assessment of psychomotor skills in medical trainees, but evidence supporting their psychometric and edumetric properties is limited.
Web-based feedback after summative assessment: how do students engage?
Online abstract click here
Christopher J Harrison, Karen D Könings, Adrian Molyneux, Lambert W T Schuwirth, Valerie Wass and Cees P M van der Vleuten
Context
There is little research into how to deliver summative assessment student feedback effectively. The main aims of this study were to clarify how students engage with feedback in this context and to explore the roles of learning-related characteristics and previous and current performance.
Methods
A website was developed to deliver feedback about the objective structural clinical examination (OSCE) in various formats: station by station or on skills across stations. In total, 138 students (in the third year out of five) completed a questionnaire about goal orientation, motivation, self-efficacy, control of learning beliefs and attitudes to feedback. Individual website usage was analysed over an 8-week period. Latent class analyses were used to identify profiles of students, based on their use of different aspects of the feedback website. Differences in learning-related student characteristics between profiles were assessed using analyses of variance (anovas). Individual website usage was related to OSCE performance.
Results
In total, 132 students (95.7%) viewed the website. The number of pages viewed ranged from two to 377 (median 102). Fifty per cent of students engaged comprehensively with the feedback, 27% used it in a minimal manner, whereas a further 23% used it in a more selective way. Students who were comprehensive users of the website scored higher on the value of feedback scale, whereas students who were minimal users scored higher on extrinsic motivation. Higher performing students viewed significantly more web pages showing comparisons with peers than weaker students did. Students who just passed the assessment made least use of the feedback.
Conclusions
Higher performing students appeared to use the feedback more for positive affirmation than for diagnostic information. Those arguably most in need engaged least. We need to construct feedback after summative assessment in a way that will more effectively engage those students who need the most help.
Evaluating the quality of medical multiple-choice items created with automated processes
Online abstract click here
Mark J Gierl and Hollis Lai
Objectives
Computerised assessment raises formidable challenges because it requires large numbers of test items. Automatic item generation (AIG) can help address this test development problem because it yields large numbers of new items both quickly and efficiently. To date, however, the quality of the items produced using a generative approach has not been evaluated. The purpose of this study was to determine whether automatic processes yield items that meet standards of quality that are appropriate for medical testing. Quality was evaluated firstly by subjecting items created using both AIG and traditional processes to rating by a four-member expert medical panel using indicators of multiple-choice item quality, and secondly by asking the panellists to identify which items were developed using AIG in a blind review.
Methods
Fifteen items from the domain of therapeutics were created in three different experimental test development conditions. The first 15 items were created by content specialists using traditional test development methods (Group 1 Traditional). The second 15 items were created by the same content specialists using AIG methods (Group 1 AIG). The third 15 items were created by a new group of content specialists using traditional methods (Group 2 Traditional). These 45 items were then evaluated for quality by a four-member panel of medical experts and were subsequently categorised as either Traditional or AIG items.
Results
Three outcomes were reported: (i) the items produced using traditional and AIG processes were comparable on seven of eight indicators of multiple-choice item quality; (ii) AIG items can be differentiated from Traditional items by the quality of their distractors, and (iii) the overall predictive accuracy of the four expert medical panellists was 42%.
Conclusions
Items generated by AIG methods are, for the most part, equivalent to traditionally developed items from the perspective of expert medical reviewers. While the AIG method produced comparatively fewer plausible distractors than the traditional method, medical experts cannot consistently distinguish AIG items from traditionally developed items in a blind review.
Predictive validity of the Dundee multiple mini-interview
Online abstract click here
Adrian Husbands and Jonathan Dowell
Context
The multiple mini-interview (MMI) is the primary admissions tool used to assess non-cognitive skills at Dundee Medical School. Although the MMI shows promise, more research is required to demonstrate its transferability and predictive validity, for instance, relative to other UK pre-admissions measures.
Methods
Applicants were selected for interview based on a combination of measures derived from the Universities and Colleges Admissions Service (UCAS) form (academic achievement, medical experience, non-academic achievement and references) and the UK Clinical Aptitude Test (UKCAT) in 2009 and 2010. Candidates were selected into medical school according to a weighted combination of the UKCAT, the UCAS form and MMI scores. Examination scores were matched for 140 and 128 first- and second-year students, respectively, who took the 2009 MMIs, and 150 first-year students who took the 2010 MMIs. Pearson’s correlations were used to test the relationships between pre-admission variables, examination scores and demographic variables, namely gender and age. Statistically significant correlations were adjusted for range restrictions and were used to select variables for multiple linear regression analysis to predict examination scores.
Results
Statistically significant correlations ranged from 0.18 to 0.34 and 0.23 to 0.50 unrestricted. Multiple regression confirmed that MMIs remained the most consistent predictor of medical school assessments. No scores derived from the UCAS form correlated significantly with examination scores.
Conclusions
This study reports positive findings from the largest undergraduate sample to date. The MMI was the most consistent predictor of success in early years at medical school across two separate cohorts. UKCAT and UCAS forms showed minimal or no predictive ability. Further research in this area appears worthwhile, with longitudinal studies, replication of results from other medical schools and more detailed analysis of knowledge, skills and attitudinal outcome markers.
Effects of morning report case presentation on length of stay and hospitalisation costs
Online abstract click here
Elham Boushehri, Mohammad E Khamseh, Amir Farshchi, Rokhsareh Aghili, Mojtaba Malek and Ameneh Ebrahim Valojerdi
Context
The primary goal of discussing patient cases during the morning report is to teach appropriate clinical decision making. In addition, the selection of the best diagnostic strategy and application of evidence-based patient care are important. Reducing hospital costs is fundamental to controlling inflation in health care costs, especially in university hospitals that are subject to budget constraints in developing countries. The goal of this study was to explore the effect of morning report case presentation on length of stay (LoS) and hospitalisation costs in a university teaching hospital.
Methods
A total of 54 patients whose cases had been presented at morning report sessions in the department of internal medicine during a 3-month period (presented group) were selected and their medical records reviewed for data on final diagnosis, hospital LoS and detailed hospital costs. A control group of 104 patients, whose cases had not been presented, were selected on the basis that their final diagnoses matched with those of the presented group. In addition, the groups were matched for age, sex, occupation, comorbidities and insurance coverage. Final diagnoses were based on International Classification of Disease 10 (ICD-10) diagnostic code criteria.
Results
The mean ± standard deviation (SD) hospital LoS was 8.32 ± 4.11 days in the presented group and 10.46 ± 4.92 days in the control group (p = 0.045). Mean ± SD hospitalisation costs per patient were significantly lower in the presented group (US$553.43 ± 92.16) than the control group (US$1621.93 ± 353.14) (p = 0.004). Although costs for paraclinical services were similar, there were very significant reductions in costs for medications used during hospitalisation and bed-days (p = 0.002).
Conclusions
Discussing clinical aspects of patient cases in morning report sessions facilitates the management process and has a significant effect on LoS and hospitalisation costs in patients admitted to hospital.
Effects of reviewing routine practices on learning outcomes in continuing education
Online abstract click here
Silvia Mamede, Sofie Loyens, Oscarina Ezequiel, Sandra Tibiriçá, Júlio Penaforte and Henk Schmidt
Context
Conventional continuing medical education (CME) has been shown to have modest effects on doctor performance. New educational approaches based on the review of routine practices have brought better results. Little is known about factors that affect the outcomes of these approaches, especially in middle-income countries. This study aimed to investigate factors that influence the learning and quality of clinical performance in CME based on reflection upon experiences.
Methods
A questionnaire and a clinical performance test were administered to 165 general practitioners engaged in a CME programme in Brazil. The questionnaire assessed behaviours related to four input variables (individual reflection on practices, peer review of experiences, self-regulated learning and learning skills) and two mediating variables (identification of learning needs and engagement in learning activities, the latter consisting of self-study of scientific literature, consultations about patient problems, and attendance at courses). Structural equation modelling was used to test a hypothesised model of relationships between these variables and the outcome variable of clinical performance, measured by the clinical performance test.
Results
After minor adjustments, the hypothesised model fit the empirical data well. Individual reflection fostered identification of learning needs, but also directly positively influenced the quality of clinical performance. Peer review did not affect identification of learning needs, but directly positively affected clinical performance. Learning skills and self-regulation did not help in identifying learning needs, but self-regulation enhanced study of the scientific literature, the learning activity that most positively influenced clinical performance. Consultation with colleagues, the activity most frequently triggered by the identification of learning needs, did not affect performance, and attendance of courses had only limited effect.
Conclusions
This study shed light on the factors that influence learning and performance improvement in continuing education based on the review of routine practices in middle-income settings. The findings support the importance of reflection on practices as an instrument for enhancing clinical performance.
Differential growth in doctor–patient communications skills
Online abstract click here
Judith M van Es, Margreet Wieringa-de Waard and Mechteld R M Visser
Context
Although doctor–patient communication is considered a core competency for medical doctors, the effect of training has not been unequivocally established. Moreover, knowledge about the variance in the growth of different skills and whether certain patterns in growth can be detected could help us to develop more efficient programmes. We therefore investigated the growth in general practitioner (GP) trainees’ doctor–patient communication skills in their first year and whether the growth was different for distinct categories of skills.
Methods
Seventy-one first-year GP trainees were invited to participate in a study aimed at measuring their consultation skills at the beginning (baseline) and at the end of their first year (follow-up). Consultation skills were assessed with the MAAS–Global rating list for consultation skills.
Results
Data on 29 general practitioner trainees were collected. MAAS–Global scores showed a significant growth on all items but one. Patient-oriented skills showed significantly more growth than task-oriented skills. Empathy as a separate skill seems to be mastered predominantly before the start of training.
Conclusions
Three patterns in the growth in skills were distinguished: (i) low baseline, relatively high follow-up, (ii) moderate baseline, moderate growth and (iii) high baseline, hardly any growth. Patient-oriented skills follow either pattern (i) or (iii), whereas task-oriented skills follow pattern (ii). These findings may help to define where the focus should lie in the training of doctor–patient communication skills.
How students deal with inconsistencies in health knowledge
Online abstract click here
Martina Bientzle, Ulrike Cress and Joachim Kimmerle
Objectives
In their work, health care professionals have to deal daily with inconsistent health information and are confronted with differing therapeutic health concepts. Medical education should prepare students to handle these challenges adequately. The aim of this study was to contribute to a better understanding of how students deal with inconsistencies in health knowledge when they are presented with either a therapeutic concept they accept or one they reject.
Methods
Seventy-six students of physiotherapy participated in this 2 × 2 experiment with health information (consistent versus inconsistent information) and therapeutic concept (congruent versus contradictory therapeutic concept) as between-group factors. The participants’ task was to improve the quality of a text about the effectiveness of stretching; participants were randomly assigned to one of four texts. Knowledge acquisition and text modification were measured as dependent variables.
Results
Students acquired more knowledge when they worked with a text containing inconsistent information. Medical information that was presented in agreement with a student’s therapeutic concept was also more readily acquired than the same information presented posing a contradictory therapeutic concept. Participants modified the contradictory text in order to adapt it to their own point of view. Disagreement resulted in a disregard or devaluation of the information itself, which in turn was detrimental to learning.
Conclusions
It is a problem when prospective health care professionals turn a blind eye to discrepancies that do not fit their view of the world. It may be useful for educational purposes to include a knowledge conflict caused by a combination of conviction and inconsistent information to facilitate learning processes.