#ClinTeach Conversation Kickstarter | Standard setting in the age of CBME

This month’s Clinical Teacher’s Toolbox is by Dr. Richard Hays of Tasmania, Australia. Dr. Hays’ summary of standard setting procedures is most definitely a worthy addition to most graduate studies courses on assessment, and yet, as an educator whom is working on competency-based medical education (CBME) projects, this article made me think:

What and how are we setting standards in the age of CBME?

As the world moves towards outcome-based medical education, this is a critical question to consider. Traditional psychometric methods (including those mentioned in Dr. Hays’ paper: Ebel, Angoff, Hofstee) require population-level data to ‘set standards’. And in the K-12, university-level and possibly undergraduate medical education realms where we have traditionally used these techniques for setting standards, these have been helpful in determining passing grades.

That said, these methods primarily are based on some key assumptions – most specifically on the idea that a post-hoc analysis of data on a given population (or a aggregate analysis of historical data of similar populations) might be used to set a standard. And perhaps, if there is no clear outcome, this the best way to do this… But in the age of outcomes, where do these standard setting procedures now belong?

The achievement of the outcome IS the standard, no?

Please share with me your thoughts below! I have pondered this question quite extensively and I’m not sure what the answer is…

Conversation Kickstarter #MedEdJ: Can spaced testing help students manage their study time and improve their marks?

Contributed by e-Council member Karen Scott, PhD

Can spaced testing help students manage their study time and improve their marks?

Kerdijk and colleagues have compared the effects of cumulative and end-of-term assessment on medical students’ ability to manage their self-study time, as well as their assessment results.1 The authors found using spaced testing throughout the course helped students devote regular time to self-study, avoid procrastination and increase their overall study time. In contrast, students preparing for end-of-term assessment tended to cram in the final two weeks of the course.

When comparing student marks between the two assessment formats, the only difference was found in marks for content studied in the final two weeks of the course, with higher marks gained by students sitting the spaced tests. The authors speculated that these students had more cognitive resources available for learning in the final two weeks because they were not cramming for the final exam.

What do you think are the pros and cons of cumulative assessment compared with end-of-term assessment?

Do you think long-term retention of knowledge could be improved through cumulative rather than end-of-term assessment?

In what other ways could teachers help students manage their study time throughout a course?

 

  1. Kerdijk W, Cohen-Schotanus J, Mulder FBF, Muntinghe FLH and Tio RA. Cumulative versus end-of-course assessment: effects on self-study time and test performance. Medical Education 2015: 49: 709–716 doi:1111/medu.12756

Link to article click here

Link to related blog post click here

 

Presence with purpose: attitudes of people with developmental disability towards health care students

By Ginette Moores, Natalie Lidster, Kerry Boyd, Tom Archer, Nick Kates and Karl Stobbe

Click here to link to article.

Context

Early clinical encounters help medical and nursing students build professional competencies. However, there is a necessary emphasis on patient autonomy and appropriate consent. Although most individuals do not object to student involvement in clinical encounters, there are occasions when personal preference and health care education conflict. Many studies have evaluated patient attitudes towards students across a variety of specialties.

Objectives

The purpose of this study was to identify the attitudes, comfort level and preferences of individuals with developmental disability (DD) towards the presence and involvement of medical and nursing students during clinical encounters.

Methods

Adults with DD across the Hamilton–Niagara region were invited to participate. Focus groups were moderated by two students with a health care facilitator and physician-educator. Participants were provided with focus group questions in advance and encouraged to bring communication aids or care providers. Data were analysed for emerging themes by two independent reviewers, who then compared results.

Results

Twenty-two individuals participated. A wide range of opinions were expressed. Some participants were positively disposed towards students and perceived better care and improved communication with the health care team. Others were indifferent to students in a clinical setting. The final group was opposed to the presence of health care students, expressing confusion over their role and purpose, uneasiness with deviation from the norm, and concerns about confidentiality. Informative introductions with confidentiality statements and the presence of a supervising clinician were seen as helpful.

Conclusions

People with DD are affected by above-average health care needs. Their input into health care planning has been limited. Their opinions on health care learners varied considerably. Themes relating to attitudes, comfort and preferences about student involvement provide impetus for health care training practices that promote person-centred approaches and improvements to the quality of care received by people with DD.

DOI: 10.1111/medu.12751

Cumulative versus end-of-course assessment: effects on self-study time and test performance

By Wouter Kerdijk, Janke Cohen-Schotanus, B Florentine Mulder, Friso L H Muntinghe and René A Tio

Link to this article here.

Context

Students tend to postpone preparation for a test until the test is imminent, which raises various risks associated with ‘cramming’ behaviours, including that for suboptimal learning. Cumulative assessment utilises spaced testing to stimulate students to study more frequently and to prevent procrastination. This randomised controlled study investigated how cumulative assessment affects time spent on self-study and test performance compared with end-of-course assessment.

Methods

A total of 78 undergraduate medical students in a Year 2 pre-clinical course were randomly assigned to either of two conditions. Students in the cumulative assessment condition were assessed in weeks 4, 8 and 10. Students in the end-of-course assessment condition were assessed in week 10 only. Each week, students reported the number of hours they spent on self-study.

Results

Students in the cumulative assessment condition (n = 25) spent significantly more time on self-study than students in the end-of-course assessment condition (n = 37) in all weeks of the course except weeks 5, 9 and 10. Overall, the cumulative assessment group spent 69 hours more on self-study during the course than did the control group, although the control group spent 7 hours more in studying during the final week of the course than did the cumulative assessment group. Students in the cumulative assessment condition scored slightly higher on questions concerning the content of the last part of the course.

Conclusions

Cumulative assessment encourages students to distribute their learning activities over a course, which leaves them more opportunity to study the content of the last part of the course prior to the final examination. There was no evidence for a short-term effect of cumulative assessment on overall knowledge gain. We hypothesise that larger positive effects might be found if retention were to be measured in the long term.

DOI: 10.1111/medu.12756

How faculty members experience workplace-based assessment rater training: a qualitative study

By Jennifer R Kogan, Lisa N Conforti, Elizabeth Bernabeo, William Iobst and Eric Holmboe

Click here to link to article.

Context

Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members’ reactions to rater training.

Objectives

This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches – performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) – to elucidate how such faculty development can be optimally designed.

Methods

In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data.

Results

Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one’s approach to assessment, and (iv) concerns about institutional culture and buy-in.

Conclusions

Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

DOI: 10.1111/medu.12733

Teaching during consultation: factors affecting the resident–fellow teaching interaction

By Eli M Miloslavsky, Jakob I McSparron, Jeremy B Richards, Alberto Puig and Amy M Sullivan

Click to link to article here.

Objectives

The subspecialty consultation represents a potentially powerful opportunity for resident learning, but barriers may limit the educational exchanges between fellows (subspecialty registrars) and residents (house officers). We conducted a focus group study of internal medicine (IM) residents and subspecialty fellows to determine barriers against and factors facilitating resident–fellow teaching interactions on the wards, and to identify opportunities for maximising teaching and learning.

Methods

We conducted four focus groups of IM residents (n = 18) and IM subspecialty fellows (n = 16) at two academic medical centres in the USA during February and March 2013. Participants represented trainees in all 3 years of residency training and seven IM subspecialties. Four investigators analysed the transcripts using a structured qualitative framework approach, which was informed by literature on consultation and the theoretical framework of activity theory.

Results

We identified two domains of barriers and facilitating factors: personal and systems-based. Sub-themes in the personal domain included fellows’ perceived resistance to consultations, residents’ willingness to engage in teaching interactions, and perceptions and expectations. Sub-themes in the systems-based domain included the process of requesting the consult, the quality of the consult request, primary team structure, familiarity between residents and fellows, workload, work experience, culture of subspecialty divisions, and fellows’ teaching skills. These barriers differentially affected the two stages of the consult identified in the focus groups (initial interaction and follow-up interaction).

Conclusions

Residents and fellows want to engage in positive teaching interactions in the context of the clinical consult; however, multiple barriers influence both parties in the hospital environment. Many of these barriers are amenable to change. Interventions aimed at reducing barriers to teaching in the setting of consultation hold promise for improving teaching and learning on the wards.

DOI: 10.1111/medu.12760

Illness and risk behaviour in health care students studying abroad

by Martin Angelin, Birgitta Evengård and Helena Palmgren

Click to link to article here.

Context

The numbers of university students studying abroad increase every year. These students are not tourists as their studies require different types of travel that expose them to different risks. Moreover, health care students (HCSs) may be exposed to even greater risks according to their travel destinations and itineraries. Clearly, research-based pre-travel advice is needed.

Methods

This study reports on a prospective survey conducted from April 2010 to January 2014 of health care and non-health care students from Swedish universities in Umeå, Stockholm and Gothenburg studying abroad.

Results

Of the 393 students included in the study, 85% responded. Over half (55%) were HCSs. Pre-travel health information was received by 79% and information on personal safety by 49% of HCSs. The rate of illness during travel was 52%. Health care students more often travelled to developing regions and were at increased risk for travellers’ diarrhoea. One in 10 experienced theft and 3% were involved in traffic accidents. One in five met a new sexual partner during travel and 65% of these practised safe sex. Half of all participants increased their alcohol consumption while abroad; high alcohol consumption was associated with increased risk for being a victim of theft, as well as for meeting a new sexual partner during travel.

Conclusions

University authorities are responsible for the safety and well-being of students studying abroad. This study supplies organisers and students with epidemiological data that will help improve pre-travel preparation and increase student awareness of the potential risks associated with studying abroad.

DOI: 10.1111/medu.12753

Factors associated with adverse clinical outcomes among obstetrics trainees

by Catherine E Aiken, Abigail R Aiken, Hannah Park, Jeremy C Brockelsby and Andrew Prentice

Click here to link to article.

Objectives

This study was conducted to determine whether UK obstetrics trainees transitioning from directly to indirectly supervised practice have a higher likelihood of recording adverse patient outcomes in operative deliveries compared with other indirectly supervised trainees, and to examine whether performing more procedures under direct supervision is associated with fewer adverse outcomes in initial practice under indirect supervision.

Methods

We examined all deliveries (13 856) conducted by obstetricians at a single centre over 6 years (2008–2013). Mixed-effects logistic regression models were used to compare estimated blood loss (EBL), maternal trauma, umbilical arterial pH, delayed neonatal respiration, failed instrumental delivery, and critical incidents for trainees in their first indirectly supervised year with those for trainees in all other years of indirect supervision. Outcomes for trainees in their first indirectly supervised 3 months were compared with their outcomes for the remainder of the year. Linear regression was used to examine the relationship between number of procedures performed under direct supervision and initial outcomes under indirect supervision.

Results

Trainees in their first indirectly supervised year had a higher likelihood of recording EBL of > 2 L at any delivery (odds ratio [OR] 1.32, 95% confidence interval [CI] 1.01–1.64; p < 0.05) and of failed instrumental delivery (OR 2.33, 95% CI 1.37–3.29; p < 0.05) compared with other indirectly supervised trainees. Other measured outcomes showed no significant differences. In the first 3 months of indirect supervision, the likelihood of operative vaginal deliveries with EBL of > 1 L (OR 2.54, 95% CI 1.88–3.20; p < 0.05) was higher than in the remainder of the first year. Performing more deliveries under direc

New Vodcast available from The Clinical Teacher (#ClinTeachJ)

Watch this new video from Helen Morgan et al
The flipped classroom for medical students

This curricular innovation project implemented a flipped classroom curriculum for the gynaecologic oncology topics of the obstetrics and gynaecology medical student clerkship, and to evaluate student satisfaction with the change. Read the accompanying article to this new vodcast, and leave your comments!

Costs and inconsistencies in US IRB review of low-risk medical education research

By Miria Kano, Christina M Getrich, Crystal Romney, Andrew L Sussman and Robert L Williams

Link to article here.

Context

Advances in communication technologies over the last two decades have transformed the way medical education research is conducted, creating opportunities for multi-institution national and international studies. Although these studies enable researchers to gain broader understandings of educational processes across institutions, they increase the need for multiple institutional review board (IRB) reviews to ensure the protection of human subjects.

Objectives

This study describes the process of obtaining multiple IRB approvals of the same protocol for a multi-site, low-risk, medical education research project in the USA. The burden of obtaining those reviews and their consistency are assessed. The associated time and costs, and implications for the research process are detailed.

Methods

Following review by the investigators’ parent institution IRB, the project team circulated a uniform protocol for conduct of a low-risk, medical education survey to the IRBs of 89 US medical schools for review. The processes and time required to obtain approvals were recorded to estimate associated research team personnel costs.

Results

Approval could not be obtained from five IRBs as a result of insurmountable procedural barriers. A total of 67 IRBs eventually deferred to the parent IRB determination. The remaining IRBs required a variety of additional procedural processes before ultimately agreeing with the original determination. The personnel costs associated with obtaining the 84 approvals amounted to US$121 344.

Conclusions

Considering the value of multi-site designs to address a range of research questions, enhance participant diversity and develop representative findings, solutions must be found to counter inefficiencies of current IRB review processes for low-risk research, such as that usually conducted in medical education. Although we acknowledge that local review is an essential protective measure for research involving identifiable communities that are uniquely susceptible to social or economic harm, this report suggests that proposals to modernise and streamline IRB review processes for low-risk research are timely and relevant.

DOI: 10.1111/medu.12693