Rubrics to Prompts: Assessing Medical Student Post-Encounter Notes with AI

Here at UT Southwestern, we’re changing the game with a revolutionary automated AI-based grading system for medical student learning, thanks to Dr. Jamieson & team, in collaboration with the UTSW Simulation Center.

Med students undergo Objective Structured Clinical Examinations (OSCE) which comprises simulated patient encounters with human actors & each student generates post-encounter notes that require accurate assessment by specialized evaluators. The challenge for assessors is the huge time and labor commitment and returning results in a timely manner.

Taking station-specific rubrics and post-encounter learner notes, the team built an AI grading system which reduced human effort by ~90% & returned assessments in days compared to the weeks it took for manual grading. The performance was rigorously evaluated retrospectively by validating against a large, real dataset with no prior exposure to any foundation model pretraining. The LLM is also adaptable to changing assessment criteria, scalable, & needs no additional training.

While the inaugural model works off of student learning notes, the team is already exploring inclusion of AI-based audiovisual analysis of student behavior during the OSCE & developing prototype multimodal AI systems. Such technology has far-reaching implications for broad use in the medical education community, poses economical benefit, enriches the learning experience, and demonstrates potential for being adapted to clinical practice.

Read the scientific article here: Rubrics to Prompts: Assessing Medical Student Post-Encounter Notes with AI

Andrew Jamieson, PhD - Faculty Profile

Jamieson Group