AI-generated Feedback in Social Robotic Virtual Patients

Summary

The goal of this quasi-experimental educational study is to learn whether AI-generated post-consultation feedback in social robotic virtual patient interactions improves medical students' clinical performance in medical history-taking. The main question it aims to answer is:

Can AI-generated feedback integrated in an AI-enhanced social robotic virtual patient platform improve medical students' clinical performance in medical history taking?

Researchers will compare results from standardised examinations following the structure of an objective structured clinical examination (OSCE), of medical students performing virtual patient interactions with AI-generated post consultation feedback compared to medical students who have not received AI-generated feedback.

Participants will perform five virtual patient cases in rheumatology using an established virtual patient platform: the Social AI-enhanced Robotic Interface (SARI). After completion of each case, students participate in follow-up seminars with consultant rheumatologists to discuss the cases. After completion of all nine cases, students take part in a OSCE based examination to evaluate medical-history taking.

Conditions

• Virtual Patient
• Social Robots
• Artificial Intelligence (AI)
• Large Language Model
• Feedback
• Medical History Taking

Interventions

DEVICE

AI post consultation feedback

A feedback algorithm which follows a two-stage design was implemented to generate post consultation feedback using large language models (LLMs) from OpenAI. The first stage of the feedback algorithm is an assessment model that evaluates student-VP dialogues using a predefined rubric developed in collaboration with consultant rheumatologists. This assessment model was iteratievly refined and validated prior to the study. The second stage of the feedback algorithm involves generating a feedback output based on the stageone assessment Students received approximately one page of structured written feedback immediately after completing each VP encounter with SARI. This feedback focused on medical history-taking within the context of rheumatology and included constructive comments with examples covering general history-taking, specific symptom enquiries, and systematic assessment of the VPs.

Sponsors & Collaborators

• Region Stockholm

  collaborator OTHER_GOV

• Karolinska Institutet

  collaborator OTHER

• Ioannis Parodis

  lead OTHER

Principal Investigators

• Ioannis Parodis, MD, PhD · Karolinska Institutet

Study Design

Allocation

NON_RANDOMIZED

Purpose

OTHER

Masking

DOUBLE

Model

PARALLEL

Eligibility

Sex

ALL

Healthy Volunteers

No

Timeline & Regulatory

Start

2025-01-27

Primary Completion

2025-06-05

Completion

2025-06-05

Countries

• Sweden

Study Locations