Research and Application of an AI Agent-Based MBBS Course Assistant

Summary

Prior to the quasi-experiment, this study performed semi-structured interviews among Bachelor of Medicine and Bachelor of Surgery(MBBS) undergraduate students from the Belt and Road International Medical College, Zhejiang University. Based on learning science, cognitive load theory and causal inference frameworks, the interview outline focused on system usability, feedback quality, learning reasoning processes, cognitive burden and instructional optimization advice. Each 20-30 minute individual interview was audio-recorded with participants' informed consent and verbatim transcribed for standardized qualitative analysis.

The quasi-experiment recruited no fewer than 60 MBBS students. The sample size was determined by intergroup statistical power analysis to guarantee around 30 participants per group for valid intergroup comparison. All eligible students were randomly assigned into two groups with balanced demographic and academic baseline characteristics. The experimental group (n=30) received structured epidemiological learning assisted by the (Socratic Agent for Guided Epidemiology) SAGE (Artificial Intelligence)AI agent with professional Socratic cognitive guidance. The control group (n=30) adopted general large language model (LLM)-based learning without systematic thinking intervention. Participants with clinically diagnosed severe mental disorders, cognitive dysfunction or inability to finish the complete research process were excluded. A baseline epidemiological knowledge pre-test confirmed no significant academic differences between the two groups via independent samples t-test, and all participants had no prior experience of AI-assisted medical learning.

The semi-structured interviews were conducted to collect students' authentic learning experiences, interactive perceptions and cognitive characteristics during the use of SAGE agent and general LLMs, providing qualitative evidence for the iterative optimization of AI teaching tools. All interviewees completed baseline assessments and preliminary AI learning trials, ensuring qualified professional foundation and genuine interactive experience. Conducted by trained researchers, the standardized interviews centered on three core themes: system usability and feedback clarity; AI-induced changes in information extraction, hypothesis formulation and causal inference; and common learning barriers including interactive obstacles, comprehension difficulties, cognitive overload and potential AI over-reliance. Transcribed interview data were analyzed through thematic analysis to summarize typical user experience patterns. Qualitative outcomes were triangulated with quantitative experimental results to revise the SAGE teaching protocol, optimize agent prompt chains and improve the interpretation of experimental findings.

The quasi-experiment consisted of three standardized stages. In the pre-test stage, all participants signed informed consent, completed a 25-item clinical epidemiology knowledge scale, an 11-item reasoning ability test and a demographic questionnaire to establish consistent baseline levels. In the intervention stage, the experimental group received standardized training in confounder identification and causal inference construction in strict accordance with the SAGE teaching protocol. The SAGE agent improved students' advanced epidemiological reasoning ability through continuous multi-round Socratic questioning and targeted cognitive guidance. The control group received equal-duration learning in the same experimental environment, only using conventional search engines and unguided LLMs for basic information retrieval without any cognitive and thinking intervention. All participants submitted screenshots to record their accurate AI tool usage duration after completing learning tasks.

In the post-test stage, all participants finished parallel-version epidemiological knowledge assessments and unified reasoning ability tests. Validated scales were adopted to evaluate students' cognitive load, system usability, learning satisfaction and academic self-confidence. Students' final scores of the Epidemiology course were collected as supplementary indicators of long-term learning effectiveness. Upon the completion of data collection, backend AI interaction logs were summarized and strictly screened. Invalid samples with insufficient interaction rounds or incomplete responses were excluded to ensure high data quality and reliable experimental conclusions.

Conditions

• an Intelligent Teaching Agent for the "Epidemiology" Course of the MBBS Program
• Students' Critical Thinking, Practical Ability and Learning Outcomes

Interventions

BEHAVIORAL

Socratic Agent for Guided Epidemiology

A total of 30 participants are assigned to the experimental group receiving the SAGE teaching model, namely the Socratic Agent for Guided Epidemiology.

BEHAVIORAL

General LLM

30 participants are allocated to the control group with teaching assistance from a general large language model (General LLM).

Sponsors & Collaborators

• The Fourth Affiliated Hospital of Zhejiang University School of Medicine

  lead OTHER

Study Design

Allocation

RANDOMIZED

Purpose

BASIC_SCIENCE

Masking

SINGLE

Model

PARALLEL

Eligibility

Min Age

16 Years

Max Age

25 Years

Sex

ALL

Healthy Volunteers

Yes

Timeline & Regulatory

Start

2026-05-15

Primary Completion

2027-01-31

Completion

2027-01-31

Countries

• China

Study Locations