Evaluating the Potential of Large Language Models for Respiratory Disease Consultations

Summary

The clinical trial aimes to evaluate multiple large language models in respiratory disease consultations by comparing their performance to that of human doctors across three major medical consultation scenarios.

The main question aims to answer are:

* How do large language models perform in comparison to human doctors in diagnosing and consulting on respiratory diseases across various clinical scenarios?

In three clinical scenarios including the online query section, the disease diagnosis section and the medical explanation section, research assistants or volunteers will be asked to cross-question all LLMs or real doctors using predefined online questions and their own issues. After each questioning session, a short washout period is implemented to eliminate potential biases.

Conditions

• Hay Fever
• Asthma
• Acute Upper Respiratory Infection
• Pulmonary Embolism
• Pneumonia
• Tuberculosis
• Acute Bronchitis
• Pulmonary Fibrosis
• Lung Cancer
• Bronchiectasis

Interventions

DIAGNOSTIC_TEST

Diagnosis by three human doctors

This intervention involves answering patient inquiries by different human doctors. Each patient is randomly assigned by the system to three doctors from different provinces in China selected from the database of doctors. The doctors all come from multiple online consultation platforms in China, and their diagnostic qualifications and medical licenses have undergone strict verification.

DIAGNOSTIC_TEST

Diagnosis by ChatGPT-3.5 (with search capabilities)

This intervention involves answering patient inquiries by ChatGPT-3.5 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by ChatGPT-3.5 (without search capabilities)

This intervention involves answering patient inquiries by ChatGPT-3.5 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by ChatGPT-4.0 (with search capabilities)

This intervention involves answering patient inquiries by ChatGPT-4.0 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by ChatGPT-4.0 (without search capabilities)

This intervention involves answering patient inquiries by ChatGPT-4.0 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by Claude instant (with search capabilities)

This intervention involves answering patient inquiries by Claude instant with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by Claude instant (without search capabilities)

This intervention involves answering patient inquiries by Claude instant without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by Claude 2 (with search capabilities)

This intervention involves answering patient inquiries by Claude 2 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by Claude 2 (without search capabilities)

This intervention involves answering patient inquiries by Claude 2 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by Gemini Pro (with search capabilities)

This intervention involves answering patient inquiries by Gemini Pro with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

DIAGNOSTIC_TEST

Diagnosis by Gemini Pro (without search capabilities)

This intervention involves answering patient inquiries by Gemini Pro without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

Sponsors & Collaborators

• University of Glasgow

  collaborator OTHER

• Nanchong Central Hospital

  collaborator OTHER_GOV

• Affiliated Hospital of North Sichuan Medical College

  collaborator OTHER

• North Sichuan Medical College

  lead OTHER

Principal Investigators

• Jiebin Xie, Doctor · North Sichuan Medical College

Study Design

Allocation

RANDOMIZED

Purpose

DIAGNOSTIC

Masking

QUADRUPLE

Model

CROSSOVER

Eligibility

Sex

ALL

Healthy Volunteers

No

Timeline & Regulatory

Start

2023-10-01

Primary Completion

2023-12-12

Completion

2024-10-12

Countries

• China

Study Locations