GPT-4 in Psychosis Psychoeducation: Strong Scores, Lower Inclusivity

Key Takeaways

• In a first-episode psychosis program, GPT-4 responses to common psychoeducation questions from patients, caregivers, and relatives were generally rated strongly across the main quality domains.
• Inclusivity was the relative weak point, suggesting less consistent attention to cultural, social, and contextual perspectives than to accuracy, clarity, and clinical relevance.
• Readability was relatively advanced, which may reduce accessibility for some patients and caregivers seeking psychosis information.
• Exploratory analyses linked clearer GPT-4 responses with greater completeness and higher clinical utility.
• The authors concluded that GPT-4 may have a limited adjunctive, clinician-guided role in psychosis psychoeducation rather than serving as a substitute for individualized care.

First-episode psychosis care often depends on repeated psychoeducation for patients, caregivers, and relatives as questions recur about diagnosis, treatment, early warning signs, hospitalization, stigma, and substance use. GPT-4 is appealing in that setting because it may generate individualized explanations at scale and potentially extend access to information between visits. The unresolved clinical question is whether those answers are accurate, understandable, inclusive, and clinically usable enough to support psychosis care.

In the cross-sectional psychosis psychoeducation study in Early Intervention in Psychiatry, investigators conducted a cross-sectional qualitative evaluation of GPT-4, a large language model (LLM), through the ChatGPT interface using 20 psychosis-related psychoeducational questions derived from real-world first-episode psychosis care. The prompts reflected common inquiries from patients, caregivers, and relatives. Each question was entered in a new independent session, and the first three questions were re-prompted three times for a qualitative check on variability. Two psychosis experts independently scored the responses for accuracy, clarity, inclusivity, completeness, clinical utility, and overall quality, then resolved disagreements by discussion and consensus. The rubric was informed by prior literature and expert consensus but was not formally validated.

Responses were generally coherent, organized, and clinically relevant, with mean scores of 2.88 ± 0.22 for accuracy, 2.93 ± 0.18 for clarity, and 4.35 ± 0.52 for clinical utility.

Other domain scores also clustered toward the higher ends of their scales, with mean scores of 2.30 ± 0.41 for inclusivity, 0.93 ± 0.18 for completeness, and 3.55 ± 0.46 for overall quality. Inclusivity remained the relative weak point.

Readability was relatively advanced, with a mean Flesch-Kincaid Grade Level (FKGL) of 15.59 ± 1.59. Average responses were 330 words long and contained 31.5 sentences.

In exploratory analyses, higher clarity was associated with greater completeness (Spearman ρ = 0.61, p = 0.004).

Higher clarity was also associated with greater clinical utility (Spearman ρ = 0.54, p = 0.013).

The authors noted several limits to interpretation: the evaluation drew on 20 questions from a single clinical setting, expert scoring was subjective, formal inter-rater reliability was not assessed, and the rubric itself was not formally validated. The cross-sectional exploratory design also did not address real-world use, patient comprehension, trust, behavioral impact, outcomes, or safety over time. Some answers lacked sufficient nuance for complex or individualized scenarios, and the combination of high reading complexity with lower inclusivity could reduce accessibility for some patients and caregivers.

According to the authors, GPT-4's adjunctive role in psychosis psychoeducation may be limited to structured, clinician-guided contexts rather than individualized clinical assessment or care. They based that interpretation on generally strong expert ratings alongside lower inclusivity, advanced reading complexity, and unanswered questions about safety and real-world implementation. Further research is needed to clarify how GPT-4 performs in real-world psychosis psychoeducation.

Clinician Questions

What kinds of psychosis questions were used to evaluate GPT-4 in first-episode care?

The prompts were clinician-developed questions informed by common real-world inquiries from patients, caregivers, and relatives in a first-episode psychosis treatment program.

How was psychoeducation quality defined in the GPT-4 psychosis evaluation?

Two psychosis experts independently assessed GPT-4 responses across six domains: accuracy, clarity, inclusivity, completeness, clinical utility, and overall quality. They then resolved disagreements by discussion and consensus. The rubric used predefined scoring scales, and formal validation of that rubric was beyond the scope of the study.

Was GPT-4 response variability checked when the same psychosis question was asked more than once?

Investigators re-prompted the first three psychosis questions three times and compared the outputs qualitatively for variability. This functioned as a limited consistency check within the study design rather than a broader performance benchmark.

Did the GPT-4 psychosis psychoeducation study test patient comprehension or real-world safety?

No. It was a cross-sectional expert evaluation with no patient follow-up and no assessment of real-world use, patient comprehension, trust, behavioral impact, or safety over time, which is why the authors limited GPT-4 to an adjunctive role in psychosis psychoeducation.