LLM Assistance Improves Carotid Plaque Classification Agreement
Key Takeaways
- In a three-institution comparison of carotid MRI reports, junior and senior radiologists had greater vessel-level agreement with expert report-text consensus when assisted by either evaluated model.
- Both models increased patient-level agreement with the expert reference, with the largest gains under DeepSeek-R1.
- Criteria-guided Qwen3-8B improved standalone report-classification agreement but remained below DeepSeek-R1.
Investigators assembled MRI reports from 433 patients at three institutions, 376 of them from one center; all reports were in Chinese. In a reader study of a stratified random subset of 200 patients (400 vessels), four radiologists—two junior (4–5 years’ experience) and two senior (7–9 years), none with neurovascular subspecialty training—classified plaques from report text unaided and then with Qwen3-8B or DeepSeek-R1 assistance. Separately, investigators benchmarked 10 models and a rule-based baseline on all 433 patients’ reports, using naive prompts or prompts explicitly encoding AHA criteria.
Among junior radiologists, vessel-level agreement with the expert report-text reference was 62% without assistance, 83% with Qwen3-8B and 88% with DeepSeek-R1. These figures describe agreement on plaque classes assigned from existing narratives, not accuracy of plaque detection on independently assessed images. The improvement was greater for junior than for senior readers, according to the investigators.
Senior radiologists began with higher vessel-level agreement with the same reference and improved under both assisted conditions; DeepSeek-R1 produced the highest agreement. All reported assisted vessel-level comparisons were statistically significant after Holm correction. The senior-reader pattern reinforces the distinction between assistance in classifying a written report and independently confirming features on the original scans.
Patient-level agreement, which required correct classification of both vessels, rose from 41% unaided to 71% with Qwen3-8B and 79% with DeepSeek-R1 among junior readers, and from 60% to 80% and 88% among senior readers. Readers’ mean quadratic-weighted κ against the reference rose from 0.231 unaided to 0.775 with DeepSeek-R1. Neither endpoint establishes that the assigned categories match independently assessed plaque features.
In the standalone model-only benchmark, encoding AHA criteria in prompts improved Qwen3-8B's agreement over naive prompting, although DeepSeek-R1 remained ahead. This result concerns models classifying written reports without a radiologist; it cannot be treated as a change in reader performance. Investigators suggested the criteria-guided smaller model could help standardize text-based classification, but the comparison did not test a deployed clinical workflow.
Assistance also introduced errors: 33 of 800 junior vessel-level readings (4.1%) were correct unaided but incorrect with AI, with a similar pattern among senior readers, mostly at borderline categories. Unaided reading always came first, so learning effects could inflate the apparent benefit of assistance. The expert reference came from the same written MRI reports, not independently reviewed images or histopathology. Greater agreement with that reference establishes neither image-level validity nor better stroke prediction, treatment decisions, patient safety or outcomes. The comparison is therefore most directly relevant to the reproducibility of report-derived classifications, not to whether assistance improves care.
Whether agreement on report text extends to classifications grounded in independent image assessment or predicts clinical outcomes remains unanswered. Those tests would address a different question from concordance with experts reading the same narratives.
Clinician Questions
Did criteria-guided Qwen3-8B carotid plaque classification require model fine-tuning?
Investigators described explicitly encoding AHA criteria in prompts as a potential route to standardizing modified AHA classifications from MRI report text without task-specific fine-tuning. This proposed use does not establish clinical deployment or document the model's previous fine-tuning history.
What reader order or blinding was specified for LLM-assisted carotid MRI report classification?
Readers completed unaided classification first, then read with each model in a balanced order (Qwen3-8B first or DeepSeek-R1 first), with a 6-week washout between the two assisted sessions. Because the unaided reading always came first, the order was not fully randomized. A post hoc analysis found no significant effect of assisted-session order.