ChatGPT Falters at Predicting Cut-Out After Femoral Nailing
Key Takeaways
- In routine postoperative surveillance after proximal femoral nailing, ChatGPT showed modest discrimination for future cut-out but very low PPV in a low-prevalence cohort.
- Point estimates were numerically higher in TFNA cases than in Gamma nail cases, although the between-group difference was not statistically significant.
- False positives far outnumbered false negatives, and raising the probability threshold traded modest specificity gains for worsening sensitivity.
- Binary yes/no calls and probability scores did not align well, with higher confidence attached more often to incorrect than correct classifications.
ChatGPT reviewed three baseline radiographs per case — the injury anteroposterior and lateral views plus an immediate postoperative control radiograph, typically obtained on postoperative days 1 to 3 — along with structured data on nail type, fixation subtype where applicable, nail length, AO/OTA fracture classification, laterality, age, sex, body mass index (BMI), American Society of Anesthesiologists (ASA) class, and days from surgery to the postoperative film; complication-related information and all follow-up radiographs were withheld. Cut-out was then adjudicated on later radiographs by a departmental review board that required agreement from at least two senior surgeons.
Overall discrimination was modest, with sensitivity 68% (95% confidence interval [CI] 48.4–82.8%) and specificity 62.7% (95% CI 59.6–65.7%); positive predictive value (PPV) 4.5% (95% CI 2.8–7.1%) and negative predictive value (NPV) 98.7% (95% CI 97.4–99.3%); and area under the curve (AUC) 0.694 (95% CI 0.580–0.790). In this low-prevalence surveillance cohort, the low PPV indicates that many positive calls were false alarms, and the high NPV likely reflects the rarity of cut-out rather than by itself establishing dependable rule-out performance, although the authors said it may suggest a potential rule-out role in very-low-risk surveillance scenarios pending prospective validation. TFNA point estimates exceeded Gamma nail estimates, but the between-group AUC difference was not statistically significant and the Gamma subgroup included only six events.
The directional error pattern was dominated by 360 false positives versus 8 false negatives. Increasing the probability cutoff did not rescue performance: at a 70% threshold, sensitivity fell to 0% while specificity reached 98.1%, and lower thresholds showed the same specificity-for-sensitivity tradeoff qualitatively. Calibration also ran counter to the binary outputs, with a median predicted probability of 9% for correct classifications versus 40% for incorrect classifications (p < 0.001).
This evaluation used one standardized prompt and one model version at a single non-U.S. institution, with cut-out uncommon overall and especially rare in the Gamma nail subgroup. There was no formal blinded human-reader comparison, and tip-apex distance plus Cleveland zone position were not retained as structured numeric variables even though they are familiar radiographic concepts in fixation assessment.
The authors said this evaluated ChatGPT workflow was not ready to replicate routine surveillance for cut-out after proximal femoral nailing. They added that task-specific development and external validation would be needed before a model of this kind could be considered for fracture follow-up pathways.
Clinician Questions
Which postoperative hip fracture cases do these ChatGPT cut-out findings apply to?
These findings come from a consecutive routine surveillance cohort of patients with intertrochanteric or subtrochanteric hip fractures treated with TFNA or Gamma nails at one institution. ChatGPT was asked to predict later cut-out from baseline injury films and the immediate postoperative radiograph before cut-out was visible, so the signal pertains to scheduled postoperative follow-up rather than to a symptomatic or selectively high-suspicion subset.
How was cut-out confirmed when ChatGPT did not see follow-up radiographs?
Cut-out after proximal femoral nailing was confirmed on subsequent follow-up radiographs by direct comparison with the immediate postoperative baseline film. A departmental orthopaedic review board made the reference-standard decision, requiring agreement of at least two senior surgeons and using consensus discussion when needed, while those follow-up images were not shown to ChatGPT.
Why can NPV look high when ChatGPT still performed poorly for cut-out surveillance?
Because cut-out was rare in this routine surveillance cohort, most postoperative hip fracture fixation cases were true negatives from the outset; that base-rate effect can make NPV appear high even when discrimination is only modest. The same analysis also reported a very low PPV and a large false-positive burden, which is why the authors did not present the NPV as dependable rule-out performance for cut-out surveillance.
What limits these cut-out prediction results from being generalized to other AI prompts or model versions?
These results characterize one standardized prompt, one GPT-5.3 workflow, and one single-centre retrospective dataset rather than all multimodal AI approaches to fracture follow-up. The authors noted that general-purpose AI systems change over time, so performance in this evaluation does not establish how different prompts, later model versions, or other institutions would perform.