Chatbots Show Similar Overall Quality in Testicular Cancer Answers

Key Takeaways

• Free-tier chatbots gave moderately rated answers to simulated questions from newly diagnosed patients with testicular cancer, without a significant overall difference among models.
• Corrected pairwise clarity testing favored Gemini over ChatGPT but did not distinguish Claude from ChatGPT.
• Claude and Gemini scored higher than ChatGPT on fertility and sexual-health questions descriptively, without an established domain-specific difference.
• Clinician raters flagged both overly conditional and overly categorical language about sperm-banking timing.
• No chatbot answer supplied citations or directions to external resources without prompting.

A patient diagnosed with testicular cancer may look for information about impending surgery, sperm banking, testosterone, sexual function, or a prosthesis before meeting a specialist. Timing and personal circumstances can change how a general answer is understood. Investigators compared chatbot answers to simulated questions arising at that point in care.

Investigators conducted a cross-sectional, blinded evaluation of free-tier web versions of ChatGPT v5.5, Gemini v3.1, and Claude Sonnet 4.6. Queried from Sydney, Australia, on 14 May 2026, the models generated 45 first, unedited responses to 15 simulated patient-perspective questions; there were no human participants. Questions came from patient-education materials, including the American Cancer Society and U.S. cancer centers, and were refined by a senior urologist rather than obtained directly from patients. They covered basics and risk, symptoms and diagnosis, treatment, prognosis and survivorship, and fertility and sexual health, including sperm banking, testosterone, sexual function, and prostheses. A common prompt identified a newly diagnosed patient and requested a brief answer. Each question began a fresh conversation with history, memory, and personalization disabled; investigators requested no references, follow-up, or regeneration. Responses were de-identified and randomized within questions. A blinded consultant urologist and three registrars gave initial written ratings before grading by consensus; individual scores were not retained. The Quality Analysis of Medical Artificial Intelligence (QAMAI) assessed accuracy, clarity, relevance, completeness, sources, and usefulness on a scale out of 30. Investigators used matched-question Friedman comparisons and Bonferroni-corrected pairwise Wilcoxon tests.

Mean overall QAMAI totals out of 30 were 21.8 for ChatGPT, 22.40 for Claude, and 22.13 for Gemini. All fell in the moderate grade; those averages alone do not establish whether the models differed.

The overall Friedman comparison was not statistically significant (p = 0.246). Clarity differed across models after correction, but only Gemini scored significantly higher than ChatGPT in corrected pairwise testing (p = 0.038). ChatGPT–Claude and Claude–Gemini clarity comparisons were not significant. Other QAMAI items showed no significant model differences.

On fertility and sexual-health questions, Claude and Gemini scored higher than ChatGPT descriptively; no domain-specific significance test was reported. Clinician raters considered ChatGPT too conditional about sperm banking before surgery and too late in raising preoperative prosthesis discussion. They considered Claude’s urgency language too categorical because clinical urgency can sometimes override preservation. No model spontaneously provided citations or directions to external resources.

One English-language, free-tier first answer per question and model on a single day captures a snapshot, not performance with later versions, different prompts, paid tiers, or continued conversations. Questions came from education sites, and consensus ratings did not preserve initial individual scores. Real patient understanding, choices, and outcomes were not tested. A possible QAMAI ceiling effect also limits interpretation of the nonsignificant overall comparison as equivalence. Because the brief prompt did not request references, absent spontaneous citations do not establish whether the models could provide them if asked. The authors concluded that chatbot answers may supplement, rather than replace, individualized specialist counseling.

Clinician Questions

Which sperm-banking elements did ChatGPT, Gemini, and Claude address in answers about newly diagnosed testicular cancer?

In answers to a simulated newly diagnosed testicular cancer patient’s presurgical sperm-banking question, all three models recommended banking before surgery or treatment and acknowledged permanent infertility risk; ChatGPT conveyed urgency only partially and framed pre-surgical banking conditionally.

Which complication after testicular cancer surgery did Claude alone mention when discussing sexual function?

In an answer about sexual function and testicular prosthesis for a simulated newly diagnosed testicular cancer patient, Claude alone mentioned retrograde ejaculation following retroperitoneal lymph node dissection. The clinician consensus panel identified this difference in the chatbot responses.