AI = Appropriate Insight? ChatGPT Appropriately Answers Parents’ Questions for Common Pediatric Orthopaedic Conditions Original Research

Abstract

Background: Artificial intelligence services, such as ChatGPT (generative pretrained transformer), can provide parents with tailored responses to their pediatric orthopaedic concerns. Chatbots are increasingly popular, therefore we undertook a qualitative study to assess the accuracy of the answer provided by ChatGPT in comparison to OrthoKids (“OK”), a patient-facing educational platform governed by the Pediatric Orthopaedic Society of North America (POSNA) for common pediatric orthopedic conditions.

Methods: A cross-sectional study was performed on May 26-June 18, 2023. OK website (orthokids.org) was reviewed and 30 existing questions were collected. The corresponding OK and ChatGPT responses were recorded. Two pediatric orthopedic surgeons assessed the answer provided from ChatGPT against the OK response. Answers were graded as: AGREE (accurate information; question addressed in full), NEUTRAL (accurate information; question not answered), DISAGREE (information was inaccurate or could be detrimental to patients’ health). The evaluators’ responses were compiled; discrepancies were adjudicated by a third pediatric orthopedist. Additional chatbot answer characteristics such as unprompted treatment recommendations, bias, and referral to a healthcare provider were recorded. Data was analyzed using descriptive statistics.

Results: The chatbot’s answers were agreed upon in 93% of questions. Two responses were felt to be neutral. No responses met disagreement. Unprompted treatment recommendations were included in 55% of its responses (excluding treatment-specific questions). The chatbot encouraged users to “consult with a healthcare professional” in all responses. It was nearly an equal split between recommending a generic provider (46%) in contrast to specifically stating a pediatric orthopedist (54%). The chatbot was inconsistent in related topics in its provider recommendations, such as recommending a pediatric orthopedist in 3 of 5 spine conditions.

Conclusion: Questions pertaining to common pediatric orthopedic conditions were accurately represented by a chatbot in comparison to a specialty society-governed website. The knowledge that chatbots deliver appropriate responses is reassuring, however the chatbot frequently offered unsolicited treatment recommendations whilst simultaneously inconsistently recommending an orthopaedic consultation. We urge caution to parents utilizing artificial intelligence without also consulting a healthcare professional.