Artificial intelligence in pediatric dental trauma: do artificial intelligence chatbots address parental concerns effectively?

Abstract

Background

This study focused on two Artificial Intelligence chatbots, ChatGPT 3.5 and Google Gemini, as the primary tools for answering questions related to traumatic dental injuries. The aim of this study to evaluate the reliability, understandability, and applicability of the responses provided by these chatbots to commonly asked questions from parents of children with dental trauma.

Methods

The case scenarios were developed based on frequently asked questions that parents commonly ask their dentists or Artificial Intelligence chatbots regarding dental trauma in children. The quality and accuracy of the information obtained from the chatbots were assessed using the DISCERN Instrument. The understandability and actionability of the responses obtained from the Artificial Intelligence chatbots were assessed using the Patient Education Materials Assessment Tool for Printed Materials. In statistical analysis; categorical variables were analyzed in terms of frequency and percentage. For numerical variables, skewness and kurtosis values were calculated to assess normal distribution.

Results

Both Artificial Intelligence chatbots performed similarly, although Google Gemini provided higher quality and more reliable responses. Based on the mean scores, ChatGPT 3.5 had a higher understandability. Both chatbots demonstrated similar levels of performance in terms of actionability.

Conclusion

Artificial Intelligence applications can serve as a helpful starting point for parents seeking information and reassurance after dental trauma. However, they should not replace professional dental consultations, as their reliability is not absolute. Parents should use Artificial Intelligence applications as complementary resources and seek timely professional advice for accurate diagnosis and treatment.

Introduction

Traumatic dental injuries (TDIs) represent a significant proportion of all head and neck injuries, particularly among children and young adults [1]. If not properly treated, dental trauma can result in both functional and aesthetic complications. Given that most negative outcomes are caused by inadequate or inappropriate emergency interventions, the timing of treatment is as crucial as the treatment itself [2, 3].

Studies indicate that TDIs most commonly occur at home in settings where children are with their families [4]. However, public awareness regarding the emergency management of TDIs remains generally insufficient [5, 6]. Moreover, many individuals face challenges in securing timely appointments and accessing specialist consultations [7]. Consequently, the trend of seeking online advice after dental trauma has notably increased in recent years [8].

In today’s technological era, artificial intelligence (AI) chatbots are increasingly being used to provide patients with access to vital information [9]. AI is defined as the ability of a system to replicate human-like intelligence [10]. AI chatbots simulate human cognitive processes using advanced algorithms and extensive datasets to generate instant, human-like responses. This capability enables AI chatbots to engage effectively with users using natural language, thus allowing for quick and accurate answers to the user’s queries [11, 12].

The integration of AI into dentistry marks a considerable advancement and are a valuable tool for predicting dental diseases, diagnosing conditions, and developing treatment plans [13]. AI chatbots can save time in emergencies by offering instant, comprehensive, and personalized responses [14]. Unlike search engines that present general information from various sources, AI-based chatbots deliver information in a conversational format and simplify complex topics for the user [15, 16]. This approach is particularly beneficial in cases of TDIs as it eliminates the need to navigate multiple web pages [14].

Chat Generative Pre-Trained Transformer (ChatGPT) is a large language model that leverages both AI and machine learning to engage in conversations with users. ChatGPT uses deep learning techniques to generate human-like responses to natural language inputs and provides meaningful and contextually relevant answers based on an extensive knowledge base [17].

Another AI application introduced as an alternative to ChatGPT is Gemini, which is an AI-powered chatbot that answers user questions, engages in conversations, and generates creative content [18]. These advanced AI applications that are free to access provide enhanced capabilities in text-based interactions, delivering quick and effective solutions to users’ requests [19, 20].

In health care, patients who cannot communicate directly with their doctor are often driven to seek information from various online sources [21, 22]. While all chatbots aim to provide accurate feedback and address concerns via user interaction, there are ongoing debates, particularly in health care, about the validity and reliability of AI chatbot responses [8]. A review of the literature identifies only one study that evaluates the performance of AI chatbots in answering hypothetical, case scenario-based questions related to dental trauma [14]. As such, there is a need for additional research that simulates real patient inquiries. This study aimed to assess and compare the accuracy and applicability of responses from two different AI chatbots, based on case scenarios derived from common questions that parents of children with dental trauma typically ask their doctors or AI chatbots.

The null hypothesis of the study is that the reliability, understandability, and actionability of the responses provided by the ChatGPT 3.5 and Gemini chatbots to questions related to TDIs are not significantly different.

Material and methods

This study focused on two AI chatbots, ChatGPT 3.5 and Gemini, as the primary tools for answering questions related to TDIs. The aim was to evaluate the reliability, understandability, and applicability of the responses provided by these chatbots to commonly asked questions from parents of children with TDIs.

Question design

The case scenarios used as queries for the chatbots were developed based on frequently asked questions that parents commonly ask their doctors or AI chatbots regarding dental trauma in children. These questions were developed based on inquiries frequently directed to pediatric dentists during clinical examinations regarding dental trauma. They were formulated through literature reviews and the insights of the pediatric dentists who authored the study.To assess the understandability of the case scenarios, five pediatric dentists who were selected independently of the study reviewed them. These pediatric dentists were selected based on having at least five years of experience and holding educational roles in university hospitals. 17 questions were sent electronically via Google Forms, and pediatric dentists were asked to provide feedback on each question as either “Clear” or “Needs Revision.” For the questions marked as “Needs Revision,” participants were asked to specify the necessary modifications in the comment section. This iterative process continued—refining and resubmitting the questions based on the participants’ feedback—until all questions were clearly understood.

Chatbot processing

The prepared questions were proposed to the AI chatbots by a single individual (one of the authors). To avoid any influence from previous searches, new accounts were created on each platform. After inputting the case scenarios into the chatbots, the questions were asked using the phrase: “What should I do in such a situation?” The “new conversation” option was selected each time, ensuring that the questions were asked independently and to prevent follow-up questions. To maintain consistency, all questions were inputted to both chatbots by the same user on the same day.

Quality assessment

The quality and accuracy of the information obtained from the AI chatbots was independently assessed by two authors using the Discern instrument (DI) and the mean scores were calculated. Before the evaluation, the authors gained proficiency in the assessment process by reviewing the literature on DI. Inter-rater and intra-rater reliability were analyzed using Cohen's kappa coefficient. The kappa coefficient ranged from 0.72 to 0.78, indicating a substantial level of agreement. During this evaluation, the accuracy and quality of the responses obtained from the chatbots were determined based on the International Association of Dental Traumatology (IADT) guidelines, and the DI scoring was conducted accordingly [23, 24]. Developed in 1999, the DISCERN tool is a 16-item measurement designed to evaluate the quality and reliability of health-care websites [25]. The DI is divided into three sections: Sect." Introduction"with eight questions that assess the reliability of the information; Sect." Material and Methods"focuses on treatment information with seven questions; and Sect." Results"includes one question that evaluates the overall quality of the information. Each question is scored from 1–5 with the total score being the sum of all 16 questions. Based on this scoring system, the total average score is used to categorize the quality of the information: a score between 16–26 is considered very poor, 27–38 is weak, 39–50 is average, 51–62 is good, and a score higher than 63 is considered excellent [26].

Evaluation of the understandability and actionability of the responses

The understandability and actionability of the responses obtained from the AI chatbots were independently assessed by two authors using the Patient Education Assessment Tool for Printed Materials (PEMAT-P) and the mean scores were calculated. Before the evaluation, the authors gained proficiency in the assessment process by reviewing the literature on PEMAT-P. PEMAT-P consists of 24 questions, divided into two sections: one that evaluates understandability (questions 1–17) and the other that evaluates actionability (questions 18–24). During the evaluation, responses are scored as 0 for “Disagree”, 1 for “Agree”, and NA for “Not Applicable”. As no visual content was included in the questions posed to the AI chatbots, the visual questions in this analysis were marked as NA. Following the PEMAT-P guidelines, the scores for understandability and actionability were calculated separately [27]. Additionally, the understandability and actionability scores were evaluated separately for primary and permanent teeth, and a comparison was made based on the type of tooth.

Ethical committee approval

Since this study did not involve humans or animals, ethical approval was not required. The data were collected in full compliance with the terms of service of the relevant AI platforms.

Statistical analysis

Data analysis was performed with IBM SPSS Statistics version 26. Categorical variables were analyzed in terms of frequency and percentage. For numerical variables, skewness and kurtosis values were calculated to assess normal distribution, as shown in Table 1. According to the rules for normal distribution, skewness values should fall between ± 1.5, and kurtosis values (calculated by dividing the statistical value by the standard error) should be between ± 7 [28]. Based on these criteria, all values presented in Table 1exhibit normal distribution. Consequently, parametric tests (independent sample T test, one-way ANOVA, and Pearson’s correlation analysis) were applied. The significance levels for the study were set at 0.05 and 0.01 [29].

Table 1 Skewness and kurtosis values of the data

Results

A total of 17 case scenario questions containing prepared inquiries were proposed to both chatbots, resulting in a total of 34 detailed responses. The complete list of responses is provided in Appendix 1.

Accuracy and quality of the responses

The quality and accuracy of the information obtained from the AI chatbots were assessed using the DI. The mean DI scores for each chatbot are presented in Table 2. The analysis revealed no statistically significant difference between the two groups (t = –0.265, p = 0.793). These results suggest that both AI chatbots performed similarly, although Gemini provided more reliable and higher quality responses.

Table 2 Comparison of DISCERN Scores by AI Chatbots

The comparison of DI scores of AI chatbots for different tooth types is presented in Table 3. No statistically significant difference was seen between the scores for permanent teeth and primary teeth for either ChatGPT 3.5 or Gemini (p > 0.05). These results suggest that both AI chatbots offer similar levels of quality and accuracy across different tooth types. However, the mean scores for Gemini were higher than ChatGPT 3.5 for both types, indicating that ChatGPT 3.5 might provide better overall quality and accuracy.

Table 3 Comparison of DI Scores by Tooth Types and AI Chatbots

Understandability and actionability of the responses

The comparison of PEMAT-P scores between the AI chatbots is presented in Table 4. The results indicate that there was no statistically significant difference between ChatGPT 3.5 and Gemini in terms of understandability or actionability (p > 0.05). However, based on the mean scores, ChatGPT 3.5 had a higher understandability score. Both chatbots showed similar levels of performance in terms of actionability.

Table 4 Comparison of PEMAT-P Scores by AI Chatbots

The comparison of PEMAT-P understandability scores of AI chatbots for different tooth types is presented in Table 5. No statistically significant difference was seen between the scores for permanent teeth and primary teeth for either ChatGPT 3.5 or Gemini (p > 0.05). These results suggest that both AI chatbots offer similar levels of understandability across different tooth types. However, the mean scores for ChatGPT 3.5 were higher than Gemini for both types, indicating that ChatGPT 3.5 might provide better overall understandability.

Table 5 Comparison of PEMAT-Understandability Scores by Tooth Types and AI Chatbots

The comparison of PEMAT-P actionability scores by AI chatbots for different tooth types is Table 6. Overall, there was no statistically significant difference between ChatGPT 3.5 and Gemini in terms of actionability scores for both permanent and primary teeth (p > 0.05). However, it is worth noting that ChatGPT 3.5 had a higher mean score for primary teeth, while Gemini achieved higher mean scores for permanent teeth. ChatGPT 3.5 demonstrated a more consistent performance, whereas Gemini showed greater variability in its results. These findings suggest that both AI chatbots exhibit similar levels of actionability, but their performance can differ depending on the tooth type included in the query.

Table 6 Comparison of PEMAT-Actionability Scores by AI Chatbots

Discussion

In health care, the inability of patients to communicate with their doctors at all times or learn from the experiences of individuals with similar medical histories often leads patients to seek information from various online sources [21, 22]. Information from the internet is valuable due to its constant accessibility and the variety of perspectives it provides [30]. Prominent examples of AI applications include large language models, such as ChatGPT 3.5 and Gemini. The accuracy and performance of these models in clinical settings are crucial for both patients and clinicians [31]. Consequently, this study evaluated the responses of AI chatbots to frequently asked questions from families regarding simulated dental trauma case scenarios.

The reliability of AI chatbots largely depends on their ability to provide accurate, clear, and contextually appropriate information. Research evaluating the responses of AI chatbots in health care scenarios, including dental trauma, has highlighted both the strengths and limitations of these technologies. While most studies assessing the validity and reliability of AI chatbots as information sources in dentistry have focused on endodontics [32, 33], research on dental traumatology remains limited. Given that dental trauma is one of the most common dental issues encountered in daily life and is frequently searched online, there is a clear need for further research in this area [34].

When evaluating a chatbot, the language used and the types of questions addressed are important; however, frequently asked questions during clinical visits also play a crucial role. These questions, which cover the types of traumas and the psychological and physiological effects of dental trauma on patients, help assess how effectively the chatbot responds to specific situations [8]. In this study, similar to the work by Güven et al., (2024) [14] simulated case scenarios were created and the responses from AI chatbots were evaluated. This approach led to more personalized answers regarding dental trauma. Furthermore, when parents express their concerns in greater detail, the quality and reliability of the responses from AI chatbots improve.

Çalışmamızda cevapların doğruluğu ve kalitesi DI ile değerlendirilmiştir. DI, başlangıçta yazılı sağlık bilgilerinin kalitesini değerlendirmek amacıyla geliştirilmiş olsa da, web siteleri, YouTube videoları ve son zamanlarda yapay zekâ sohbet botları gibi çevrim içi hizmetlerin değerlendirilmesi için de uyarlanmıştır [14]. İnternetteki tıbbi bilgilerin güvenilirliğini, makullüğünü ve kullanılabilirliğini belirlemek amacıyla HONcode ve JAMA kriterleri ile Ensuring Quality Information for Patient (EQIP) aracı da kullanılmaktadır. McCool ve arkadaşları tarafından yapılan çalışmalarda DI, Ensuring Quality Information for Patients (EQIP) aracına kıyasla daha yüksek iç tutarlılık ve gözlemciler arası güvenilirlik, daha iyi uyum ve daha hassas değerlendirme yeteneği sergilemiştir [35]. Doubleday ve ark. HONcode'un web tabanlı bilgilerin kalitesini değerlendirmek için yeterli bir araç olmadığını çalışmalarında belirtmiştir. Ayrıca HONcode ise 15 Aralık 2022 tarihinden itibaren kalıcı olarak sonra erdirilmiş ve artık güncellenmemektedir [36]. JAMA kriterlerinin de dereceli bir ölçek yerine evet/hayır şeklinde ikili sorular kullanmasından dolayı, bazı web siteleri için ne derece geçerli olduğu konusunda da soru işaretleri mevcuttur [37].

When assessing the quality of responses according to the DI categorization, responses from ChatGPT 3.5 were classified as “moderate”, while Gemini responses were “good”. These results suggest higher quality and more reliable responses from Gemini compared with ChatGPT 3.5, although the data remains somewhat limited. A similar study by Güven et al., (2024) found that Gemini had higher DI scores when compared with three other AI chatbots [14]. In contrast, a study by Behers et al., (2024) evaluating the quality of patient education materials on cardiac catheterization from AI chatbots found that ChatGPT had higher DI scores [38]. AI chatbots effectively engage with users by processing text and previous dialogue data, adopting a dialogue-based approach, and learning from user interaction. However, continuous learning and training on the latest information are essential for improving the quality of chatbot responses [39, 40]. As AI technology evolves, the continuous development of response generation capabilities might explain the variations in response quality and reliability observed in studies conducted at different times [41].

Tariq et al., (2024) emphasized the need for a standardized evaluation framework to assess AI models in health care. This framework should not only measure the accuracy and scope of the information but also ensure that responses address patients’ diverse needs and backgrounds, and that they are both understandable and actionable [42]. In this study, PEMAT-P was used to evaluate the understandability and actionability of responses. No statistically significant differences were observed in terms of understandability, although ChatGPT 3.5 provided better understandability. Regarding actionability, both AI applications showed similar levels of competence. In contrast to our findings, Güven et al., (2024) [14] reported that the understandability of ChatGPT 3.5 was lower compared with other AI chatbots, while actionability results were consistent with those observed in this study.

Suárez et al., (2024), in their assessment of ChatGPT's accuracy and consistency, concluded that while AI applications show promise, they are not yet sufficient for clinical decision making [33]. In this study, a detailed evaluation of the responses from ChatGPT 3.5 and Gemini revealed several shortcomings. For instance, neither chatbot addressed the role of an open or closed apex in root canal treatment for dental trauma. Additionally, in question 16, while GG suggests that the hardness felt in the lip may result from a broken tooth fragment embedded in the lip, ChatGPT does not mention this possibility. Furthermore, in question 4, ChatGPT 3.5 explained that: “For a 4-year-old child, since the anterior teeth are most likely primary teeth, replantation is generally not recommended” while Gemini did not mention about the replantation procedure of primary teeth. However, upon comparative evaluation, GG’s responses were found to be more comprehensive and valid than those of ChatGPT 3.5. Güven et al., (2024) also noted that ChatGPT mistakenly recommended the replantation of primary teeth in cases of avulsion injuries [14].

These differences are believed to arise from the AI chatbots interpreting open-ended clinical questions in different ways. Additionally, responses to the same questions might vary when asked at different times. Özden et al., (2024) also reported that Gemini did not achieve a sufficient level of consistency and provided different responses at various times [43]. Furthermore, large language models are not specifically trained in specialized fields, such as endodontics and dental traumatology, which can affect the accuracy and applicability of their responses in clinical settings. These models are typically trained with general language and knowledge datasets, limiting their ability to provide detailed information on specific dental subspecialties.

Although AI chatbots generally achieve highly valid scores, critical errors in some responses and the potential to mislead the public on certain topics can lead to considerable issues, especially in health care contexts. ChatGPT 3.5 is more popular among patients than other AI platforms because of its free access, lack of registration requirements, and its longer presence in the market compared with other AI platforms [44]. However, since this model was trained on a broad range of online sources, including books, scientific articles, and websites up until 2021, there is a risk of it providing outdated or limited medical advice [17]. In contrast, while Gemini is free, it requires logging in with a Google account [44]. Med-Gemini, a health care-specialized model developed by Google, holds greater potential for the future. Equipped with the ability to seamlessly integrate web searches and an evolving database, Med-Gemini is seen as a promising tool for accurate medical dialogues, medical research, and health education [45]. These advancements could considerably improve the validity and reliability of Gemini in providing medical advice, positioning it as a more effective AI platform than ChatGPT 3.5.

Güven et al., (2024) noted that ChatGPT 3.5 struggled to accurately assess the relationship between a pediatric patient's age and their dental development stage based on information provided by parents, resulting in more superficial responses [14]. In this study, the questions were specifically tailored to the patients’ ages, and the AI chatbots were expected to accurately identify dental types based on age and provide appropriate suggestions. In this context, the understandability and actionability of responses related to primary and permanent teeth were evaluated separately. Although no statistically significant difference was found, ChatGPT 3.5 showed more consistent performance across both primary and permanent teeth, whereas Gemini showed more variability in its responses. This finding suggests that both AI chatbots have similar capabilities in terms of actionability, but their performance might differ depending on the type of teeth that is queried.

This study has several limitations. Although the case scenarios were developed based on real expressions from patients’ relatives to accurately reflect the patient’s condition, AI chatbots might not capture all the nuances of the situation, which typically requires a clinical assessment by a dentist. Furthermore, the AI chatbots used in this study were not specifically trained in dental or dental trauma topics, meaning that their responses might lack the depth and accuracy needed for clinical contexts. Finally, while the authors followed specific guidelines when evaluating the responses, subjective judgments could have influenced the scoring.

Dental trauma, especially in children, can be a major source of concern for parents. However, consulting a specialist is not always possible. In such cases, parents might turn to AI-powered chatbots to alleviate their concerns and obtain information. While AI chatbots have the potential to guide parents regarding dental trauma and possible complications; however, certain shortcomings have been observed in the responses provided.

This comparison offers valuable insights into the effectiveness of AI chatbots and contributes to a deeper understanding of their applications in dentistry. Specifically, the data highlights the potential advantages and limitations of AI in dental traumatology and general dental practices. These findings underscore the importance of further research to improve and refine AI technologies in the health-care sector, ensuring that their integration leads to more reliable, accurate, and actionable information for both practitioners and patients.

Conclusion

AI applications can serve as a helpful starting point for parents seeking information and reassurance after dental trauma. However, they should not replace professional dental consultations as their reliability is not absolute. Parents should use AI applications as a complementary resource and seek timely professional advice for accurate diagnosis and treatment. Additionally, parents should be trained by pediatric dentists on dental trauma.