US Doctors Review ChatGPT and Google Translate for Medical Translations

A group of physicians from multiple Boston, USA, area hospitals researched the use of ChatGPT and Google Translate in a small study, reporting their findings in a paper published on June 11, 2024. Specifically, the physicians wanted to find out how well these technologies work for the translation of pediatric discharge instructions compared to professional translation services.

As the basis for their research, the paper authors argue that machine translation (including large language models, LLMs) has “the potential to expand access to translation services.”

The experiment consisted of translating twenty standardized discharge instructions in the pediatric conditions domain into Spanish, Brazilian Portuguese, and Haitian Creole using professional translation services, and machine translation (MT) via Google Translate and ChatGPT-4.0.

The translations were then evaluated for what the authors labeled as “adequacy” (preserved information), “fluency” (grammatical correctness), “meaning” (preserved connotation), and “severity” (clinical harm), as well as an “assessment of overall preference” (defined as the comparison to reveal the preferred translation out of the different methods).

The ratings showed that Google Translate and ChatGPT performed at a similar level in the domain compared to professional translations for Spanish and Portuguese. For Haitian Creole, however, professional translations had higher scores, and ChatGPT and Google Translate had more of what the evaluators considered “potentially clinically significant errors.” 

The physicians concluded that machine translation platforms “have comparable performance to professional translations for Spanish and Portuguese but shortcomings in quality, accuracy, and preference persist for Haitian Creole.”

The authors also added that “diverse multilingual training data are needed, along with regulations ensuring safe and equitable applications of machine translation in clinical practice.”

The topic of translation of medical forms and other documentation continues to be the subject of studies at the organizational level. Many physicians have identified the lack of translated forms as a recurrent issue in their own clinical settings.

For example, back in August 2023, Slator covered the results of a consent form translation study by clinicians at the University of California and Beth Israel Deaconess Medical Center. The researchers explained that most patients in both sponsored and non-sponsored clinical studies signed consent documents in a language that wasn’t their primary language.

The Research Gate repository of scholarly papers also has quite a few reports of similar studies, including about a dozen just on discharge instructions published since 2023.