The potential for artificial intelligence (AI) models, such as ChatGPT, to recommend harmful medications like thalidomide to pregnant women poses serious ethical and health risks.Thalidomide, notorious for causing severe birth defects, has been withdrawn from general use since 1961, except for specific treatments like multiple myeloma.
Source:
pubmed.ncbi.nlm.nih.govHowever, recent studies indicate that AI models can be easily manipulated to suggest this dangerous drug under certain conditions.A recent study published in JAMA Network Open explored how susceptible various large language models (LLMs) are to prompt injection, a technique where malicious actors alter user queries to elicit harmful recommendations.In scenarios where users sought advice for pregnancy-related nausea, researchers found that LLMs could be tricked into recommending thalidomide, with alarming success rates.For instance, when using evidence-fabrication injection, models like ChatGPT recommended thalidomide 100% of the time in the tested scenarios.
Source:
medscape.comThe implications of this are profound.Pregnant women often seek medical advice from AI due to its perceived accessibility and empathy.However, the reality is that these models can produce dangerously misleading information.In a separate study, ChatGPT was found to have a significant number of responses with critical safety omissions, particularly regarding over-the-counter medications used during pregnancy.This raises concerns about the reliability of AI as a standalone resource for medical guidance.
Source:
mdpi.comThe thalidomide tragedy of the 1950s serves as a historical reminder of the consequences of inadequate drug testing in pregnant populations.At least 8,000 babies were born with severe birth defects due to thalidomide, which was prescribed for morning sickness without proper testing on pregnant women.
Source:
sph.brown.eduThis historical context underscores the importance of rigorous safety evaluations and the ethical considerations surrounding drug recommendations for pregnant individuals.Moreover, the growing reliance on AI for medical advice highlights a gap in the healthcare system.While AI can provide general information, it lacks the nuanced understanding required for safe clinical guidance.The risk of misinformation is particularly acute in vulnerable populations, such as pregnant women, who may face life-altering decisions based on AI-generated advice.The potential for AI to induce unnecessary terminations of pregnancy due to misinformation about teratogenic risks is a significant concern that warrants further investigation.
Source:
pubmed.ncbi.nlm.nih.govAs AI technology continues to evolve, it is crucial to establish guidelines and safeguards to prevent the misuse of these models.The integration of human oversight in AI-generated health information is essential to ensure patient safety.Healthcare professionals must remain involved in the decision-making process, particularly when it comes to prescribing medications during pregnancy.In conclusion, while AI models like ChatGPT offer promising tools for accessing medical information, their current limitations and vulnerabilities pose significant risks, especially in sensitive contexts like pregnancy.The potential for these models to recommend harmful drugs like thalidomide underscores the urgent need for caution, regulation, and human oversight in the use of AI in healthcare.As we navigate this evolving landscape, prioritizing patient safety and ethical considerations must remain at the forefront of discussions surrounding AI in medicine.
