Your doctor has 7 minutes. ChatGPT has all night. What could go wrong?
As millions of Canadians turn to AI for health advice, doctors grapple with a technology that’s already reshaping care, whether they’re ready or not.
TLDR:
The Surge: Over 230 million people now ask ChatGPT health questions weekly, driven by a 313% jump in Canadian specialist wait times since 1993.
The Successes: For some, AI acts as a life-saving safety net, catching drug interactions and symptoms (like strokes) that overworked doctors miss.
The Risks: Studies show up to 13% of AI medical responses are actively unsafe, with rare disease diagnostic accuracy as low as 4%.
The Bias: 84% of clinical AI models fail to report race in training data, potentially hard-coding existing medical biases into the “solution.”
Last August, a 60-year-old Australian man arrived at an emergency room hallucinating and stuporous. Doctors quickly diagnosed him with bromide poisoning. He’d asked ChatGPT for dietary advice, and the bot suggested replacing his table salt with sodium bromide, a toxic chemical compound. He’d been following that advice for weeks.
From San Francisco, 7,000 miles away, Ayrin Santoso had a different experience. Her mother woke up in Jakarta unable to see out of one eye, and local doctors suggested she simply rest. Instinctively, Santoso typed her mom’s symptoms into ChatGPT. The response was immediate and alarming: This could be a stroke. Get to a hospital now. Her mother was hospitalized that day and recovered 95% of her vision.
Two cases, two outcomes, both real. And millions of us are making the same bet Santoso did. We’re typing symptoms into an AI and hoping we’re one of the lucky ones.
When OpenAI announced ChatGPT Health a few weeks ago, physicians felt uneasy. Dr. Jonathan Block, a urologist in New York, called it “a mix of elation and fear”. But for the 230 million people who already ask ChatGPT health questions every week, this isn’t a new behaviour, it’s an acknowledgment of a reality already woven into how we navigate illness.
The wellness survival strategy we didn’t ask for
ChatGPT Health, rolling out in the coming weeks, wants to be your all-in-one medical archive. You’ll be able to upload your records and sync data from fitness trackers and apps like Apple Health and MyFitnessPal into an encrypted chat space. Unlike standard ChatGPT conversations, health chats won’t be used to train OpenAI’s models, the company promises.
The demand is massive: OpenAI says that more than 5% of all global ChatGPT prompts are already health-related. But there is a glaring, legal-sized contradiction at the heart of the product. OpenAI’s own terms of service explicitly state the tool is “not intended for use in the diagnosis or treatment of any health condition.”
The company is marketing a dedicated health product while explicitly disclaiming medical responsibility, which places all the risk on users who may not understand the fine print.
The arithmetic of waiting for care: 60 weeks for a specialist, 3 seconds for a bot
The math of Canadian healthcare right now leaves a lot to be desired. The launch of ChatGPT Health itself is an acknowledgment of a massive, systemic gap in how we and others around the world access care.
6.5 million Canadians lack a family doctor
The national average wait to see a specialist has hit 15.3 weeks, a 313% jump since 1993
Median wait times from GP referral to specialist treatment have reached 28.6 weeks, up 208% since 1993
In New Brunswick, the wait for treatment stretches to 60.9 weeks. In PEI, it’s nearly 50 weeks. When your symptoms are screaming but your appointment is a year away, a chatbot that answers in three seconds can feel like the only option.
The second opinion: When AI finds what an overworked doctor missed
For OpenAI’s CEO of Applications, Fidji Simo, it’s personal. When a doctor prescribed her an antibiotic, she used ChatGPT to cross-reference it with her history. The AI flagged that the drug could reactivate a serious past infection. The doctor, overworked and rushed, had missed it.
“She explained that she only has about 5 minutes per patient when making rounds, and that health records aren’t organized in a way that would make this sort of risk clear,” Simo wrote on her Substack in her post “ChatGPT Health and what AI can do for a broken system.”
Many of use AI to connect the dots in a system that wasn’t built to see a full picture of our health. For some, like Santoso’s mom, that second opinion is life-saving.
For patients like Simo (educated, technologically fluent, with access to care), AI is a safety net. But for others, it represents something more desperate.
Women navigating reproductive health issues describe using ChatGPT to decode medical reports filled with jargon like “focal adenomyosis” and “adhesions,” terminology their doctors provide without explanation. Andoeni Ruezga used ChatGPT-4 this way, asking the bot to explain where endometrial tissue was growing and why she was experiencing debilitating pain.
“We’re at a point where the language barrier between medical professionals and regular people is very high,” Ruezga said on TikTok after trying to decipher her endometriosis diagnosis.
Patients turn to AI not for diagnosis, but for translation, and for the vocabulary to make doctors listen. In Canada, it takes 5 to 9 years on average to get an endometriosis diagnosis. AI has become one way to find the words that legitimize our pain to doctors.
The bias built into the code: When confident advice becomes unsafe
But the tool we’re using to escape medical dismissal might be carrying the same biases we’re trying to outrun. For Black patients, the endometriosis timeline is twice as long as for white patients: 1.34 years versus 0.67 years after first reporting pelvic pain. Black women are three times more likely to be misdiagnosed with fibroids instead of endometriosis.
With the discrimination some groups face in the healthcare system, turning to AI for advocacy feels like an act of self-preservation. But 84% of clinical AI models failed to report race or ethnicity in their training data
The mental health gap
The mental health gap is perhaps where the “helpful” nature of AI becomes most dangerous. LLMs are designed to flatter, so they validate rather than question. They encourage rather than redirect.
The American Psychological Association has issued a “do not use” advisory for generic AI bots in mental health care. Yet, OpenAI reports that more than a million users a week express suicidal intent to the bot.
Anthony Tan, a Toronto developer, spent three weeks in psychiatric care after ChatGPT conversations about simulation theory fuelled a psychotic break. The bot validated his delusions, calling his thoughts “historically important.”
Allan Brooks spent three hundred hours with a bot that compared him to Galileo while encouraging mathematically nonsensical work. Cases like these led to the Human Line Project, a support group for AI-fuelled delusions.
The accuracy problem
LLMs predict likely responses, not correct answers. They hallucinate, fabricating plausible-sounding but false information. One study by physicians found that up to 43% of AI responses were problematic and 13% were actively unsafe.
The rate of problematic responses varies from 21.6% (Claude) to 43.2% (Llama), with unsafe responses varying from 5% (Claude) to 13% (GPT-4o, Llama).
ChatGPT correctly diagnosed common conditions like carpal tunnel 100% of the time but succeeded in only 4% in rare conditions, another study found.
When NBC Washington asked one chatbot for anxiety help, it claimed to be a real California doctor and provided a stolen medical licence number. California’s Medical Board is investigating.
“What worries me is that it’s not obvious where general information ends and medical advice begins, especially when responses sound confident even if they mislead,” Alex Ruani, a health misinformation researcher at University College London, told The Guardian after the sodium bromide poisoning case.
The regulatory patchwork
While the FDA in the US just announced it will ease regulation of AI-enabled digital health products to promote adoption, Canada occupies an uncertain middle ground. Health Canada has issued only non-enforceable guidelines. ChatGPT Health sidesteps even these minimal guardrails by positioning itself as a “wellness tool” rather than a medical device, a distinction that leaves millions of users without regulatory protection.
This means ChatGPT Health will launch with no requirement to prove it works equally well for Black and white patients, no mandate to detect and redirect mental health crises, and no obligation to flag when it’s hallucinating. The company sets its own rules, writes its own disclaimers, and leaves users to navigate the risks.
The smart patient’s guide to AI
If you’re navigating the system and using AI as a tool, experts suggest these hard boundaries:
✅ Use it for the “prep work”: Let AI help you generate questions for your doctor or translate confusing lab results into simple summaries. Use it to find the clinical vocabulary to describe your pain more precisely
✅ Stop if it’s an emergency: If you have chest pain, sudden vision changes, or thoughts of self-harm, call 911 or the 988 Suicide Crisis Helpline immediately
✅ Watch for hallucinations: If the bot gives conflicting answers across different sessions, it’s making things up
✅ Keep receipts: If a doctor dismisses you, ask them to document their refusal to test or refer you in your chart
✅ Report dangerous advice: Screenshot concerning AI responses and report them to Health Canada’s MedEffect program
Where this leaves us
We’ve built a healthcare system so broken that a chatbot that hallucinates, can’t detect crisis, and was trained on biased data, feels like progress. That’s not an AI problem. That’s a societal failure.
The technology is here. Patients are using it. But the system that made a chatbot feel necessary is still failing them. And no amount of algorithmic innovation will fix that.
Disclosure: I lead AI communications at Manulife, a life insurance company. All views expressed in this newsletter are my own and do not represent my employer.
AI in the news
An OpenAI safety research lead departed for Anthropic (The Verge) Andrea Vallone, who led OpenAI’s work on how chatbots should respond to users in mental health distress, joined Anthropic’s alignment team amid growing concern that safety is lagging behind product speed. Her move underscores a deeper industry tension: as AI systems become emotional confidants, the hardest problems are no longer technical, but ethical, human, and unresolved.
Malaysia will take legal action against Musk’s X and xAI over misuse of Grok chatbot (CTV) Elon Musk’s AI chatbot Grok sparked an international outcry for generating sexually explicit deepfake images, including one of children. Malaysia and Indonesia were the first countries to block Grok, citing “grossly offensive” and non-consensual content, and Britain’s regulator opened an investigation into Musk’s platform X. Musk’s team is brushing off media queries, but pressure is mounting.
Google wins big in AI deal with Apple, which highlights the iPhone maker’s own AI struggles and could spell trouble for OpenAI (Fortune) Apple announced a multi-year deal to use Google’s Gemini AI models to power a much smarter Siri and other future iPhone features. Experts note this raises questions about competition and privacy, but it also signals how even rival tech giants are teaming up in AI. It’s a development to watch as it could reshape the balance of power in consumer AI platforms going forward.
This is a hell of a write‑up — and a reminder that AI in health isn’t some shiny gadget problem, it’s an everyday‑people problem.
What struck me reading this wasn’t the tech, but the math of desperation: weeks‑long waits, five‑minute doctor visits, and a system so backlogged that a chatbot with a confidence problem feels like a lifeline.
We can debate safety, bias, and regulation (and we should), but there’s a simpler truth underneath it all: people will use whatever shows up fastest when they’re scared, confused, or in pain. That’s not hype. That’s human nature.
And that’s why “usage” ends up being the only metric that matters. Not because it proves the AI is good — but because it proves the need is real.
If we don’t fix the system that’s driving people to AI in the first place, we’re just arguing about which bucket to use while the boat keeps taking on water.
AI will keep showing up. People will keep using it.
The question is whether our institutions catch up — or keep pretending this isn’t already happening at scale.
Thanks Nicolle for this nuanced take on an extremely difficult and complex issue. I was very much against using ChatGPT for health advice because of data extraction and missing guardrails but as some other substackers pointed out what about those people who could never afford to see a doctor in the first place? If only BigTech were as committed to the Hippocratic Oath as they were to profits...