Artificial intelligence has been in use in some Yukon clinics for months — and according to the territory’s health department, they’ve already become a valuable resource for health care providers.
But others argue governments have embraced AI too quickly without fully understanding potential impacts on patients.
Two AI tools have been deployed in the territory: Heidi, which listens to conversations between providers and patients and spits out a transcript and key bullet points, and Spotlight, which uses those clinical notes to auto-populate referral forms and other paperwork.
The services can only be used with the consent of patients. Clinicians are expected to check in with patients at the beginning of each appointment.
Alethea Stobbe, director of integrated health services with the Yukon government, said she has heard positive feedback from providers about how the tools are working in clinics.
“People can actually be present with their clients because they don’t have to document while they’re in the appointment,” she said.
But Marcus Castillo, whose company provides advice on AI adoption, wondered why the tools were adopted without policy guidelines in place around the use of AI in clinics.
“If you don’t know what you’re doing with AI tools, especially with something so sensitive, you’re going to commit mistakes,” he said.
Chirag Jadhwani, Spotlight’s founder, told CBC his goal when he created the tool was to help clinicians avoid doing just that. While working for the Yukon government in Whitehorse, he got terrible news from family back in Mumbai.
“My grandmother, she was one of the people who raised me with my mom,” he said. “When she passed away, it was because of her referrals being delayed and bounced back because there was a mistake in her files.”
The experience drove him to try and find a solution.
“I quit my job and started cold-calling people, hospitals, crisis lines, all the people who make referrals and also manage people who make referrals in the healthcare system,” he said.
He learned providers at the territory’s walk-in clinic require as many as four visits to complete a referral package for a single patient, and three hours to complete the paperwork. In response, Jadhwani designed software in partnership with Vector, the research institute co-founded by Geoffrey Hinton, the so-called “godfather of AI.”
“Now, two clicks, and a draft is ready for review,” he said.
Stobbe said this has helped fill a critical gap in the Yukon’s health-care system, which heavily relies on locum doctors and nurses who may not be familiar with all referral options both in and outside of the territory.
She says as many as 60 per cent of clinicians now regularly use the AI scribe tool Heidi, and 90 per cent use Spotlight. Both tools are in use across Canada, with Spotlight now deployed in clinics in Ontario, B.C., two U.S. states and England.
The Yukon information gathered by Heidi is stored on servers in Quebec before being automatically deleted after 10 days. Spotlight put servers in clinics, and relies on a virtual private network rather than third-party cloud-based storage.
Report finds medical AI tools hallucinate
In May, Ontario Auditor General Shelley Spence released a report which found AI tools intended for use by doctors provided incorrect and incomplete information or demonstrated “hallucinations.” It also found significant gaps in the evaluation and testing process.
Stobbe acknowledged Yukon providers have seen Heidi generate false information, but said once they asked the tool not to make assumptions and only use the information discussed in the clinical encounter, errors dropped significantly. She said the government completed privacy impact and security threat risk assessments before deploying the software.
The Auditor General’s report also detailed issues with Ontario’s procurement process, which it said could result in selection of vendors whose AI tools “produce inaccurate or biased medical records” or “lack adequate protection to safeguard personal health information.”
The Yukon government’s procurement process circumvented an assessment of various AI vendors and issued direct awards to Heidi and Spotlight last year. This can only be done under unique circumstances, such as an emergency or if the contract is under a certain cost threshold. The health department did not respond to a request for more information from CBC.
Spence made a number of recommendations for Ontario, such as mandatory live demonstrations to ensure the effectiveness of AI systems before implementing them and mandatory training to ensure users understand the risks.
“We’re not saying stop progress,” said Spence. “We’re just saying, put controls around the environment to ensure we’re not running off into the sunset with something that isn’t working.”
A Heidi spokesperson did not return a request for comment by deadline, but referred CBC to a section on its website outlining the company’s commitments to safety and performance, which read, in part, “the bar for how clinical AI is evaluated rises faster when the work is held in the open.”
In the Yukon, the evaluation phase has involved using the tools with real patients since the tools were introduced last year. A report on that rollout is now complete. The Yukon government has not released it.
‘Health systems are strained’
“So much of our time as clinicians ends up going to repetitive tasks,” said Dr. Puneet Seth, a Vancouver-based family doctor who says he was an early adopter of scribe tools in his clinical practice. “Health systems are incredibly strained.”
In April, Carney announced a reduction to the Canada Health Transfer, which pays for provincial and territorial healthcare systems, as well as cuts to programs like the Emergency Treatment Fund, which funds several Yukon programs.
Ken Kirkwood, an associate professor of bioethics at Western University, fears AI could be used to justify further cuts to healthcare spending.
“When you have governments who don’t fund healthcare adequately, you can have a situation where the seeming benefits and necessity are actually manufactured,” he said.
Stobbe said clinicians are expected to review AI-produced notes and paperwork for accuracy, and added providers are accountable for any errors.
But Kirkwood worries that offloads liability onto already-overworked physicians. He says AI tools should undergo robust testing like any other drug or medical device before being deployed.
Jadhwani says his company has put significant effort into ensuring patient safety and confidentiality.
“But definitely there is a risk when using tools like Spotlight — it’s software at the end of the day, it can make mistakes,” he said. “Our goal is that instead of spending three, four hours getting paperwork done, we can bring it down to 10 minutes.”
