Clinicians Are Asking AI About Treatment. Are We Evaluating the Wrong Tasks?
New JAMA research reveals the questions clinicians bring to AI. Before expanding access, compare those tasks with the evidence in your approval packet.
Treatment and management accounted for 33.1% of physician queries in a new JAMA study of an AI clinical-information platform. Diagnosis accounted for 13.9%. The authors contrast those use patterns with the diagnostic reasoning and knowledge tasks emphasized in prior AI evaluations. JAMA, October 15.
That creates a specific question for the next AI approval meeting: does the evidence cover the job clinicians want the tool to do?
A strong diagnostic benchmark does not, by itself, establish performance on treatment selection, medication questions or interpreting evidence for an individual patient. We recommend requiring a task-by-task match between intended use and evaluation before endorsing broader clinical use. A product-level approval is too blunt when different uses require different evidence.
New Evidence About Questions, Not Proof About Care
The cross-sectional study examined OpenEvidence query logs from January 2024 through June 2026: 952,008 NPI-verified US clinicians submitted 271 million queries. Query-purpose estimates used an AI-classified, stratified sample of 38.8 million queries, not manual review of every question.
This is one platform's use, not a census of hospital AI. Researchers could not establish whether questions arose during encounters, preparation or other settings. The findings do not show that clinicians followed the answers, or establish patient benefit or harm. Classification can mislabel questions. Several authors were OpenEvidence employees; the company supplied data and contributed to analysis. Methods, limitations and disclosures.
The executive implication is ours: investigate whether your evaluation matches intended and reported use. The study does not establish that your hospital has approved the wrong product, that its clinicians use this platform, or that they use AI without authorization.
Look Past the Headline Score
Start with the decision under consideration. Are you approving a resource for general medical reference, evidence retrieval, patient-specific decision support, or documentation? Do not treat those descriptions as interchangeable.
Ask the vendor to map its evidence to each requested use. Then ask clinical leaders which uses staff are actually bringing forward. A tool demonstrated on a clear diagnostic vignette may face a very different test when a clinician asks what to do after the diagnosis is known, with incomplete information and competing considerations.
This is not an argument to discard diagnostic evaluation. It is an argument against allowing one category of success to carry an untested category into production.
Joint Commission/CHAI guidance recommends seeking validation relevant to the deployment context and establishing local monitoring. That is high-level guidance, not validation of any product or of the exercise below. Responsible-use guidance, ongoing quality monitoring.
One Practical Example: After the Diagnosis Is Known
Consider this proposed evaluation exercise, not a real patient case or an observed product result. Use synthetic information in an approved environment; clinical reviewers define the appropriate response before testing. This is not treatment advice.
Choose an atrial-fibrillation scenario in which the diagnosis is already established. The question for the tool is: what stroke-prevention options should the clinician compare, and what information could change their suitability?
The reviewers prepare one complete case containing the information they judge necessary, including relevant kidney function, medication history and bleeding history. These examples are not an exhaustive clinical checklist. They also prepare a matched version with a necessary detail withheld, such as kidney function when their selected question requires it.
The test is not whether the tool can name the diagnosis. Reviewers assess whether it uses the available information appropriately, identifies missing context before making a patient-specific recommendation, and supports its statements with applicable evidence. A general educational answer is different from a specific recommendation to act.
Suppose the incomplete-case response gives a confident recommendation without addressing the missing information. The worked approval decision is to withhold endorsement for that patient-specific use pending correction and retesting. That response does not establish that the product fails every task. Equally, a good answer to the complete case does not clear it for every treatment question.
Put the Intended Task Beside Its Evidence
Use this proposed review record for the example. It is an ALHQ evaluation tool, not the JAMA study's instrument or a universal clinical standard.
Evaluation question: Does the test represent the requested use?
Review owner: Clinical informatics with the service lead
Evidence to retain: Intended-use statement, synthetic cases and exact prompts.
Decision use: Identify untested permissions before approval.
Evaluation question: Does the response handle necessary context?
Review owner: Relevant clinician and pharmacist
Evidence to retain: Complete/incomplete responses and a predefined clinical rubric.
Decision use: Restrict a use that relies on unsupported assumptions.
Evaluation question: Do citations support the advice and apply to the case?
Review owner: Clinical evidence reviewer
Evidence to retain: Source passages, dates and applicability judgments.
Decision use: Separate a plausible answer from a supported one.
Evaluation question: What effort and failures does use create?
Review owner: Operations with clinical quality
Evidence to retain: Review/correction time, failed attempts and adjudicated safety concerns.
Decision use: Size support and determine whether further evaluation is justified.
Preserve the tested product configuration, available sources, date and full response. Review cases independently and adjudicate disagreements. Report results by task and case variant rather than hiding a weak group in a pooled score.
If reporting a success rate, use successful attempts divided by all prespecified attempts in the same task group. Include timeouts and failed runs, report them separately, and identify unscorable responses rather than silently dropping them. Record serious concerns individually. A small test set can expose a problem; it cannot certify rare-event safety or prove better patient outcomes.
Compare the proposed workflow with the current approved information resource using matched cases and information. Do not assume that retaining human review means the new workflow adds no review burden. Track that effort rather than treating it as free capacity.
Change the Approval, Not Just the Presentation
At the next meeting, ask for a clear disposition for each proposed use: supported for a defined scope, continue controlled evaluation, or not supported by the current evidence. Clinical leadership sets the acceptance criteria and safety response in advance.
The CIO/CAIO should bind the decision to the tested configuration. Operations and finance should account for integration, training, subscriptions, review and ongoing monitoring before expansion. Faster information retrieval is not automatically less total work, and time returned is not booked savings without an actual financial mechanism.
The immediate action is modest: put your intended-use list beside the evaluation packet and identify the first task the evidence does not cover. Fund the missing evaluation before funding wider reliance.
Download the Task-to-Evidence Workbook and Evidence File to use alongside this review.
Which clinical question is missing from your AI evaluation? Reply with a general example, without patient or identifying details.