89.3% Semantic Agreement Is Not ROI: The Hospital AI Scribe Scorecard
What this week's evidence changes, and what your hospital still needs to prove.
A September 23 Scientific Reports paper reports 89.3% weighted mean semantic agreement for an AI scribe. That is a documentation measure, not a return on investment.
The retrospective Spanish-network study covers more than 2.3 million outpatient consultations across 45 specialties. Its detailed quality assessment uses 300 reports across 15 specialties. The abstract reports better readability and structured quality, but supplies no measured labor savings, net financial return, or patient outcomes. All authors work for the network that developed and evaluated the system; they declare no personal financial interests in the technology. These findings describe associations, not established causation. Publisher source.
This week's evidence warrants investigating documentation performance, not booking savings. We recommend a bounded local evaluation before adding seats or making an irreversible expansion commitment.
What the Number Cannot Decide
Semantic agreement is not a clinical error rate. Do not present 89.3% as the percentage of clinically correct notes, or its complement as the percentage containing errors. Without the full metric definition, comparison reference, and weighting method, an executive cannot even establish that a vendor's similarly named score measures the same thing.
Similar, readable notes can still misstate medications or omit important findings. Test clinical content against authorized encounter evidence, not merely compare wording.
Nor does a Spanish outpatient result establish performance in your US hospital. Language, specialty mix, EHR workflows, documentation obligations, and reimbursement arrangements require local testing. Treat inpatient and emergency workflows as separate expansion decisions, not automatic extensions of an outpatient evaluation.
This Week's Investment Decision
Bring a capped evaluation budget to the next approval meeting. Specify the specialties, seat count, evaluation end date, and contract exit or resizing terms. Do not enter clinician time multiplied by salary as cash savings unless spending actually falls.
The CFO should require reduced paid expenses or incremental collected contribution. The COO should test whether schedules, rooms, staffing, and demand can convert available minutes into visits. The CIO/CAIO should establish the evaluated system version and whether audit records, review controls, and data protections work.
Reduced evening documentation can justify investment as workforce relief. Label it that way. Capacity without conversion is not cash; employee relief does not need a fictional revenue claim.
Take This Scorecard to the Meeting
This is our proposed local tool, not the study's evaluation instrument. Report baseline, evaluation result, missing observations, and variation by specialty and clinician. Do not let a pooled average hide a failing workflow.
Before launch, fix eligibility by encounter type, specialty, and supported language, independent of actual use. Fix encounter dates and a signing deadline relative to each encounter, allowing equal follow-up. Include failures and opt-outs; deduplicate by encounter ID. Report late or unsigned notes separately.
| Measure and definition/denominator | Accountable owner | Local data source | Decision use |
|---|---|---|---|
| Usable adoption: unique eligible encounters with at least one scribe-assisted note signed within the fixed window / all unique eligible encounters; separate attempt, failure, and opt-out counts. | Ambulatory operations | Encounter roster, scribe logs, EHR linkage | Size seats to actual use; prevent duplicate-note inflation. |
| Clinical content: notes with clinically consequential errors / audited notes, separately for drafts and signed notes; complete required elements / applicable required elements. | Clinical quality lead | Authorized encounter evidence, chart, independent audit | Detect introduced errors and failures that survive review. |
| Net documentation work: documentation-active person-minutes of clinicians and documentation-support staff / observed completed encounters, versus comparable baseline; report each labor group and after-hours minutes separately. | Clinical informatics | Time observation, schedules, EHR activity logs | Test work reduction versus transfer, including failed-attempt fallback. |
| Realized capacity: incremental completed visits / comparable staffed sessions; adjust for case mix and scheduling changes. | COO | Scheduling, staffing, encounter records | Test conversion of time into throughput. |
| Realized financial benefit: verified expense reductions plus incremental collected contribution, in dollars per evaluation period. | CFO/controller | Payroll, general ledger, collections, visit costs | Separate observed cash benefit from forecasts. |
| Incremental cost: total dollars and dollars / eligible encounter; separate one-time, recurring cash, and staff opportunity costs. | Finance with CIO | Contract, invoices, implementation and support time | Establish full cost and volume sensitivity. |
| Control exceptions: unreviewed released notes / released scribe-assisted notes; failed required audit checks / required checks performed; privacy incidents as distinct counts by severity, with an exposure denominator only when meaningful and defined. | CIO/CAIO with privacy lead | Release logs, incident register, version history | Escalate serious events regardless of aggregate rates. |
Count each person's active documentation time once across setup, review, correction, and failed-attempt fallback. Passive capture concurrent with care adds no work minutes; overlapping review/correction counts once. Sum simultaneous work by different people. Report active after-hours documentation person-minutes per clinician-day, using scheduled hours, not elapsed login time.
Cost licenses, usage, interfaces, training, support, audits, security review, and fallback. Measure correction minutes once; include incremental paid correction labor once in cash costs, and existing salaried correction effort once only in the opportunity-cost view. Measuring minutes and costing them are not double counting.
For the cash case, subtract incremental cash costs from verified expense reductions and incremental collected contribution. Deduct added visit-delivery costs within contribution; deduct scribe costs separately, once. Separate uncollected forecasts. The opportunity-cost view adds existing staff effort valued at agreed loaded rates, without recounting labor already in cash costs. Show steady-state and first-year economics separately.
An Illustrative 30-Day Evaluation
The following timeline and samples are proposals, not study findings or statistically validated requirements.
Days 1-5: Select two outpatient specialties and approve counting rules and acceptance criteria before seeing results. Establish baseline using the same clinicians' prior encounters and, where feasible, a concurrent non-scribe group. Record case mix, language, staffing, and schedule changes. Clinical quality defines severity, escalation triggers, and mandatory review. Finance sets cost and break-even assumptions.
Days 6-25: Maintain clinician review of every draft before release. As a proposed starting audit, independently review 100 scribe-assisted and 100 comparator notes, stratified across clinicians, specialties, languages, and complexity. Oversample risky encounters and report those results separately. Preserve drafts and signed versions under approved data controls; include failed attempts in operational reporting. Audit a subset twice to resolve reviewer disagreements. This sample can identify problems, not certify rare-event safety.
Days 26-30: Reconcile quality, work, adoption, cost, and capacity. Separate early training effects from later performance. Document comparison-group differences rather than attributing every change to the scribe. Collections may not mature within 30 days: mark the financial result provisional and schedule reconciliation. A short operational evaluation is not a patient-outcome trial.
Expand only in tested workflows that meet pre-agreed local quality and control criteria, show credible net-work improvement, and have an affordable, explicit benefit case. Expansion for workforce relief is a different authorization from expansion for financial return.
Continue within the cap when quality controls hold but adoption, time estimates, or collections remain uncertain. Set the next evidence deadline and the specific gap to close. Do not renew indefinitely because the pilot feels promising.
Pause and escalate the affected workflow when a locally predefined serious-error or privacy trigger occurs, regardless of a low aggregate rate, when mandatory review fails, or when a version change invalidates the evaluation. Resume after documented remediation and retesting. Use approved local thresholds, not a universal similarity cutoff.
The board question is straightforward: Are we authorizing this spend for better documentation, workforce relief, added capacity, or realized financial return, and what local evidence supports that specific claim?
Put the scorecard beside the contract. Approve the benefit you can defend, and fund the evaluation needed to establish the rest.
Reply with the decision your hospital faces: add seats, renew, or continue evaluation, and which scorecard measure is still missing from the approval packet.