Where AI Output Needs a Human Check: A Review-Level Framework for Mid-Market Owners and Operators

Where AI Output Needs a Human Check: A Review-Level Framework for Mid-Market Owners and Operators

2026-10-23

The takeaway: Don’t put a human reviewer on every AI output, and don’t put one on none. Sort each AI use by who sees the result and what a mistake would cost you, then assign one of four review levels. That way your people’s limited review time goes to the outputs that can actually hurt you.

Why “a human will check it” isn’t a plan

Most AI policies include a line like “employees must review AI output.” It sounds responsible and means almost nothing. Review what? How closely? Who signs off?

Two well-documented cases show what happens when nobody answers those questions.

In Mata v. Avianca, lawyers filed a brief citing court opinions that ChatGPT had made up. On June 22, 2023, U.S. District Judge P. Kevin Castel of the Southern District of New York sanctioned the two lawyers and their firm, imposing a $5,000 penalty and ordering them to send letters to each judge falsely named as the author of a fake opinion (Opinion and Order on Sanctions). Judge Castel was clear that using AI wasn’t the offense: “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance.” The offense was skipping the gatekeeping the rules already required, then standing by the fake opinions after the court questioned them.

In Moffatt v. Air Canada (decided February 14, 2024), the airline’s website chatbot told a grieving customer they could apply for a bereavement fare after travel. Air Canada’s own bereavement page said otherwise. Christopher C. Rivers, a member of British Columbia’s Civil Resolution Tribunal, rejected the airline’s argument that it wasn’t liable for its chatbot: “It should be obvious to Air Canada that it is responsible for all the information on its website.” The tribunal ordered Air Canada to pay $812.02 (Moffatt v. Air Canada, 2024 BCCRT 149).

Small dollars, big lesson: if AI says it in your company’s name, your company said it.

Why wrong output looks right

NIST’s Generative AI Profile (NIST AI 600-1, published July 26, 2024) calls this risk “confabulation”: “confidently stated but erroneous or false content,” which most people call hallucinations (NIST AI 600-1). The same document flags automation bias, which it describes as “excessive deference to automated systems,” and notes it can make confabulation risks worse. Translation: the more polished and dependable a tool seems, the more likely your team is to wave its mistakes through.

There’s field evidence, too. In a preregistered experiment with 758 Boston Consulting Group consultants, Fabrizio Dell’Acqua of Harvard Business School and co-authors found AI improved performance on tasks inside its capabilities. But on a task designed to fall outside them, consultants using AI were 19 percentage points less likely to reach a correct solution than those working without it (HBS Working Paper 24-013, September 2023). The catch is that you usually can’t tell from the output which side of that line a task falls on.

The four review levels

Score each AI use on two questions:

  1. Who sees it? Only your team, or customers, regulators, courts, or the public?
  2. What does a mistake cost? A few minutes of rework, or money, legal exposure, safety, or someone’s job?

Then assign a level.

Level 1: Ship it

Internal, low stakes, and errors are easy to spot. Think meeting summaries for people who were in the meeting, internal brainstorm lists, and first drafts of notes. The reader is the reviewer, so no formal check is needed.

Level 2: Spot-check

Internal work that feeds routine decisions, or high-volume work where any single error is cheap. Think ticket categorization, call notes that feed the CRM, or draft job descriptions. A named owner samples a set share of outputs each week and tracks the error rate. If errors creep up, the use moves up a level.

Level 3: Review before it leaves the building

Anything customer-facing, and anything containing a number, a citation, a price, a policy, or a promise. Think proposals, customer emails, support answers, and marketing claims. A person checks every output against the source of truth before it goes out. For live chat, where you can’t review each answer, limit the bot to content you’ve already approved and route policy questions to a person. That’s the Air Canada lesson.

Level 4: Expert sign-off

Legal, financial, HR, safety, or regulatory output. Think contract language, filings, pay decisions, and anything that could end up in front of a judge. A qualified person verifies every fact and citation and puts their name on it. Mata v. Avianca is the reason “the AI drafted it” is not a defense.

How to roll it out without a six-month committee

  • List your AI uses. Start with what people actually use today, not what the policy assumes.
  • Give each use a level and an owner. One row per use in a shared sheet is enough.
  • Define “checked” for each level. For Level 3, that might be “every figure, name, and policy statement confirmed against the system of record.”
  • Watch for level creep. An internal summary that starts getting pasted into client emails just moved from Level 1 to Level 3.
  • Re-test when tools change. A new model or a new data source is a reason to check again, not a reason to relax.

The bottom line

Review effort is a budget. Spend it where AI output is public, costly, or hard to reverse, and let low-risk internal work move fast. That’s how you get the productivity without the sanctions order.

If you’d like help mapping your AI uses to review levels and standing up the guardrails behind them, that’s the kind of work Yellow Coop does through fractional CTO and AI solutions engagements. See what we do or get in touch.

Sources