The Three Checks Before I Trust Any AI Output
Three checks, one real mistake, and the review process I promised last week.
Yesterday, a tool I built told me a LinkedIn post had gone live. It printed “POSTED.” The screenshot it saved as proof showed my home feed, and other people’s posts, but nothing that actually confirmed my post existed anywhere.
I didn’t mark it done. I opened my own profile’s activity page and looked for the post myself. It was there, and the header image had rendered correctly, so I moved on. But sit with the gap for a second: a system reported success, and the success it reported was just a log line that happened to be true this time, coincidentally.
Last week I told you the scarce skill left over once AI gets good is judgment, not generation. The short version, if you missed it: AI writes clean, well-structured, confident-sounding output now by default, at zero cost and infinite volume. That used to be a decent proxy for someone knowing what they were doing. It isn’t anymore, because the model can fake the surface for free. What it can’t fake is the read that catches what’s wrong underneath the polish.
I promised to come back with how I actually run that review instead of just describing it from a distance. Here are the three checks, in the order I actually use them, plus the mistake that reminded me why check three exists.
Check one: is the mistake reversible?
I wrote about this back in April without knowing I’d need it again this week, in a note about deciding where to place my trust in AI. The real question is what happens if AI got it wrong.
A bad line of code gets caught in code review before it ships. A bad paragraph gets edited before anyone reads it. Both of those are cheap to undo, so I let AI move fast there and I review lightly. A LinkedIn post that goes out with the wrong link in its first comment costs real time to fix, and a lot of it is spent apologizing. A database write that silently duplicates a row nobody notices for three weeks costs even more, and it’s usually discovered by accident, weeks after the fact. The scrutiny I apply scales with how expensive a mistake is to reverse. How much I happen to trust the tool that produced the work has nothing to do with it.
I built this directly into the system behind this newsletter. Before it writes anything to my content ledger, it checks whether the row already exists. If it does, it updates. If it doesn’t, it inserts. The tool that schedules my Substack Notes does the same thing on the way in: it checks whether a note is already sitting in the queue before adding it, so running the import twice never creates two copies of the same post. Neither check is clever. Both of them mean a re-run, a retry, or a script I fire twice by accident can’t quietly cost me an afternoon of cleanup I wouldn’t have noticed I needed.
You don’t need a database to use this. A “solopreneur” running AI-drafted invoices can let the tool guess at line-item wording all day, because a clumsy sentence costs thirty seconds to fix. The total at the bottom is a different animal. Get that wrong and you’re either under-billing yourself or asking a client to fix your math for you. Same tool, same afternoon, two different reversibility levels depending on which field you’re looking at.
Check two: is the consequence proportional to how hard I’m checking?
I turn this dial per task. Code that touches production, I turn it up high, because a broken function costs me hours and maybe a client’s trust in the work. A first draft of a Note, I turn it down, because the worst case is I rewrite three sentences before I post it.
Most people I talk to run one setting for everything instead. Some check every output like it’s a nuclear launch code, burn out by week three, and quietly stop checking anything at all. Others stop checking after the first ten outputs look clean, and the eleventh one is the one that costs a client, or ships a number that was never real. An operator automating invoices needs the dial turned up on the total and turned down on the greeting line. A non-technical builder using AI to draft a contract needs it turned up on every clause with a dollar figure in it and turned down on the formatting.
I see the opposite failure most often in people who love the tools the most. The AI enthusiast who’s tried every new model the week it ships tends to trust the newest one the hardest, because it’s smarter than last month’s, and that’s true on average and useless for any single task in front of you. A smarter model still doesn’t know that your biggest client hates being called a “partner” in an email, or that the number in row 40 of your spreadsheet came from a source you stopped trusting last quarter. That context doesn’t come bundled with more capability, no matter how new the model is.
The dial has to move task by task. Check one, reversibility, is how I decide where to set it.
Check three: did I verify it myself, or did I trust the tool’s report of itself?
This is the one nobody talks about, and it’s the one that got me yesterday.
A tool telling you it succeeded and the thing actually succeeding are two separate facts. Written down like that, it sounds obvious. It stopped feeling obvious around 7am, with eleven items queued and the fastest path being to read the word “done” in a log and move to the next one.
The fix is a habit: for anything that matters, go look at the real state of the world instead of the tool’s description of it. A newer model doesn’t buy you out of this one. The same system that publishes my LinkedIn posts arms a follow-up comment five minutes after each one goes live, and every time, it re-opens the actual post and re-reads the actual comment thread before calling the job finished. That habit has caught real gaps. A comment the log marked as posted that never landed. A note scheduled for the wrong month because a date picker assumed the wrong one. None of those showed up anywhere in a success log. All of them showed up the moment I checked the real thing instead of the report about the real thing.
This isn’t only a developer problem. If AI drafts your client email and your inbox shows “sent,” that’s the tool’s report. Whether the client actually opened it, whether the attachment actually came through: the real inbox holds those answers. The confirmation screen is designed to make you feel done. Feeling done and being done are not the same event.
Why this matters more as models get better
I run more of my work through AI models today than I did a year ago, and that number keeps climbing every quarter. This is about exactly where trust has to stop: at the line between “the tool said it worked” and “I confirmed it worked.” Everything on the generation side of that line is fast, cheap, and getting cheaper by the month. Everything on the verification side of it is still yours to do, and it’s the part that doesn’t show up in a demo video.
That’s the part last week’s piece was pointing at without naming it directly. These three questions are what judgment actually looks like in practice, asked in order, on the thing in front of you, before you ship it or believe it.
As models get better, this gets more important. A worse model fails in ways you notice. It writes an awkward sentence, or the code doesn’t compile, and the failure announces itself. A better model fails quietly. The sentence reads fine. The code compiles and passes its tests and does something subtly different from what you actually needed. The confidence of the output stops correlating with how much you should trust it, and the three checks are what fill the gap that used to be filled by “well, it looked wrong.” Nothing looks wrong anymore. That’s exactly why you still have to look.
Do this in the next ten minutes
Pick one thing an AI tool did for you today that you marked done without checking. Run it through the three questions. Was the mistake reversible if the tool got it wrong? Was the consequence proportional to how little you actually checked? Did you verify the real result, or did you verify the tool’s report of the result?
If the honest answer to that last one is “the report,” go look at the real thing right now, while it’s still fresh enough to fix.
Next week, I’ll hand over the actual checklist file I run this against.
PS. Building the specific checks and systems that keep your judgment in the loop, instead of a tool’s word for it, is what I help operators do. If that’s the muscle you want built into your own work, the details are at coaching.g8n.ai.