Technology05 Aug 2026

AI can be confidently wrong

By Alan Goh · 2 min read

Before you rely on it

Check what carries weight

  • Numbers and dates
  • Clause numbers and sources
  • Names
  • Anything leaving the company
Short answer

AI writes by predicting what fits, so it can give an answer that is specific, fluent and wrong. Treat every AI answer as a draft, and check the parts that carry weight, such as numbers, references and names, before you rely on it.

A manager asks AI to summarise a supplier contract before a renewal meeting. The summary is clear and well organised. It notes that clause 14.2 allows prices to be revised once a year, with thirty days’ notice.

The contract has no clause 14.2.

Everything else in the summary was right. That one line read exactly like the rest.

Why it happens

AI writes its answers by predicting what most likely comes next. It is remarkably good at this, which is why it drafts, summarises and explains so well.

The same ability lets it produce something that sounds right and isn’t. A clause number, a figure, a name, a reference: each is the kind of detail a good answer contains, so AI can supply one that fits the pattern. It arrives in the same calm, fluent tone as everything around it.

Detail is a style

With people, specific detail usually earns trust. Someone who quotes the exact clause has normally read the contract.

With AI, specific detail tells you how the answer was written. Whether it is true is a separate question.

Check what carries weight

Checking every word would cancel the time saved. Check the parts that would cause a problem if they were wrong.

  • Numbers: totals, prices, dates, percentages.
  • References: clause numbers, regulations, sources, quotes. Open the original and find the line.
  • Names: people, companies, products.
  • Anything that leaves the company: a reply to a customer, a figure for the bank, a submission to an authority.

One habit helps: ask AI where each key fact came from, then open that source and look for yourself.

Make the check part of the work

Every AI answer is a draft until a person has checked it.

For one-off questions, that check is a habit. For work that repeats, such as a weekly report or a standard reply, make it a step with an owner: who checks, against what, before it goes out.

Keep using it

AI remains one of the most useful tools a team can have. Draft with it freely, and check before you rely on it.

Before you use an AI answer, ask one question: which part of this would hurt if it were wrong? Check that part.

In your department

  • Accounts & Finance: Check every figure, total and date AI produces against the ledger before it reaches the bank, the auditor or a tax filing.
  • Operations & Delivery: When AI summarises a supplier contract or a delivery schedule, open the original and find every clause number and date it quotes before you act on it.
  • HR & Admin: When AI quotes the Employment Act or a policy clause, open the source and find the line before it goes into a letter.

Want to try this with your team?

Tell me what your team is working on, and I'll suggest where to start. I read and reply to every message myself.

More notes

  • 15 Jul 2026

    Brief AI the way you'd brief a person

    Search engines taught us to type a few keywords. AI works from meaning, so it gives better answers when you describe the situation and what the result is for, in plain sentences, the way you would brief a person.

    Read it →
  • 01 Jul 2026

    Build what only you can build

    AI now lets almost anyone build software, senior leaders included. For a leader, the most valuable hours usually still go into direction: deciding what should be built, and why.

    Read it →

All the notes →