How to Fact-Check AI-Generated Facts

To fact-check a fact written by an AI tool, treat it as a lead, not as a source. Split it into single claims, find where each claim would have been recorded first, open that source yourself, and compare the wording number by number. If you cannot find a source, do not repeat the claim. The seven steps below are the method we use on this site before any fact is published. It works just as well for a sentence produced by a chatbot.

Why AI facts need checking

An AI language model produces text that sounds right. It is built to continue a passage in a plausible way, and it will give you a confident, well-formed sentence whether or not a source stands behind it. It may also offer a reference that does not exist, or one that exists but does not say what is claimed.

None of that makes AI useless. It makes it a starting point. On this site AI helps to draft explanations and to organize content, but it is never the source of a fact, and every fact is checked by a person against its source before it is published.

Step 1: Treat the answer as a lead

Read the AI’s answer the way a reporter reads a tip. It tells you what to look for. It does not tell you that the thing is true. Nothing is settled until you have seen a source.

Step 2: Split it into single claims

A sentence like “Saturn, the second largest planet, has 274 moons and rings 30 feet thick” contains several claims. Take them apart: the size ranking, the moon count, the thickness of the rings. Mark every number, unit, date, name and superlative such as “largest”, “first” or “only”. Those are the parts that can be wrong, and each needs its own check.

Step 3: Ask where it would be recorded first

For each claim, ask who would know first-hand. A planet’s measurements come from the space agency that measured them. A historical date comes from the archive that holds the document. An animal’s size comes from the body that studies the species. A new scientific finding comes from the published study. That is the primary source, and it is where you want to end up.

Step 4: Open the source yourself

Do not ask the AI for a link and stop there. Go to the organization’s own website, find the page, and read the sentence that supports the claim. If the AI gave you a link, open it and check that the page exists and that it says what the AI says it does.

Here is a claim that is repeated almost everywhere, with what the primary source says about it.

Step 5: Compare the wording exactly

This is where most errors show up. Put the claim next to the source and compare:

• the number and the unit;
• the date, including any “as of”;
• the scope: one country or the world, one species or all of them;
• the small words: “about”, “up to”, “may”, “at least”.

Two examples from our own collection show why this matters. The first keeps the date the source attached to its figure. The second corrects a date that many people hold with complete confidence.

Step 6: Look for a second source, and for conflicts

If your first source is not a primary one, find a second that is independent of it. Two websites that copied the same line do not count as two sources.

Expect to find disagreements. When we checked the size of the Pacific Ocean, two pages published by the same agency gave different areas: about 63 million square miles on one, about 59 million on the other. Our fact uses neither figure and keeps to what both pages agree on.

The same thing happened with the giant Martian volcano Olympus Mons. The height on the source page did not match the figures given elsewhere, so our fact describes the size of the volcano’s base and leaves the height out. A fact that says less but is certain is worth more than one that says more and might be wrong.

Step 7: Write down what you checked, and know when to stop

Record the source, the address of the page and the date you read it. If you publish the fact, show them. Then other people can check you, which is the point.

And if you cannot find a source after an honest search, stop. The claim is unconfirmed. Do not soften it, do not add “reportedly” and publish it anyway, and do not ask the AI whether it is sure. Leave it out.

The patterns to watch for

Check enough statements and the same kinds of error keep appearing:

• a famous myth, repeated because it appears in so much of what people have written;
• a correct number with the wrong unit, or a metric figure labelled as imperial;
• a figure that was right some years ago, such as a count of moons;
• a hedge that has been removed, so that “may live over 200 years” becomes “lives 200 years”;
• a source that is real but does not contain the claim.

If you would rather start from facts that have already been through these steps, our random fact generator shows one at a time, and How we verify facts describes our process in full.

Questions and answers