Why AI Checkers Give Different Scores for the Same Essay

Intro

Two browser tabs, one essay, and two very different answers. The first checker says 8 percent. The second says 61. Nothing in the text changed between pastes, you didn't edit a word, and now you're not sure which screen to believe.

Neither one is lying. They're answering the question differently, and understanding why takes the panic out of the moment. This piece explains where the gap comes from, how to compare results sensibly and what a disagreement is really telling you.

Different training, different instincts

Every checker is built from its own recipe. Some lean on perplexity, a measure of how predictable your wording is to a language model. Others weigh burstiness, the variety in sentence length and structure. Many combine these with classifiers trained on labelled examples of human and machine text, and what those examples contain shapes everything the tool notices afterwards.

Phrasly's detector page says its training draws on over 1 million real human articles, a vendor claim worth knowing about because it illustrates the point: a tool trained mostly on news articles will judge an academic essay differently from one trained on student writing. Two checkers can read the same sentence and reach different conclusions because they learned from different libraries of writing.

What the percentage means depends on the tool

Even the headline figure isn't standard. One checker reports the probability that the entire text is machine-written. Another reports the share of sentences it thinks are machine-written. A third averages sentence scores. They all print a percentage, and they aren't measuring the same thing. Thresholds differ as well. One tool may call anything above 50 percent likely machine-written while another waits until 80, so identical raw output can become opposite labels.

That's why a bare number from two sources can't be compared directly. Look for how each tool describes its score, and prefer ones that show highlighted sentences and probabilities. When you can see the sentences behind the figure, you can compare what the tools actually flagged rather than the labels they attached.

Model updates leave gaps

AI writing tools change constantly, and detectors lag behind. One 2026 explainer from Centric DXB describes the problem as distribution shift: a classifier trained on older models loses accuracy when newer ones arrive, until it is retrained. A checker updated last month and one updated last year can handle the same essay very differently. None of this makes older checkers useless. It means you should check a tool's recent update notes the way you'd check an app's release date, and treat silence as a reason for extra caution.

Coverage is a factor too. Phrasly's page names ChatGPT, Claude, Gemini, Perplexity, DeepSeek and Grok among the models it targets, and supports English, Spanish, French and German. A tool with narrower coverage may miss patterns another picks up, or flag human writing it simply doesn't understand.

The same essay, read differently

There's also the human side of the text. A 2023 study led by Stanford researchers tested seven widely used detectors on 91 TOEFL essays by non-native English speakers, and an average of 61.3 percent were wrongly labelled as AI-generated (Liang et al., Patterns, 2023). Plain, careful, predictable phrasing is what a perplexity-based tool treats as machine-like.

If your essay is written in a second language, in a heavily structured academic style or around a standard template, expect higher scores and more disagreement among tools. The disagreement isn't random. It reflects how sensitive each checker is to a style of writing that's common among honest writers. Subject matter plays in as well. Technical writing, with its fixed terms and repeated structures, gives a detector less variation to work with than a personal reflection does.

A sensible way to compare results

When two tools disagree, resist the temptation to keep pasting until you get a number you like. Use a plan instead. Don't overlook length. Paste the same full essay each time, because a checker given only your introduction is solving a different problem from one given ten pages.

• Check the same full text, with no edits between scans.

• Note how each tool defines its score.

• Compare highlighted sentences rather than headline figures.

• Mark any sentence flagged by both, since that overlap is the most useful clue.

• Rewrite those lines with something specific and original, then re-run.

Pick one AI checker as your baseline and use it consistently, so changes in the score reflect changes in your writing instead of changes in the tool. Consistency gives you a trend you can learn from, while jumping between tools gives you noise. Keep a small log if you do this often: date, tool, score and which sentences lit up. After a few essays you'll know how each tool tends to behave on your style, which is worth more than any single reading.

When disagreement is the information

Here's a point many people miss. If two reputable tools disagree sharply on your text, that's evidence of ambiguity. The writing sits in a gray area that software cannot settle, and neither number deserves to be treated as proof. In 2023 OpenAI withdrew its own AI Text Classifier after reporting a 26 percent true-positive rate and a 9 percent false-positive rate (OpenAI, 2023), a reminder that even well-resourced teams struggle to draw a firm line.

So the right response to a split verdict is to rely on your process. Keep drafts, outlines and notes, and be ready to explain your argument in your own words. If an instructor or editor asks about a score, those records speak louder than any single scan. If you must report a number to someone, report the range and name the tools.

Agreement Is Rare, and That's Useful to Know

Different tools, trained differently, built for different models and measuring slightly different things, will rarely produce matching scores. Knowing that lets you stop chasing a single correct number and start doing something better: reading the highlights, strengthening any generic sentence and keeping a record of how you wrote. A gap between two checkers isn't a crisis. It's a prompt to look more closely.

If you'd like to understand more about how detection scores are produced and where they fall short, further reading on the Phrasly blog explores these questions in more depth.