TL;DR: An AI detector score is a classifier output: proxy features (predictability, uniformity, stock connectors) and a threshold someone chose. Careful second-language English pushes those same features toward "AI", and a 2023 peer-reviewed study measured how much. Below: the features, the confound, the evidence, and what responsible output looks like.
Imagine a teacher sees a hypothetical high AI score on a student's essay. It reads like a measurement. It is not. It is the output of a classifier built on proxy features, with a threshold someone chose, evaluated on data that may look nothing like the essay in front of them.
This article approaches AI text detection the way you would approach any binary classifier in production: what are the features, what is the decision boundary, where are the errors, and who absorbs them. The running example is one of the best-documented failure cases: non-native English writers.
The problem framing
Formally, a detector maps a document d to a score s(d) and flags it when s(d) > τ. The true label, whether a human or a model produced the words, is never observed by the detector. It only sees the surface text.
That makes every detector a proxy classifier. It does not test for authorship; it tests for properties that happened to differ between human and machine text in whatever data it was tuned on.
The proxy features
Statistical detectors typically draw on three families of features.
Token predictability. Score the document with a reference language model. If each next token is assigned high probability, the text has low perplexity. Model-generated text, decoded toward likely continuations, tends to land here.
Distributional uniformity. Compute something per sentence (length, or per-sentence predictability) and look at its spread. Human text tends to be "bursty"; generated text is often more even.
Lexical markers. Counts of stock transitions ("Moreover", "In conclusion"), formulaic list structures and generic phrasing.
Each of these is a reasonable feature. The engineering question is what else correlates with them.
The confound: proficiency
In ML terms, second-language writing is a confounder. A learner's prose shifts all three features toward the "machine" side for reasons that have nothing to do with machines:
| Feature | Typical generated text | Careful L2 writing | Direction |
|---|---|---|---|
| Token predictability | High (common continuations) | High (safe, frequent vocabulary) | Same |
| Sentence uniformity | High | High (trusted templates) | Same |
| Stock connectors | Frequent | Frequent (taught explicitly) | Same |
A classifier trained to separate fluent native writing from model output has learned a boundary that L2 writing sits on the wrong side of. Two honest sentences make the point:
"Social media is important for young people. It has many advantages and disadvantages."
"My cousin runs her whole bakery from Instagram, which still surprises our grandmother."
The first is common words in a textbook pattern; it is easy to predict. The second is specific and personal. A predictability-based detector may well score the first as more "AI-like", though both are perfectly human.
Evidence of the distribution shift
Liang et al. (2023, Patterns) effectively ran the out-of-distribution test. Several widely used AI-text detectors were applied to 91 human-written TOEFL essays and 88 US eighth-grade essays:
- US essays: classified accurately.
- TOEFL essays: a large share of those essays wrongly flagged as AI-written.
- TOEFL essays flagged by all of those detectors had significantly lower perplexity than the rest.
The neat part is the intervention. Simplifying the vocabulary of the native essays, pushing them toward the L2 distribution, substantially increased misclassification. Manipulate the confounder, and the label moves.
The authors flag the limits honestly: a pilot study, small samples, and detectors (early 2023) mostly using an older, smaller language model as their backbone. Newer systems differ in detail; whether they still lean on the same family of signals is worth checking for any tool you use.
Ensemble disagreement as an uncertainty signal
If you treat those detectors as an ensemble, their disagreement is informative. In the Stanford data, most TOEFL essays were flagged by at least one detector, but only a minority by all of them. When independent models disagree that much, the inputs are near or across their decision boundaries: high uncertainty, presented to users as confident percentages.
Error costs are asymmetric
In most product classifiers you tune τ against the relative cost of false positives and false negatives. In education, a false positive can mean a misconduct accusation against an honest student. Base rates make this worse at scale. In August 2023 Vanderbilt University disabled the AI-detection feature of its commercial plagiarism checker, noting that even a small false-positive rate claimed by the vendor could have meant hundreds of wrongly labelled papers among those submitted in 2022.
What responsible output looks like
If a detector's job is to inform a human decision, the interface should expose the evidence and the uncertainty:
- Localised signals. Highlight which sentences drove the score so a reader can see whether it is a formulaic intro or definition, a known false-positive zone.
- Human-in-the-loop by design. Pair the score with process evidence: drafts, version history, a conversation with the writer.
That is the approach we take with the Coda One AI Detector, which highlights the sentences behind its score rather than giving only a number, and it is the framing we would apply to any detector: detection is evidence, not proof. Coda One is a free AI writing toolkit: next to the AI Detector, it offers writing tools such as a Grammar Checker. It is there to help you review your own writing before someone else reads it.
Summary
Treat a detector score like any classifier output: a function of proxy features, a chosen threshold, and a training distribution. Second-language writing is a known confounder that shifts those features toward "AI". Build and read detectors accordingly, and keep the final decision with a person who has looked at more than one number.
Over to you: if you have shipped a classifier whose mistakes land on people, how did you show its uncertainty to the people reading the output? I'd like to hear in the comments.
Sources
- Liang, W., Yuksekgonul, M., Mao, Y., Wu, E. & Zou, J. (2023). Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779 (preprint: arXiv:2304.02819). The detectors tested are not named in this article.
- Vanderbilt University, blog post announcing that the university was disabling AI detection, August 2023. As reported in: Tom Carter, "Universities ditch AI detectors over fears students could be falsely accused of cheating", Business Insider, 22 September 2023. https://www.businessinsider.com/universities-ditch-ai-detectors-over-fears-students-falsely-accused-cheating-2023-9
Drafted with AI assistance and edited by the author.
About the author: I work with the small team in London behind Coda One, a free AI writing toolkit — codaone.ai
