I plugged a short paragraph of pure AI slop into an open-source text detector running on my own machine and watched it come back 95 percent “human.” That one result told me more about the whole AI-detection industry than any marketing page could.
The timing mattered. Just as the internet’s trust problem was reaching a peak, Pangram’s co-founder and CEO Max Spero took the stage in a TechCrunch interview to argue that AI detection is fundamentally not a “real or fake” question. Platforms are being flooded with AI-generated job applications, product reviews, and even insurance claims, and the tools built to police that flood are forced to answer a binary question they were never shaped to answer well.
Here’s the practical side of that conversation: you can build a detector yourself — a real one, with a fine-tuned language model from Hugging Face — in under half an hour. And once you do, you’ll understand exactly why Spero is right, and why treating a probability score like a verdict is asking for trouble.
The “Real or Fake” trap
The earliest detectors used two statistics borrowed from linguistics: perplexity and burstiness. Perplexity measures how unexpected a stretch of text is. Burstiness measures how the surprise spikes and dips. AI writing tends to be steady and low-narrative-variation, so early tools could split human from AI text with claimed accuracies of 95 to 99 percent.
That sounds great until you notice the cracks. A familiar document like the Declaration of Independence scores as low perplexity to a language model, so it gets flagged as machine-written. A non-native English speaker writes more simply, which also reads as low perplexity — and gets flagged too. The approach was good in a lab and harmful in the real world.
So the modern tools dropped the heuristics and trained a deep-learning classifier instead. As WIRED described it, Pangram uses something called “synthetic mirroring” — taking real human writing and having language models generate near-matches — plus “hard negative mining” to retrain on its own false positives. The result is a probability and a boundary estimate, not a stamp. That nuance is the whole point.
What the honest benchmarks show
Independent monthly benchmarks keep landing on the same conclusion. The April 2026 edition from aidetectors.io tested ten commercial detectors against 500 texts from five different models. Average accuracy across all of them: about 85.7 percent. The gap between the best and worst for Claude-generated text was over 22 points. And the average false-positive rate on genuine human writing still hovered near nine percent.
The numbers are honest about how far this is from a solved problem. Pangram itself calibrates to keep false positives near one in ten thousand, and its CEO says that choice means real AI-written work will sometimes slip through. A Notre Dame working paper found that simply running AI text through a “humanizer” tool dropped Pangram’s detection of it below four percent. Every improvement on the detection side is met by an improvement on the evasion side.
Build your own local detector
Enough theory. Let’s actually build one. You’ll need Python and a machine with any amount of RAM — I ran this on a CPU-only laptop, no GPU required.
Step 1: Install the pieces
Install the transformers library and CPU build of PyTorch. On a Linux machine (including WSL), that looks like:
pip install transformers
pip install torch --index-url https://download.pytorch.org/whl/cpuStep 2: Load an open detector model
I used Hello-SimpleAI/chatgpt-detector-roberta, a RoBERTa base fine-tuned on the HC3 dataset — the same corpus from the 2023 paper “How Close is ChatGPT to Human Experts?” It’s roughly 500 MB and downloaded in a few seconds.
from transformers import pipeline
pipe = pipeline("text-classification",
model="Hello-SimpleAI/chatgpt-detector-roberta")On my box the model loaded in about 32 seconds. From there, every check was a single line.
Step 3: Run it on real samples
First, a paragraph I’d written myself, in my own voice:
I remember sitting in our ICT division office with a pile of paperwork that should have taken an afternoon and quietly wishing I had automated half of it years ago.
Verdict: Human, 99.85 percent. Good, that one’s obvious.
Then the test that actually matters. I wrote a deliberately robotic paragraph stuffed with the tired phrases that litter so much AI-generated marketing copy:
In today’s rapidly evolving digital landscape, leveraging cutting-edge artificial intelligence technology has become essential for organizations seeking to unlock their full potential and drive transformative growth.
Verdict: Human, 95.04 percent.
Re-read that. The model classified an obviously AI-sounding paragraph, full of the exact cliches every human reader would groan at, as human-written with high confidence. That isn’t a bug I hit by accident — it’s the reality of detection. My “gotcha” sample was short, generic, and written in the same register the training data treated as ordinary prose. The detector had every reason to be fooled.
Why it failed, and what that teaches you
The model is a 2023-era checkpoint. It was trained to recognize the ChatGPT of January 2023 — verbose, hedging, listicle-heavy output — not today’s frontier models with stronger writerly preferences. As Spero himself noted, ironically the hardest-to-detect models are the oldest ones, because they were trained to imitate the full human distribution rather than to prefer any particular style.
Two more lessons worth keeping:
- Short text is the weak spot. The detectors work on windows of a few hundred tokens. A single paragraph — exactly what gets copy-pasted into an email or a cover letter — is the hardest input to judge. Pangram openly concedes its tool gets worse under about 100 words.
- The confidence number is not a lie-detector. A score is a starting point for a human decision, not proof. It’s a probability that a classifier trained on a particular era of AI text assigned to a snippet of that era’s style. Treat it like a hint, not a verdict.
That’s the same honest framing I’ve reached covering AI models that hallucinate package names — the tools are powerful, and the trust boundary is on you.
Putting a detector to work without going overboard
So should you run one of these locally? Maybe — but only if you treat it as one signal among several, not as an enforcement hammer. A few ground rules from everything I’ve read and tested:
- Never auto-reject on a single score. The false-positive bias is real: detectors disproportionately flag non-native English writers and neurodiverse writers. A borderline number on human work damages real people.
- Cross-check with more than one tool before you believe a result. The independent benchmarks show huge disagreement between detectors on the same text.
- Remember the humanizer problem. Anyone determined to hide AI text can, in under a minute, with free tools. Detection is an arms race you cannot win by scoring alone.
- For most people, the fix is policy, not software. Decide, openly, what level of AI assistance is acceptable in a given context, and say so. Then offense — a rewrite or a pass — becomes a trust conversation rather than a forensic battle.
The same arms race plays out with images, where I dug into Google’s SynthID spotting a viral fake photo of Senator Mitch McConnell. And it’s all happening because the feed is filling with AI-generated summaries burying real links — the two problems feed each other.
Bottom line
Building my own detector was the fastest way I’ve found to internalize this. The tool worked; it loaded fast and answered instantly. And it still gave me a confident, completely wrong answer on a deliberately terrible paragraph. That’s not a reason to throw detection away, but it’s a reason to never let a single probability score make a decision about a person by itself.
Spero’s pitch is that detection belongs alongside spam filtering and fraud detection — a background signal that never fully finishes, and the platforms are starting to agree. Substack now shows readers a Pangram-driven AI score for the authors they follow. The infrastructure is coming whether we like it or not. The smart move is to understand what the numbers actually mean before you start trusting them with other people’s futures.