I want to talk about the new A.I. feature Substack has inserted into it’s platform, because there are some very real, very problematic things happening in A.I. detectors, specifically with Pangram, the A.I. detector Substack uses.
First, I wanted to put it to the test. So, I took a piece I wrote in 1997, called, The War Within, describing my unhealed childhood trauma and how it was going to metastasize into my early adulthood, and I ran it through Pangram.
The results came back 52% A.I. drafted - not assissted... DRAFTED. That’s impossible.
So, I began running other pieces through it. Some from the New York Times, The Intercept, and The Guardian. Every piece came back with results of either A.I. assisted or a percentage of A.I. drafted. But they weren’t. How can A.I. possibly draft breaking news? So, I dug deeper and found several other examples how unreliable A.I detection got a novel pulled from a major publisher before release this year after Pangram flagged it as A.I.-generated, a Modern Love column went viral after getting flagged, the Pope’s own speech was flagged as 46% A.I.-generated and even The United States Constitution was flagged. One of the biggest issues I have with “A.I.” is that there is no such thing as artificial intelligence. If something is artificial, how can it be intelligent? And if something is intelligent, how can it be artificial? What we call A.I. is actually machine learning. And when it comes to “breaking news” a machine cannot draft what it has not learned yet. So, what in the hell was going on here?
Pangram is a probabilistic classifier, which means it works in terms of likelihood rather than absolute judgments. It looks for stylistic signals that tries to separate human writing from A.I, such as how uniform the sentence lengths are, how predictable the word choices feel, and how steady or “flat” the rhythm of the prose is, then it makes a calculated guess based off those criteria. A.I. detectors even consider copy and pasting as A.I. assisted or drafted, because they also use individual key logging as a measurement tool as well. In other words, if you write your draft in Word, then copy and paste it to Substack, it could categorize that as A.I. assisted or drafted.
It pre-declares that real human writing shows more uneven pacing and structure with some sentences short, some long, and occasional quirky phrases, whereas machine learning output often has smoother, more predictable patterns. Pangram also pays attention to generic, overused transitional phrases that show up disproportionately in A.I.-generated text. Machine learning is pattern-matching at an absurd scale.
Instead of declaring “this is A.I.” or “this is human,” Pangram assigns a percentage score that reflects how confident it is that the text was authored by a person vs created or heavily assisted by an A.I system.
The issue is that these detectors can and do misfire, often and repeatedly. They sometimes flag entirely human work as A.I-generated simply because the writing is clean, well-organized, or stays in a steady, professional tone from start to finish.
In other words, the very qualities people are encouraged to aim for in good writing — clarity, structure, and a coherent voice — are often the same signals that cause some tools to wrongly suspect A.I. involvement.
The skepticism around Pangram isn’t just hearsay-based. The Atlantic has highlighted its accuracy limits, and even Substack’s own announcement about the tool acknowledged that it doesn’t guarantee correctness. They ensured to maintain a legal disclosure because when you put out a shit product, it’s important to make sure nobody can come back and sue you. The real pain point is that heavily edited or unusually polished human prose, exactly the kind many professionals and careful writers produce, gets flagged far more often than it should.
In practice, this means clean, well-structured writing can look “too smooth” to these detectors, triggering false positives even when zero machine learning was involved. For founders and teams building or relying on A.I. compliance tools, that gap between claimed performance and real-world behavior is a core accountability issue.
The A.I. Smear Campaign
Over the last couple of days, since Substack rolled out this bullshit, I have seen countless creators praising the product because, “now we will see which creators are just using A.I. slop.” Hang on. Pump the brakes.
Let’s be so real about what actually just launched. I’m not just claiming that Substack’s tool is flawed, but it’s important to understand that Substack outsourced a trust-and-safety function to a small, VC-funded startup with less than 20 employees and let it become the referee of whether your writing counts as your own.
Substack’s CEO, Chris Best, keeps telling us it’s a badge of transparency, and not so much a warning label. That he didn’t want Substack turning into LinkedIn before fixing the problem, so he launched what he calls, tools to make things “clearer.” Creators can write a “how I make this” disclosure statement. They can also switch off the scan for their own posts if they’d rather not participate. Which, when writers do this, they immediately get persecuted of faking the funk anyway. So, it’s a lose-lose situation with the critics who constantly complain about A.I. on this app.
Personally, I disabled it. Be mad. IDGAF.
But that’s Chris’s whole pitch. “Transparency. Trust. Protecting the humans.” A little grit, a smidgen of edge, and a dash of conscience. I find that real fuckin rich coming from a guy who helped build a company that built its entire brand identity on being hands-off about who gets to publish and profit, invoking free speech and open discourse as the north star, while simultaneously building and shipping a machine that scrutinizes incredibly talented writers’ word choices for the crime of possibly using a tool to help draft a sentence. Nazis, rapists and a plethora of hate groups get a philosophy of restraint. And hard-working writers get a confidence score from a 20-person startup… Got it.
If a paid subscription isn’t in the cards right now, I completely get it — truly. But if you’ve ever read something here and thought “damn, she needs to keep doing this,” you can throw a few bucks in the tip jar over at Buy Me a Coffee.
No pressure. No guilt. Just me, asking nicely, with my hand out like a very earnest raccoon who writes controversial essays.
☕ buymeacoffee.com/eeviebateman
Thank you for being here. Seriously. You make this worth doing.
~Eevie
It's absolutely ludicrous that we think we could use a machine to tell if another machine was responsible for creating a thing.
So glad you’re talking about this shit. Let’s expand.