How I built an open-source AI detector after my own writing got flagged as artificial

In 2011, I finished my PhD dissertation. Thousands of hours of research, writing, rewriting, and defending my ideas in front of a committee that seemed personally offended by my comma usage. That document sat on a university server for over a decade, bothering nobody. Then ChatGPT happened in late 2022, and suddenly everyone with an internet connection became an expert on what “sounds like AI.”

For a gag, I ran my dissertation through one of the popular commercial detectors on a whim. It came back flagged. I really didn’t think much of it until recently, as more and more folks say I sound too robotic.

Then I started questioning things. My own writing. From 2011. A full decade before large language models could string a coherent paragraph together. Apparently “likely AI-generated.” (PS – I don’t have some secret time machine either).

Fine. Maybe academic writing just looks robotic. I get it. So I tried fiction I’d written in 2019, something looser, more personal. Also flagged. At this point I was less offended than curious. What exactly were these tools measuring? Because whatever it was, it wasn’t whether a human wrote the text.

Here’s what made it worse. I was job hunting at the time. Commercial detectors wanted $30, $50, $100 a month for the privilege of being told my writing was fake. They’d give you a score with zero explanation. A number. Maybe a color. Red means bad, green means good, and you’re supposed to just trust it. No breakdown of what triggered the flag, no way to understand why, no recourse if you disagreed. Just pay up and hope the algorithm likes you today.

So I did what any reasonable person with too much free time and a grudge would do. I built my own.

GhostBusters In The Shell started as a weekend project to scratch that itch. It turned into something much bigger. A local AI detection tool that runs entirely on your machine, explains every score it gives, and costs nothing. But before I tell you how it works, let me tell you what I learned about how AI actually writes. Because once you see it, you really can’t unsee it.

What AI Writing Looks Like

I call it the Red Car Test. Someone tells you to watch for red cars on your drive home. Suddenly they’re everywhere. They were always there. You just weren’t looking.

AI writing works the same way. Once somebody points out that ChatGPT loves the word “delve,” you’ll spot it in every AI output for the rest of your life. Once you notice the construction “This is not just X, it’s Y,” you can’t miss it. These patterns were always there. We just needed to catalog them.

So I cataloged them. Over 68 detection rules across sentence, paragraph, and document tiers. Not by reading academic papers (though I did that too). By reading thousands of AI outputs and thousands of human texts and asking myself: what’s actually different here?

Some of it is vocabulary. AI has favorite words. “Delve” and “utilize” and “it’s worth noting” show up at rates that would make a human writer blush. The phrase “in today’s rapidly evolving” is practically an AI signature. These aren’t subtle tells. They’re neon signs.

But the interesting patterns go deeper than word choice. AI writes in lists of exactly three items. Always three. “Speed, efficiency, and reliability.” “Clear, concise, and engaging.” Humans do two things, or four, or five. AI does three because the training data is saturated with that rhetorical structure.

AI paragraphs follow predictable arcs. Topic sentence, supporting detail, supporting detail, conclusion. Every single time. Human paragraphs wander. They start in the middle of a thought. They end abruptly. They sometimes contain a single sentence that just sits there, doing nothing except creating rhythm.

At the document level, AI text has an eerie consistency. The vocabulary stays uniform from start to finish. The sentence lengths barely vary. A human writer might use twelve short sentences in a row during an angry passage, then shift to long, meandering constructions when they’re being contemplative. AI maintains a steady, pleasant monotony throughout. It’s the written equivalent of smooth jazz elevator: technically competent, completely soulless.

The detection engine scores all of this. Every sentence gets evaluated against every applicable rule, weighted by severity. You get a number, sure, but you also get the full breakdown. Which rules triggered, on which sentences, and why. Perhaps the most important thing about the tool is that it shows its work. No black boxes.

Your Voice Is Your Fingerprint

Here’s where it gets personal (and where I think most tools get it wrong).

Say you run your text through a detector and it comes back flagged. What do you do? Most people reach for a paraphrasing tool. “Make this sound more human.” The problem is that paraphrasing tools are also AI. You’re replacing one set of machine patterns with a slightly different set of machine patterns. The output still sounds like a robot wrote it, just a different robot.

I needed a different approach. Instead of telling a rewriter “make this sound human” (which is meaningless, since AI already thinks it sounds human), you tell it “make this sound like this specific human.”

That’s what voice profiles do. You feed the system at least 500 words of your actual writing, though 2000 words or more gives much better results. It extracts 65 or more style elements from that sample: readability metrics, sentence structure distributions, vocabulary richness scores, punctuation habits, tone markers, even your ratio of passive to active voice. Your writing fingerprint, quantified.

When the system rewrites flagged text, it doesn’t aim for generic “human-sounding” output. It aims for your voice specifically. And then it checks its own work. After every rewrite, evasion metrics calculate divergence scores using cosine similarity on sentence embeddings and n-gram overlap via fuzzy matching. These numbers tell you whether the rewrite genuinely restructured the text or just shuffled words around. If the divergence score comes back below 0.15, a second rewrite pass fires automatically with more aggressive structural changes. No babysitting required.

The result is text that reads like you wrote it, not like a machine approximation of “human writing” as a concept.

Testing Against the Best

This is the part that changed how I think about AI detection entirely.

At some point I had to face the obvious question. If my heuristics are so transparent and explainable, how do they stack up against the serious players? The ones with millions of training samples and billion-parameter neural networks?

Enter Pangram Labs. Their platform runs a Mistral NeMo classifier, roughly 12 billion parameters, trained on over 28 million documents. Their technical report (arXiv:2402.14873) describes hard negative mining with cross-model data augmentation. Two companion papers flesh out the approach: EditLens (arXiv:2510.03154, accepted at ICLR 2026) introduces a regression model scoring text on a continuous scale from fully human to fully AI, and DAMAGE (arXiv:2501.03437) tests 19 different humanizer tools and reports 93.2% true positive rate. These are not hobbyists. This is serious research.

So I tested. First, I took a voice-profiled rewrite that my own system scored at 11.5, firmly in the “clean” zone. Every heuristic said this text was fine. I submitted it to Pangram.

The verdict: 100% AI Generated.

Okay. Humbling. Maybe my heuristics missed something. But then I tried something that should have been impossible to get wrong.

I submitted the opening chapter of a novel I started writing in 2019. My own fiction, written three years before ChatGPT existed. No AI involvement whatsoever. Not edited by AI, not prompted by AI, not touched by anything more sophisticated than Microsoft Word and a glass of bourbon.

Pangram’s verdict: 100% AI Generated.

Let that land for a second. Human fiction, written years before the technology existed to generate it, flagged with absolute confidence as machine-written.

Third test. Fiction from 2005, a different style entirely. More baroque, more idiosyncratic, stranger vocabulary choices. Pangram scored it 100% Human Written. Correct.

The pattern was clear, and it’s a problem that goes way beyond one detector. Modern polished human writing occupies the same embedding space as AI-generated text, because the AI was trained on modern human text. We taught the machines to write like us, and now the detectors can’t tell us apart. Older writing with stranger rhythms and weirder vocabulary registers as human precisely because it’s unusual enough to fall outside the AI distribution. That temporal bias seems baked into every neural detector, not just Pangram’s.

(And passing it opinion pieces from the NYT and Washington Post fared no better – on average 32% of the articles came back as highly AI).

Think about what that means for everyone writing today. Students submitting essays. Job applicants writing cover letters. Journalists filing stories. Bloggers publishing posts (like this one). If you write clearly and concisely in 2025 or 2026, your prose might sit in the same statistical neighborhood as ChatGPT output. Not because you used AI, but because AI learned to sound like you.

The takeaway wasn’t that Pangram is bad. Their classifier caught real patterns in AI text that my heuristics completely missed. The takeaway was that neural detection alone isn’t enough. Neither is heuristic detection alone. You need both approaches checking each other’s blind spots; that realization drove the entire next phase of the project.

Running Detection On Your Own Machine

After getting schooled by Pangram’s classifier, the next question was obvious. Could I bring that kind of neural detection local? No cloud calls, no per-scan fees, no sending my text to someone else’s server.

Pangram made this possible by open-sourcing their EditLens RoBERTa-Large model on HuggingFace. That same model from the ICLR 2026 paper can run inference locally. I designed it as an optional sidecar container, because not everyone has the hardware or the interest to run a 1.3 GB transformer model.

The scoring now uses a triple blend when all components are available. Heuristic analysis contributes 35%, Claude AI analysis contributes 35%, and RoBERTa classification contributes the remaining 30%. (why those weightings? Because after passing it over thousands of documents in the corpus, that gave us the best discrimination and output – sound enough like a human to matter, but no so gung ho on the neural detection that you’re throwing out everything). But the system degrades gracefully. If you don’t have Claude API access, heuristics carry the full weight. Add Claude and you get a 50/50 split. Add RoBERTa and all three share the load. You’re never locked out of functionality because you lack one component.

(Quick tangent: the embeddings sidecar also enabled three new Pangram-inspired heuristics that weren’t possible before. Semantic embedding monotony, which measures mean pairwise similarity across sentence embeddings. AI text clusters tightly in embedding space. Human text scatters. That single metric turned out to be one of the most reliable signals in the whole system.)

One `docker compose up -d` gets you the full stack. Flask backend, React frontend, PostgreSQL database, and the embeddings sidecar. Everything binds to localhost. Your text never leaves your machine. The RoBERTa sidecar is opt-in via `–profile research` for those who want neural detection on top.

Beyond the web interface, there are 20 or more MCP tools for integration with AI coding assistants, plus a Chrome extension for analyzing text on any webpage. The whole thing is open source under MIT license.

I started this project because a machine told me my own writing wasn’t mine. That felt wrong in a way I couldn’t articulate at first, but I think I can now. These detection tools are making consequential decisions about people’s work, their grades, their job applications, their professional reputations. And most of them can’t explain why they flagged something. They just did. Trust us…

So here’s what I’d say. Try it yourself. It’s free. It runs on your machine. Your text stays private. And when it flags something, it tells you exactly why.

github.com/kenpodragon/GhostBustersInTheShell

For the technical deep-dive, including the full heuristic catalog and architecture details, see the research article on GitHub.