Scanning for Pangram errors

I scanned over 2 million words into Pangram looking for false positives

Pangram is a new AI detection tool that has made waves in various reader circles for catching literary prize winners using AI generated prose. Pangram claims it can help its users detect AI generated writing. AI detection isn’t new, it has been around as long as AI. Unlike previous generations of detection tools though, Pangram boasts a false positive rate of 1 in a 10,000. That is, if Pangram says a block of text is AI there is only a one in ten thousand chance that it was written by a human.

I run a small romance book community. Right now many niche book communities are facing an onslaught of unlabeled AI works, mine included. To solve this problem, I turned to Pangram to identify works as AI written so that I could focus my moderation effort on those works.

Pangram isn’t uncontroversial, and many online doubt its accuracy claims. The public has been burned by overconfident claims by AI detection vendors before¹. I have approached detection with a dose of skepticism myself, and so I tested Pangram in a number of ways.

I ran a large number of stories published to RoyalRoad² through Pangram, and found the tool did not detect any AI on stories before the release of ChatGPT. This is what we expect to find in a functioning detector, and is an encouraging signal that Pangram doesn’t needlessly flag human writing. I plan to write up my RoyalRoad results later, but that isn’t the focus of this post. In this post I test a different aspect of Pangram: memorization.

Pangram is a machine learning system, and one concern with machine learning systems is that they might “memorize” their training set. When training a machine learning model like Pangram, the scientists and engineers give the model many labeled examples of AI and human writing, and the hope is that the model learns what AI writing “looks like” if you give it enough examples. But there is the risk Pangram is simply remembering every example from training and regurgitating its answers, rather than learning to spot AI in the wild.

Concerns around memorization can be found in online discussions of Pangram. When discussing the Commonwealth prize, one redditor pointed out that the work of previous prize winners didn’t hit as AI written while the 2026 winner did. Another commenter responded.

The Commonwealth Prize is fairly well known. Pangram has almost definitely trained on it and all the other major literary prizes. It would be absolutely insane not to, and these aren't stupid people.

The implication is that it is possible Pangram has memorized previous prize winning works, and knows not to label them as AI written. Could Pangram have simply memorized every story published to RoyalRoad too?

In the very worst possible case, Pangram’s algorithm could be simply be “Label as ‘Human’ for works we know are published before ChatGPT, otherwise label it as ‘AI’.” Most works were written before AI existed, so Pangram could feasibly achieve a 1 in 10,000 false positive rate by performing well on old memorized works.

Fortunately there is an easy way to test this; by providing examples that Pangram has never seen before! The easy way to do this is by feeding Pangram your own writing. But I wanted to try something different.

I found old books that didn’t exist in digital form on the internet, they had no ebook on Amazon nor on archive.org, I scanned them and then sent them through Pangram. Even if Pangram was memorizing old books from its training data, it would have missed these works as they were never digitized. In that case we should expect to see very high false positive rates on these old books.

Ebay is filled with lots of old books and magazines that never made it into digital formats. These works are past their commercial life and largely forgotten. I bought a few of these old forgotten books, disbound them, then fed them in to a sheet fed scanner, and then converted them into text using OCR. In total I scanned forty-five books and then sent then sent their entire contents through Pangram.

Doing surgery on old books. Double Duty Nurse with its spine removed. The pages are loose and ready to be fed through an ADF scanner.

The process turned out to be very simple, only requiring a box cutter and metal ruler to guide the knife. The repetitive motions of slicing out the glue and spine became a sort of meditative process for me, allowing me time to think or listen to Dungeon Crawler Carl as my hands worked.

A summary of the results are below, you can find a more complete table at the end:

The dataset has 2,883,616 words and Pangram broke them up into into 8,029 segments for an average segment length of 359 words. Of the segments, the vast majority are labeled as human. But you might notice that there are some AI segments. There appear to be three “AI assisted” segments and one “AI segment”. Does that mean Pangram made errors?

In this case, something interesting happened! My scanner produces an image of each page, and Pangram requires text. To convert image to text I used Mistral’s OCR service. Mistral OCR is itself AI, and in this case it Mistral hallucinated entire tables onto blank pages when they didn’t exist!

Below you can see that page 2’s OCRed text of Double Duty Nurse contains a table in Chinese, when the original image shows the page is blank.

Pangram saw this table and was able to determine that Mistral’s AI had erroneously added the table in to the book. In fact, all of the “AI” grades that Pangram gave came from Mistral hallucinations.

In this case Pangram did what it was designed to do. But that it did highlights that you should always have a human review Pangram results if you want to use it for anything important. Human review is something the Pangram team recommends as well.

Overall, the results give me confidence Pangram is not memorizing its training set.

If I’m honest I’m a little disappointed I didn’t find a false positive for my efforts. But 1 in 10,000 is quite rare. Consider that with Pangram’s 1/10,000 false positive rate, if an author wrote as many words as Stephen King³ they would expect to have three segments come up as “AI” in their entire career⁴.

---

1. In one case, ZeroGPT flagged the Texas Declaration of Independence as AI written

2. RoyalRoad is a large fiction serialized fiction website where authors share their works. It has existed in its current form since at least 2018.

3. King has stated that he tries to write 2,000 words a day, and is one of the most prolific America authors.

4. Estimates put King’s word count at around 11 million. This website has his word count at ~11,400,000. If a segment is 359 words on average, that means you would expect ~32,000 segments from Pangram for that word count.