I’ve seen numerous mentions online of plagiarism checkers, whether by a writer getting feedback from a client, someone noting what’s “obviously AI” or people wondering if artificial intelligence-powered plagiarism tools are accurate.
I recently found Pangram AI, a tool designed to detect whether text has been generated by AI — some or all — in comparison to human-generated content. Pangram boasts that it is the only AI detector that outperforms human experts.
Don’t miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
Founded in 2023 by Max Spero and Bradley Emi, the tool supports multiple languages and integrates with platforms like Google Classroom and Google Chrome as an extension. Spero shared that their “ML model looks at millions of examples of AI and human writing and pulls out the key stylistic tendencies of AI-generated text.”
Pangram’s model supposedly only predicts AI if it’s sure that the text is AI. The company has also conducted various studies on this approach, including testing 60,000 news articles in one day.
Spero also shared that the process looks like comparing examples, one written by a human and a “synthetic mirror,” which is an AI-generated document that matches the human document. Lastly, they train the model over and over again to improve its accuracy.
So, I put Pangram to the AI-plagiarism test myself with three text samples.
How Pangram works
Pangram is set up so you can try it out for free, which provides five credits per day — one credit per 500 words — before you need to upgrade to a paid account, which starts at $12.50/month.
When you sign up for an account, you’ll be prompted with a guide on how the platform works: Enter or upload text (up to 500 words) and in response, you are given a score with areas noted that may or may not be AI-written. You can learn more about Pangram’s full process on its blog.
I tested this in three ways:
- Uploading an article where I had used AI to sort out direct quotes.
- Uploading a fully AI-written draft.
- Uploading a published article I wrote that I then asked AI to refine for me.
My hope was to see how the tool went about finding plagiarism and comparing it to my three text samples, since I understood the reality of how much AI-written content they included.
My first attempt came back at 13% AI use. When I scrolled down to where it found the seemingly most “AI plagiarized content,” I found that it flagged writing I felt was most unique to me, which was odd.
But I also noticed it was where a couple of direct quotes from a human-written document lived. Since I had AI help me sort and place these direct quotes out of a transcript, I wonder if that’s what helped trigger this on Pangram’s end, but I couldn’t be sure.
Pangram said it was confident that 13% of this writing sample was AI-written. In reality, it was fully written by me, except that I had asked an AI tool to sort through human quotes to include in it.
My next attempt was a purely AI document — with a caveat. I first prompted ChatGPT to explain how I write based on our previous conversations. It answered, “conversational, reflective, a little cheeky but thoughtful.”
Then, I asked ChatGPT to write a 500-word article on technology in “my voice.” I uploaded my results to Pangram and was hit with a note that said Pangram was “strongly confident this document contains AI-generated writing,” at 99.9%. Accurate, yes. But I looked more closely at why it noted my writing as AI output.
Pangram’s results stated that words like “creativity,” “story” and “narrative” show up 10-20x more often in AI-written output. (Are those not some of the most basic writing-adjacent words used in everyday written and verbal communication?) So while Pangram was correct in that my 500-word output was AI, its context didn’t really match reality.
I asked ChatGPT to write a 500-word article on technology in “my voice,” and Pangram AI picked up that it was 99.9% likely to be AI-written.
(Disclosure: Ziff Davis, CNET’s parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
Lastly, I uploaded 500 words of a published article into ChatGPT and asked it to refine it for me, then I uploaded the resulting text into Pangram. This was also labeled as an almost fully AI response, at 99.3% certainty. But Pangram again pointed to phrases and words that were either part of everyday speech or part of the original, human-written article that the AI-rewrite decided to keep. (This included part of the company’s own slogan.)
Pangram pegged this writing sample as being 99.3% likely to be AI-written. It was written by me but “refined” by ChatGPT before I uploaded it into Pangram.
I can’t help but wonder if Pangram’s platform is triggered by copy written by AI vs. words that an AI chatbot is likely to often spit out. Take my third example — there were still many remnants of my original article woven into the AI response, but Pangram labeled the whole thing as being AI-written. Although 100% of my Pangram upload came from ChatGPT, ChatGPT’s output wasn’t 100% written by ChatGPT — it was just refined there. Do you see what I mean?
Should you try Pangram?
This answer really depends on who is using Pangram, why they are using it and whether they double-check its work before making final judgment calls. I believe Pangram can find AI-generated output in bulk and also be completely inaccurate around what equates to or is similar to AI-generated output. This inevitably leads to diagnosing seemingly AI-generated writing based on pattern recognition rather than context.
Pangram aims to solve the problem that “human content moderation is an expensive and time-consuming task.” But hear me out: Using the platform still comes back to you, as a human, to make your best judgment on whether someone’s writing is AI-plagiarized.
For example, questioning whether someone’s writing seems a bit more formal than usual… or if it’s unlikely someone spells “resume” as “résumé” multiple times across a 1,000-word article.
Writing is such a sensitive, subjective art form that, in my opinion, no technical software can truly understand it.
Furthermore, data is subjective. So when creating a platform that aims to separate writing into extreme binaries, we end up missing the point of writing and the true abilities of writers.
While I can respect Pangram’s attempt to dismantle misinformation as a business model, I see a need for a nuanced approach rather than large, blanketed statements made from data analysis that lack the subjective nature that any type of writing, inevitably, creates.
