An AI detector developed by the University of Kansas can detect AI-generated content in academic papers with a 99% accuracy rate.
This is one of the few detectors on the market developed specifically for academic writing.
Background
OpenAI’s chatbot, ChatGPT, was opened to the public in November 2022.
It rapidly gained popularity, having surpassed the million-users mark in less than a week.
People used it for things like creating code and writing essays.
However, its popularity led to scrutiny over whether students were writing genuine papers by themselves or not.
Several schools either indefinitely or temporarily banned the software, including New York Public Schools, Seattle Public Schools and the Los Angeles Unified School District.
The process of detection
A team of researchers selected 64 perspectives (a type of article) and used them to make 128 articles using ChatGPT, which was then used to train the AI detector.
The detector could detect with 100% accuracy what were human-created articles from AI-generated ones.
It also had a 92% accuracy rate for identifying specific paragraphs within the text.
Lead author statement
“Right now, there are some pretty glaring problems with AI writing,” lead author Heather Desaire said in a statement.
“One of the biggest problems is that it assembles text from many sources and there isn’t any kind of accuracy check—it’s kind of like the game Two Truths and a Lie.”
AI detectors and accuracy
It should be noted that AI detectors have not proven to be 100% accurate.
For example, a University of California at Davis student alleged she was falsely accused of cheating with AI.
She pleaded her case and ultimately won, using time stamps to prove she wrote the paper.
The point is that this wasn’t the only time educators falsely labeled assignments as AI-generated.
Existing programs and services
There are several programs which promise to identify AI-written content.
TurnitIn: An AI detection tool for papers which shows an overall percentage of the amount of work AI software generated within the paper.
The company claims 98% accuracy at spotting AI-written work.
Copyleaks: This software can detect AI content in multiple languages, including Spanish, Russian, French, Dutch and German.
It claims to have an AI detection accuracy rate of 99%.
OpenAI’s Classifier: Though these are the same makers behind ChatGPT, the tool isn’t very accurate.
It has a success rate of around 26% and incorrectly labels human work as AI work 9% of the time.
However, it is claimed that its accuracy increases as the length of the text increases along with it.
Otherwise It’s “very unreliable” on texts with 1,000 characters or less.
The company itself warns against using this tool as the primary decision making tool and suggests using it along with other methods of detection.
