Are AI Content Detectors Accurate? A Test of Originality AI

Are AI content detectors accurate?

That’s the question I have on my mind as I delve into the world of AI writing.

As a blogger and a writer myself, I can’t help but wonder if these tools are truly effective in distinguishing between human-written content and content generated by an AI.

One such tool is Originality AI, which claims to be able to detect whether a piece of content is created by an AI or a human.

But how accurate is it? Are AI content detectors really that accurate?

To answer these questions, I decided to put Originality AI to the test.

I submitted several pieces of content, some written by me, some generated using an AI writing tool, and others mixed.

What Is an Ai Content Detector Tool and Is It Useful?

Before continuing with our test, I want to tell you what an AI content detector is.

Essentially, an AI content detector is a piece of software that analyzes written content and determines whether it was created by a human or generated by an algorithm.

One such tool is Originality AI.

Now, the big question is: can these tools accurately differentiate between a well-written article crafted by a skilled writer and one generated by a sophisticated AI writer?

The answer is not so straightforward.

After using AI content detectors for several months now I can confidently say that in some cases, AI content detectors can accurately identify machine-generated content.

For example, when I tested some of the articles produced by ChatGPT or GPT-3.5, a popular language model developed by OpenAI, I noticed that most AI content detectors would easily flag the content as machine-generated.

However, I also noticed that things get trickier when it comes to more sophisticated AI writing tools like Jasper AI or other popular AI writing tools in the market right now, that can produce high-quality content.

In fact, I have even heard that a few reputable news outlets have been publishing AI-written articles without readers realizing they were written by an algorithm.

So, I believe while AI content detector tools can be useful in identifying obvious cases of machine-generated content, they may not always be accurate in detecting more advanced forms of AI writing. And that’s the reason why we are conducting our little test to find out the accuracy of these AI content detectors.

How Is This AI Content Detector Accuracy Test Going to Work?

I’ve gathered 8 articles for this AI content detector accuracy test – 3 of these articles are AI written, 3 written by us and 2 are mixed content.

My goal is to see how accurate Originality AI is at detecting AI-generated content. I’m hoping that this experiment will give us a better understanding of whether the so-called AI content detectors are accurate or not.

Let’s see what the results say!

Here’s how the test is going to work. For AI written content I am using Jasper AI.

Why Jasper?

Because in one of my earlier AI content tests where I tested content generated by different AI tools with Originality AI, surprisingly, only one of the six AI tools that I tested was rated as human content – Jasper.

And that’s why I want to use Jasper in this test as well.

Then I am going to use 3 articles written by us.

And then I am going to mix our content with Jasper written content and create 2 more articles and test them with Originality AI. That’s a total of 8 articles.

Now, I have already prepared the 8 articles I am going to test with Originality.

So, let’s begin our testing.

Test Results – Are AI Content Detectors Accurate?

I am only going to check for AI content in the given samples and not for plagiarism.

1st Sample

The 1st sample I am going to test is the article we wrote. Let me paste it in the Originality AI tool and click ‘scan now’.

And, Originality AI scored our content as 45% original and 55% AI written. Sadly, that is completely wrong. We didn’t use any AI tools to write this article. It was completely handwritten. So the score should have been 100% original content.

2n Sample

Now, the next content I am going to test is the AI-written article. This particular article has been written 100% using Jasper’s long form editor. Let’s see what Originality AI scores for this article.

And Originality AI scored the content 51% original and 49% AI content which is again wrong.

3rd Sample

Let’s move on to the next one. This time I am going to test the content which was written around 60% by us and 40% using Jasper AI.

And again, Originality has missed the mark. I am not sure what’s happening here. It scored 4% original content and 96% AI which should have been at least 50% each.

4th Sample

Next, I am going to test the second human-written article. And Originality scored 51% original content and 49% AI content. The score should have been 100% original content.

5th Sample

Let’s move on to the next sample, which is the Ai content we created using Jasper’s short-form templates.

Let us copy and paste it into the tool.

The score should have been 100% AI-written but Originality scored it 91% human written.

6th Sample

Next, I am going to test the 2nd article created by us, and using Jasper. It should be around 50% human content and 50% AI.

Let’s see how it goes.

And the score is 96% original content and 4% AI.

7th Sample

Let’s check 2 more samples and finish this test.

Next is the article completely written by us. Let’s test.

And Originality scored the content as 98% original and 2% AI.

This is the first time Originality got the score almost right. It should have been 100% original but a 98% score is still good.

8th Sample

Now the last article we are going to test is 100% AI written.

Let’s see the score.

And the score is 100% original content, which is WRONG!

Verdict

And this concludes our test.

All I can say here is, Originality AI’s performance has been disappointing. Out of 8 articles I tested, it got only 1 right.

This is a clear indication that Originality AI still has a long way to go before it can be considered a reliable tool for detecting AI-generated content.

It is important to note that this tool is still in its early stages and there is still room for improvement.

Another takeaway from this test for me is that Jasper’s content is really good in fooling Originality.

We had already seen that in our last video, where except for Jasper all the other AI writer tools failed the AI content detection test. And this test confirms it.

I mean, don’t get me wrong, Originality AI is a great tool for detecting plagiarism and ensuring content originality. But when it comes to distinguishing between human-written and AI-generated content, it still has some way to go.

And who knows, maybe in the future we’ll see more advanced AI detectors that can accurately tell the difference. But right now, I feel they are not ready.