AI Image Detection: Evaluating the Accuracy of the Most Popular Tools

Image generation using artificial intelligence (AI) became prevalent beginning in 2022 with the advent of tools like Stable Diffusion and Midjourney. Since then, AI image generation has been on the fast track, with most graphic design platforms integrating AI either for image creation or editing.

This includes software designed for casual users and professionals alike. Canva, one of the most accessible graphic design tools (with over 16 million paid subscribers and 170 million users total), incorporates AI for image generation and editing. Meanwhile, Adobe Photoshop — the most popular program for design professionals — offers Generative Fill to allow users to quickly remove or add items from a photo.

However, the growing sophistication of AI-generated images — which makes them potentially indistinguishable from real photos — has raised significant questions about the technology’s impact on misinformation, authenticity, and creative ethics. This creates a need for accurate AI image detection tools.

At Website Planet, we help people safely navigate their digital presence with data-driven insights and research. As such, we wanted to test and compare popular AI tools to evaluate their accuracy in detecting AI-generated visual content and differentiating between AI-generated content and real digital photographs.

By providing an in-depth analysis of AI image detection tools, we hope to help creatives, tech analysts and researchers, AI enthusiasts, and the general public learn how to safely use AI while mitigating its negative effects on our digital lives.

Research Overview and Methodology

In this research, we explored the strengths and limitations of the most accessible and widely used AI image detection tools. We also looked at the significance of image metadata in improving detection accuracy.

Our experiment tested 7 AI platforms: 3 multifunctional chatbots and 4 dedicated image detection tools:

The experiment was limited to 3 chatbots (ChatGPT, Google Gemini, and Microsoft Copilot) because we wanted to test popular tools with image analysis capabilities. Other chatbots, such as Perplexity, were excluded because they lack this feature.

On the other hand, the 4 dedicated AI image detection platforms were selected off the top of Google’s search results, presuming that these tools are the most widely accessible and commonly used.

Our test used 31 digital photographs and 42 AI-generated images. We identified 6 categories and generated 1 image for each category using detailed prompts across 7 platforms:

Check out the appendix for the full breakdown of the images used in this research.

Results: Accuracy of Each AI Detection Tool

We tested the accuracy of different tools in detecting AI images (with and without metadata) and digital photographs. For the batch of images without metadata, we also resized the images to a standard 1024 x 1024 pixels.

ChatGPT

ChatGPT struggled to identify Society and Lifestyle images as having been AI-generated, getting only 3 out of 7 correct. This is likely due to this category’s typical elements of natural settings and realistic human interactions, which closely mimic real photographs.

In terms of image generators, there was only a marginal accuracy difference. ChatGPT correctly identified 5 out of 6 AI photos from over half of the generators, while the rest had perfect accuracy.

The platform’s overall identification score improved from 90% to 93% with the inclusion of metadata. It correctly recognized one additional AI-generated image under the Society and Lifestyle category after metadata was included.

Google Gemini

Google Gemini struggled with detecting AI-generated images in categories that typically include depictions of real people, including Politics and Government, History and Culture, and Society and Lifestyle. It also scored only 43% in Technology and Innovation, while it scored better in abstract categories like Nature and Environment and Fantasy and Mythology.

In February 2024, Google Gemini paused its image generation of people following outcry regarding racially biased and historically inaccurate photos. The company relaunched the feature 6 months later, but the results of this test likely reveal Gemini’s continuing limitations in properly processing and evaluating images of people.

When it came to accuracy scores per image generator, Google Gemini only correctly identified up to half of the AI-generated images. Moreover, the detection percentage only marginally improved with the addition of metadata.

Microsoft Copilot

Microsoft Copilot showed similar patterns as ChatGPT, performing well for most categories except Society and Lifestyle. Meanwhile, results were slightly more balanced when evaluated per image generator. It correctly identified 5 out of 6 AI images as having been AI-generated from 4 tools and 3 to 4 images from the other 3.

The inclusion of metadata only improved the detection accuracy for one image. Copilot was still unable to correctly identify Society and Lifestyle photos as having been AI-generated.

Notably, Microsoft Copilot only detected 5 out of 6 photos generated by Microsoft Designer.

Is It AI?

‘Is It AI?’ (isitai.com) was found to be the best-performing in terms of average accuracy, correctly identifying AI-generated images (with or without metadata) across all categories and from all generators. It also got a 100% score for real photos.

However, we must note that ‘Is It AI?’ only provides up to 15 free credits per month. We elected not to pay for more credits, as it could be seen as violating our ethical standards and prejudicing our results. Results may vary for larger datasets, but the platform’s consistent performance across categories, generators, and test parameters suggests a greater accuracy than other platforms.

*1 image tested without metadata / 2 images tested with metadata 

†1 image tested without metadata / 3 images tested with metadata 

‡No image tested without metadata / 3 images tested with metadata 

*2 images tested without metadata / 6 images tested with metadata 

†1 image tested without metadata / 6 images tested with metadata

‡1 image tested without metadata / No images tested with metadata  

§1 image tested without metadata / 2 images tested with metadata

AI or Not

AI or Not correctly identified AI-generated images in 5 out of 6 categories, only failing in Technology and Innovation. However, similar to ‘Is It AI?’, we only tried a limited number of images due to the platform’s monthly cap of 10 free tests.

When metadata was included, AI or Not performed marginally better than it had without metadata, detecting AI images across all categories.

*1 image tested without metadata / 2 images tested with metadata 

*1 image tested without metadata / 6 images tested with metadata 

†1 image tested without metadata / 3 images tested with metadata

Sightengine

Sightengine had the second-highest average accuracy of all the detector tools, but it’s arguably the most accurate across larger datasets. It only failed to correctly identify one AI image (in the Society and Lifestyle category from Freepik) and one real image.

The platform also maintained the same level of performance with and without metadata, which suggests that its algorithm and testing parameters are already optimized for the most accurate results.

AI Image Detector

AI Image Detector recorded high accuracy in detecting AI-generated images. It only missed two images — one each in the Technology and Innovation and Nature and Environment categories and from DaVinci and Freepik. When metadata was added, it matched the performance of Sightengine, accurately detecting 41 out of 42 AI-generated photos.

However, the platform’s accuracy dropped significantly when it came to real images. It was only able to correctly recognize 4 real images, misidentifying 27 as AI-generated. It had the lowest accuracy for real photo recognition and is the only platform that failed to correctly classify more than one image.

This brings to question AI Image Detector’s ability to differentiate AI-generated images from real digital photographs. Depending on the user’s need, this may make the tool less than ideal.

Overall Performance Comparison of the Tools Tested

Sightengine had a standout performance, scoring near-total accuracy across a large dataset and receiving the second-highest average accuracy. It also has the most consistent evaluations, with or without metadata. Sightengine is followed closely by ‘Is It AI?’, which recorded full accuracy across the board.

On the other hand, Google Gemini was the worst performer at detecting AI-generated images. It wasn’t able to correctly detect AI-generated photos in half of the image categories, and the inclusion of image metadata recorded only a marginal improvement (7% increase or 3 additional photos identified). Still, it was able to correctly recognize 30 out of 31 real images.

AI Image Detector’s behavior was the opposite. It scored a high accuracy percentage (95%) for AI-generated images but mistook 27 real images as AI-generated.

The platform is the only tool in our experiment that recorded such a poor score for the real-images test. This performance brings to question its ability to distinguish AI-generated images from real digital photographs, as opposed to mislabeling all images as AI-generated.

Notably, most of the AI detection tools we tested are more reliable at confirming the authenticity of real images, even without metadata. Two tools correctly identified 30 out of 31 images (97%), and the rest scored perfectly.

The inclusion of metadata in images (and retention of original image sizes) only marginally improved the detection accuracy of 5 out of 6 tools that didn’t get a perfect score in the round of tests with metadata removed. Only Sightengine didn’t see an improvement, staying at 98% across all categories.

On average, the tools had an accuracy of 83% when metadata was removed and 87% when it was retained. Given the already high accuracy score of most tools when identifying real digital photographs, and to protect the privacy of the photos’ owners, we didn’t test them with metadata.

Overall, most AI image detection tools more accurately identify authentic photos, likely due to the differences in texture, lighting, and other visual elements that may be hard for AI image generators to replicate.

Society and Lifestyle photos were the most difficult for detection tools to correctly recognize as AI-generated. The average accuracy is only 61% for images with no metadata and 63% for those with.

On the other hand, Fantasy and Mythology images were the most easily identifiable, logging an average score of 100%, with and without metadata.

While most categories saw only slight increases (2% to 4%) in detection accuracy after the inclusion of metadata, AI photos in Technology and Innovation saw a 16% hike in average scores.

When testing without metadata, AI images generated from Freepik were detected correctly only 58% of the time. That said, photos from the platform also saw the biggest increase in accuracy (+15%) when tested with metadata. This was followed by Canva with a 7% increase.

Notably, the average accuracy scores for images from DaVinci (-1%), Microsoft Designer (-3%), and Imagine.Art (-4%) dropped slightly after testing them with metadata.

The Future of AI Image Generation and Detection

Overall, advancements in AI algorithms have allowed more sophisticated tools to recognize AI-generated images with a fair amount of accuracy, but we’re yet to find a tool that can accurately differentiate AI-generated images from real digital photographs with 100% accuracy over large datasets.

Given the relative novelty of generative AI and its continuing development, a perfectly accurate detection tool may not be available in the foreseeable future. The presence of image metadata has a marginal impact on accuracy scores.

That said, there are already tools — such as Sightengine and ‘Is It AI?’ — that offer relatively reliable results. However, even these may not be accurate enough for an organization to base important decisions on.

Moving forward, AI detection programs need to keep pace with the growth and progression of generative AI to more accurately help organizations and individuals distinguish between authentic and AI-generated content.

Appendix

We generated 42 AI images using the following prompts:

We used ExifTool for metadata removal and Canva for image resizing.

We also followed a structured approach in having the AI detection tools evaluate whether our collected images were AI-generated or not. Our standardized prompt was:

“In a grid, tell me if these images are AI-generated or non-AI:

Grid: Image Name, AI-Generated or Real, Explain answer (in short).”

The uniformity in our prompts and image formats ensured that all tools were tested using the same set of parameters and made it easier to compare and analyze the results.