Why AI Image Detectors Fail on Damage Claims

The appeal of a detector is obvious. You already have a queue of images and a problem with some of them, and a detector promises a score you can threshold. Drop it in, flag anything above the line, done.

The evidence against that plan is unusually strong right now, and it comes from sources with no product to sell you.

A detector is trained on the output of generators that already exist. A new generator is, by definition, not in that training set. This is not an implementation problem that better engineering fixes; it is the shape of the problem.

Microsoft Research’s January 2026 report Media Integrity and Authentication: Status, Directions, and Futures (arXiv 2602.18681, 56 pages, authors including Eric Horvitz, Microsoft’s Chief Scientific Officer, and Andrew Jenks, former C2PA chair) states it directly: “Because AI generators and detectors will always be in a continual ‘cat-and-mouse’ race, we cannot rely on detectors for high-confidence assurance; they cannot be 100% reliable.”

The largest out-of-the-box study to date (arXiv 2602.07814, February 2026) tested 16 detection methods across 291 generators and 2.6 million images. Mean accuracy:

Two things follow. First, the numbers people quote for detector accuracy usually come from benchmarks built on older generators, and that gap is now enormous. Second, detector rankings are unstable across datasets: there is no universal winner you can pick and stop thinking about.

This is the finding that matters most operationally, and it gets the least attention. AIGIBench (arXiv 2505.12335, NeurIPS 2025 Datasets and Benchmarks Track) re-ran detectors on images saved at JPEG quality 50. CNNDetection’s fake-detection accuracy went from 95.3% to 0.2%. SAFE went from 63.0% to near zero.

Note the direction of the failure. The detector does not throw an error or return low confidence. It starts answering “real” to everything. A queue running an automated check would show a clean dashboard and catch nothing, and nobody would receive a signal that anything had changed.

Now consider what happens to a photo in a real claims flow. A customer photographs an item on a phone, uploads it through a web form that resizes it, your CMS generates a derivative, your ticketing system attaches a compressed copy. By the time an image reaches whatever you would run a detector on, it has been re-encoded several times. Detector accuracy measured on pristine files is not the accuracy you will get.

Whole-image classifiers assume the image is either wholly real or wholly synthetic. The attack people actually run is neither.

In a documented Allianz UK case, a fraudster took a photograph of a van scraped from the victim’s own business social media, used AI to add damage to it, and submitted it with a £1,000 invoice. The base image was authentic, publicly available, and belonged to the victim. Ninety-something percent of the pixels are real photography. A classifier looking for global generative artifacts has almost nothing to work with.

At the other end of the effort spectrum, Boll & Branch received a return claim for $489 sheets with an AI-generated damage photo that still carried a visible AI watermark. Both ends of the same attack are now routine, and only the low-effort end is easy to catch.

NewsGuard’s May 2026 audit of commercial AI-image detection tools found a 13.33% false-positive rate across tools on authentic wire-service photography, with one tool at 40% (NewsGuard, 8 May 2026).

Do the arithmetic for a returns desk. Ten thousand claims a month, a genuine fraud rate of a few percent, and a 13% false-positive rate means you have flagged well over a thousand honest customers to catch a few hundred bad ones — and your reviewers, drowning, start rubber-stamping. The economics never work.

The deeper harm is the one Microsoft names: “the failures — particularly false negatives — from the highest-confidence detectors are likely to be the most trusted and, therefore, the most devastating.”

And there is the liar’s dividend. Once “that photo is AI” is a credible-sounding claim, it becomes a defence. A customer with genuine damage, flagged by your detector, is now in an argument they cannot win with evidence. An unreliable detector does not merely fail to stop fraud; it manufactures a new way to dispute the truth.

If you cannot reliably tell whether an arbitrary image is real, stop starting from an arbitrary image. Microsoft again: “The only trustable protection against claims that a real image or media file is fake is the use of a digital signature per provenance tools on the original image or media file.” The UK NCSC agrees: “AI tools designed to detect synthetic or inauthentic data can be ineffective and unreliable, meaning other technical measures are required.”

In practice, capture-side control means some combination of:

• Photos come from your app’s camera, not a gallery picker. If there is no upload path, an injected image has no way in through the normal flow.
• The capture is bound to a session, order, or job server-side. A photo that is not tied to a specific transaction is much easier to reuse.
• The timestamp is your server’s, not the device’s. Device clocks are trivially changed; a server-side created_at is not.
• The result is an immutable record with an audit log. What was submitted, when, against which policy, and what the evaluation returned.
• Reuse detection across submissions, where you have it, because recycling a genuine old photo is the cheapest attack of all.

What VerifyAI does today, precisely. Our SDK captures from the live camera inside the host app, with no gallery-picker path in the scanner. Each verification is timestamped server-side, so it does not depend on the device clock. The result is stored as an immutable verification record with an audit log. A vision-language model then evaluates the image against a policy you declare and returns a compliance decision, a confidence value, and the reasons for any violation. That is workflow-enforced live capture plus an independent server-side record. It is not cryptographic provenance, we do not describe it as such, and VerifyAI does not today perform device attestation, image hashing, tamper detection, or synthetic-image detection. Anyone selling you those four should be asked to demonstrate them.

A piece arguing “provenance replaces detection” would be vendor spin, and anyone technical would spot it. Capture-side control has real holes:

• The analogue hole. Nothing stops someone pointing a controlled camera at a high-quality screen showing a fabricated image. Capture control proves when and where a photo was taken, not that the scene in front of the lens was honest.
• You often do not control the capture. Marketplaces, third-party sellers, insurance claims from the general public, and anything arriving by email are exactly the cases where you cannot mandate an app.
• Provenance standards have their own problems. A UMBC and NSA security analysis of C2PA (IACR ePrint 2026/804) documents weaknesses in the leading standard, and Nikon suspended C2PA on the Z6 III over an authentication issue. Provenance is a supply chain, and supply chains break.
• Adoption is partial and will stay that way for years. Most images arriving in most queues today carry no provenance at all.

Which leaves a sensible ordering rather than a silver bullet: control capture where you can, layer human review at the value thresholds that justify it, and reserve detection for content that arrives with no provenance and no controlled path — used as a weak triage signal, never as an automated decision. Truepic, which sells provenance, says detection and provenance are complementary. That is the right frame, and it is also why anyone selling you either one as a complete answer is overselling.