not.bot

Why not.bot: Deepfakes and Inauthentic Content

Updated

This document opens the Why not.bot series, which presents the public evidence for what not.bot exists to solve. This one covers fabricated content: cloned voices, synthetic video, and the theft of a known person's credibility at scale. Every figure below carries its source and date. The incidents are on the public record.

Making a fake costs almost nothing

Researchers at the Oxford Internet Institute analyzed 35,000 deepfake image generators available for public download. Building a new one requires as few as 20 photographs of the target, 24 GB of video memory, and about 15 minutes on a consumer computer ("Deepfakes on Demand," FAccT 2025). Microsoft Research demonstrated in January 2023 that its VALL-E system synthesizes personalized speech from a 3-second recording of a voice it has never heard. OpenAI built a tool that clones a voice from 15 seconds of audio, then declined to release it, judging the risk of misuse too high (March 2024). When the company that built the tool will not ship it, the threat assessment is settled.

The capability is cheap enough to demonstrate as theater. At Berkshire Hathaway's May 2026 shareholder event, Greg Abel played a video deepfake of Warren Buffett with Buffett sitting in the audience. "That was done with zero input from Warren," Abel told shareholders. "We were able to obtain that with information that's out there and replicate those actions and that voice" (Money, May 6, 2026).

Documented incidents

A US$25.6 million wire through a deepfaked video call. In January 2024, a finance employee at Arup, the British engineering firm, joined a video call with the company's CFO and several colleagues. Every other participant on the call was a deepfake recreation of a real coworker. The employee made 15 transfers totaling HK$200 million (US$25.6 million) to five bank accounts (CNN, February 4 and May 16, 2024; South China Morning Post, May 2024). The funds were not recovered.

Voice clones against bank call centers. In spring 2023, a software-generated copy of a Florida investor's voice called his Bank of America representative and tried to redirect a large transfer (The New York Times, August 30, 2023). The banker caught it. The damage arrived anyway: she stopped trusting calls and emails from the real customer, and it took ten days and an in-person visit to restore the relationship. A fraud that fails still breaks the trust the relationship ran on.

A coordinated campaign impersonating real doctors. The New York Times documented (September 5, 2025) a global operation that hijacked the likenesses of physicians at UCSF, Stanford, Harvard, and other institutions, using AI video and voice to sell unproven health products. One Stanford scientist's face fronted at least six YouTube channels carrying hundreds of AI-narrated videos. One physician's fake was convincing enough that her own mother believed it.

A single fake ad with 35 million views. CNN's chief medical correspondent Sanjay Gupta described fake videos of himself selling cures for Alzheimer's disease and diabetes (CNN Podcasts, September 23, 2025). One had been viewed 35 million times. His own former professor was fooled. Victims paid hundreds of dollars for products that never arrived, then wrote to the real Gupta in anger and shame.

An €830,000 romance scam. A French interior designer lost €830,000 over 18 months to scammers using AI-generated images and messages of Brad Pitt (TF1, January 2025; Euronews, January 15, 2025). When her story aired she faced a wave of public ridicule, and the broadcaster withdrew the interview to protect her.

Sexualized fakes of a head of government. AI-generated lingerie images of Italian Prime Minister Giorgia Meloni circulated as authentic in May 2026 and drew public criticism of her before they were exposed as fakes (Forbes, May 6, 2026). Meloni's response named the asymmetry: "Deepfakes are a dangerous tool, because they can deceive, manipulate and strike anyone. I can defend myself. Many others cannot."

Fabricated endorsements from active footballers. The Guardian documented (July 5, 2026) two unlicensed betting operators presenting famous players as brand partners. Nightwin bought Instagram advertising carrying a made-up BBC story announcing that Jude Bellingham had launched "Bellingham Bet," with a logo built from the stylized signature his sponsor uses; the app it led to displayed a fabricated 4.9 rating and 1.9 million downloads. QH88 went further with Bruno Fernandes, building a site around a fictitious partnership and producing an AI-generated video of the player signing an ambassadorial contract at Old Trafford. Both operators run from offshore jurisdictions behind shell companies whose owners are shielded by local law. The Guardian's assessment of the remedy on offer: "Cease-and-desist letters will be ignored. Legal action? Against whom? You can't sue ghosts."

A regulator warning about deepfaked financial journalists. South Africa's Financial Sector Conduct Authority warned the public on June 5, 2026 about a Facebook video that used AI to impersonate the personal-finance journalist Maya Fisher-French and the broadcaster Bruce Whitfield, promoting an investment in "Quantum AI," a company the video attributed to Elon Musk, at returns of R60,000 a week on a R4,200 stake. Both journalists confirmed the video was a deepfake and that they had no association with it. The regulator added that the people behind it may be providing financial services without authorization.

The scale

  • Deloitte's Center for Financial Services projects that generative AI will drive US fraud losses from US$12.3 billion in 2023 to US$40 billion by 2027 under its aggressive adoption scenario, and cites a 700% increase in fintech deepfake incidents during 2023 (May 2024).
  • The UK Home Office states that an estimated 8 million deepfakes were shared in 2025, up from 500,000 in 2023 (February 2026).
  • Scams of all kinds cost Americans US$158 billion a year, a figure that prompted the first coordinated US national anti-scam strategy (Aspen Institute National Task Force on Fraud and Scam Prevention, September 30, 2025).
  • Of 2,000 deepfake generator models analyzed in the Oxford study, 96% target identifiable women without any suggestion of consent, and 99% of sexual deepfakes target women and girls (FAccT 2025, citing Internet Matters, 2024).
  • The threat-intelligence firm Revelum counted at least 10,000 deepfake scam ads impersonating footballers worldwide over twelve months, and confirmed 2,736 across six players it analyzed in depth: 1,300 or more for Cristiano Ronaldo, 450 or more for Lionel Messi, 164 unique videos for Luis Díaz, 142 in a single campaign against Moisés Caicedo, and 57 for James Rodríguez (July 2, 2026). The 2026 World Cup accelerated it from June 11: Neymar's daily rate rose 1,700%, Luis Díaz's 413%, Caicedo's 70%, Ronaldo's 69%, and James Rodríguez's 41%. Messi's fell 66%, having peaked earlier. The ads sold gambling apps and investment schemes, and targeting concentrated on Spain, Argentina, Colombia, Costa Rica, Ecuador, and Brazil.
  • FINRA's Annual Regulatory Oversight Report describes investment club scams in which bad actors "post fraudulent social media advertisements, often using the likeness of well-known finance personalities or financial professionals unaffiliated with the scam, to direct victims to purported 'investment clubs' on encrypted messaging applications," then pump and dump thinly traded securities. The same report names deepfake selfies as a way fraudsters defeat the selfie checks firms use to verify customers (December 2025).
  • UK Finance recorded £1.28 billion in UK payment fraud losses in 2025, of which authorized push payment fraud accounted for £576.4 million across 248,070 cases, up 19% in a year. Investment fraud was the largest single category at £221.5 million, up 40%, and two-thirds of authorized push payment fraud originated online (Annual Fraud Report 2026).
  • The World Economic Forum's Cybercrime Atlas surveyed 25 face-swap and camera-injection tools selling for US$10 to US$3,000 and documented the chain that defeats remote identity checks: AI-generated identity documents, face-swapped video matching those documents, and camera injection feeding the synthetic video into live biometric verification (January 2026). Its summary: "identity itself has become synthetic, scalable and weaponizable."

Detection loses the arms race

Detection fails in six separate ways, and each has a documented case behind it.

Technical: generators train against detectors. Deloitte describes deepfake systems built to keep checking and updating their ability to fool computer-based detection, and notes that for audio, "the technology industry is behind in developing tools to identify fake content" (May 2024). The World Economic Forum reaches the same end point: deepfake fraud "may never be fully eliminated," only "contained, deterred and made economically unviable" (January 2026).

Durability: an invisible mark does not survive ordinary editing. Meta released Content Seal on July 7, 2026, an invisible watermark carried by images and video from its Muse models, with a public page where anyone can check a file. Reuters tested it three days later. The tool identified all 40 unmodified Muse images and missed 55% of the same images after they were cropped to between a third and a half of their original size (Reuters, July 10, 2026). Meta's response was that the tool is a preview, that the watermark is built to hold through common edits, and that the signal can be lost when an image is heavily cropped. A crop to a third is not what most people would call heavy. This is a harder failure than metadata stripping, because the mark survived stripping and still did not survive a crop, on its own maker's images, under independent test.

Operational: takedown does not scale. In the doctor-impersonation campaign, a Stanford team spent hours reporting fake videos and posting warnings under them; the warnings were deleted within a minute. One impersonated oncologist, after her reports and a legal letter went unanswered, paid US$260 to a takedown service that failed. The platform said it was unaware of the fakes until a reporter called (The New York Times, September 5, 2025). Measure any takedown clock against a fake that reached 35 million viewers.

Latency: detection arrives after the audience. Between April 19 and 21, 2026, an AI-generated video showing two men helping President Trump out of the Walter Reed National Military Medical Center spread across Facebook. The detectors had everything they needed. PolitiFact ran the clip through Google Gemini, which found SynthID, the watermark Google embeds in output from its own AI tools. The hospital's communications office confirmed the logo was not theirs and the signage did not match the hospital's. Zooming in showed the text on the sign rendering as gibberish. PolitiFact rated it False and published on April 21 (PolitiFact, April 21, 2026). The earliest post it found was from April 19, so the correction landed two days into the video's run, across a weekend, on a claim about the president's health. No one published a figure for how far it travelled in those two days. The detection was right, and it ran downstream of distribution, which is where it always runs.

Disclosure: a label does not stop the harm. In late February 2026 the White House TikTok account posted an AI-altered clip of Ottawa Senators captain Brady Tkachuk appearing to insult Canadians in vulgar terms after the Olympic gold-medal game. The clip carried a "Contains AI-generated media" label and passed 12 million views (Sportico, March 2026). Tkachuk's answer was that it was "clearly fake, because it's not my voice, not my lips moving," and that he was "not in control" of the account that posted it and "can't do anything about" the clip. A label reaches whoever happens to see it, and a label is not proof.

Epistemic: a correct finding does not settle the question. In early July 2026, as speculation spread that Senator Mitch McConnell was dead or dying, his office published a photograph of him beside his wife Elaine Chao on his Senate homepage and his official Facebook page. X's chatbot Grok told users the photograph was AI-generated, that it carried a SynthID watermark from Gemini or OpenAI, that fact-checkers including the Cincinnati Enquirer had confirmed this using OpenAI's verification tool, and that McConnell's office had released only a text statement. Every one of those claims was false, and the Enquirer had been covering a different fake image (Snopes, July 13, 2026).

Snopes then ran the checks Grok had invented. Google's Gemini found no SynthID. OpenAI's verification tool found no SynthID and no C2PA manifest. Four detection tools split between no AI and a low probability of AI, and Snopes noted in the same report that detection software is imperfect and its results deserve skepticism. Two forensics researchers, Matthew Stamm at Drexel University and Hany Farid at GetReal Security, examined pixels, faces, lighting, and shadows, and found no evidence of AI generation (PolitiFact and Poynter, July 13, 2026). The date came from the newspaper in McConnell's right hand: the sports section of the July 12, 2026 Washington Post, with "Noskova claims title in all-Czech showdown" legible beside his fingers.

Snopes left the claim unrated.

Every one of these produces, at best, a negative finding about a fake. None produces a positive claim about an authentic item, and a negative finding cannot be checked by the audience without trusting whoever reported it. "No watermark found" is not evidence of authenticity, because almost no authentic photograph carries a watermark and plenty of fakes carry none either. Walter Reed and McConnell are the two ends of this. Both were checked with the same instrument, Google's SynthID detector, three months apart. It found the watermark in one and found none in the other. Finding it settled the Walter Reed video, two days after the video started circulating. Finding nothing settled nothing, because no watermark is what an authentic photograph and an unwatermarked fake both look like. That is why the McConnell claim stayed unrated after two forensics researchers found no evidence of AI generation and the date sat legible in his own hand. His office was left arguing for its own picture, after the fact, through intermediaries, while an AI system asserted the opposite with a fabricated citation. Detection asks whether content looks fake, and that question gets harder with every model generation. The question that stays answerable is whether content carries proof of authenticity.

Regulation arrives, and creates demand for verification

  • The US TAKE IT DOWN Act, signed May 19, 2025, requires platforms to remove non-consensual intimate imagery, including AI "digital forgeries," within 48 hours of a valid request. FTC enforcement began May 2026, with penalties up to US$53,088 per violation.
  • Italy's Law 132/2025, in force October 10, 2025, is the first comprehensive national AI law in the EU. It criminalizes disseminating, without consent, AI-altered images, video, or voice capable of misleading as to authenticity and causing unjust harm, with penalties of one to five years.
  • 46 US states have laws addressing non-consensual intimate deepfakes, and 31 regulate election deepfakes (Public Citizen tracker, July 2026). The limits are visible too: a federal court struck down Hawaii's election-deepfake law on First Amendment grounds (D. Haw., January 30, 2026).
  • EU AI Act Article 50 applies from August 2, 2026, and deployers must disclose deepfakes from that date. The obligation on providers to mark synthetic content in machine-readable form moved to December 2, 2026 for systems placed on the market before August 2, under the amendments EU legislators agreed on May 7, 2026. Of the duties this article creates, the one closest to a working instrument is the one that slipped.
  • ETSI TS 119 461, the EU identity-proofing standard under eIDAS 2.0, names "deep fakes" as a technique imposters use and requires accredited-lab testing against injection and presentation attacks before the end of 2026 (February 2025).
  • Denmark has proposed copyright-style rights in a person's own face, voice, and body, the first proposal of its kind in Europe (June 2025; before parliament as of June 2026).

These laws assign duties: remove within 48 hours, disclose synthetic content, test identity checks against injection attacks. None of them can prove a given piece of content authentic. Regulation raises the cost of fakery and creates compliance demand for verification infrastructure. The authenticity question stays open.

Who bears the cost

Financial institutions. Voice clones target call centers and the fraud trajectory points at US$40 billion by 2027 (Deloitte, May 2024). The losses come with liability fights: "customer relationships may be tested when determining whether a fraud loss is to be borne by customers or their financial institutions" (Deloitte, May 2024). A Federal Reserve survey of more than 360 US institutions found scams the most common fraud type and concern about mule-account activity up 12 points in a year (2024).

Enterprises. The Arup pattern is repeatable: contact through a messaging app, an urgent video call, a deepfaked executive, a wire. The identity checks that gate remote onboarding are themselves a target; the WEF documents the bypass chain end to end (January 2026).

Public figures, executives, and clinicians. Anyone whose voice and face exist in abundance online is raw material. The cost lands as reputation damage, lost audience trust, and takedown labor that does not end.

Athletes, and student-athletes most of all. A single viral fake endorsement costs sponsorship opportunities and standing with a team, and the betting-operator campaigns documented above put active players in breach-shaped situations they never chose. Student-athletes are the acute case, because their marketability is early and thin and they hold no reservoir of public trust to spend, so brief exposure carries lasting consequences. The remedy is slow and arrives after the fact: courts have set no uniform standard for when a synthetic likeness becomes actionable misappropriation, no federal statute governs name, image, and likeness, and claimants are left with right-of-publicity and Lanham Act false-endorsement theories. Some states are moving on their own, and Florida strengthened § 836.13 to cover AI deepfakes and nudify tools.

Financial influencers and creators. Credibility is the asset, and credibility is what gets stolen. The attacker here is not evading a check. They are borrowing a real person's audience, which is why a system that proves a human is present does nothing about it. FINRA describes the pattern from the regulator's side, and the FSCA warning names two journalists whose reputations were the product being sold.

Women. 96% of deepfake generator models target identifiable women (Oxford Internet Institute, FAccT 2025). Non-consensual intimate imagery is the largest single category of deepfake harm, and the reason most of the new criminal statutes exist.

Platforms. Statutory removal duties now carry per-violation penalties and active FTC enforcement, against a moderation problem that defeats both automated detection and manual review.

Consumers. Romance scams, fake cures, products that never arrive. Victims report shame more than anger, and shame suppresses reporting, so official figures undercount the harm.

The deepest cost: authentic content loses its authority

A fake does damage twice. The first harm lands on whoever it defrauds or defames. The second lands on everything authentic, because every real recording now competes with the possibility that it is fake. In the documented incidents above, the people fooled included a physician's mother and a neurosurgeon's former professor: people with personal knowledge of the target. Gupta's conclusion from inside the problem: "all content, even good content, ultimately gets tainted by this... Nobody believes anything. Everybody's suspicious of each other. And unless you can touch the person or talk to them directly, they are also suspect, because it could be an AI impersonator" (CNN Podcasts, September 23, 2025).

Meloni offered the public a rule: "verify before believing, and believe before sharing" (Forbes, May 6, 2026). The rule is correct. Today the ordinary reader has no way to follow it.

Nobody had the instrument

Walter Keane signed and sold his wife Margaret Keane's "big eyes" paintings as his own for years. Who had painted them was settled in a federal courtroom in Honolulu in 1986, in a libel suit Margaret brought against Walter and the publisher of USA Today over an article naming him as the artist. The judge asked both of them to paint a big-eyed child in court. Margaret finished hers in 53 minutes. Walter declined, saying his shoulder hurt too much to paint. After a three-week trial the jury awarded her US$4 million. On appeal the Ninth Circuit affirmed in part and reversed in part, and the award did not survive (893 F.2d 1338, January 18, 1990). She never collected it. "It's been worth it," she said, "even if I don't see any of that four million dollars."

The obvious reading is that a signature can lie. Walter's signature was genuine, and it was never evidence that he had painted anything. The useful reading is the other one. Margaret's problem was that she had no way to sign her own work. At the moment she finished each painting she held no instrument for attaching a dated claim to it, so the question could be answered only the expensive way: years later, through a federal court, a paint-off under oath, and an award that was cut on appeal and never paid.

McConnell's office reached for a newspaper. Margaret Keane painted in a courtroom. Both were people made to prove something after the fact, through institutions, on long timescales, because nothing existed to attach a checkable claim at the moment that mattered. Forty years apart, the same gap.

Signing would not have made Margaret's case airtight, and that is the honest limit. A signature at publication gives a creator a dated, checkable claim, and a later signer cannot displace an earlier dated signature. It is still not proof of creation. What a signature claims is who stands behind this, and from when.

What an adequate solution requires

The evidence defines the requirement set:

  1. Deterministic proof. A verifier needs a yes-or-no answer about authenticity. Probabilistic detection decays as generators improve; a cryptographic check does not.
  2. Binding to a human, and to a time. Proof must tie content to an accountable person and carry the date it was made, so the claim can be checked later without a court. Devices, accounts, and communication channels are all spoofable, as the Arup call showed.
  3. Survival through distribution. Proof must survive screenshots, re-encoding, cropping, and forwarding, because content travels stripped of its metadata.
  4. Verification open to anyone. Checking must be free and require no enrollment. The people who most need to verify are the audience, and the audience cannot be asked to subscribe first.
  5. Speed at distribution scale. A fake reaches 35 million viewers faster than any takedown process. Authenticity must be checkable at the moment of viewing, by the viewer.
  6. No new surveillance. An authenticity layer that tracks who signs, who verifies, and what they look at would trade one harm for another. The proof must work without anyone watching.

Content Provenance and Digital Signatures (Doc #2) describes how not.bot meets these requirements with visible cryptographic signatures (patent pending) that travel with the content itself.

View this page as Markdown