CLIP bias analysis: hand-crafted features beat CLIP on non-CLIP generators#53
Open
wfproc wants to merge 1 commit intodarkshapes:mainfrom
Open
CLIP bias analysis: hand-crafted features beat CLIP on non-CLIP generators#53wfproc wants to merge 1 commit intodarkshapes:mainfrom
wfproc wants to merge 1 commit intodarkshapes:mainfrom
Conversation
Confirms that CLIP-based detection is biased toward generators that use CLIP internally. Tested on Defactify MS-COCOAI dataset (96K images, 5 labeled generators, semantically matched captions): Generator Uses CLIP? Hand-crafted CLIP Delta SD 2.1 YES 86.5% 96.1% +9.6pp SDXL YES 93.5% 99.0% +5.5pp SD 3 YES 85.4% 97.5% +12.1pp Midjourney v6 Unknown 88.5% 99.5% +11.0pp DALL-E 3 NO 98.7% 98.2% -0.5pp CLIP advantage on CLIP generators: +9.1pp average CLIP advantage on non-CLIP generators: -0.5pp (hand-crafted wins) Replaces per-experiment PDFs with single consolidated research report (negate_research_report.pdf) covering all experiments, scaling analysis, CLIP bias findings, and recommended next steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigates whether CLIP embeddings detect genuine AI artifacts or just recognize their own latent fingerprint in images from CLIP-based generators.
Finding: CLIP detection is biased. On the Defactify MS-COCOAI dataset (96K images, 5 labeled generators, semantically matched captions):
CLIP's advantage disappears entirely on DALL-E 3 (which uses T5, not CLIP). Hand-crafted features (artwork + style, 64 total) actually beat CLIP on non-CLIP generators.
As generators move away from CLIP-based architectures, CLIP detection will become less reliable. The hand-crafted features are the more robust long-term signal.
What's in this PR
tests/test_clip_bias_defactify.py— per-generator CLIP vs hand-crafted benchmarktests/generate_final_report.py— generates consolidated PDFresults/negate_research_report.pdf— single 5-page report replacing all prior PDFsHow to reproduce
See
results/EXPERIMENTS.mdfor full write-up with code links.