Deepfake Detection Arena (DFD) Leaderboard
๐ฏ The Open Benchmark for Detecting AI-Generated Images
DFD-Arena is the first benchmark to address the open-source computer vision community's need for a comprehensive evaluation framework for state-of-the-art (SOTA) detection of AI-generated images.
While previous studies have focused on benchmarking the SOTA on content-specific subsets of the deepfake detection problem, e.g. human face deepfake benchmarking via DeepfakeBench, these benchmarks do not adequately account for the broad spectrum of real and generated image types seen in everyday scenarios.
๐ Explore DFD-Arena
Learn how the framework evaluates on diverse, content-rich images with semantic balance between real and generated data:
โ๏ธ Authorship
Both DFD-Arena and novel synthetic image datasets used for evaluation are created by BitMind.
Average Performance Metrics
- "headers": [
- "Detector",
- "Accuracy",
- "Precision",
- "Recall",
- "F1-Score",
- "MCC"
- "data": [
- [
- "NPR",
- 0.7169,
- 0.9193,
- 0.5996,
- 0.7258,
- 0.5044
- [
- "UCF",
- 0.7229,
- 0.9436,
- 0.592,
- 0.7275,
- 0.5285
- [
- "CAMO",
- 0.7555,
- 0.9442,
- 0.647,
- 0.7679,
- 0.5707
- [
- "metadata": null
Dataset-specific Accuracy
Detector | CelebA-HQ | Flickr30k | ImageNet | DiffusionDB | CelebA-HQ-SDXL | CelebA-HQ-Flux | Flickr30k-SDXL | MS-COCO-Flux |
|---|---|---|---|---|---|---|---|---|
CAMO | 0.987 | 0.916 | 0.834 | 0.876 | 0.386 | 0.846 | 0.302 | 0.588 |
Submission Instructions
๐๏ธ 1. Ensure that your detector code follows the DFD Arena detectors format.
The best way to guarantee compatibility is to develop and test your detector within a local copy of the DFD Arena repository, with dependencies, detector file, and configs in relative locations similar to how we structured our implementations of UCF, NPR, and CAMO detectors.
โ Check list:
- Your code should work with detector dependencies (architecture and training code) imported from a dependencies directory a level above the detector directory.
- e.g.,
arena/detectors/deepfake_detectors/ucf_detector.pyrelies on a dependency folder atarena/detectors/UCF/ - Our automated benchmarking pipeline will reconstruct the required directory at evaluation time
- e.g.,
- Implement a
.pyfile inarena/detectors/deepfake_detectors/containing aDeepfakeDetectorsubclass with PascalCase naming convention, registered as a module in the dfd-arenaDETECTOR_REGISTRY.- e.g., in
myCustomDetector.py,
@DETECTOR_REGISTRY.register_module(module_name='MyCustomModuleName') class MyCustomDetector(DeepfakeDetector): # implementation- The module name should match the detector name you want to appear on the leaderboard
- e.g., in
- Create a config YAML file that the DeepfakeDetector loads in
arena/detectors/deepfake_detectors/configs/.
๐ค 2. Create a HuggingFace model repo with the detector `.py` file, config `.yaml`, and dependencies in the same root level.
Check out our Sample Leaderboard Submission Repo for Reference!
๐ค 3. Fill out the form below with the correct paths and submit!
The results will be processed after a code review by the BitMind team, and an automated test/benchmarking stage.
โ ๏ธ Note: The Detector Name must match the name of the registered detector module in the dfd arena detector registry. This will be the name of your detector on our leaderboard.
- For example, using the Sample Submission Repo provided, you would submit:
- Detector Name:
test - Hugging Face Model Repo:
caliangandrew/submit_test - Path to detector
.py:test_detector.py - Path to config
.YAML:test_config.yaml
- Detector Name:
You can monitor your submission via the companion detectors and results datasets.