HABench is a benchmark tool designed to systematically leverage MLLMs across all dimensions relevant to video generation assessment in generative models. By incorporating few-shot scoring and chain-of-query techniques, HA-Video-Bench provides a structured, scalable approach to generated video evaluation.

⭐Overview | 📒Leaderboard | 🤗HumanAlignment | 🛠️Installation | 🗃️Preparation | ⚡Instructions | 🚀Usage | 📭Citation | 📝Literature

• Overview
• Leaderboard
• Installation
• HumanAlignment
• Installation
  • Installation Requirements
  • Pip Installation
  • Download From Huggingface
• Praparation
• Instruction
• Usage
  • Standard Mode
  • Custom Mode
  • Videos and Annotations
• Citation
• Literature

Notes:

• Higher scores indicate better performance.
• The best score in each dimension is highlighted in bold.

Notes:

• Higher scores indicate better performance.
• The best score in each dimension is highlighted in bold.

• Python >= 3.8
• OpenAI API access Update your OpenAI API keys in config.json:

  {
      "GPT4o_API_KEY": "your-api-key",
      "GPT4o_BASE_URL": "your-base-url",
      "GPT4o_mini_API_KEY": "your-mini-api-key",
      "GPT4o_mini_BASE_URL": "your-mini-base-url"
  }

• Install with pip

  pip install HAbench

• Install with git clone

  git clone https://github.com/yourusername/Video-Bench.git
  cd Video-Bench
  pip install -r requirements.txt

wget https://huggingface.co/Video-Bench/Video-Bench 

or

curl -L https://huggingface.co/Video-Bench/Video-Bench 

Please organize your data according to the following data structure:

# Data Structure
/Video-Bench/data/
├── color/                           # 'color' dimension videos
│   ├── cogvideox5b/
│   │   ├── A red bird_0.mp4
│   │   ├── A red bird_1.mp4
│   │   └── ...
│   ├── lavie/
│   │   ├── A red bird_0.mp4
│   │   ├── A red bird_1.mp4
│   │   └── ...
│   ├── pika/
│   │   └── ...
│   └── ...
│
├── object_class/                    # 'object_class' dimension videos
│   ├── cogvideox5b/
│   │   ├── A train_0.mp4
│   │   ├── A train_1.mp4
│   │   └── ...
│   ├── lavie/
│   │   └── ...
│   └── ...
│
├── scene/                           # 'scene' dimension videos
│   ├── cogvideox5b/
│   │   ├── Botanical garden_0.mp4
│   │   ├── Botanical garden_1.mp4
│   │   └── ...
│   └── ...
│
├── action/                          # 'action' 'temporal_consistency' 'motion_effects' dimension videos
│   ├── cogvideox5b/
│   │   ├── A person is marching_0.mp4
│   │   ├── A person is marching_1.mp4
│   │   └── ...
│   └── ...
│
└── video-text consistency/             # 'video-text consistency' 'imaging_quality' 'aesthetic_quality' dimension videos
    ├── cogvideox5b/
    │   ├── Close up of grapes on a rotating table._0.mp4
    │   └── ...
    ├── lavie/
    │   └── ...
    ├── pika/
    │   └── ...
    └── ...

Video-Bench provides comprehensive evaluation across multiple dimensions of video generation quality. Each dimension is assessed using a specific scoring scale to ensure accurate and meaningful evaluation.

Video-Bench supports two modes: standard mode and custom input mode. Video-Bench only supports assessments of the following dimensions: 'aesthetic_quality', 'imaging_quality','temporal_consistency', 'motion_effects','color', 'object_class', 'scene', 'action', 'video-text consistency'

The Standard Mode assesses videos generated by various video generation models using the prompt suite defined in our HAbench_full.json.

It allows users to organize three sets of video data for the seven provided models or add three sets for other models following the data structure. It also supports using only one set of video data for all models. Please ensure that the number of data sets is consistent across all models within the data structure.

To evaluate videos, simply specify the models to be tested in the --models parameter. For example, if you want to evaluate videos under modelname1 and modelname2, use the following commands with --models modelname1 modelname2

python evaluate.py \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --config_path ./config.json \
 --models modelname1 modelname2

or

HAbench \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --config_path ./config.json \
 --models modelname1 modelname2

This mode allows users to evaluate videos generated from prompts that are not included in the Video-Bench prompt suite.

You can provide prompts in two ways:

1. Single prompt: Use --prompt "your customized prompt" to specify a single prompt.
2. Multiple prompts: Create a JSON file and use --prompt_file $json_path. Create a JSON file containing your prompts and use --prompt_file $json_path to load them. The JSON file can follow this format:

{
    0: "prompt1",
    1: "prompt2",
    ...
}

python evaluate.py \
 --dimension $DIMENSION \ 
 --videos_path ./data/ \
 --mode custom_nonstatic \
 --config_path ./config.json \
 --models modelname1 modelname2

or

HAbench \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --mode custom_nonstatic \
 --config_path ./config.json \
 --models modelname1 modelname2

python evaluate.py \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --mode custom_static \
 --config_path ./config.json \
 --models modelname1 modelname2

or

HAbench \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --mode custom_static \
 --config_path ./config.json \
 --models modelname1 modelname2

You can obtain the video data and human annotations in two ways:

• Videos dataset: Video-Bench/Video-Bench_videos
• Human annotations: Video-Bench/Video-Bench_human_annotation

# Download videos
git clone https://huggingface.co/datasets/Video-Bench/Video-Bench_videos
# Download annotations  
git clone https://huggingface.co/datasets/Video-Bench/Video-Bench_human_annotation

The human annotations can also be found in the local directory:

./data/human_anno/

If you use our dataset, code or find Video-Bench useful, please cite our paper in your work as:

@article{ni2023content,
  title={Video-Bench: Human Preference Aligned Video Generation Benchmark},
  author={Han, Hui and Li, Siyuan and Chen, Jiaqi and Yuan, Yiwen and Wu, Yuling and Leong, Chak Tou and Du, Hanwen and Fu, Junchen and Li, Youhua and Zhang, Jie and Zhang, Chi and Li, Li-jia and Ni, Yongxin},
  journal={arXiv preprint arXiv:xxx},
  year={2024}
}