Designed to track and analyze eye movements in real-time, by leveraging computer vision and gaze estimation algorithms based on the MPIIGaze and MPIIFaceGaze papers to determine the gaze direction.

Note that this is an implementation of existing project.

# Create Python Virtual Environment
python -m venv .venv

# Install requirements
pip install -r requirements.txt

# Download MPIIGaze - dataset focus on cropped eye images
bash scripts/download_mpiigaze_dataset.sh
python tools/preprocess_mpiigaze.py --dataset datasets/MPIIGaze -o datasets/

# Download MPIIFaceGaze - dataset focus on full face images
bash scripts/download_mpiifacegaze_dataset.sh
python tools/preprocess_mpiifacegaze.py --dataset datasets/MPIIFaceGaze_normalized -o datasets/

To run the gaze estimation from a webcam:

1. Download the dlib pretrained model for landmark detection

   bash scripts/download_dlib_model.sh

2. Calibrate the camera for better accuracy (optional)

   Save calibration result in the same format as the sample file data/calib/sample_params.yaml

3. Run demo

   Specify the model path and the path of the camera calibration results in the configuration file as in configs/demo_mpiigaze_resnet.yaml

   # Running mpiigaze resnet - about 16fps
   python demo.py --config configs/demo_mpiigaze_resnet.yaml
   
   # Running mpiifacegaze resnet - about 12fps
   python demo.py --config configs/demo_mpiifacegaze_resnet_simple_14.yaml

By running the following code, you can train a model using all the data except the person with ID 0, and run test on that person.

python train.py --config configs/mpiigaze/lenet_train.yaml
python evaluate.py --config configs/mpiigaze/lenet_eval.yaml

Run all training and evaluation for LeNet and ResNet-8 with default parameters, using scripts/run_all_mpiigaze_lenet.sh and scripts/run_all_mpiigaze_resnet_preact.sh.

• structure - Specifies model parameters in YAML file e.g. configs/mpiigaze/lenet_train.yaml
• data - camera calibration parameters, saved model files (alexnet pretrained weights not provided as it is too large), and dlib files
• dataset - MPIIGaze and MPIIFaceGaze dataset downloaded earlier
• experiments - logs from model train & evaluation
• gaze_estimation - all the code data, with some notable folders:
  • gaze_estimator - base classes for gaze estimation specific tasks (e.g. head post estimation, face modelling)
  • models - neural network model definitions
• others - other custom code
• scripts - bash scripts to automate downloading of data, executing model training & evaluation using all data, as well as reading the mean test angle error and training time
• tools - tools for data preprocessing (used for data download section)

Summary

Specific Details

The training time is the value when using RTX3090.

There is a slight difference with the training done by the sample code, as we did not set cudnn benchmark = False, cudnn deterministic = True

Summary

Specific Details

The training time is the value when using RTX3090.

There is a slight difference with the training done by the sample code, as we did not set cudnn benchmark = False, cudnn deterministic = True