Skip to content

[TITS 2023] Method-Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways

License

Notifications You must be signed in to change notification settings

gy65896/DeepSORVF

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

[TITS 2023] DeepSORVF: Deep Learning-based Simple Online and Real-Time Vessel Data Fusion Method


Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways
Yu Guo, Ryan Wen Liu* , Jingxiang Qu, Yuxu Lu, Fenghua Zhu* , Yisheng Lv
(* Corresponding Author)
IEEE Transactions on Intelligent Transportation Systems

Introduction: In this work, we first extract the AIS- and video-based vessel trajectories, and then propose a deep learning-enabled asynchronous trajectory matching method (named DeepSORVF) to fuse the AIS-based vessel information with the corresponding visual targets. In addition, by combining the AIS- and video-based movement features, we also present a prior knowledge-driven anti-occlusion method to yield accurate and robust vessel tracking results under occlusion conditions.


video

Flowchart

Figure01_Flowchart

The architecture of the proposed deep learning-based simple online and real-time vessel data fusion method.

Figure03_Video

The flowchart of anti-occlusion tracking method for video-based vessel trajectory extraction.

Environment

  • Python3.7
  • easydict 1.11
  • geopy 2.4.1
  • pyproj 3.2.1
  • fastdtw 0.3.4
  • pytorch 1.13.1
  • cuda 11.7
  • pandas 1.3.5
  • numpy 1.21.6

Running

  • Save ckpt.t7 to DeepSORVF/deep_sort/deep_sort/deep/checkpoint/ folder.
  • Save YOLOX-final.pth to DeepSORVF/detection_yolox/model_data/ folder.
  • Set data dir by parser.add_argument("--data_path", type=str, default = './clip-01/', help='data path').
  • Run main.py.

draw_org.py is used to simultaneously visualize the ais-based trajectory (blue line), target detection box (red box), and fusion results (black text). It can be enabled by modifying import draw in main.py to import draw_org.

Test Data: clip-01

FVessel: Benchmark Dataset for Vessel Detection, Tracking, and Data Fusion

The FVessel benchmark dataset is used to evaluate the reliability of AIS and video data fusion algorithms, which mainly contains 26 videos and the corresponding AIS data captured by the HIKVISION DS-2DC4423IW-D dome camera and Saiyang AIS9000-08 Class-B AIS receiver on the Wuhan Segment of the Yangtze River. To protect privacy, the MMSI for each vessel has been replaced with a random number in our dataset. As shown in Figure 1, these videos were captured under many locations (e.g., bridge region and riverside) and various weather conditions (e.g., sunny, cloudy, and low-light).

Figure04_FVessel

Some samples of the FVessel dataset, which contains massive images and videos captured on the bridge region and riverside under sunny, cloudy, and low-light conditions.

Performance on FVessel_V1.0

Name MOFA (%) IDP (%) IDR (%) IDF (%)
video-01 79.94 89.35 90.76 90.05
video-02 73.19 83.27 91.60 87.23
video-03 96.45 99.23 97.20 98.20
video-04 98.08 99.45 98.63 99.03
video-05 89.19 93.46 95.91 94.67
video-06 91.17 96.04 95.08 95.56
video-07 96.81 99.59 97.21 98.39
video-08 82.28 99.64 82.58 90.31
video-09 98.45 100.00 98.45 99.22
video-10 88.74 90.42 99.26 94.63
video-11 97.66 99.29 98.36 98.83
video-12 95.45 99.06 96.36 97.69
video-13 84.82 94.82 89.72 92.20
video-14 93.10 97.82 95.22 96.50
video-15 95.88 97.19 98.74 97.96
video-16 98.68 100.00 98.68 99.33
video-17 90.02 93.80 96.39 95.08
video-18 74.49 83.57 92.72 87.91
video-19 96.62 98.31 98.31 98.31
video-20 96.74 98.66 98.07 98.36
video-21 76.43 87.03 89.82 88.40
video-22 96.82 99.35 97.45 98.39
video-23 94.71 98.91 95.77 97.31
video-24 94.70 98.34 96.33 97.32
video-25 91.49 97.66 93.73 95.66
video-26 97.44 99.11 98.32 98.72
Average 91.13 95.90 95.41 95.59

Acknowledgements

We deeply thank Jianlong Su from the School of Computer and Artificial Intelligence in Wuhan University of Technology who performs the data acquisition and algorithm implementation works.

Citation

@article{guo2023asynchronous,
  title={Asynchronous trajectory matching-based multimodal maritime data fusion for vessel traffic surveillance in inland waterways},
  author={Guo, Yu and Liu, Ryan Wen and Qu, Jingxiang and Lu, Yuxu and Zhu, Fenghua and Lv, Yisheng},
  journal={IEEE Transactions on Intelligent Transportation Systems},
  volume={24},
  number={11},
  pages={12779--12792},
  year={2023}
}

The DeepSORVF is available for non-commercial research purposes only. If you have any questions, please get in touch with me (guoyu65896@gmail.com).

Reference

https://github.com/bubbliiiing/yolox-pytorch

https://github.com/dyh/unbox_yolov5_deepsort_counting/tree/main/deep_sort

Visitor count

About

[TITS 2023] Method-Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published