• An OAK-D device
• Supported operating systems: Windows 10 or Linux (e.g., Ubuntu 20+)
• Supported Python versions 3.6+
• Linux only: set up OAK-D udev rules

Install the Python pacakge: pip install spectacularAI

To install dependecies for all examples you can use: pip install -r requirements.txt. On Linux, you may also need to sudo apt install python3-tk to run the Matplotlib-based visualizations.

• Minimal example. Prints 6-DoF poses as JSON text: python vio_jsonl.py

• Basic visualization. Interactive 3D plot / draw in the air with the device: python vio_visu.py

• 3D pen. Draw in the air: cover the OAK-D color camera to activate the ink. python pen_3d.py

• Advanced Spatial AI example. Spectacular AI VIO + Tiny YOLO object detection. See depthai_combination.py for additional dependencies that also need to be installed.

• Mixed reality. In less than 130 lines of Python, with the good old OpenGL functions like glTranslatef used for rendering. Also requires PyOpenGL_accelerate to be installed, see mixed_reality.py for details.

• Remote visualization over SSH. Can be achieved by combining the vio_jsonl.py and vio_visu.py scripts as follows:

   ssh user@example.org 'python -u /full/path/to/vio_jsonl.py' | python -u vio_visu.py --file=-
  

  Here user@example.org represents a machine (e.g., Raspberry Pi) that is connected to the OAK-D, but is not necessarily attached to a monitor. The above command can then be executed on a laptop/desktop machine, which then shows the trajectory of the OAK-D remotely (like in this video).

https://spectacularai.github.io/docs/sdk/python/latest

The SDK uses the following coordinate conventions, which are also elaborated in the diagram below:

• World coordinate system: Right-handed Z-is-up
• Camera coordinate system: OpenCV convention (see here for a nice illustration) which is also right-handed

These conventions are different from, e.g., Intel RealSense SDK (cf. here), ARCore, Unity and most OpenGL tutorials, most of which use an "Y-is-up" coordinate system, often different camera coordinates systems, and sometimes different pixel (or "NDC") coordinate conventions.

Rarely, the OAK-D device factory calibration may be inaccurate, which may cause the the VIO performance to be always very bad in all environments. If this is the case, the device can be recalibrated following Luxonis' instructions (see also our instructions for fisheye cameras for extra tips).

It is possible to fit certain OAK-D models with fisheye lenses. These are supported from Spectacular AI SDK version 0.16 onwards, but require the following spectacularAI.Configuration changes to be applied in spectacularAI.Pipeline:

meshRectification = True
depthScaleCorrection = True

These settings are work-arounds that also (currently) work with normal OAK-D lenses, but may stop working with future DepthAI versions, and increase initialization time, and therefore they are not recommended for general use. The settings are enabled in all examples in the oak-fisheye branch of this repository.

Also calibrate the camera according to these instructions, if you have changed the lenses or the device did not include a factory calibration.

Currently require setting certain parameters manually:

• OAK-D-Pro-W (dev): IMU-to-camera matrix needs to be changed, see this branch: https://github.com/SpectacularAI/sdk-examples/tree/oak-d-pro-w-dev

For more info, see the main README.md.