Skip to content
Nicolas P. Rougier edited this page Mar 2, 2021 · 11 revisions

Primitive Visuals and Visual Backends

WARNING: This is a draft/proposal for future changes to VisPy 1.0 and above.

Overview

Current VisPy (0.7 and below) is written with only OpenGL in mind. The world has evolved and there are other visualization technologies to consider (Vulkan, WebGPU, etc). This page discusses a new separation of responsibilities, a redesign, of the VisPy library in an attempt to leverage these new technologies while still providing the same (or very similar) high level interfaces. These high level interfaces include the plotting API, the SceneCanvas, and Visuals. The goal of this redesign is to rely on other lower-level libraries (ex. datoviz) for a specific set of "primitive" Visuals and for VisPy to build off of these. The below sections define the basic set of Visual primitives and the functionality they must provide to replace existing VisPy functionality.

NOTE: Due to VisPy's dependence on OpenGL there are some major changes and backwards incompatibilities that will be involved in order to accomplish this redesign. There are some low-level OpenGL interfaces that have "leaked" into the higher level interfaces of VisPy. Additionally VisPy's OpenGL 2/ES shaders will likely need to be rewritten in GLSL 4+ to be accepted by these newer technologies (no small feat).

Questions

  1. Can 2D Visuals be used in 3D and vice versa? For example, Markers may look best in a 2D view, but are still usable in a 3D view.
  2. Does we need to separate GUI backends and visual "backends"? Or does it become 'pyqt_gl' and 'pyqt_datoviz' or 'pyqt_vulkan'?

Shared Functionality

  1. Linear and non-linear transforms (MVP versus geographic transformations)
  2. Ability to update data after creation.
  3. Ability to update uniforms/properties (preferably without huge performance penalties)
  4. Ability to provide custom shaders (subclass?)

Canvas

  1. One or more Canvas classes to draw to. Multiple classes can be used to support different GUI framework backends.
  2. Access to underlying "native" GUI object (ex. .native in current VisPy for access to Qt QWidget)
  3. Native object is embed-able in larger GUI application
  4. Event or callback system for mouse, keyboard, and resize events.
  5. Application or Canvas .run() equivalent to control when event loop starts.
  6. Bonus: Ability to have multiple Canvas objects in the same application. Optionally, allow these different "views" to view the same Visuals. For example, a 2D view and a 3D view of the same Volume. This includes allowing different transforms to be applied to both if possible. May require idea of a lightweight "view" of a Visual.
  7. Ability to draw multiple Visuals. Add new Visuals after creation and remove Visuals if desired. Removed Visuals must not persist in CPU or GPU memory.
  8. Ability to use cameras (or controllers) to control transform/view of the visualization. Bonus: non-linear cameras like MagnifyCamera in VisPy.
  9. Set Canvas/viewport size on creation or after creation. 10: Picking: Convert screen/pixel coordinates from mouse event to clicked Visual. Bonus: Return nearest vertex/face on that Visual.

Primitives - 2D

Line

  1. Position: Vertex coordinates of the line. a. line strip: series of connected vertices b. line segments: pairs of connected points c. line indexes (optional): Provided a series of coordinates and a series of indexes, index into the coordinates to get the final vertices in-GPU. This is optional as it can be reproduced with line strip if really needed.
  2. Color (RGB/RGBA) a. single color b. per-vertex: Colors are interpolated between vertices
  3. Width: a. scalar/constant b. per-vertex
  4. Antialias (optional): Ability to turn off anti-aliasing if possible. Optional as a good looking line is a good looking line. If there are performance benefits to turning it off then some users may want that.

Image

  1. data: 3D array (row, col, bands) a. Bonus: CUDA/OpenCL buffer
  2. colormap (for single band data)
  3. color limits
  4. gamma correction
  5. in-GPU storage format: Integers versus floats. Normalized versus not. Avoid unnecessary copies
  6. interpolation method: nearest, bilinear, etc.
  7. mask for missing data
  8. iso-curves (number + color or sequence of values / colors)

Markers

  1. Position
  2. Symbol
  3. Size
  4. Face Color a. Single color b. Per-vertex
  5. Edge color a. Single color b. Per-vertex
  6. Edge Width a. Bonus: Per-vertex or relative to marker size
  7. Scale to zoom (optional): Keep size of markers relative to number of pixels in the Scene (always the same size to the user) or relative to coordinate system.
  8. Orientation (2D or 3D, for example arrows on the surface of a sphere)

2D Axes

  1. X limits
  2. Y limits
  3. Tick labels
  4. Axis labels
  5. Label font/size
  6. Easily and quickly update limits
  7. Log, semi-log, linear

Text

  1. Text to draw (str)
  2. Font: Face, bold, italic
  3. Size
  4. X Alignment/Anchor (left, right, center)
  5. Y Alignment/Anchor (top, bottom, middle)
  6. Color
  7. Rotation
  8. Orientation (or always facing camera)

Primitives - 3D

Mesh

// TODO

Volume

// TODO

3D Axes

// TODO

Clone this wiki locally