Differentiable Swift based sensor fusion library.

Think factor graphs a la GTSAM coupled with deep learning, Jax/TensorFlow style. Based on Swift for TensorFlow.

Still very early, but feel free to explore! Subject to massive change :-)

Using XCode is the easiest way to develop on Mac. Unfortunately, the current version of Xcode 12 has a bug, and you need to work with the XCode beta, available from Apple here.

Follow the instructions to install Swift for TensorFlow on MacOS. Installing the latest development snapshot is recommended.

To use experimental toolchain:

export PATH="/Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.11.xctoolchain/usr/bin/:$PATH"

To re-generate XCode project:

swift package generate-xcodeproj

Requirements: Ubuntu 18.04 (if you use GPU).

Follow the instructions to install Swift for TensorFlow on Linux. Installing the latest development snapshot is recommended.

Download toolchain here, and then follow Linux instructions.

To check whether everything works you can run all the tests by changing to the SwiftFusion directory and

swift test --enable-test-discovery

swift run -c release -Xswiftc -cross-module-optimization SwiftFusionBenchmarks

To enable autocomplete in VSCode, install the plugin vknabel.vscode-swift-development-environment, and set the following plugin settings:

• "sde.languageServerMode": "sourcekit-lsp",
• "sourcekit-lsp.serverPath": "/usr/bin/sourcekit-lsp",
• "sourcekit-lsp.toolchainPath": "",
• "swift.path.swift_driver_bin": "/usr/bin/swift",

Debugging within VS code is easiest via the CodeLLDB plugin so you can debug in vscode. You need to set the following setting: "lldb.library": "/swift-tensorflow-toolchain/usr/lib/liblldb.so"

A sample launch.json file:

{
"version": "0.2.0",
"configurations": [
{
"type": "lldb",
"request": "launch",
"name": "Debug",
"program": "${workspaceFolder}/.build/x86_64-unknown-linux-gnu/debug/SwiftFusionPackageTests.xctest",
"args": ["--enable-test-discovery"],
"cwd": "${workspaceFolder}"
}
]
}

The main code is in Sources/SwiftFusion, which as a number of sub-directories:

The main protocols on which SwiftFusion is built:

• Vector.swift: protocol that formalizes a Euclidean vector space with standard orthonormal basis, and defines a great number of default methods for it. Also defines the ScalarsInitializableVector and FixedSizeVector protocols, and a collection StandardBasis<V: Vector> for the orthogonal bases.
• Manifold.swift: protocol that inherits from Differentiable that adds retract and localCoordinate, which convert between a Manifold structure and its LocalCoordinate
• LieGroup.swift: protocol that formalizes making a differentiable manifold into a Lie group with composition operator *. Also defines LieGroupCoordinate protocol.

Some specific Vector-like data structures:

• VectorN.swift: generated by VectorN.swift.gyb, implements small fixed vector structs conforming to AdditiveArithmetic, Vector, and FixedSizeVector.
• FixedSizeMatrix.swift: matrix whose dimensions are known at compile time. Conforms to Equatable, KeyPathIterable, CustomStringConvertible, AdditiveArithmetic, Differentiable, FixedSizeVector

A number of utilities:

• DataTypes.swift: for now, just implements LieGroup protocol for Vector5
• Dictionary+Differentiable.swift: makes Dictionary differentiable
• MathUtil.swift: for now just pinv (pseudo-inverse), using Tensor.svd
• TensorVector.swift: a view of a Tensor that conforms to the Vector protocol.
• TrappingDouble.swift: a wrapper for Double that traps instead of allowing NaN.
• Tuple+Vector.swift: (undocumented)
• TypeKeyedArrayBuffers.swift: related to storage of Values/Factors

• FactorGraph.swift: the main FactorGraph struct, which stores factors
• Factor.swift: defines the core factor protocol hierarchy GaussianFactor : LinearizableFactor : VectorFactor : Factor, as well as the VariableTuple and DifferentiableVariableTuple protocols, as well as extensions to make Tuple conform to them.
• GaussianFactorGraph.swift: A factor graph whose factors are all GaussianFactors. A factor graph stores factors as a storage array of type [ObjectIdentifier: AnyFactorArrayBuffer], where ObjectIdentifier is a builtt-in Swift identifier type, and AnyFactorArrayBuffer is defined in FactorsStorage.swift. It's complicated.

• GradientDescent.swift: a basic gradient descent optimizer
• CGLS.swift: Conjugate Gradient solver for Least-squares problems
• LM.swift: a Levenberg-Marquardt nonlinear optimizer

• RandomWalkMetropolis.swift: implements a Metropolis sampler modeled after its equivalent in TensorFlow Probability

For 2D and 3D points we just use Vector2 and Vector3, but here we define 2D and 3D rotations, and 2D and 3D poses. We use the GTSAM naming scheme: Rot, Pose2, Rot3, and Pose3.

• ArrayImage.swift: Differentiable multi-channel image stored as [Double]. Has a differentiable tensor method and update function.
• OrientedBoundingBox.swift: a rectangular region of an image, not necessarily axis-aligned.
• Patch.swift: a differentiable patch method for ArrayImage, that returns a resampled image for the given OrientedBoundingBox.

Mostly reading of G2O files

Copyright 2020 The SwiftFusion Team

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.