Hi folks,

Right now most of Synapse's dependencies only declare a minimum version bound, like Twisted>=18.9.0 (cite). This means that every time we build release artifacts or install Synapse from source, we unconditionally pull in the latest versions of our dependencies at that moment.

This creates unnecessary risk, as our dependencies can change out from under us without warning. For example, installing Synapse 1.49.0 may work today but fail tomorrow if one of our dependencies releases a new version overnight.

This exact scenario bit us with the release of attrs 21.1.0 (synapse#9936) on May 6th. We were forced to release Synapse 1.33.1 less than a day after 1.33.0 as the attrs release broke our ability to install Synapse from source, build working Debian packages, or create functioning Docker images, even though nothing in our repositories had changed.

The absence of locked dependencies also impedes our ability to pursue continuous delivery, maintain LTS releases, or easily backport security fixes as we cannot recreate older release artifacts without also implicitly updating all of the dependencies included therein.

Definition of Done

• For any given Synapse commit, it is possible to repeatably create identical virtualenvs.

Further Discussion / Constraints

Resolving this implies that it must be possible to enumerate exact versions of all dependencies included in any given upstream release of Synapse, using only a clone of the Synapse repository. This is important for auditing, as it allows us to easily answer questions like "did we ever ship a release with a vulnerable version of that dependency?"

Moreover, any solution must record hash digests to protect against remote tampering, such as with pip's hash-checking mode.

To ease maintenance burden (and avail of GitHub's supply chain security features), it would be nice if whatever solution we arrived at integrated with Dependabot. Supported package ecosystems for Python are requirements.txt (pip / pip-tools), pipfile.lock (Pipenv), and poetry.lock (Poetry).