Some general notes about hardware-backed quorum models, based on our experience on PyPI when attempting to deploy TUF on PyPI (i.e. PEP 458):

1. Hardware (i.e. HSM) backed tokens are difficult to operationalize over expected key lifetimes:
   • You'll need to establish a chain of trust for the hardware itself (some, but not all, HSM models have baked-in device attestations).
   • You'll need to perform a secure offline signing and enrollment ceremony (we designed a runbook for PyPI, but it's pretty old at this point and was also constrainted by HSM vendor limitations that should be re-evaluated)
   • You'll need a testable compromise and roation process, to prevent/limit normalization of deviance around key management and enrollment into the quorum.
   • Each quorum party will ideally physically secure their HSM in a way that stymies a medium-complexity physical adversary: PyPI chose tamper-evident bags, and in practice each quorum member will need multiple bags and a tag-in-tag-out procedure for removing their key from their bag if occasional key operations are expected.
2. This section is currently a little light on cryptographic details: it specifies the size of the quorum, but it doesn't say which types and sizes of keys are permitted in the root set, or how the community will verify that a key is actually enrolled within a particular HSM. For PyPI we stipulated a mix of P-256 and P-384 keys due to the limitations of HSMs at the time, but it might be possible to do Ed25519 keypairs with current commercial HSMs. We also prepared HSM-level attestations of key possession, although in practice hardware limitations meant that only the YubiHSMs actually supported root key attestation (versus attestation of the HSM itself).

As a whole, these were nontrivial issues to address in our initial attempt to implement TUF on PyPI, and IMO they're a large part of why TUF (in the form of PEP 458) hasn't materialized on PyPI. So as part of this RFC I recommend the Rust's Leadership Council think about the over/under on a hardware quorum model versus something simpler (e.g. a soft root key in a cloud HSM) or even a different, smaller-footprint architecture (like a transparency scheme). I'll leave a separate comment on the latter.

To add to what @woodruffw wrote, for our HSM enrolment we require each key-holder to upload a device and key attestation that can be verified up to the manufacturer's root CA. Verification of this is done via GitHub actions, so for each new key added we get automatic verification.

We also hook this into the TUF verification process, so that each time a TUF metadata document is updated we run a workflow that verify the TUF signatures via HSM's attested device key. This may be slightly overkill, but makes it simple and foolproof to see that no extra keys has been added that is not known and approved.