Hello,

I found this source code confusingly written (and downright scary at times) and the specification woefully underspecified and inexplicit, so it's entirely possible my understanding of the handshake is inaccurate. But on the off-chance that 5 minutes of source code review at 4am yielded something accurate, here is my understanding of the handshake:

Peer A (Alice) has the longterm static keypair (S_A^{pub}, S_A^{priv}). Peer A has the session-generated ephemeral keypair (E_A^{pub}, E_A^{priv}). Peer B (Bob) has the longterm static keypair (S_B^{pub}, S_B^{priv}). Peer B has the session-generated ephemeral keypair (E_B^{pub}, E_B^{priv}).

Message 1: A -> B

XAEAD(key=ECDH(S_A^{priv}, S_B^{pub}), payload=E_A^{pub})

Message 2: B -> A

XAEAD(key=ECDH(S_B^{priv}, S_A^{pub}), payload=E_B^{pub})

Session Key Derivation

ECDH(E_A^{priv}, E_B^{pub}) = ECDH(E_B^{priv}, E_A^{pub})

Is this an accurate representation of the handshake? If so, keep reading. If not, you may safely stop reading here, close the issue, and accept my apologies for the misunderstanding.

The issue is that this naive handshake is vulnerable to key-compromise impersonation, something that basically all modern authenticated key exchanges (AKEs) are designed to protect against. Concretely, the issue is that if A's longterm static private key is stolen, an attacker can impersonate anybody to A without A realizing. Let's say that Mallory, M, has stolen A's private key and wants to pretend to be B:

Message 1: M -> A

XAEAD(key=ECDH(S_A^{priv}, S_B^{pub}), payload=E_M^{pub})

Message 2: A -> M

XAEAD(key=ECDH(S_A^{priv}, S_B^{pub}), payload=E_A^{pub})

Session Key Derivation

ECDH(E_A^{priv}, E_M^{pub}) = ECDH(E_M^{priv}, E_A^{pub})

A now thinks he is talking to B, but is actually talking to M.

Perhaps Tox doesn't care about this, or about many of the threat models that modern AKEs are designed to protect against, in which case, probably it's fine to continue using your homebrewed crypto. But if you actually desire some kind of high assurance security, I strongly recommend not building your own protocols and instead use something designed by an educated expert, such as Noise.

This is just what immediately leaped out at me after a few short minutes of review. I haven't even begun to look at key derivation and other interesting aspects (are you guys really just using raw ECDH results as keys?).

Again, apologies if this doesn't actually represent the handshake you're using; I'm not 100% certain. But in case it does, then let this be a wake-up call to developers not to roll your own crypto, as well as a wake-up call to users not to rely on crypto software written by non-experts.