-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Single-use token used to create a snapshot reappears in the restored database and is irrevocable #522
Comments
@mossblaser Thanks again for the very detailed reproducer! Let me know if you want to collaborate on particular areas or if you're happy sending random reports. :-) Interestingly, I would've expected:
to fix it, but indeed, it is still valid:
In the logs, I see:
Strictly, I suppose there's an ambiguous scenario (from a tidy PoV) where the snapshot cannot be detected as being different from a wrap/unwrap to the cubbyhole. I wonder what the proper behavior there is, if an operation wraps to the cubbyhole but is a single-use token... It seems like the token might need to live longer? It looks like the answer is no however, it doesn't matter:
So even using an (expired) parent token to read the unwrap seems fine. Interesting! My thoughts then is, perhaps this warrants changes to |
A pleasure, thanks for your work maintaining this! I'm very happy sending in reports if they're helpful; everything works too well for me to go on a systematic bug hunt :).
That is a fun one isn't it: it would probably surprised me whichever way it turned out. On the one hand being valid when the parent isn't sounds obviously bad. But if it didn't work that way it would make the wrapping feature more awkward to use! In any case, many thanks for looking into this! |
Describe the bug
If a single use (e.g.
-use-limit=1
) token is used to create a raft snapshot, the expired token reappears when the snapshot is restored. The expired token shows anum_uses
of -1 and appears to not be usable. The restored token also appears to be irrevocable and in some kind of stuck state.Because the reappearing token (appears) not to be usable, this doesn't present an immediate security problem. It does, however, seem to be leading to some kind of "impossible" database state which might be concerning from the perspective of database integrity. It also complicates auditing of tokens in a live system as these "stuck and unusable" tokens must be manually eliminated from any checks.
To Reproduce
Steps to reproduce the behavior:
1. Create an openbao cluster using raft based storage.
Sample minimal openbao config (click to expand)
Setup a quick-and-dirty openbao server:
Initialise, unseal and authenticate:
2. Create a single-use token with the ability to hit the snapshot endpoint. For example:
3. See that the token exists:
4. Create snapshot using the single-use token:
5. Verify that token has expired:
6. Restore snapshot:
7. Observe that the expired token now exists again (though it isn't usable as num_uses is -1):
8. Also observe that the token cannot be revoked.
Expected behavior
I would expect that a single-use token used to create a snapshot would not appear in the backup (I would expect that it would have been deleted as part of the checking process before the snapshot started).
I would also expect to be able to revoke the back-from-the-dead token after restoring a snapshot (even if it is in an unusable state).
Extra detail: time and use limited tokens
In case it is useful, I have observed that in the case of a single-use token also having a finite TTL, the stuck token in the restored database will eventually be removed when the TTL expires.
1. Create token with single-use limit and a short TTL:
2. Create snapshot using the token and then restore from that snapshot.
3. Observe that the token comes back to life but is unusable and cannot be revoked, as before.
4. Wait for TTL to expire and observe that the token is deleted as expected.
5. If restored again after the TTL has expired, notice that the expired token is also briefly present (though unusable) for a few seconds before being cleaned up.
Environment:
OpenBao v2.0.1 (88383de), built 2024-09-03T19:57:34Z
Ubuntu 22.04 x86_64
Extra context:
The same bug also exists in Hashicorp Vault.
The text was updated successfully, but these errors were encountered: