Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single-use token used to create a snapshot reappears in the restored database and is irrevocable #522

Open
mossblaser opened this issue Sep 12, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@mossblaser
Copy link

Describe the bug
If a single use (e.g. -use-limit=1) token is used to create a raft snapshot, the expired token reappears when the snapshot is restored. The expired token shows a num_uses of -1 and appears to not be usable. The restored token also appears to be irrevocable and in some kind of stuck state.

Because the reappearing token (appears) not to be usable, this doesn't present an immediate security problem. It does, however, seem to be leading to some kind of "impossible" database state which might be concerning from the perspective of database integrity. It also complicates auditing of tokens in a live system as these "stuck and unusable" tokens must be manually eliminated from any checks.

To Reproduce
Steps to reproduce the behavior:

1. Create an openbao cluster using raft based storage.

Sample minimal openbao config (click to expand)

Setup a quick-and-dirty openbao server:

$ cat << EOF > config.hcl
   storage "raft" {
     path = "data"
     node_id = "bao"
   }
   listener "tcp" {
     address = "127.0.0.1:8200"
     cluster_address = "127.0.0.1:8201"
     tls_disable = true
   }
   api_addr = "http://127.0.0.1:8200"
   cluster_addr = "https://127.0.0.1:8201"
   disable_mlock = true
EOF
$ mkdir data
$ bao server -config config.hcl

Initialise, unseal and authenticate:

$ export BAO_ADDR="http://127.0.0.1:8200"
$ bao operator init -key-shares=1 -key-threshold=1
$ bao operator unseal <unseal key printed during init>
$ bao login <root token printed during init>

2. Create a single-use token with the ability to hit the snapshot endpoint. For example:

$ bao token create -policy=root -use-limit=1
Key                  Value
---                  -----
token                s.umEj7CId7GgLybIgfFwY2Czb
token_accessor       mSF2fv8BHdCgM3v55f1t4aC9
...

3. See that the token exists:

$ bao token lookup -accessor mSF2fv8BHdCgM3v55f1t4aC9
Key                 Value
---                 -----
accessor            mSF2fv8BHdCgM3v55f1t4aC9
...

4. Create snapshot using the single-use token:

$ BAO_TOKEN=s.umEj7CId7GgLybIgfFwY2Czb bao operator raft snapshot save raft.snap

5. Verify that token has expired:

$ bao token lookup -accessor mSF2fv8BHdCgM3v55f1t4aC9
Error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup-accessor
Code: 400. Errors:

* 1 error occurred:
        * invalid accessor

6. Restore snapshot:

$ bao operator raft snapshot restore raft.snap

7. Observe that the expired token now exists again (though it isn't usable as num_uses is -1):

$ bao token lookup -accessor mSF2fv8BHdCgM3v55f1t4aC9
Key                 Value
---                 -----
accessor            mSF2fv8BHdCgM3v55f1t4aC9
creation_time       1726150164
creation_ttl        0s
display_name        token
entity_id           n/a
expire_time         <nil>
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T15:09:24.438032422+01:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           false
ttl                 0s
type                service
$ BAO_TOKEN=s.umEj7CId7GgLybIgfFwY2Czb bao token lookup
Error looking up token: Error making API request.

URL: GET http://127.0.0.1:8200/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

8. Also observe that the token cannot be revoked.

$ bao token revoke -accessor mSF2fv8BHdCgM3v55f1t4aC9
Error revoking token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/revoke-accessor
Code: 400. Errors:

* token not found
$ bao token lookup -accessor mSF2fv8BHdCgM3v55f1t4aC9
Key                 Value
---                 -----
accessor            mSF2fv8BHdCgM3v55f1t4aC9
...

Expected behavior

I would expect that a single-use token used to create a snapshot would not appear in the backup (I would expect that it would have been deleted as part of the checking process before the snapshot started).

I would also expect to be able to revoke the back-from-the-dead token after restoring a snapshot (even if it is in an unusable state).

Extra detail: time and use limited tokens

In case it is useful, I have observed that in the case of a single-use token also having a finite TTL, the stuck token in the restored database will eventually be removed when the TTL expires.

1. Create token with single-use limit and a short TTL:

$ bao token create -policy=root -use-limit=1 -ttl=5m
Key                  Value
---                  -----
token                s.LGyUbmS9c29REWxqMNPW5yq8
token_accessor       hU13dwiqoj23Bw0kShUjyRlE
token_duration       5m
token_renewable      true
token_policies       ["root"]
identity_policies    []
policies             ["root"]

2. Create snapshot using the token and then restore from that snapshot.

$ BAO_TOKEN=s.LGyUbmS9c29REWxqMNPW5yq8 bao operator raft snapshot save raft.snap
$ bao operator raft snapshot restore raft.snap

3. Observe that the token comes back to life but is unusable and cannot be revoked, as before.

$ bao token lookup -accessor hU13dwiqoj23Bw0kShUjyRlE
Key                 Value
---                 -----
accessor            hU13dwiqoj23Bw0kShUjyRlE
creation_time       1726150326
creation_ttl        5m
display_name        token
entity_id           n/a
expire_time         2024-09-12T15:17:06.873084796+01:00
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T15:12:06.873089291+01:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           true
ttl                 3m43s
type                service
$ bao token revoke -accessor hU13dwiqoj23Bw0kShUjyRlE
Error revoking token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/revoke-accessor
Code: 400. Errors:

* token not found
$ bao token lookup -accessor hU13dwiqoj23Bw0kShUjyRlE
Key                 Value
---                 -----
accessor            hU13dwiqoj23Bw0kShUjyRlE
...

4. Wait for TTL to expire and observe that the token is deleted as expected.

$ bao token lookup -accessor hU13dwiqoj23Bw0kShUjyRlE

5. If restored again after the TTL has expired, notice that the expired token is also briefly present (though unusable) for a few seconds before being cleaned up.

$ bao operator raft snapshot restore raft.snap
$ bao token lookup -accessor hU13dwiqoj23Bw0kShUjyRlE
Key                 Value
---                 -----
accessor            hU13dwiqoj23Bw0kShUjyRlE
creation_time       1726150326
creation_ttl        5m
display_name        token
entity_id           n/a
expire_time         2024-09-12T15:17:06.873084796+01:00
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T15:12:06.873089291+01:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           false
ttl                 -33s
type                service
$ sleep 5
$ bao token lookup -accessor hU13dwiqoj23Bw0kShUjyRlE
Error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup-accessor
Code: 400. Errors:

* 1 error occurred:
        * invalid accessor

Environment:

OpenBao v2.0.1 (88383de), built 2024-09-03T19:57:34Z

Ubuntu 22.04 x86_64

Extra context:

The same bug also exists in Hashicorp Vault.

@mossblaser mossblaser added the bug Something isn't working label Sep 12, 2024
@cipherboy
Copy link
Member

@mossblaser Thanks again for the very detailed reproducer! Let me know if you want to collaborate on particular areas or if you're happy sending random reports. :-)


Interestingly, I would've expected:

$ bao write -force auth/token/tidy
WARNING! The following warnings were returned from Vault:

  * Tidy operation successfully started. Any information from the operation
  will be printed to OpenBao's server logs.

to fix it, but indeed, it is still valid:

$ bao token lookup -accessor ViQs6PxZj1QrUVlBSOiGbEpu
Key                 Value
---                 -----
accessor            ViQs6PxZj1QrUVlBSOiGbEpu
creation_time       1726170798
creation_ttl        0s
display_name        token
entity_id           n/a
expire_time         <nil>
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T15:53:18.509003804-04:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           false
ttl                 0s
type                service

In the logs, I see:

2024-09-12T15:55:16.565-0400 [INFO]  token: beginning tidy operation on tokens
2024-09-12T15:55:16.566-0400 [INFO]  token: number of entries scanned in parent prefix: count=1
2024-09-12T15:55:16.566-0400 [INFO]  token: number of entries deleted in parent prefix: count=0
2024-09-12T15:55:16.566-0400 [INFO]  token: number of tokens scanned in parent index list: count=1
2024-09-12T15:55:16.566-0400 [INFO]  token: number of tokens revoked in parent index list: count=0
2024-09-12T15:55:16.566-0400 [INFO]  token: number of accessors scanned: count=2
2024-09-12T15:55:16.566-0400 [INFO]  token: number of deleted accessors which had empty tokens: count=0
2024-09-12T15:55:16.566-0400 [INFO]  token: number of revoked tokens which were invalid but present in accessors: count=0
2024-09-12T15:55:16.566-0400 [INFO]  token: number of deleted accessors which had invalid tokens: count=0
2024-09-12T15:55:16.566-0400 [INFO]  token: number of deleted cubbyhole keys that were invalid: count=0
2024-09-12T15:55:16.566-0400 [INFO]  token: finished tidy operation on tokens

Strictly, I suppose there's an ambiguous scenario (from a tidy PoV) where the snapshot cannot be detected as being different from a wrap/unwrap to the cubbyhole. I wonder what the proper behavior there is, if an operation wraps to the cubbyhole but is a single-use token... It seems like the token might need to live longer?

It looks like the answer is no however, it doesn't matter:

$ bao kv put secret/dev username="alex" password="ivebeenherethewholetime"
Success! Data written to: secret/dev
$ bao read secret/dev
Key                 Value
---                 -----
refresh_interval    768h
password            ivebeenherethewholetime
username            alex
$ bao token create -policy=root -use-limit=1
Key                  Value
---                  -----
token                s.cf0u3UUov6MEIJj2YIlavSFe
token_accessor       L4joHLLqLIRosTyE3cwgcWRs
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
$ BAO_TOKEN=s.cf0u3UUov6MEIJj2YIlavSFe bao kv get -wrap-ttl=10000m secret/dev
Key                              Value
---                              -----
wrapping_token:                  s.CR14CkzhxeMnEsiy36Fi0M6n
wrapping_accessor:               pRQv5YYPN57igO9FlVUugFrG
wrapping_token_ttl:              166h40m
wrapping_token_creation_time:    2024-09-12 16:09:56.285900265 -0400 EDT
wrapping_token_creation_path:    secret/dev
$ WRAP_TOKEN=s.CR14CkzhxeMnEsiy36Fi0M6n
$ BAO_TOKEN=$WRAP_TOKEN bao unwrap
Key                 Value
---                 -----
refresh_interval    768h
password            ivebeenherethewholetime
username            alex

So even using an (expired) parent token to read the unwrap seems fine. Interesting!

My thoughts then is, perhaps this warrants changes to tidy and suggest running tidy after snapshot restore?

@mossblaser
Copy link
Author

@mossblaser Thanks again for the very detailed reproducer! Let me know if you want to collaborate on particular areas or if you're happy sending random reports. :-)

A pleasure, thanks for your work maintaining this!

I'm very happy sending in reports if they're helpful; everything works too well for me to go on a systematic bug hunt :).

So even using an (expired) parent token to read the unwrap seems fine. Interesting!

That is a fun one isn't it: it would probably surprised me whichever way it turned out. On the one hand being valid when the parent isn't sounds obviously bad. But if it didn't work that way it would make the wrapping feature more awkward to use!

In any case, many thanks for looking into this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants