-
Notifications
You must be signed in to change notification settings - Fork 36.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rest: Reject truncated hex txid early in getutxos parsing #30482
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsNo conflicts as of last run. |
🚧 At least one of the CI tasks failed. HintsMake sure to run all tests locally, according to the documentation. The failure may happen due to a number of reasons, for example:
Leave a comment here, if you need help tracking down a confusing failure. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approach ACK 9984bdc
Straightforward improvement to make the code more robust, very nice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice cleanup, please see my questions and observations
@paplorinc the first 3 commits here come from pending PR #30436, so comments on those changes are probably better to make there. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to see the added functional test conditions for truncated txids!
a9d1df0
to
e8d8b98
Compare
Pushed and replied to all feedback, except for #30482 (comment), which I'll do later |
🚧 At least one of the CI tasks failed. HintsMake sure to run all tests locally, according to the documentation. The failure may happen due to a number of reasons, for example:
Leave a comment here, if you need help tracking down a confusing failure. |
This avoids a hex-decoding and makes the next commit smaller.
SetHex is fragile, because it accepts any non-hex input or any length of input, without error feedback. This can lead to issues when the input is truncated or otherwise corrupted. Document the problem by renaming the method. In the future, the fragile method should be removed from the public interface. -BEGIN VERIFY SCRIPT- sed -i 's/SetHex/SetHexDeprecated/g' $( git grep -l SetHex ./src ) -END VERIFY SCRIPT-
fa459e6
to
aaaa66d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK aaaa66d
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Without seeing an exploratory branch where SetHexDeprecated
is removed, I think it might be better to call it SetHexDiscouraged
after all, in case it turns out to be really problematic to remove it for some unforeseen reason.
Thank you for the suggestion. However, outside of I appreciate the feedback, but when it comes to trivial style nits, I don't consider it worth it to spend days on it myself. |
Fair. |
fac13e9
to
fa93c83
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK fa93c83
Left a couple of suggestions but nothing blocking.
Returning an early "Parse error" for malformed txids instead of silently ignoring them is a better user interface, and the underlying hex parse refactoring is a very welcome improvement.
This is a safe replacement of the previous SetHex, which now returns an optional to indicate success or failure. The code is similar to the ParseHashStr helper, which will be removed in a later commit.
This is needed for the next commit.
No need to have two functions with different names that achieve the exact same thing.
re-ACK fac0c3d - only doc and test updates to address review comments, thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -86,7 +86,7 @@ BOOST_AUTO_TEST_CASE(CheckProofOfWork_test_negative_target) | |||
uint256 hash; | |||
unsigned int nBits; | |||
nBits = UintToArith256(consensus.powLimit).GetCompact(true); | |||
hash.SetHex("0x1"); | |||
hash = uint256{1}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Man, I was worried for a moment that this modification of switching to the uint8_t
ctor was setting the other end of the base_blob
array, but it's the same. Will be glad to see the fragile function go. 👍
@@ -66,6 +72,7 @@ using Txid = transaction_identifier<false>; | |||
/** Wtxid commits to all transaction fields including the witness. */ | |||
using Wtxid = transaction_identifier<true>; | |||
|
|||
/** DEPRECATED due to missing length-check and hex-check, please use the safer FromHex, or FromUint256 */ | |||
inline Txid TxidFromString(std::string_view str) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If someone is looking for follow-up work, a test-only change could be to replace TxidFromString(a)
with Txid::FromHex(a).value()
in src/test
.
git grep TxidFromString ./src/test/
git grep WtxidFromString
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've started working on a branch that removes (W)TxidFromString
entirely, it seems the qt callsites would work well with FromHex
too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in #30532. Also started working on a similar branch to remove uint256S
.
…of transaction_identifier::FromHex() f553e6d refactor: remove TxidFromString (stickies-v) 285ab50 test: replace WtxidFromString with Wtxid::FromHex (stickies-v) 9a0b2a6 fuzz: increase FromHex() coverage (stickies-v) 526a87b test: add uint256::FromHex unittest coverage (stickies-v) Pull request description: Since fab6ddb, `TxidFromString()` has been deprecated because it is less robust than the `transaction_identifier::FromHex()` introduced in [the same PR](#30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. In this PR, `TxidFromString` is removed completely to clean up the code and prevent further unsafe usage. Unit and fuzz test coverage on `uint256::FromHex()` and functions that wrap it is increased. Note: `TxidFromSring` allowed users to prefix strings with "0x", this is no longer allowed for `transaction_identifier::FromHex()`, so a helper function for input validation may prove helpful in the future _(this overlaps with the `uint256::uint256S()` vs `uint256::FromHex()` future cleanup)_. It is not relevant to this PR, though, besides the fact that this unused (except for in tests) functionality is removed. The only users of `TxidFromString` are: - `test`, where it is straightforward to drop in the new `FromHex()` methods without much further concern - `qt` coincontrol. There is no need for input validation here, but txids are not guaranteed to be 64 characters. This is already handled by the existing code, so again, using `FromHex()` here seems quite straightforward. Addresses @maflcko's suggestion: #30482 (comment) Also removes `WtxidFromString()`, which is a test-only helper function. ### Testing GUI changes To test the GUI coincontrol affected lines, `regtest` is probably the easiest way to quickly get some test coins, you can use e.g. ``` alias cli="./src/bitcoin-cli -regtest" cli createwallet "coincontrol" # generate 10 spendable outputs on 1 address cli generatetoaddress 10 $(cli -rpcwallet=coincontrol getnewaddress) # generate 10 spendable outputs on another address cli generatetoaddress 10 $(cli -rpcwallet=coincontrol getnewaddress) # make previous outputs spendable cli generatetoaddress 100 $(cli -rpcwallet=coincontrol getnewaddress) ``` ACKs for top commit: maflcko: re-ACK f553e6d 🔻 hodlinator: ACK f553e6d paplorinc: ACK f553e6d TheCharlatan: Nice, ACK f553e6d Tree-SHA512: c1c7e6ea4cbf05cf660ba178ffc4f35f0328f7aa6ad81872e2462fb91a6a22e4681ff64b3d0202a5a9abcb650c939561585cd309164a69ab6081c0765ee271ef
This is a safe replacement of the previous SetHex, which now returns an optional to indicate success or failure. The code is similar to the ParseHashStr helper, which will be removed in a later commit. Github-Pull: bitcoin#30482 Rebased-From: fad2991
This is needed for the next commit. Github-Pull: bitcoin#30482 Rebased-From: fab6ddb
Github-Pull: bitcoin#30482 Rebased-From: fa90777
18d65d2 test: use uint256::FromUserHex for RANDOM_CTX_SEED (stickies-v) 6819e5a node: use uint256::FromUserHex for -assumevalid parsing (stickies-v) 2e58fdb util: remove unused IsHexNumber (stickies-v) 8a44d7d node: use uint256::FromUserHex for -minimumchainwork parsing (stickies-v) 70e2c87 refactor: add uint256::FromUserHex helper (stickies-v) 85b7cbf test: unittest chainstatemanager_args (stickies-v) Pull request description: Since fad2991, `uint256S` has been [deprecated](fad2991#diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138) because it is less robust than the `base_blob::FromHex()` introduced in [the same PR](#30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. _(see also #30532)_ This PR carves out the few `uint256S` callsites that may potentially prove a bit more controversial to change because they deal with user input and backwards incompatible behaviour change. The main behaviour change introduced in this PR is: - `-minimumchainwork` will raise an error when input is longer than 64 hex digits - `-assumevalid` will raise an error when input contains invalid hex characters, or when it is longer than 64 hex digits - test: the optional RANDOM_CTX_SEED env var will now cause tests to abort when it contains invalid hex characters, or when it is longer than 64 hex digits After this PR, the remaining work to remove `uint256S` completely is almost entirely mechanical and/or test related. I will open that PR once #30560 is merged because it builds on that. ACKs for top commit: hodlinator: re-ACK 18d65d2 l0rinc: ACK 18d65d2 achow101: ACK 18d65d2 ryanofsky: Code review ACK 18d65d2. Very nice change that cleans up the API, adds checking for invalid values, makes parsing of values more consistent, and adds test coverage. Tree-SHA512: ec118ea3d56e1dfbc4c79acdbfc797f65c4d2107b0ca9577c848b4ab9b7cb8d05db9f3c7fe8441a09291aca41bf671572431f4eddc59be8fb53abbea76bbbf86
8756ccd scripted-diff: Replace ParseHex[<std::byte>]("str") -> "str"_hex[_u8] (Hodlinator) 9cb6873 refactor: Prepare for ParseHex -> ""_hex scripted-diff (Hodlinator) 50bc017 refactor: Hand-replace some ParseHex -> ""_hex (Hodlinator) 5b74a84 util: Add consteval ""_hex[_v][_u8] literals (l0rinc) dc5f6f6 test refactor: util_tests - parse_hex clean up (Hodlinator) 2b5e6ef refactor: Make XOnlyPubKey tolerate constexpr std::arrays (Hodlinator) 403d86f refactor: vector -> span in CCrypter (Hodlinator) bd0830b refactor: de-Hungarianize CCrypter (Hodlinator) d99c816 refactor: Improve CCrypter related lines (Hodlinator) 7e1d9a8 refactor: Enforce lowercase hex digits for consteval uint256 (Hodlinator) Pull request description: Motivation: * Validates and converts the hex string into bytes at compile time instead of at runtime like `ParseHex()`. * Eliminates runtime dependencies: #30377 (comment), #30048 (comment) * Has stricter requirements than `ParseHex()` (disallows whitespace and uppercase hex digits) and replaces it in a bunch of places. * Makes it possible to derive other compile time constants. * Minor: should shave off a few runtime CPU cycles. `""_hex` produces `std::array<std::byte>` as the momentum in the codebase is to use `std::byte` over `uint8_t`. Also makes `uint256` hex string constructor disallow uppercase hex digits. Discussed: #30560 (comment) Surprisingly does not change the size of the Guix **bitcoind** binary (on x86_64-linux-gnu) by 1 single byte. Spawned already merged PRs: #30436, #30482, #30532, #30560. ACKs for top commit: l0rinc: ACK 8756ccd stickies-v: Rebase re-ACK 8756ccd, no changes since a096215 ryanofsky: Code review ACK 8756ccd, just rebasing since last review and taking advantage of CScript constructors in #29369, also tweaking a code comment Tree-SHA512: 9b2011b7c37e0ef004c669f8601270a214b388916316458370f5902c79c2856790b1b2c7c123efa65decad04886ab5eff95644301e0d84358bb265cf1f8ec195
43cd83b test: move uint256_tests/operator_with_self to arith_uint256_tests (stickies-v) c6c994c test: remove test-only uint160S (stickies-v) 62cc465 test: remove test-only uint256S (stickies-v) adc00ad test: remove test-only arith_uint256S (stickies-v) f51b237 refactor: rpc: use uint256::FromHex for ParseHashV (stickies-v) Pull request description: _Continuation of #30569._ Since fad2991, `uint256S()` has been [deprecated](fad2991#diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138) because it is less robust than the `base_blob::FromHex()` introduced in #30482. Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. (see also #30532) This PR removes `uint256S()` (and `uint160S()`) completely, with no non-test behaviour change. Specifically, the main changes in this PR are: - the (minimal) last non-test usage of `uint256S()` in `ParseHashV()` is removed without behaviour change, which can partially be verified by cherry-picking and/or modifying [this test commit](stickies-v@1f2b0fa)). - the test usage of `uint{160,256}S()` is removed, largely replacing it with `uint{160,256}::FromHex()` where applicable, potentially modifying the test by removing non-hex characters or dropping the test entirely if removing non-hex characters makes it redundant - the now unused `uint{160,256}S()` functions are removed completely. - unit test coverage on converting `uint256` <-> `arith_uint256` through `UintToArith256()` and `ArithToUint256()` is beefed up, and `arith_uint256` tests are moved to `arith_uint256_tests.cpp`, removing the `uint256_tests.cpp` dependency on `uint256h`, mirroring how the code is structured. _Note: `uint256::FromUserHex()` exists to more leniently construct uint256 from user input, allowing "0x" prefixes and too-short-input, as safer alternative to `uint256S()` where necessary._ ACKs for top commit: l0rinc: reACK 43cd83b hodlinator: re-ACK 43cd83b ryanofsky: Code review ACK 43cd83b. Only code change is a small refactoring which looks good. The rest of the PR is all test changes, which I only lightly reviewed, but seem to be positive and do what's described Tree-SHA512: 48147a4c6af671597df0f72c1b477ae4631cd2cae4645ec54d0e327611ff302c9899e344518c81242cdde82930f6ad23a3a7e6e0b80671816e9f457b9de90a5c
In
rest_getutxos
truncated txids such asaa
orff
are accepted. This is brittle at best.Fix it by rejecting any truncated (or overlarge) input.
Review note: This also starts a major refactor to rework hex parsing in Bitcoin Core, meaning that a few refactor commits are included as well. They are explained individually in the commit message and the work will be continued in the future.