Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rest: Reject truncated hex txid early in getutxos parsing #30482

Merged
merged 7 commits into from
Jul 25, 2024

Conversation

maflcko
Copy link
Member

@maflcko maflcko commented Jul 19, 2024

In rest_getutxos truncated txids such as aa or ff are accepted. This is brittle at best.

Fix it by rejecting any truncated (or overlarge) input.


Review note: This also starts a major refactor to rework hex parsing in Bitcoin Core, meaning that a few refactor commits are included as well. They are explained individually in the commit message and the work will be continued in the future.

@DrahtBot
Copy link
Contributor

DrahtBot commented Jul 19, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK stickies-v, hodlinator

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

@DrahtBot DrahtBot changed the title rest: Reject truncated hex txid early in getutxos parsing rest: Reject truncated hex txid early in getutxos parsing Jul 19, 2024
@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed.
Debug: https://github.com/bitcoin/bitcoin/runs/27652526405

Hints

Make sure to run all tests locally, according to the documentation.

The failure may happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

Copy link
Contributor

@stickies-v stickies-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach ACK 9984bdc

Straightforward improvement to make the code more robust, very nice.

src/test/pow_tests.cpp Show resolved Hide resolved
src/uint256.h Outdated Show resolved Hide resolved
src/rest.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@l0rinc l0rinc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup, please see my questions and observations

src/util/transaction_identifier.h Outdated Show resolved Hide resolved
src/rest.cpp Outdated Show resolved Hide resolved
src/rest.cpp Outdated Show resolved Hide resolved
src/uint256.cpp Outdated Show resolved Hide resolved
src/uint256.cpp Show resolved Hide resolved
src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
src/uint256.cpp Outdated Show resolved Hide resolved
src/uint256.cpp Outdated Show resolved Hide resolved
src/uint256.cpp Outdated Show resolved Hide resolved
src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
@hodlinator
Copy link
Contributor

@paplorinc the first 3 commits here come from pending PR #30436, so comments on those changes are probably better to make there.

Copy link
Contributor

@hodlinator hodlinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to see the added functional test conditions for truncated txids!

src/util/transaction_identifier.h Show resolved Hide resolved
src/rest.cpp Outdated Show resolved Hide resolved
src/rest.cpp Outdated Show resolved Hide resolved
@maflcko maflcko force-pushed the 2407-rest-txid branch 3 times, most recently from a9d1df0 to e8d8b98 Compare July 23, 2024 14:29
@maflcko
Copy link
Member Author

maflcko commented Jul 23, 2024

Pushed and replied to all feedback, except for #30482 (comment), which I'll do later

@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed.
Debug: https://github.com/bitcoin/bitcoin/runs/27809611813

Hints

Make sure to run all tests locally, according to the documentation.

The failure may happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

src/rest.cpp Show resolved Hide resolved
src/test/blockfilter_tests.cpp Show resolved Hide resolved
src/uint256.h Outdated Show resolved Hide resolved
src/uint256.h Outdated Show resolved Hide resolved
@maflcko maflcko marked this pull request as ready for review July 24, 2024 07:12
MarcoFalke added 2 commits July 24, 2024 09:14
This avoids a hex-decoding and makes the next commit smaller.
SetHex is fragile, because it accepts any non-hex input or any length of
input, without error feedback. This can lead to issues when the input is
truncated or otherwise corrupted.

Document the problem by renaming the method.

In the future, the fragile method should be removed from the public
interface.

-BEGIN VERIFY SCRIPT-
 sed -i 's/SetHex/SetHexDeprecated/g' $( git grep -l SetHex ./src )
-END VERIFY SCRIPT-
@maflcko maflcko force-pushed the 2407-rest-txid branch 2 times, most recently from fa459e6 to aaaa66d Compare July 24, 2024 07:40
Copy link
Contributor

@hodlinator hodlinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK aaaa66d

src/bitcoin-tx.cpp Show resolved Hide resolved
src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
src/uint256.h Outdated Show resolved Hide resolved
Copy link
Contributor

@hodlinator hodlinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without seeing an exploratory branch where SetHexDeprecated is removed, I think it might be better to call it SetHexDiscouraged after all, in case it turns out to be really problematic to remove it for some unforeseen reason.

src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
@maflcko
Copy link
Member Author

maflcko commented Jul 24, 2024

Without seeing an exploratory branch where SetHexDeprecated is removed, I think it might be better to call it SetHexDiscouraged after all, in case it turns out to be really problematic to remove it for some unforeseen reason.

Thank you for the suggestion. However, outside of src/qt, and internally, the function is used in a single place uint256S, so the name shouldn't matter much. There is already a lengthy bike-shedding thread about the naming in #30482 (comment), with you concluding that the current name is fine. So I don't really have the motivation and time to go back and forth on this style nit, unless there is a convincing reason. Even if it was "problematic to remove it for some unforeseen reason", a 9 line scripted-diff can be applied then.

I appreciate the feedback, but when it comes to trivial style nits, I don't consider it worth it to spend days on it myself.

@hodlinator
Copy link
Contributor

I appreciate the feedback, but when it comes to trivial style nits, I don't consider it worth it to spend days on it myself.

Fair.

src/test/uint256_tests.cpp Show resolved Hide resolved
src/uint256.h Show resolved Hide resolved
src/uint256.h Outdated Show resolved Hide resolved
src/rest.cpp Show resolved Hide resolved
src/rest.cpp Show resolved Hide resolved
src/test/blockfilter_tests.cpp Show resolved Hide resolved
src/test/blockfilter_tests.cpp Show resolved Hide resolved
src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
@maflcko maflcko force-pushed the 2407-rest-txid branch 2 times, most recently from fac13e9 to fa93c83 Compare July 24, 2024 10:20
Copy link
Contributor

@stickies-v stickies-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fa93c83

Left a couple of suggestions but nothing blocking.

Returning an early "Parse error" for malformed txids instead of silently ignoring them is a better user interface, and the underlying hex parse refactoring is a very welcome improvement.

src/test/uint256_tests.cpp Outdated Show resolved Hide resolved
src/util/transaction_identifier.h Outdated Show resolved Hide resolved
src/uint256.h Show resolved Hide resolved
src/rest.cpp Show resolved Hide resolved
@DrahtBot DrahtBot requested a review from hodlinator July 24, 2024 15:08
MarcoFalke added 5 commits July 24, 2024 17:38
This is a safe replacement of the previous SetHex, which now returns an
optional to indicate success or failure.

The code is similar to the ParseHashStr helper, which will be removed in
a later commit.
This is needed for the next commit.
No need to have two functions with different names that achieve the
exact same thing.
@stickies-v
Copy link
Contributor

re-ACK fac0c3d - only doc and test updates to address review comments, thanks!

Copy link
Contributor

@hodlinator hodlinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fac0c3d

make check passed and test_runner.py passed.

Happy you went with FromHex()!

@@ -86,7 +86,7 @@ BOOST_AUTO_TEST_CASE(CheckProofOfWork_test_negative_target)
uint256 hash;
unsigned int nBits;
nBits = UintToArith256(consensus.powLimit).GetCompact(true);
hash.SetHex("0x1");
hash = uint256{1};
Copy link
Contributor

@hodlinator hodlinator Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Man, I was worried for a moment that this modification of switching to the uint8_t ctor was setting the other end of the base_blob array, but it's the same. Will be glad to see the fragile function go. 👍

@fanquake fanquake merged commit 30e8a79 into bitcoin:master Jul 25, 2024
16 checks passed
@maflcko maflcko deleted the 2407-rest-txid branch July 25, 2024 12:55
@@ -66,6 +72,7 @@ using Txid = transaction_identifier<false>;
/** Wtxid commits to all transaction fields including the witness. */
using Wtxid = transaction_identifier<true>;

/** DEPRECATED due to missing length-check and hex-check, please use the safer FromHex, or FromUint256 */
inline Txid TxidFromString(std::string_view str)
Copy link
Member Author

@maflcko maflcko Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If someone is looking for follow-up work, a test-only change could be to replace TxidFromString(a) with Txid::FromHex(a).value() in src/test.

git grep TxidFromString ./src/test/
git grep WtxidFromString

Copy link
Contributor

@stickies-v stickies-v Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've started working on a branch that removes (W)TxidFromString entirely, it seems the qt callsites would work well with FromHex too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #30532. Also started working on a similar branch to remove uint256S.

glozow added a commit that referenced this pull request Aug 1, 2024
…of transaction_identifier::FromHex()

f553e6d refactor: remove TxidFromString (stickies-v)
285ab50 test: replace WtxidFromString with Wtxid::FromHex (stickies-v)
9a0b2a6 fuzz: increase FromHex() coverage (stickies-v)
526a87b test: add uint256::FromHex unittest coverage (stickies-v)

Pull request description:

  Since fab6ddb, `TxidFromString()` has been deprecated because it is less robust than the `transaction_identifier::FromHex()` introduced in [the same PR](#30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters.

  In this PR, `TxidFromString` is removed completely to clean up the code and prevent further unsafe usage. Unit and fuzz test coverage on `uint256::FromHex()` and functions that wrap it is increased.

  Note: `TxidFromSring` allowed users to prefix strings with "0x", this is no longer allowed for `transaction_identifier::FromHex()`, so a helper function for input validation may prove helpful in the future _(this overlaps with the `uint256::uint256S()` vs `uint256::FromHex()` future cleanup)_. It is not relevant to this PR, though, besides the fact that this unused (except for in tests) functionality is removed.

  The only users of `TxidFromString` are:
  - `test`, where it is straightforward to drop in the new `FromHex()` methods without much further concern
  - `qt` coincontrol. There is no need for input validation here, but txids are not guaranteed to be 64 characters. This is already handled by the existing code, so again, using `FromHex()` here seems quite straightforward.

  Addresses @maflcko's suggestion: #30482 (comment)

  Also removes `WtxidFromString()`, which is a test-only helper function.

  ### Testing GUI changes

  To test the GUI coincontrol affected lines, `regtest` is probably the easiest way to quickly get some test coins, you can use e.g.

  ```
  alias cli="./src/bitcoin-cli -regtest"
  cli createwallet "coincontrol"
  # generate 10 spendable outputs on 1 address
  cli generatetoaddress 10 $(cli -rpcwallet=coincontrol getnewaddress)
  # generate 10 spendable outputs on another address
  cli generatetoaddress 10 $(cli -rpcwallet=coincontrol getnewaddress)
  # make previous outputs spendable
  cli generatetoaddress 100 $(cli -rpcwallet=coincontrol getnewaddress)
  ```

ACKs for top commit:
  maflcko:
    re-ACK f553e6d 🔻
  hodlinator:
    ACK f553e6d
  paplorinc:
    ACK f553e6d
  TheCharlatan:
    Nice, ACK f553e6d

Tree-SHA512: c1c7e6ea4cbf05cf660ba178ffc4f35f0328f7aa6ad81872e2462fb91a6a22e4681ff64b3d0202a5a9abcb650c939561585cd309164a69ab6081c0765ee271ef
luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Aug 1, 2024
This is a safe replacement of the previous SetHex, which now returns an
optional to indicate success or failure.

The code is similar to the ParseHashStr helper, which will be removed in
a later commit.

Github-Pull: bitcoin#30482
Rebased-From: fad2991
luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Aug 1, 2024
This is needed for the next commit.

Github-Pull: bitcoin#30482
Rebased-From: fab6ddb
luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Aug 1, 2024
achow101 added a commit that referenced this pull request Aug 27, 2024
18d65d2 test: use uint256::FromUserHex for RANDOM_CTX_SEED (stickies-v)
6819e5a node: use uint256::FromUserHex for -assumevalid parsing (stickies-v)
2e58fdb util: remove unused IsHexNumber (stickies-v)
8a44d7d node: use uint256::FromUserHex for -minimumchainwork parsing (stickies-v)
70e2c87 refactor: add uint256::FromUserHex helper (stickies-v)
85b7cbf test: unittest chainstatemanager_args (stickies-v)

Pull request description:

  Since fad2991, `uint256S` has been [deprecated](fad2991#diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138) because it is less robust than the `base_blob::FromHex()` introduced in [the same PR](#30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. _(see also #30532)_

  This PR carves out the few `uint256S` callsites that may potentially prove a bit more controversial to change because they deal with user input and backwards incompatible behaviour change.

  The main behaviour change introduced in this PR is:
  - `-minimumchainwork` will raise an error when input is longer than 64 hex digits
  - `-assumevalid` will raise an error when input contains invalid hex characters, or when it is longer than 64 hex digits
  - test: the optional RANDOM_CTX_SEED env var will now cause tests to abort when it contains invalid hex characters, or when it is longer than 64 hex digits

  After this PR, the remaining work to remove `uint256S` completely is almost entirely mechanical and/or test related. I will open that PR once #30560 is merged because it builds on that.

ACKs for top commit:
  hodlinator:
    re-ACK 18d65d2
  l0rinc:
    ACK 18d65d2
  achow101:
    ACK 18d65d2
  ryanofsky:
    Code review ACK 18d65d2. Very nice change that cleans up the API, adds checking for invalid values, makes parsing of values more consistent, and adds test coverage.

Tree-SHA512: ec118ea3d56e1dfbc4c79acdbfc797f65c4d2107b0ca9577c848b4ab9b7cb8d05db9f3c7fe8441a09291aca41bf671572431f4eddc59be8fb53abbea76bbbf86
ryanofsky added a commit that referenced this pull request Aug 31, 2024
8756ccd scripted-diff: Replace ParseHex[<std::byte>]("str") -> "str"_hex[_u8] (Hodlinator)
9cb6873 refactor: Prepare for ParseHex -> ""_hex scripted-diff (Hodlinator)
50bc017 refactor: Hand-replace some ParseHex -> ""_hex (Hodlinator)
5b74a84 util: Add consteval ""_hex[_v][_u8] literals (l0rinc)
dc5f6f6 test refactor: util_tests - parse_hex clean up (Hodlinator)
2b5e6ef refactor: Make XOnlyPubKey tolerate constexpr std::arrays (Hodlinator)
403d86f refactor: vector -> span in CCrypter (Hodlinator)
bd0830b refactor: de-Hungarianize CCrypter (Hodlinator)
d99c816 refactor: Improve CCrypter related lines (Hodlinator)
7e1d9a8 refactor: Enforce lowercase hex digits for consteval uint256 (Hodlinator)

Pull request description:

  Motivation:
  * Validates and converts the hex string into bytes at compile time instead of at runtime like `ParseHex()`.
  * Eliminates runtime dependencies: #30377 (comment), #30048 (comment)
  * Has stricter requirements than `ParseHex()` (disallows whitespace and uppercase hex digits) and replaces it in a bunch of places.
  * Makes it possible to derive other compile time constants.
  * Minor: should shave off a few runtime CPU cycles.

  `""_hex` produces `std::array<std::byte>` as the momentum in the codebase is to use `std::byte` over `uint8_t`.

  Also makes `uint256` hex string constructor disallow uppercase hex digits. Discussed: #30560 (comment)

  Surprisingly does not change the size of the Guix **bitcoind** binary (on x86_64-linux-gnu) by 1 single byte.

  Spawned already merged PRs: #30436, #30482, #30532, #30560.

ACKs for top commit:
  l0rinc:
    ACK 8756ccd
  stickies-v:
    Rebase re-ACK 8756ccd, no changes since a096215
  ryanofsky:
    Code review ACK 8756ccd, just rebasing since last review and taking advantage of CScript constructors in #29369, also tweaking a code comment

Tree-SHA512: 9b2011b7c37e0ef004c669f8601270a214b388916316458370f5902c79c2856790b1b2c7c123efa65decad04886ab5eff95644301e0d84358bb265cf1f8ec195
ryanofsky added a commit that referenced this pull request Sep 10, 2024
43cd83b test: move uint256_tests/operator_with_self to arith_uint256_tests (stickies-v)
c6c994c test: remove test-only uint160S (stickies-v)
62cc465 test: remove test-only uint256S (stickies-v)
adc00ad test: remove test-only arith_uint256S (stickies-v)
f51b237 refactor: rpc: use uint256::FromHex for ParseHashV (stickies-v)

Pull request description:

  _Continuation of #30569._

  Since fad2991, `uint256S()` has been [deprecated](fad2991#diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138) because it is less robust than the `base_blob::FromHex()` introduced in #30482. Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. (see also #30532)

  This PR removes `uint256S()` (and `uint160S()`) completely, with no non-test behaviour change.

  Specifically, the main changes in this PR are:
  - the (minimal) last non-test usage of `uint256S()` in `ParseHashV()` is removed without behaviour change, which can partially be verified by cherry-picking and/or modifying [this test commit](stickies-v@1f2b0fa)).
  - the test usage of `uint{160,256}S()` is removed, largely replacing it with `uint{160,256}::FromHex()` where applicable, potentially modifying the test by removing non-hex characters or dropping the test entirely if removing non-hex characters makes it redundant
  - the now unused `uint{160,256}S()` functions are removed completely.
  - unit test coverage on converting `uint256` <-> `arith_uint256` through `UintToArith256()` and `ArithToUint256()` is beefed up, and `arith_uint256` tests are moved to `arith_uint256_tests.cpp`, removing the `uint256_tests.cpp` dependency on `uint256h`, mirroring how the code is structured.

  _Note:  `uint256::FromUserHex()` exists to more leniently construct uint256 from user input, allowing "0x" prefixes and too-short-input, as safer alternative to `uint256S()` where necessary._

ACKs for top commit:
  l0rinc:
    reACK 43cd83b
  hodlinator:
    re-ACK 43cd83b
  ryanofsky:
    Code review ACK 43cd83b. Only code change is a small refactoring which looks good. The rest of the PR is all test changes, which I only lightly reviewed, but seem to be positive and do what's described

Tree-SHA512: 48147a4c6af671597df0f72c1b477ae4631cd2cae4645ec54d0e327611ff302c9899e344518c81242cdde82930f6ad23a3a7e6e0b80671816e9f457b9de90a5c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants