-
Notifications
You must be signed in to change notification settings - Fork 36.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backport wtxid relay to v0.20 #19606
Conversation
7ee4769 [0.20] lint: fix shellcheck URL in CI install (fanquake) Pull request description: This is causing the tests to fail for backports i.e #19606. If you look in the Travis logs there, the output is: ```bash You are downloading ShellCheck from an outdated URL! Please update to the new URL: https://github.com/koalaman/shellcheck/releases/download/v0.6.0/shellcheck-v0.6.0.linux.x86_64.tar.xz For more information, see: koalaman/shellcheck#1871 PS: Sorry for breaking your build. The hosting costs were getting out of hand :( ``` ACKs for top commit: MarcoFalke: review ACK 7ee4769 Tree-SHA512: 62470291e53954ab541a7109e530390410d9b8a4d3ed6f4128ab8807d2225f368b8c984342f92de802a60dd082292cb59557599b4112413a29fc9ad8e8bcd0ee
e9dcffe
to
afb6a20
Compare
Rebased to pick up the shellcheck fix |
afb6a20
to
91646ff
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code review ACK 91646ff
All differences either related to the unbroadcast set work not existing, some inner help functions for mempool data retrieval, or adapting tests.
Did some basic spot-checking of the functional testing even though it matched the master PR, and fails when expected.
@@ -343,6 +345,7 @@ def on_reject(self, message): pass | |||
def on_sendcmpct(self, message): pass | |||
def on_sendheaders(self, message): pass | |||
def on_tx(self, message): pass | |||
def on_wtxidrelay(self, message): pass | |||
|
|||
def on_inv(self, message): | |||
want = msg_getdata() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: two below here does not get changed to checking for
+ if (i.type == MSG_TX) or (i.type == MSG_WTX):
as in master PR. Tests seem to still work when I make the change.
Looks like wallet_resendwallettransactions.py is having the sads sometimes in travis:
|
91646ff
to
4daad79
Compare
Concept ACK |
Concept ACK on backporting for a future 0.20.2 or later concomitant with 0.21. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK 4daad79 modulo that removed test, although I am not sure if this is a blocker for a backport.
Checked the diffs with git range-diff 2b4b90aa8f0440deacefb5997d7bd1f9f5c591b3~1..0a4f1422cd1c20e12a05d7ff1a2ef1d5e7c654bb 0615a589dd70afd7280921b7e32f96c1e018cb99~1..4daad798f3b5ea258cb183932956101e8df24e94
and looked at the commits with relevant changes in more detail.
@@ -1295,8 +1316,6 @@ def test_tx_relay_after_segwit_activation(self): | |||
# Node will not be blinded to the transaction | |||
self.std_node.announce_tx_and_wait_for_getdata(tx3) | |||
test_transaction_acceptance(self.nodes[1], self.std_node, tx3, True, False, 'tx-size') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In 59f38ad
Hm, here I deleted that test by accident which @instagibbs added back, see #19649. Might want to not remove it here as well.
Backport looks ok to me ACK 4daad79 |
4daad79
to
f5d90ea
Compare
Rebased Specifically, there were conflicts around the conditionals for adding wtxid/txids to recent rejects in commits Add wtxids to recentRejects and Make TX_WITNESS_STRIPPED its own rejection reason. If it's too difficult to review those changes, we could revert #19680 so these are merged into V0.20 in the same order as master. |
Code review ACK f5d90ea Confirmed relevant commits to review: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK f5d90ea on non-python commits. Verified by cherry-picking commits one-by-one from #18044 on top of the 0.20
branch.
This piece of code
bitcoin/src/net_processing.cpp
Lines 1971 to 1975 in f5d90ea
if (orphan_state.GetResult() == TxValidationResult::TX_INPUTS_NOT_STANDARD && orphanTx.GetWitnessHash() != orphanTx.GetHash()) { | |
// We only add the txid if it differs from the wtxid, to | |
// avoid wasting entries in the rolling bloom filter. | |
recentRejects->insert(orphanTx.GetHash()); | |
} |
bitcoin/src/net_processing.cpp
Lines 2755 to 2757 in f5d90ea
if (state.GetResult() == TxValidationResult::TX_INPUTS_NOT_STANDARD && tx.GetWitnessHash() != tx.GetHash()) { | |
recentRejects->insert(tx.GetHash()); | |
} |
f5d90ea
to
73b74e0
Compare
This adds a field to CNodeState that tracks whether to relay transactions with that peer via wtxid, instead of txid. As of this commit the field will always be false, but in a later commit we will add a way to negotiate turning this on via p2p messages exchanged with the peer.
When sent to and received from a given peer, enables using wtxid's for announcing and fetching transactions with that peer.
Using both txid and wtxid-based relay with peers means that we could sometimes download the same transaction twice, if announced via two different hashes from different peers. Use a heuristic of delaying txid-peer-getdata requests by 2 seconds, if we have at least one wtxid-based peer.
Previously, TX_WITNESS_MUTATED could be returned during transaction validation for either transactions that had a witness that was non-standard, or for transactions that had no witness but were invalid due to segwit validation rules. However, for txid/wtxid-relay considerations, net_processing distinguishes the witness stripped case separately, because it affects whether a wtxid should be able to be added to the reject filter. It is safe to add the wtxid of a witness-mutated transaction to the filter (as that wtxid shouldn't collide with the txid, and hence it wouldn't interfere with transaction relay from txid-relay peers), but it is not safe to add the wtxid (== txid) of a witness-stripped transaction to the filter, because that would interfere with relay of another transaction with the same txid (but different wtxid) when relaying from txid-relay peers. Also updates the comment explaining this logic, and explaining that we can get rid of this complexity once there's a sufficient deployment of wtxid-relaying peers on the network.
This new p2p protocol version allows to use WTXIDs for tx relay.
Also cleans up some doublicate lines in the rest of the test. co-authored-by: Anthony Towns <aj@erisian.com.au>
ef0569e
to
d4a1ee8
Compare
Rebased to pick up appveyor fixes |
re-ACK d4a1ee8 Confirmed no changes since my last review: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK d4a1ee8, only rebased since my previous review:
$ git range-diff master f5d90eadc182a2a3d1cffa4cc01b4fb68f9dc567 d4a1ee8f1d4c46ab726be83965bd86bace2ec1ec
1: 004b0e1b9 = 1: 004b0e1b9 build: Bump version to 0.20.0
2: cd0c99890 = 2: cd0c99890 doc: Update manpages pre-rc1
3: cd1f5bf1d = 3: cd1f5bf1d qt: Update translations pre-rc1
4: ade4185e6 = 4: ade4185e6 gitian: Add missing automake package to gitian-win-signer.yml
5: 842b13a5f = 5: 842b13a5f Avoid non-trivial global constants in SHA-NI code
6: 1d1e3585f = 6: 1d1e3585f build: Set libevent minimum version to 2.0.21
7: 6986b2634 = 7: 6986b2634 build: fix ASLR for bitcoin-cli on Windows
8: 54d2063d1 = 8: 54d2063d1 Do not expose and consider -logthreadnames when it does not work
9: a9ca65bd2 = 9: a9ca65bd2 Fix naming of macOS SDK and clarify version
10: 7f7548d82 = 10: 7f7548d82 rpc: Do not advertise dumptxoutset as a way to flush the chainstate
11: 59d57f6c1 = 11: 59d57f6c1 build: Ensure source tarball has leading directory name
12: 7d87ba0e0 = 12: 7d87ba0e0 travis: Remove valgrind
13: aa7c6858e = 13: aa7c6858e travis: Remove s390x
14: 315ae14f3 = 14: 315ae14f3 gui: Fix itemWalletAddress leak when not tree mode
15: fb821731e = 15: fb821731e [net processing] ignore tx GETDATA from blocks-only peers
16: 1e73d7248 = 16: 1e73d7248 [net processing] ignore unknown INV types in GETDATA messages
17: 011532e38 = 17: 011532e38 [test] test that an invalid GETDATA doesn't prevent processing of future messages
18: a3fe458a1 = 18: a3fe458a1 [docs] Improve commenting in ProcessGetData()
19: ca4dac48c = 19: ca4dac48c rpc: Add mutex to guard deadlineTimers
20: 251e321ad = 20: 251e321ad rpc: Relock wallet only if most recent callback
21: ed0afe8c1 = 21: ed0afe8c1 test: Add test for conflicted wallet tx notifications
22: ff4dc2075 = 22: ff4dc2075 gui: Fix manual coin control with multiple wallets loaded
23: 37a620748 = 23: 37a620748 test: Add unregister_validation_interface_race test
24: cc7d34465 = 24: cc7d34465 miner: Avoid stack-use-after-return in validationinterface
25: cf2a6e2a3 = 25: cf2a6e2a3 test: Remove const to work around compiler error on xenial
26: 6161c94a6 = 26: 6161c94a6 [net processing] Only send a getheaders for one block in an INV
27: 9a8fb4cf4 = 27: 9a8fb4cf4 fuzz: Remove enumeration of expected deserialization exceptions in ProcessMessage(...) fuzzer
28: 245c862cf = 28: 245c862cf test: disable script fuzz tests
29: 0793eca3a = 29: 0793eca3a qt: Pre-rc2 translations update
30: 6f7f94a27 = 30: 6f7f94a27 build: Bump RC to rc2
31: 1dfad4259 = 31: 1dfad4259 doc: Merge 0.20.0 release notes from wiki
32: 808c8d15f = 32: 808c8d15f build: Set rc to 0 for -final
33: a62f0ed64 = 33: a62f0ed64 doc: Manpages update pre-final
34: 384d3f991 = 34: 384d3f991 Add missing QPainterPath include
35: 0596a6eeb = 35: 0596a6eeb util: Don't reference errno when pthread fails.
36: c219d2163 = 36: c219d2163 build: improved output of configure for build OS
37: 5c7151a60 = 37: 5c7151a60 gui: update Qt base translations for macOS release
38: febebc4ea = 38: febebc4ea Fix WSL file locking by using flock instead of fcntl
39: 654420d6d = 39: 654420d6d wallet: Minimal fix to restore conflicted transaction notifications
40: 27786d072 = 40: 27786d072 trivial: Suggested cleanups to surrounding code
41: 68e0e6f85 = 41: 68e0e6f85 rpc: show both UTXOs in decodepsbt
42: ed5ec3080 = 42: ed5ec3080 psbt: Allow both non_witness_utxo and witness_utxo
43: 3228b59b1 = 43: 3228b59b1 psbt: always put a non_witness_utxo and don't remove it
44: cf0b5a933 = 44: cf0b5a933 tests: Check that segwit inputs in psbt have both UTXO types
45: c9b49d285 = 45: c9b49d285 wallet: Handle concurrent wallet loading
46: eb6b82a55 = 46: eb6b82a55 qa: Test concurrent wallet loading
47: e7f06f9b0 = 47: e7f06f9b0 test: remove Cirrus CI FreeBSD job
48: 047734805 = 48: 047734805 Replace automatic bans with discouragement filter
49: 2b79ac740 = 49: 2b79ac740 Clean up separated ban/discourage interface
50: fa3998859 = 50: fa3998859 doc: Clear release notes for minor release
51: 888886ed5 = 51: 888886ed5 doc: move-only release notes for 0.20.1
52: faf5e256c = 52: faf5e256c appveyor: Remove clcache
53: fae0e9383 = 53: fae0e9383 Remove cached directories and associated script blocks from appveyor CI configuration.
54: bad9cf8f4 = 54: bad9cf8f4 Increment input value sum only once per UTXO in decodepsbt
55: 8b4093749 = 55: 8b4093749 qt: Fix QFileDialog for static builds
56: cd34ff546 = 56: cd34ff546 build: Bump version to 0.20.1rc1
57: 5e21c55ef = 57: 5e21c55ef doc: Regenerate man pages for 0.20.1rc1
58: cac7a9809 = 58: cac7a9809 qt: Translation update for 0.20.1rc1
59: 58feb9ecb = 59: 58feb9ecb doc: Add changelog and authors to release notes for 0.20.1
60: 7ee4769cd = 60: 7ee4769cd [0.20] lint: fix shellcheck URL in CI install
61: 7ff64311b = 61: 7ff64311b build: Bump version to 0.20.1-final
62: 7c1c15329 = 62: 7c1c15329 doc: Update 0.20.1 release notes with psbt changes
63: 06f9c5c3b = 63: 06f9c5c3b Add txids with non-standard inputs to reject filter
64: 107cf1515 = 64: 107cf1515 test addition of unknown segwit spends to txid reject filter
-: --------- > 65: ad99777b5 Set appveyor vm version to previous Visual Studio 2019 release.
-: --------- > 66: 498b7cb6f Update the vcpkg checkout commit ID in appveyor config.
65: 220a18c3c = 67: 4df3d139b Add a wtxid-index to the mempool
66: 7cedceb60 = 68: f7833b5bd Just pass a hash to AddInventoryKnown
67: b6a583851 = 69: 365493767 Add a wtxid-index to mapRelay
68: 735be4534 = 70: 606755b84 Add wtxid-index to orphan map
69: c695be95a = 71: 73845211d Add wtxids of confirmed transactions to bloom filter
70: 36c340238 = 72: be1b7a891 Add wtxids to recentRejects
71: 91b2e8074 = 73: 2599277e9 Add support for tx-relay via wtxid
72: ab2ce25b5 = 74: 93826726e ignore non-wtxidrelay compliant invs
73: 3ae27adca = 75: 181ffadd1 Add p2p message "wtxidrelay"
74: fa93551fe = 76: c1d6a1003 Delay getdata requests from peers using txid-based relay
75: 76de3f897 = 77: 879a3cf2c Make TX_WITNESS_STRIPPED its own rejection reason
76: f53543078 = 78: e364b2a2d Rename AddInventoryKnown() to AddKnownTx()
77: 5e33fe096 = 79: 6be398b6f test: Update test framework p2p protocol version to 70016
78: 9551605e5 = 80: e48168196 test: Add tests for wtxid tx relay in segwit test
79: ebd00f4af = 81: 22effa51a test: Use wtxid relay generally in functional tests
80: cbb581bdb = 82: f082a13ab Disconnect peers sending wtxidrelay message after VERACK
81: f5d90eadc = 83: d4a1ee8f1 Further improve comments around recentRejects
@instagibbs @laanwj - do you mind rereviewing? |
reACK d4a1ee8 |
@laanwj: should be ok for merge if you reACK |
re-ACK d4a1ee8 |
fa074d2 Revert "Merge #19606: Backport wtxid relay to v0.20" (MarcoFalke) Pull request description: The 0.20 branch has bugfixes that should be released. However, a tag can currently not be created because the latest merge introduced a regression and is not a bugfix (#20317 (comment), #20317 (comment)). Fix that by reverting the last merge. Can be reviewed by re-doing the revert or calling `git diff HEAD HEAD~2 | wc` and observing an empty diff. ACKs for top commit: laanwj: Code review ACK fa074d2 Tree-SHA512: 1a1314b9bb85f44696dc307845e80292998d6c9c000e7386c48405e74400d9cd22be6996e555f198da917e04024a1c8e609dfd830759a27fe4070168b0d272bb
We want wtxid relay to be widely deployed before taproot activation, so it should be backported to v0.20.
The main difference from #18044 is removing the changes to the unbroadcast set (which was only added post-v0.20). The rest is mostly minor rebase conflicts (eg connman changed from a pointer to a reference in master, etc).
We'll also want to backport #19569 after that's merged.