Tags: stellar/stellar-core
Tags
Update testnet soroban settings file (#4569) # Description The cost type format expected in p22 has changed. This file needs to be updated for the testnet reset. <!--- Describe what this pull request does, which issue it's resolving (usually applicable for code changes). ---> # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Remove extra arg in LedgerCloseData constructor call for non buildtest ( #4563) # Description #4536 broke the non `BUILD_TEST` builds as a `BUILD_TEST` version of `LedgerCloseData`'s constructor was called from non `BUILD_TEST` code. We don't build non `BUILD_TEST` versions in github CI, and I had built preview packages with a revision before I added the `BUILD_TEST` conditionals, so this wasn't caught pre merge. ## Testing: * locally build with `./configure CXXFLAGS="-DBUILD_TESTS=0"` * run package builder on this branch https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-core-package-builder/2186/ ## Error: From https://buildmeister-v3.stellar-ops.com/job/Core/job/stellar-core-package-builder/2185/consoleText ``` n file included from catchup/ApplyCheckpointWork.cpp:5: In file included from ./catchup/ApplyCheckpointWork.h:7: In file included from ./herder/LedgerCloseData.h:7: In file included from ./herder/TxSetFrame.h:7: In file included from ./herder/SurgePricingUtils.h:7: In file included from /usr/lib/llvm-12/bin/../include/c++/v1/set:429: In file included from /usr/lib/llvm-12/bin/../include/c++/v1/__tree:15: /usr/lib/llvm-12/bin/../include/c++/v1/memory:2596:37: error: no matching constructor for initialization of 'stellar::LedgerCloseData' ::new ((void*)__get_elem()) _Tp(_VSTD::forward<_Args>(__args)...); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/lib/llvm-12/bin/../include/c++/v1/memory:3360:55: note: in instantiation of function template specialization 'std::__shared_ptr_emplace<stellar::LedgerCloseData, std::allocator<stellar::LedgerCloseData>>::__shared_ptr_emplace<unsigned int &, std::shared_ptr<const stellar::TxSetXDRFrame> &, stellar::StellarValue &, std::optional<xdr::opaque_array<32>>, std::optional<stellar::TransactionResultSet> &>' requested here ::new ((void*)_VSTD::addressof(*__guard.__get())) _ControlBlock(__a, _VSTD::forward<_Args>(__args)...); ^ /usr/lib/llvm-12/bin/../include/c++/v1/memory:3369:19: note: in instantiation of function template specialization 'std::allocate_shared<stellar::LedgerCloseData, std::allocator<stellar::LedgerCloseData>, unsigned int &, std::shared_ptr<const stellar::TxSetXDRFrame> &, stellar::StellarValue &, std::optional<xdr::opaque_array<32>>, std::optional<stellar::TransactionResultSet> &, void>' requested here return _VSTD::allocate_shared<_Tp>(allocator<_Tp>(), _VSTD::forward<_Args>(__args)...); ^ catchup/ApplyCheckpointWork.cpp:328:17: note: in instantiation of function template specialization 'std::make_shared<stellar::LedgerCloseData, unsigned int &, std::shared_ptr<const stellar::TxSetXDRFrame> &, stellar::StellarValue &, std::optional<xdr::opaque_array<32>>, std::optional<stellar::TransactionResultSet> &, void>' requested here return std::make_shared<LedgerCloseData>( ^ ./herder/LedgerCloseData.h:27:5: note: candidate constructor not viable: requires at most 4 arguments, but 5 were provided LedgerCloseData( ^ ./herder/LedgerCloseData.h:24:7: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 5 were provided class LedgerCloseData ^ ./herder/LedgerCloseData.h:24:7: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 5 were provided depbase=`echo catchup/IndexBucketsWork.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ clang++-12 -std=c++17 -DHAVE_CONFIG_H -I. -I.. -isystem ".." -I"../src" -I"../src" -isystem /build/stellar-core-22.0.1/lib/libsodium/src/libsodium/include -isystem ../lib/libsodium/src/libsodium/include -isystem ../lib/libsodium/src/libsodium/include/sodium -isystem /build/stellar-core-22.0.1/lib/xdrpp -isystem /build/stellar-core-22.0.1/lib/xdrpp -isystem ../lib/libmedida/src -isystem ../lib/soci/src/core -isystem ../lib/sqlite -DSQLITE_CORE -DSQLITE_OMIT_LOAD_EXTENSION=1 -DASIO_SEPARATE_COMPILATION=1 -DASIO_STANDALONE -isystem ../lib/asio/asio/include -isystem "../lib" -isystem "../lib/autocheck/include" -isystem "../lib/cereal/include" -isystem "../lib/util" -isystem "../lib/fmt/include" -isystem "../lib/soci/src/core" -isystem "../lib/tracy/public/tracy" -isystem "../lib/spdlog/include" -isystem "../rust/src" -DUSE_POSTGRES=1 -I/usr/include/postgresql -I"../src/protocol-curr" -DCEREAL_THREAD_SAFE -DUSE_SPDLOG -Wdate-time -D_FORTIFY_SOURCE=2 -stdlib=libc++ -fno-omit-frame-pointer -isystem /usr/include/libcxxabi -ggdb -O3 -fstack-protector-strong -Wformat -Werror=format-security -pthread -DFMT_HEADER_ONLY=1 -Wall -Wno-unused-command-line-argument -Qunused-arguments -Wno-unused-local-typedef -Wno-unknown-warning-option -Werror=unused-result -MT catchup/IndexBucketsWork.o -MD -MP -MF $depbase.Tpo -c -o catchup/IndexBucketsWork.o catchup/IndexBucketsWork.cpp &&\ mv -f $depbase.Tpo $depbase.Po ``` # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Build system changes for v22.0.0 (#4522) This is a sequence of cherry picks of (mostly) miscellaneous build-system infrastructure that I've recently landed on master transplanted onto the v22.0.0 branch. Each somewhat improves the build situation (eg. enabling tracy, disabling memory tracking, fixing asan, cleaning more thoroughly, forcing rust toolchain version identity) but none of them are essential for making the v22.0.0 release. You could take any or none of them, it'll be ok either way. There is also 72c5f9d which removes some stale cost-type adjustments in BUILD_TESTS-only code that's only active if you do multi-protocol replay -- the thing where we can run one tx twice, once on p21 and once on p22 -- and it might be nice to have that fixed so that we can do some more-confidence-inspiring side-by-side testing of p21 txs on the p22 codebase before the upgrade. But again it's really just a nice-to-have, none of this is essential.
JSON file for proposing phase5 network settings (#4491) ### Proposing an increase to the following network settings ``` Contract limit: contract_max_size_bytes 64KiB -> 128KiB Tx limit: Tx_max_write_bytes 65 KiB -> 129 KiB tx_max_size_bytes 70 KiB -> 129 KiB Ledger limit: Ledger_max_write_bytes 70 KiB -> 140 KiB Ledger_max_txs_size_bytes 70 KiB -> 130 KiB State archival limit: eviction_scan_size 100 KB -> 500 KB ``` Mainnet arm command: ` http-command 'upgrades?mode=set&upgradetime=YYYY-MM-DDT17:00:00Z&configupgradesetkey=V8RF//I9PaN2SnxMzvhxPGPgYKoh6Vq9HzRpqZeH9/MPiB7eTlpp1K/KrZdwabcdiJLhhhBIhhWwDSfnyyK+gA==' ` Output of the proposed settings on testnet, ``` curl -G 'http://localhost:11626/dumpproposedsettings' --data-urlencode 'blob=FwrrdNbCXOcY6HQ6+brP0HT/oZixrwPR6Z+8aM2CvDDD2B5syC79NSEMZMr0pcK9evrBfO1nQklXUAhAtp7DaQ==' { "ConfigSettingsEntries": [ { "configSettingID": "CONFIG_SETTING_CONTRACT_MAX_SIZE_BYTES", "contractMaxSizeBytes": 131072 }, { "configSettingID": "CONFIG_SETTING_CONTRACT_LEDGER_COST_V0", "contractLedgerCost": { "ledgerMaxReadLedgerEntries": 200, "ledgerMaxReadBytes": 500000, "ledgerMaxWriteLedgerEntries": 125, "ledgerMaxWriteBytes": 143360, "txMaxReadLedgerEntries": 40, "txMaxReadBytes": 200000, "txMaxWriteLedgerEntries": 25, "txMaxWriteBytes": 132096, "feeReadLedgerEntry": 6250, "feeWriteLedgerEntry": 10000, "feeRead1KB": 1786, "bucketListTargetSizeBytes": 300000000, "writeFee1KBBucketListLow": 9836, "writeFee1KBBucketListHigh": 12116, "bucketListWriteFeeGrowthFactor": 5000 } }, { "configSettingID": "CONFIG_SETTING_CONTRACT_BANDWIDTH_V0", "contractBandwidth": { "ledgerMaxTxsSizeBytes": 133120, "txMaxSizeBytes": 132096, "feeTxSize1KB": 1624 } }, { "configSettingID": "CONFIG_SETTING_STATE_ARCHIVAL", "stateArchivalSettings": { "maxEntryTTL": 3110400, "minTemporaryTTL": 17280, "minPersistentTTL": 2073600, "persistentRentRateDenominator": 2103, "tempRentRateDenominator": 4206, "maxEntriesToArchive": 1000, "bucketListSizeWindowSampleSize": 30, "bucketListWindowSamplePeriod": 64, "evictionScanSize": 500000, "startingEvictionScanLevel": 7 } } ] } ```
Testutils bug(s) (#4489) This makes 3 changes: 1. Stops passing `--features testutils` to soroban when building with `BUILD_TESTS` 2. Starts passing it to librust_stellar_core.a (where it enables the side-by-side execution feature) 3. Fixes a couple minor typos in the code that was gated by that The key part here is the first; evidently having `testutils` turned on on soroban makes the cost structure observably different and thereby causes divergence during replay. This seems wrong to me, and something we should track down and eliminate, but in the meantime this at least (a) gets rid of that divergence and (b) does what we did before I accidentally started passing `testutils` in the recent soroban-submodule change (#4456)
Binary fuse filter (#4389) # Description Addresses #4393. This PR adds a Binary Fuse Filter library and switches BucketListDB to use the new filter. The Binary Fuse Filter library is based on [this](https://github.com/FastFilter/xor_singleheader) library, but I ended up essentially rewriting it in C++. The original library had a bug where some filter bytes were non-deterministic. While this did not affect correctness or the false positive rate, we need determinism because State Archival will hash these filters into the ledger. Additionally, the library did not support 32 bit filters, which are required for State Archival. Currently a draft until the (non protocol breaking) associated XDR changes are merged: stellar/stellar-xdr#195 While Binary Fuse Filters are necessary for State Archival, they also significantly improve BucketListDB. Compared to the older bloom filter implementation, Bucket indexing time has decreased by 67% (which is blocking on startup in some cases), decreases index memory size by 20%, and reduces redundant disk reads by ~130x. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Binary fuse filter (#4389) # Description Addresses #4393. This PR adds a Binary Fuse Filter library and switches BucketListDB to use the new filter. The Binary Fuse Filter library is based on [this](https://github.com/FastFilter/xor_singleheader) library, but I ended up essentially rewriting it in C++. The original library had a bug where some filter bytes were non-deterministic. While this did not affect correctness or the false positive rate, we need determinism because State Archival will hash these filters into the ledger. Additionally, the library did not support 32 bit filters, which are required for State Archival. Currently a draft until the (non protocol breaking) associated XDR changes are merged: stellar/stellar-xdr#195 While Binary Fuse Filters are necessary for State Archival, they also significantly improve BucketListDB. Compared to the older bloom filter implementation, Bucket indexing time has decreased by 67% (which is blocking on startup in some cases), decreases index memory size by 20%, and reduces redundant disk reads by ~130x. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Update overlay survey script with lessons learned during testnet run (#… …4358) # Description This change makes a few tweaks to the overlay survey script to fix some small things I noticed after running it on testnet: * Changes the script's end condition to depend only on responses, and not requests. Without this it was possible for the survey script to run for the full duration of the collecting phase (2 hours) if a node with more than 25 peers stopped responding after the surveyor received the first set of peers. * Downgrades the severity of "node already in backlog" messages from `error` to `debug`. This is an expected condition that I simply forgot to special-case before. * Modifies the simulator to occasionally return "node already in backlog" messages to test the script against that case. * Adds a `--fast` option to the `simulate` mode that skips any `sleep` calls. This makes the script much nicer to test. * Fixes naming of graphml fields to match JSON result fields. * I did most of this in the V2 script update, but missed a couple spots. * Most of this change is in the simulator to support the new field names. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
PreviousNext