A v3 snapshot carries the UTXO set and raw block data (blk0001.dat) but
does NOT rebuild the per-tx index (txindex) that maps CTransaction hashes
to CDiskTxPos. Without it, any new PoS block referencing a pre-snapshot
transaction fails CheckProofOfStake with 'read txPrev failed':
CTransaction::ReadFromDisk(txdb, prevout, txindex) // src/main.cpp:714
if (!txdb.ReadTxIndex(prevout.hash, txindexRet)) // empty!
return false;
This stalls the node at the snapshot height and triggers DoS=100 on
every inbound peer feeding canonical blocks, masking as a network
misbehavior issue. DNS3 was stuck at 2,214,547 for this reason despite
the UTXO and block data being present.
Fix: after extracting blk0001.dat, walk it linearly and call
txdb.UpdateTxIndex(hash, CTxIndex(CDiskTxPos, nVout)) for every
transaction. O(N) over the historical block range, batched every
5000 txs. Adds ~30-60s to snapshot load on modern hardware.
This is the third leg of the v3 self-contained snapshot story:
v3 field source purpose
----------- ------------------------- -------------------------
headers last 2000 block headers block index continuity
utxos 22k unspent outputs UTXO set at tip
blocks blk0001.dat raw bytes on-disk block storage
setStakeSeen last 5000 PoS seen stakes stake collision dedup
txindex [this commit] PoS signature verification
Future 'v4 snapshot' work should consolidate all five into a single
load pass with progress reporting.
The writer seek calculation (contentHashPos - sizeof(numUtxos)) was
correct for v1/v2. In v3 the layout inserted numBlocks between
numUtxos and numStakeSeen, so the seek landed on the numBlocks field
and the updated count was written there, corrupting both fields.
Fix: compute the offset relative to contentHashPos, skipping the
contentHash, numStakeSeen, and numBlocks fields inserted in v2/v3.
Adds v3 snapshot format that includes the last 5000 PoS block
(prevoutStake, nStakeTime) pairs so a snapshot-loaded node has its
stake-collision set restored without walking the block index.
v2 readers still load v3 snapshots (the extra field is past numBlocks
and the loader checks version >= 3 to read numStakeSeen).
Bumps version to 6.1.1.
Per Sami's vision: 'I want to carry over the whole block inside the
UTXO.' The snapshot is now self-contained: a fresh node loading it
has everything needed (headers + UTXOs + all block bodies) without
needing a separate bootstrap tarball.
Format change (UTXO_SNAPSHOT_VERSION 1 → 2):
v1 HEADER (88 bytes):
magic, version, network, height, blockHash, moneySupply,
numHeaders, numUtxos, contentHash
v2 HEADER (92 bytes):
same + numBlocks (between numUtxos and contentHash)
v2 CONTENT (after v1's headers + utxos sections):
blocks[numBlocks] ← raw blk0001.dat bytes, SHA256 included
DumpSnapshot changes:
- Walks ALL blocks from pindexBest to pindexGenesisBlock (was: last
N=2000). The nHeaders arg is honored only when caller passes a
count smaller than the full chain for v1-compat diagnostic snapshots.
- After headers + utxos sections, streams GetDataDir()/blk0001.dat
bytes into the snapshot, chunked (64 KB), content-hashed.
- Header now writes numBlocks between numUtxos and contentHash.
LoadSnapshot changes:
- Reads numBlocks after numUtxos when version >= 2.
- After UTXOs section, streams numBlocks bytes from the snapshot
into dataDir/blk0001.dat (uses GetDataDir() since the param dataDir
is intentionally unnamed in this function).
- v1 snapshots still load via the partial-load path (no numBlocks in
header, no blk0001.dat written).
- Empty snapshot check loosened to (numHeaders && numUtxos && numBlocks)
— all three must be zero to be considered empty.
Total v2 snapshot size: ~1.9 GB (headers + blocks + UTXOs).
Generation on the operator machine: a few minutes. Download on
reasonable connection: a few minutes.
This supersedes the earlier v2 attempt (commit 69529ea) which had
compile bugs from using an unnamed dataDir parameter and had wrong
snapshot file layout.
Per Sami's vision: 'I want to carry over the whole block inside the
UTXO.' The snapshot should be self-contained so a fresh node is fully
usable — can serve blocks to peers, fully verify the chain, validate
txs, and resume syncing forward. Replaces the legacy tri-bootstrap.tar.gz.
Format change (UTXO_SNAPSHOT_VERSION 1 → 2):
v1 HEADER:
magic, version, network, height, blockHash, moneySupply,
numHeaders, numUtxos, contentHash (88 bytes)
v2 HEADER:
same + numBlocks (92 bytes) ← new field
v2 CONTENT (after v1's headers + utxos sections):
blocks[numBlocks] ← raw blk0001.dat bytes, SHA256 included
DumpSnapshot changes:
- Walks ALL blocks from pindexBest to pindexGenesisBlock (was: last
N=2000). The nHeaders arg is honored only when 0 < nHeaders < chain
height for v1-compat diagnostic snapshots.
- After writing headers + utxos sections, streams GetDataDir()/blk0001.dat
bytes into the snapshot, chunked (64 KB), content-hashed.
- Header now writes numBlocks between numUtxos and contentHash.
LoadSnapshot changes:
- Reads numBlocks after numUtxos when version >= 2.
- After the UTXOs section, streams numBlocks bytes from the snapshot
into dataDir/blk0001.dat.
- v1 snapshots (no numBlocks in header) still load via the partial
path: headers + UTXOs only, no blk0001.dat written. The 'block
verification skipped for snapshot-sourced chains' hack stays
for v1, becomes unnecessary for v2.
Total v2 snapshot size: ~1.9 GB (550 MB headers + 1.3 GB blocks + 50 MB UTXOs).
Generation on the operator machine: a few minutes. Download on
reasonable connection: a few minutes.
This commit is format-only — signature verification, auto-rebuild,
and the LoadBlockIndex crash fix from PR #8 still apply unchanged.
After LoadSnapshot, the daemon has headers + UTXOs but the raw block
bodies haven't been downloaded yet — they'll arrive via P2P as the
node syncs past the snapshot tip. LoadBlockIndex's verification
loop tries to read the last 50 block bodies from disk and fails
with 'OpenBlockFile failed' because the data isn't on disk yet.
Add fLoadedFromSnapshot global, set true at the end of successful
LoadSnapshot. In both txdb-leveldb.cpp and txdb-rocksdb.cpp LoadBlockIndex
verification loops, when ReadFromDisk fails AND fLoadedFromSnapshot is
true, log a warning and continue (the UTXO set itself was already
content-hash verified during LoadSnapshot, so we have strong evidence
the chain state is correct).
For non-snapshot chains (full blk0001.dat downloaded, normal IBD), the
ReadFromDisk failure remains a fatal error as before.
Combined with the prior fix in utxosnapshot.cpp that sets
fSerializeChainTrust=true before writes, the full snapshot path now
works end-to-end on a fresh datadir.
THE BUG: CDiskBlockIndex serialization is gated by a static flag
fSerializeChainTrust. LoadBlockIndex later sets this flag to true
based on dbformat >= 2 and tries to read nChainTrust as part of every
CDiskBlockIndex record.
But LoadSnapshot runs FIRST and writes CDiskBlockIndex records while
the static is still at its default value (false). The records are
written WITHOUT nChainTrust. Then LoadBlockIndex reads with flag=true,
expects nChainTrust, runs off the end of the buffer → 'CDataStream::read():
end of data: iostream error' → AppInit() exception.
This bug affected every fresh snapshot load: the snapshot's headers
and UTXOs loaded correctly (the per-record writes work), then the
post-load LoadBlockIndex crashed. Sami identified this as the
'format mismatch' blocker; the signature verification work went in
first but the underlying serialization bug remained.
Fix: explicitly set fSerializeChainTrust=true at the top of LoadSnapshot
before any CDiskBlockIndex writes. Then writes include nChainTrust.
Then LoadBlockIndex reads with the same flag set → matches.
The snapshot FILE format itself is unchanged — old snapshots produced
by daemons that wrote with flag=false will still fail to load (their
records don't have nChainTrust). New snapshots produced by daemons
that always write with flag=true (i.e. always include nChainTrust)
will load cleanly.
Local file snapshots (init.cpp) skip the known-checkpoint gate; P2P-delivered
snapshots (bootstrap.cpp) keep it. Rationale: the checkpoint gate exists to
prevent malicious peers from injecting fake UTXO sets. Local file loads come
from operator-trusted sources (filesystem access already grants equal power),
so the gate is unnecessary friction.
LoadSnapshot previously opened LevelDB directly at <datadir>/txleveldb to
write the snapshot in. Refactored to use the CTxDBBase abstraction:
- WipeChainDataDir() removes the configured backend's chain DB dir
- MakeChainDB("c+") opens fresh via the factory
- High-level methods (WriteBlockIndex, WriteUtxo, WriteHashBestChain,
WriteVersion, WriteDbFormat) replace manual key/value construction
- TxnBegin/Commit cycles every 1000 headers / 50000 UTXOs preserve the
prior batching cadence
The IsRocksDbChainBackend() guard added in 76579e3 is dropped — snapshot
loading now works on either backend.
Two adjacent paths in init.cpp also hardcoded "txleveldb": the snapshot
auto-load guard (Step 6c) and the -reindex datadir wipe. Both updated to
GetChainDataDir() / WipeChainDataDir() so they pick the right directory
for the configured backend.
Helpers added to txdb.h / txdb-factory.cpp:
- GetChainDataDir(): on-disk path of the configured backend's chain DB
- WipeChainDataDir(): rm -rf the same path
Bootstrap archive paths (bootstrap.cpp lines 721+) intentionally still
reference txleveldb specifically — the prebuilt-index distribution
remains LevelDB-format until that pipeline is ported separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DumpSnapshot reached into the LevelDB backend's internal handle via
`extern leveldb::DB *txdb`, which silently broke under -chaindb=rocksdb.
Switched to the backend-agnostic CTxDBBase::NewIterator() interface; the
function now works against either backend.
LoadSnapshot is more involved (writes directly into a fresh txleveldb/
directory) and is bundled with the eventual LevelDB retirement. Added an
IsRocksDbChainBackend() helper and an explicit guard at LoadSnapshot's
entry: refuse to load with a clear error message rather than silently
creating a leveldb tree alongside an active rocksdb chain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migration from boost to std-library equivalents and removal of unreachable
code paths. Touches infrastructure only — no consensus rule or wallet
serialization changes.
Dead code removed:
- IRC bootstrap (irc.cpp/h, 417 lines): orphan from pre-Tor era, no callers.
- Alert system (alert.cpp/h + sendalert RPC + Qt UI signal, ~500 lines):
retired post-V5 fork; old peers' alert messages now hit the unknown-cmd
default branch, logged + ignored.
- Legacy P2P handlers in main.cpp: "checkpoint" (already a no-op stub since
V5 fork master-key removal), "checkorder"/"reply" (2010-era Receive-by-IP
feature), plus their unused supporting structures (CRequestTracker,
PushRequest overloads, mapRequests/cs_mapRequests, mapReuseKey).
- Unreachable RPCs clearwallettransactions and scanforalltxns (~175 lines):
defined in rpcwallet.cpp but never registered in the dispatch table.
- Stale -alertnotify CLI help text (option was advertised but never wired).
boost::filesystem -> std::filesystem (C++17):
- 30 source files, 5 headers. namespace fs = boost::filesystem swapped to
namespace fs = std::filesystem; boost::filesystem::ifstream/ofstream
replaced with std::ifstream/ofstream (path-aware in C++17);
fs::system_complete -> fs::absolute; boost::filesystem::filesystem_error
-> std::filesystem::filesystem_error.
- Build system: dropped Boost::filesystem from link libs and Boost
components; PCH includes updated.
- Added explicit <filesystem> includes where types were previously
available only transitively (db.h, rpcblockchain.cpp).
boost::thread -> std::thread (12 files):
- sync.h CCriticalSection/CWaitableCriticalSection now alias
std::recursive_mutex/std::mutex. boost::unique_lock and
boost::condition_variable / boost::mutex::scoped_lock swapped to std
equivalents; sync.cpp boost::thread_specific_ptr -> thread_local
std::unique_ptr.
- init.cpp boost::thread_group rewritten as std::vector<std::thread> with
manual join loop. boost::thread::hardware_concurrency ->
std::thread::hardware_concurrency.
- main.cpp/wallet.cpp -blocknotify/-walletnotify shell-out threads now use
std::thread(...).detach() — fixes a latent bug where modern boost::thread
destructor would call std::terminate on the joinable thread.
- util.cpp NewThread now catches std::system_error.
- No interruption_point/interrupt usage anywhere — pure mechanical swap.
boost::chrono / boost::posix_time -> std::chrono (3 of 5 files):
- util.h: MilliSleep, GetTimeMillis, GetTimeMicros rewritten on std::chrono
(system_clock for epoch math, sleep_for for delays).
- snapshotnet.cpp: sleep_for swapped.
- DoS_tests.cpp: timing harness uses steady_clock.
- Skipped: rpcdump.cpp (boost::posix_time::time_input_facet has no clean
std::get_time equivalent) and qt/qtipcserver.cpp (locked to
boost::posix_time by boost::interprocess::message_queue::timed_receive).
Other housekeeping:
- Dropped unnecessary "using namespace boost;" from txdb-leveldb.cpp,
txdb-rocksdb.cpp, walletdb.cpp, db.cpp (verified no unqualified boost
names in those TUs).
- Removed unused extern declaration for clearwallettransactions.
Build fixes for non-unity builds on MinGW64/GCC 15:
- net.cpp: dropped stale #include "irc.h".
- addrman.cpp + main.cpp: explicit <cmath> include for sqrt/pow (was
arriving transitively via boost headers).
- rpcblockchain.cpp + init.cpp: defensive #undef STRICT/ADVISORY/PERMISSIVE
since windows.h macros collide with the Checkpoints:: enum values when
std headers reorder include flow.
- tor_embed_hooks.cpp: triangles_tor_check_interrupted now polls fShutdown
instead of boost::this_thread::interruption_requested (we never used
boost interruption — the hook was always effectively a no-op).
- snapshotnet.cpp: fs::remove error handle uses std::error_code.
- serialize.h: added <ios> for std::ios::badbit/failbit (was relying on
transitive include via boost).
Note: unity builds currently fail on this branch due to std::byte (C++17)
colliding with COM 'byte' typedef from shlobj.h when 'using namespace std;'
from earlier files in the unity slice leaks into util.cpp's parse of
shlobj.h. Build with -DENABLE_UNITY_BUILD=OFF (the default).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The extern declaration for the global leveldb::DB *txdb was inside
namespace UtxoSnapshot{}, causing the linker to look for
UtxoSnapshot::txdb instead of the global ::txdb defined in
txdb-leveldb.cpp.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add UTXO snapshot dump/load system (utxosnapshot.cpp/h) for fast initial sync
- Add dumputxoset RPC command to create snapshots from current chain state
- Add script verification cache (sigcache.h) to skip re-verifying scripts
already validated during mempool acceptance
- Bootstrap: try UTXO snapshot first (fast path), fall back to full bootstrap
- Support manual utxo-snapshot.bin loading on startup
- Tune sync parameters for Tor: increase timeouts, reduce buffer sizes
- Header sync cache: TTL-based eviction instead of full cache clear
- Reduce orphan block limits and script check batch size for lower memory usage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Krystie
sami7777