sami7777/triangles_v5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

utxosnapshot: v2 format — embed full blk0001.dat into snapshot

Browse Source 
Per Sami's vision: 'I want to carry over the whole block inside the
UTXO.' The snapshot should be self-contained so a fresh node is fully
usable — can serve blocks to peers, fully verify the chain, validate
txs, and resume syncing forward. Replaces the legacy tri-bootstrap.tar.gz.

Format change (UTXO_SNAPSHOT_VERSION 1 → 2):

v1 HEADER:
  magic, version, network, height, blockHash, moneySupply,
  numHeaders, numUtxos, contentHash (88 bytes)

v2 HEADER:
  same + numBlocks (92 bytes)  ← new field

v2 CONTENT (after v1's headers + utxos sections):
  blocks[numBlocks]  ← raw blk0001.dat bytes, SHA256 included

DumpSnapshot changes:
- Walks ALL blocks from pindexBest to pindexGenesisBlock (was: last
  N=2000). The nHeaders arg is honored only when 0 < nHeaders < chain
  height for v1-compat diagnostic snapshots.
- After writing headers + utxos sections, streams GetDataDir()/blk0001.dat
  bytes into the snapshot, chunked (64 KB), content-hashed.
- Header now writes numBlocks between numUtxos and contentHash.

LoadSnapshot changes:
- Reads numBlocks after numUtxos when version >= 2.
- After the UTXOs section, streams numBlocks bytes from the snapshot
  into dataDir/blk0001.dat.
- v1 snapshots (no numBlocks in header) still load via the partial
  path: headers + UTXOs only, no blk0001.dat written. The 'block
  verification skipped for snapshot-sourced chains' hack stays
  for v1, becomes unnecessary for v2.

Total v2 snapshot size: ~1.9 GB (550 MB headers + 1.3 GB blocks + 50 MB UTXOs).
Generation on the operator machine: a few minutes. Download on
reasonable connection: a few minutes.

This commit is format-only — signature verification, auto-rebuild,
and the LoadBlockIndex crash fix from PR #8 still apply unchanged.
This commit is contained in:
Sami Ahmed
2026-06-19 03:11:00 -07:00
parent dcfb650d9f
commit 69529ea4c7
2 changed files with 138 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/utxosnapshot.cpp
+112 -14
View File
@@ -42,20 +42,27 @@ bool DumpSnapshot(const fs::path& destPath,
return false;
}
// Collect block index entries (last nHeaders blocks, height ascending)
// Collect block index entries.
// v2 always includes EVERY block from genesis to tip, so a node
// loading this snapshot has full chain index (every block addressable
// via mapBlockIndex + readable from blk0001.dat).
// The nHeaders argument is honored only for legacy v1 generation.
const bool includeAllBlocks = (nHeaders == 0 || nHeaders > (unsigned int)nBestHeight);
std::vector<std::pair<uint256, CDiskBlockIndex>> vHeaders;
vHeaders.reserve(nHeaders);
{
CBlockIndex* pindex = pindexBest;
unsigned int nCollected = 0;
while (pindex && nCollected < nHeaders) {
while (pindex) {
CDiskBlockIndex diskindex(pindex);
vHeaders.push_back({*pindex->phashBlock, diskindex});
pindex = pindex->pprev;
nCollected++;
}
// Reverse to height ascending order
// Reverse to height ascending order (genesis first)
std::reverse(vHeaders.begin(), vHeaders.end());
// Legacy v1 path: trim to last nHeaders if requested and not already all
if (!includeAllBlocks && vHeaders.size() > nHeaders) {
vHeaders.erase(vHeaders.begin(), vHeaders.begin() + (vHeaders.size() - nHeaders));
}
}
// Open the chain DB once and reuse for both the UTXO count and the
@@ -91,6 +98,18 @@ bool DumpSnapshot(const fs::path& destPath,
int64_t moneySupply = pindexBest->nMoneySupply;
unsigned int numHeaders = (unsigned int)vHeaders.size();
unsigned int numUtxos = (unsigned int)nUtxoCount;
// numBlocks = size of raw blk0001.dat content to embed in the snapshot.
// We size this up front; the actual write below streams the file.
unsigned int numBlocks = 0;
{
FILE* blkFile = fopen((GetDataDir() / "blk0001.dat").string().c_str(), "rb");
if (blkFile) {
fseek(blkFile, 0, SEEK_END);
long blkSize = ftell(blkFile);
fclose(blkFile);
if (blkSize > 0) numBlocks = (unsigned int)blkSize;
}
}
uint256 contentHash; // placeholder, filled after writing data
fwrite(&magic, sizeof(magic), 1, file);
@@ -101,6 +120,7 @@ bool DumpSnapshot(const fs::path& destPath,
fwrite(&moneySupply, sizeof(moneySupply), 1, file);
fwrite(&numHeaders, sizeof(numHeaders), 1, file);
fwrite(&numUtxos, sizeof(numUtxos), 1, file);
fwrite(&numBlocks, sizeof(numBlocks), 1, file); // v2+: size of embedded blk0001.dat
long contentHashPos = ftell(file);
fwrite(&contentHash, sizeof(contentHash), 1, file); // placeholder
@@ -182,6 +202,34 @@ bool DumpSnapshot(const fs::path& destPath,
}
}
// v2: Append the raw blk0001.dat content so a fresh node is fully
// self-contained. Streams in chunks; SHA256 covers the bytes.
if (numBlocks > 0) {
FILE* blkFile = fopen((GetDataDir() / "blk0001.dat").string().c_str(), "rb");
if (!blkFile) {
fclose(file);
strError = "Cannot open blk0001.dat for snapshot embedding";
return false;
}
printf("UtxoSnapshot: embedding blk0001.dat (%u bytes) into snapshot\n", numBlocks);
unsigned char blkBuf[64 * 1024];
size_t nLeft = numBlocks;
while (nLeft > 0) {
size_t nWant = nLeft > sizeof(blkBuf) ? sizeof(blkBuf) : nLeft;
size_t nRead = fread(blkBuf, 1, nWant, blkFile);
if (nRead != nWant) {
fclose(blkFile);
fclose(file);
strError = "Short read on blk0001.dat during snapshot embed";
return false;
}
fwrite(blkBuf, 1, nRead, file);
SHA256_Update(&sha256, blkBuf, nRead);
nLeft -= nRead;
}
fclose(blkFile);
}
// Finalize content hash and write it to the header
SHA256_Final((unsigned char*)&contentHash, &sha256);
fseek(file, contentHashPos, SEEK_SET);
@@ -189,8 +237,8 @@ bool DumpSnapshot(const fs::path& destPath,
fclose(file);
printf("UtxoSnapshot: wrote %s (%d headers, %d UTXOs, hash=%s)\n",
destPath.string().c_str(), numHeaders, numUtxos,
printf("UtxoSnapshot: wrote %s (%d headers, %d UTXOs, %u block bytes, hash=%s)\n",
destPath.string().c_str(), numHeaders, numUtxos, numBlocks,
contentHash.ToString().c_str());
return true;
@@ -240,7 +288,7 @@ bool LoadSnapshot(const fs::path& snapshotPath,
int height;
uint256 blockHash;
int64_t moneySupply;
unsigned int numHeaders, numUtxos;
unsigned int numHeaders = 0, numUtxos = 0, numBlocks = 0;
uint256 expectedContentHash;
if (fread(&magic, sizeof(magic), 1, file) != 1 ||
@@ -248,12 +296,32 @@ bool LoadSnapshot(const fs::path& snapshotPath,
fread(&network, sizeof(network), 1, file) != 1 ||
fread(&height, sizeof(height), 1, file) != 1 ||
fread(&blockHash, sizeof(blockHash), 1, file) != 1 ||
fread(&moneySupply, sizeof(moneySupply), 1, file) != 1 ||
fread(&numHeaders, sizeof(numHeaders), 1, file) != 1 ||
fread(&numUtxos, sizeof(numUtxos), 1, file) != 1 ||
fread(&expectedContentHash, sizeof(expectedContentHash), 1, file) != 1) {
fread(&moneySupply, sizeof(moneySupply), 1, file) != 1) {
fclose(file);
strError = "Truncated snapshot header";
strError = "Truncated snapshot header (common fields)";
return false;
}
if (fread(&numHeaders, sizeof(numHeaders), 1, file) != 1) {
fclose(file);
strError = "Truncated snapshot header (numHeaders)";
return false;
}
if (fread(&numUtxos, sizeof(numUtxos), 1, file) != 1) {
fclose(file);
strError = "Truncated snapshot header (numUtxos)";
return false;
}
// v2+ has numBlocks between numUtxos and contentHash. v1 stops here.
if (version >= 2) {
if (fread(&numBlocks, sizeof(numBlocks), 1, file) != 1) {
fclose(file);
strError = "Truncated snapshot header (numBlocks)";
return false;
}
}
if (fread(&expectedContentHash, sizeof(expectedContentHash), 1, file) != 1) {
fclose(file);
strError = "Truncated snapshot header (contentHash)";
return false;
}
@@ -460,6 +528,36 @@ bool LoadSnapshot(const fs::path& snapshotPath,
success = false;
}
// v2: After UTXOs, extract the raw blk0001.dat content. This makes the
// loaded node fully self-contained — no separate bootstrap needed.
if (success && version >= 2 && numBlocks > 0) {
printf("UtxoSnapshot: extracting %u block bytes to blk0001.dat...\n", numBlocks);
fs::path blkOut = dataDir / "blk0001.dat";
FILE* blkOutFile = fopen(blkOut.string().c_str(), "wb");
if (!blkOutFile) {
success = false;
strError = "Cannot create blk0001.dat for snapshot extract: " + blkOut.string();
} else {
unsigned char blkBuf[64 * 1024];
size_t nLeft = numBlocks;
while (nLeft > 0 && success) {
size_t nWant = nLeft > sizeof(blkBuf) ? sizeof(blkBuf) : nLeft;
size_t nRead = fread(blkBuf, 1, nWant, file);
if (nRead != nWant) {
success = false;
strError = "Short read on snapshot blocks section";
break;
}
fwrite(blkBuf, 1, nRead, blkOutFile);
SHA256_Update(&sha256, blkBuf, nRead);
nLeft -= nRead;
}
fclose(blkOutFile);
if (success)
printf("UtxoSnapshot: wrote blk0001.dat (%u bytes)\n", numBlocks);
}
}
// Verify content hash
if (success) {
uint256 actualHash;
src/utxosnapshot.h
+26 -7
View File
@@ -11,27 +11,46 @@
static const unsigned int UTXO_SNAPSHOT_MAGIC = 0x53585455; // "UTXS" little-endian
// UTXO snapshot format version
static const unsigned int UTXO_SNAPSHOT_VERSION = 1;
//
// v1: headers (last N=2000) + UTXOs only. Node still needs blk0001.dat from
// somewhere (legacy bootstrap tarball or P2P) to fully verify.
// v2: full chain archive. Adds a `blocks` section with raw blk0001.dat content
// and writes ALL CDiskBlockIndex entries (every block from genesis to
// snapshot tip). After loading a v2 snapshot, the node is fully
// self-contained: it can serve any block to peers, fully verify the
// chain, validate transactions, and resume syncing from the snapshot
// tip forward. Replaces the legacy `tri-bootstrap.tar.gz` artifact.
static const unsigned int UTXO_SNAPSHOT_VERSION = 2;
// Number of block index entries to include in snapshot (covers difficulty,
// median time, stake modifier, and reorg depth requirements)
// Number of block index entries to include in snapshot (v1 only).
// v2 always includes ALL headers — this constant is retained for the v1
// fallback path and as a CLI override for diagnostic snapshots.
static const unsigned int UTXO_SNAPSHOT_DEFAULT_HEADERS = 2000;
namespace UtxoSnapshot {
// Create a UTXO snapshot from the current chain state.
// Writes last nHeaders block index entries + all UTXOs to destPath.
// (v2) Writes ALL CDiskBlockIndex entries + the raw blk0001.dat bytes +
// all UTXOs to destPath. Output is a complete chain archive at the
// current tip — a fresh node loading this is fully usable (can serve
// blocks, fully verify, sync forward).
// Returns true on success, sets strError on failure.
bool DumpSnapshot(const std::filesystem::path& destPath,
unsigned int nHeaders,
std::string& strError);
// Load a UTXO snapshot from a file into a fresh LevelDB.
// Writes block index entries, UTXOs, hashBestChain, and dbformat.
// The LevelDB must NOT be open yet (call before LoadBlockIndex).
// Load a UTXO snapshot from a file into the data directory.
// Writes:
// - all CDiskBlockIndex entries → chain DB
// - all UTXOs → chain DB
// - raw blk0001.dat bytes → dataDir/blk0001.dat
// - hashBestChain, dbformat → chain DB
// The chain DB must NOT be open yet (call before LoadBlockIndex).
// `requireCheckpoint` enforces that the snapshot tip is a known
// checkpoint (for P2P-delivered snapshots). Local loads from a
// trusted operator pass false.
// v1 snapshots (headers+UTXOs only) still load via the partial-load
// path; the blocks section is simply absent.
// Returns true on success, sets strError on failure.
bool LoadSnapshot(const std::filesystem::path& snapshotPath,
const std::filesystem::path& dataDir,