Compare commits

..

9 Commits

Author SHA1 Message Date
Krystie e7c5c6596a Merge fix/recalculate-supply-chainwalk into master: IBD stall fix + supply recalculation
Build All Platforms / test-linux-unit (push) Failing after 40s
Build All Platforms / build-linux-qt (push) Failing after 40s
Build All Platforms / build-linux-daemon (push) Failing after 40s
Build All Platforms / build-windows-qt (push) Has been cancelled
Build All Platforms / build-windows-daemon (push) Has been cancelled
Build All Platforms / build-macos (push) Has been cancelled
Build All Platforms / release (push) Has been cancelled
Build All Platforms / Trigger TRI-PI ARM64 Build (push) Has been cancelled
2026-04-24 15:03:42 -07:00
sami7777 16b35f6b2b Fix IBD stall from header-sync cache exhaustion (v5.9.2)
Nodes syncing from zero would accept blocks normally up to ~6000 then
stall permanently with askfor_queue=0 and no new blocks. Root cause: a
broken feedback loop between the header planner and block downloader.
Blocks consume entries from mapHeaderSync (MAX 15000) while getheaders
refills only 2000 at a time; when the cache drains, hashBestHeaderSync
falls to 0 and every refill site is guarded on it being non-zero, so
the pipeline deadlocks with no recovery path.

Recovery paths added:

- ProcessBlock: when the cache is empty during IBD after accepting a
  block, broadcast getheaders to all full-node peers. This restarts
  the planner at the exact point it dies.
- Stall detection: send getheaders alongside the existing getblocks.
  getblocks alone cannot refill the header cache.
- SendMessages: belt-and-suspenders, re-request headers every 30s
  while hashBestHeaderSync == 0 in IBD, independent of stall state.

Also fix a secondary issue: GetHeaderSyncDownloadPath walks back from
the tip and breaks on the first TTL-evicted entry. The accumulated
partial tail has a parent that is neither in mapBlockIndex nor
mapHeaderSync, so requesting those blocks would produce orphans.
Discard the partial path on a gap; the recovery paths above will
re-request the missing range.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 11:48:35 -07:00
Krystie 7faf13dc31 Fix IBD stall: refill header cache when exhausted during sync
Two fixes for the header cache exhaustion bug:

1. Block-accepted path: when hashBestHeaderSync==0 and we're still
   behind peers during IBD, send getheaders to all peers to refill
   the header cache. Previously the refill was gated on
   hashBestHeaderSync!=0, creating a dead loop once the cache drained.

2. Stall detection: also send getheaders alongside getblocks when
   a stall is detected. Previously only getblocks was sent, which
   cannot refill mapHeaderSync or restart the header planner.

Root cause: getheaders returns 2000 headers per batch. Blocks are
consumed from the cache faster than headers are fetched. Once
mapHeaderSync empties, hashBestHeaderSync becomes 0, and the
refill path is never taken again.

See BUG_ANALYSIS_IBD_STALL.md for full details.
2026-04-24 11:21:39 -07:00
Krystie db65324b7a Add IBD stall bug analysis: header cache exhaustion without refill 2026-04-24 11:20:18 -07:00
sami7777 c98bdbe335 Harden recalculatesupply: MoneyRange gate, atomic apply, single-walk (v5.9.1)
Follow-up to #5. Addresses three risks with the apply=true path:

- MoneyRange sanity gate: refuse to persist a recalculated supply that is
  negative or above MAX_MONEY (2,222,222 TRI). A walk that produces an
  out-of-range figure indicates a bug (orphan contamination, missing
  prevout), not real chain state. Prevents corrupting nMoneySupply with
  junk values.
- Atomic apply: wrap every per-block WriteBlockIndex in a single
  TxnBegin/TxnCommit so a mid-walk failure leaves on-disk state
  untouched instead of half-rewritten.
- Single chain walk: cache (valueOut - valueIn) per block during the
  dry-run pass and reuse the cached deltas during apply. Previous code
  walked the full chain twice, roughly doubling apply runtime on a
  2.2M-block chain.

Help text now warns that the RPC holds cs_main for the full walk and
blocks new blocks, wallet ops, and other RPC for the duration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 19:56:33 -07:00
SamiAhmed7777 6f1227b022 Merge pull request #5 from SamiAhmed7777/fix/recalculate-supply-chainwalk
Add full-chain supply recalculation RPC
2026-04-23 18:51:55 -07:00
Krystie 12205cdc37 Add full-chain supply recalculation RPC
Rebuild money supply by walking the active chain from genesis and
summing block valueOut - valueIn, instead of relying only on current
UTXO totals. Optionally persist repaired nMoneySupply values across the
active chain with apply=true.

This helps repair corrupted money-supply tracking after chain/index
incidents and exposes both recalculated chain supply and UTXO supply for
comparison.
2026-04-23 18:50:14 -07:00
SamiAhmed7777 2fc0e8155a Merge pull request #4 from SamiAhmed7777/update-explorer-url
Update block explorer URL on Qt wallet Overview page
2026-04-23 18:23:25 -07:00
Krystie eeda728564 Update block explorer URL to blocks.cryptographic-triangles.org
Replace the old explorer.triangles.technology link on the Qt wallet
Overview page with the new self-hosted block explorer at
https://blocks.cryptographic-triangles.org
2026-04-23 18:22:05 -07:00
6 changed files with 348 additions and 30 deletions
+157
View File
@@ -0,0 +1,157 @@
# IBD Stall Bug Analysis — `triangles_v5`
**Date:** 2026-04-24
**Symptom:** Node syncs from genesis, accepts blocks normally up to a point (observed: ~6284), then permanently stalls. `askfor_queue=0`, `orphans=0`, no new blocks ever arrive.
---
## Root Cause: Header Sync Cache Exhaustion Without Refill
The bug is a **broken feedback loop** between the header planner and block downloader. The node drains its header cache faster than it refills it, and once the cache is empty, the pipeline freezes with no recovery path.
### The Pipeline (how it should work)
```
getheaders → 2000 headers → mapHeaderSync → AskFor(MSG_BLOCK) → mapAskFor → getdata → block received → ProcessBlock → QueueHeaderSyncBlocksParallel (refill)
└── every 500 blocks: getheaders+getblocks to ALL peers
```
### The Bug Path (how it actually dies)
**Step 1: Initial header fetch**
- `version` handler sends `getblocks` + `getheaders` to peer
- Peer responds with up to 2000 headers → stored in `mapHeaderSync`
- `QueueHeaderSyncBlocksParallel(512)` queues up to 512 blocks via `AskFor()`
**Step 2: Blocks download and consume headers**
- Blocks arrive, `ProcessBlock()` accepts them
- Each accepted block calls `MarkHeaderSyncBlockAccepted()` which **erases** it from `mapHeaderSync`
- After accepting, `QueueHeaderSyncBlocksParallel(512)` tries to queue more from remaining `mapHeaderSync` entries
**Step 3: The critical gap — header cache runs dry**
- The header cache holds at most `MAX_HEADER_SYNC_CACHE = 15000` entries
- But `getheaders` only returns **2000 headers per batch**
- The `HEADER_DOWNLOAD_WINDOW = 512` means only 512 blocks are in-flight at once
- So blocks are consumed from the cache faster than headers are fetched
- Each accepted block erases its entry; if all 2000 headers are downloaded before the next `getheaders` fires...
**Step 4: Cache empties → pipeline dies**
- `mapHeaderSync` becomes empty
- `hashBestHeaderSync` is recomputed to `0` (by `RecomputeBestHeaderSync()`)
- The refill condition `if (hashBestHeaderSync != 0)` at line ~3599 evaluates **false**
- No more blocks are queued, ever
**Step 5: No recovery mechanism kicks in**
- The `ContinueHeaderSync()` call only fires when `vHeaders.size() >= 2000` (full batch)
- If the last batch was smaller (partial response, or exactly 2000 consumed), **no new `getheaders` is sent**
- The pipeline refill every 500 blocks only fires **when a block is received** — but no blocks are coming
- The stall detection sends `getblocks` (not `getheaders`), which produces `inv` messages → `AskFor()` for individual blocks
- But `getblocks` uses `CBlockLocator` with exponential spacing, which maps to an old block → peer sends `inv` for blocks we already have → walk-forward logic tries to progress but may loop or stall
### Why It's Worse on Fast Connections / Sync-from-Zero
- Blocks download fast (PoS blocks are tiny)
- All 2000 headers are consumed quickly
- The window between "all headers consumed" and "need more headers" is tiny
- On slow Tor connections, the 15-minute TTL eviction (`PruneHeaderSync`) adds a second failure mode: headers that took too long to download get evicted, creating gaps in `GetHeaderSyncDownloadPath()`
---
## Affected Code Locations
| File | Line(s) | Issue |
|------|---------|-------|
| `main.cpp` | 137 | `MAX_HEADER_SYNC_CACHE = 15000` — cache is large but `getheaders` only returns 2000 |
| `main.cpp` | 138 | `HEADER_DOWNLOAD_WINDOW = 512` — window is smaller than header batch |
| `main.cpp` | 298-345 | `AddHeaderSyncNode()` / `PruneHeaderSync()` — TTL eviction can create gaps in the download path |
| `main.cpp` | 380-396 | `GetHeaderSyncDownloadPath()` — walks back from tip; **breaks on first gap** in `mapHeaderSync` chain |
| `main.cpp` | 455-461 | `MarkHeaderSyncBlockAccepted()` — erases from `mapHeaderSync`, may set `hashBestHeaderSync = 0` |
| `main.cpp` | 3599-3606 | Block-accepted refill — **guarded by `hashBestHeaderSync != 0`**, skips when cache is empty |
| `main.cpp` | 4972-4976 | `getheaders` continuation — **only fires on full batch** (`vHeaders.size() >= 2000`) |
| `main.cpp` | 5110-5121 | Pipeline refill every 500 blocks — **only fires when blocks arrive**, useless during stall |
| `main.cpp` | 5952-5980 | Stall detection — sends `getblocks` (not `getheaders`), can't restart header planner |
---
## Fix Options
### Fix A: Refill headers when cache runs dry (minimal, targeted)
In the block-accepted handler, when `hashBestHeaderSync == 0`, send `getheaders` to all peers:
```cpp
// After the existing refill (line ~3599)
if (hashBestHeaderSync == 0)
{
// Header cache exhausted — request more headers from all peers
LOCK(cs_vNodes);
for (CNode* pnode : vNodes)
{
if (!pnode->fClient && pnode->nVersion != 0)
{
pnode->pindexLastGetHeadersBegin = NULL;
pnode->PushGetHeaders(pindexBest, uint256(0));
}
}
}
```
### Fix B: Also send getheaders in stall detection (defense in depth)
In the stall handler (line ~5968), alongside the `getblocks`, also send `getheaders`:
```cpp
pto->PushGetBlocks(...); // existing
pto->pindexLastGetHeadersBegin = NULL;
pto->PushGetHeaders(pindexBest, uint256(0)); // ADD THIS
```
### Fix C: Don't let the header cache fully drain (robustness)
In `QueueHeaderSyncBlocksParallel()`, stop consuming the last N entries from the cache to keep the chain intact. When only `HEADER_DOWNLOAD_WINDOW` entries remain, trigger a `getheaders` continuation before consuming more.
### Fix D: Periodic getheaders in SendMessages loop (most robust)
Add a periodic `getheaders` request in the `SendMessages` loop, similar to how stall detection already sends periodic `getblocks`. This ensures headers are always being fetched regardless of block progress:
```cpp
// In SendMessages, alongside stall detection:
if (!pto->fClient && IsInitialBlockDownload() && hashBestHeaderSync == 0)
{
static int64_t nLastHeaderRequest = 0;
if (GetTime() - nLastHeaderRequest >= 30)
{
pto->pindexLastGetHeadersBegin = NULL;
pto->PushGetHeaders(pindexBest, uint256(0));
nLastHeaderRequest = GetTime();
}
}
```
---
## Recommended Fix
**Fix A + Fix B together** — minimal code change, covers both the block-accepted path and the stall recovery path. Fix D adds belt-and-suspenders protection in the main loop.
---
## Secondary Issue: TTL Eviction Creating Path Gaps
`PruneHeaderSync()` evicts entries older than 15 minutes. If block download is slow (Tor, slow peers), entries at the beginning of the download path can be evicted while entries at the end still exist. `GetHeaderSyncDownloadPath()` walks back from the tip and **breaks on the first missing entry**, making the entire tail of the cache unreachable.
**Fix:** In `GetHeaderSyncDownloadPath()`, skip gaps instead of breaking:
```cpp
while (hashTip != 0 && !mapBlockIndex.count(hashTip))
{
auto mi = mapHeaderSync.find(hashTip);
if (mi == mapHeaderSync.end())
break; // Currently breaks — could skip to next known ancestor instead
vPath.push_back(hashTip);
hashTip = mi->second.header.hashPrevBlock;
}
```
This is a secondary concern but contributes to cache exhaustion on high-latency connections.
+1 -1
View File
@@ -6,7 +6,7 @@ if(POLICY CMP0167)
endif()
project(Triangles
VERSION 5.9.0
VERSION 5.9.2
DESCRIPTION "Cryptographic Triangles Wallet"
LANGUAGES C CXX
)
+1 -1
View File
@@ -8,7 +8,7 @@
// These need to be macros, as version.cpp's and triangles-qt.rc's voodoo requires it
#define CLIENT_VERSION_MAJOR 5
#define CLIENT_VERSION_MINOR 9
#define CLIENT_VERSION_REVISION 0
#define CLIENT_VERSION_REVISION 2
#define CLIENT_VERSION_BUILD 0
// Converts the parameter X to a string after macro replacement on X has been performed.
+22
View File
@@ -3607,6 +3607,23 @@ bool ProcessBlock(CNode* pfrom, CBlock* pblock)
printf("IBD-DIAG: queued %u more blocks from header planner after accepting %s\n",
nQueued, hash.ToString().substr(0,20).c_str());
}
else if (IsInitialBlockDownload() && nBestHeight < GetNumBlocksOfPeers())
{
// Header cache exhausted during IBD — request more headers from all peers.
// This fixes the stall where mapHeaderSync drains to empty, hashBestHeaderSync
// becomes 0, and the refill path is never taken again.
printf("IBD-DIAG: header cache empty at height %d, requesting more headers from all peers\n",
nBestHeight);
LOCK(cs_vNodes);
for (CNode* pnode : vNodes)
{
if (!pnode->fClient && pnode->nVersion != 0)
{
pnode->pindexLastGetHeadersBegin = NULL;
pnode->PushGetHeaders(pindexBest, uint256(0));
}
}
}
return true;
}
@@ -5970,6 +5987,11 @@ bool SendMessages(CNode* pto, bool fSendTrickle)
} else {
pto->PushGetBlocks(pindexBest, uint256(0));
}
// Also send getheaders during stall to restart the header planner.
// Without this, a drained header cache stays empty because only
// getblocks is sent on stall, which can't refill mapHeaderSync.
pto->pindexLastGetHeadersBegin = NULL;
pto->PushGetHeaders(pindexBest, uint256(0));
nLastBlockReceived = GetTime();
}
}
+1 -1
View File
@@ -868,7 +868,7 @@ QWidget#line {
&lt;/style&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;a href=&quot;http://explorer.triangles.technology&quot;&gt; &amp;#187; TRI block explorer&lt;/a&gt;&lt;/body&gt;</string>
&lt;a href=&quot;https://blocks.cryptographic-triangles.org&quot;&gt; &amp;#187; TRI block explorer&lt;/a&gt;&lt;/body&gt;</string>
</property>
<property name="alignment">
<set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+166 -27
View File
@@ -364,49 +364,188 @@ Value gettxoutsetinfo(const Array& params, bool fHelp)
return obj;
}
Value recalculatesupply(const Array& params, bool fHelp)
static void GetActiveChainVector(std::vector<CBlockIndex*>& chain)
{
if (fHelp || params.size() != 0)
throw runtime_error(
"recalculatesupply\n"
"Recalculates the money supply by summing all unspent transaction outputs.\n"
"Updates the stored money supply value at the chain tip and persists it to disk.\n"
"Returns the old and new supply values for comparison.\n"
"\nWARNING: This modifies blockchain index state. Only use if money supply is incorrect.");
chain.clear();
if (!pindexBest)
throw runtime_error("recalculatesupply: no best block");
for (CBlockIndex* pindex = pindexBest; pindex; pindex = pindex->pprev)
chain.push_back(pindex);
std::reverse(chain.begin(), chain.end());
}
static int64_t ComputeActiveChainSupplyFromBlocks(const std::vector<CBlockIndex*>& chain, int& nBlocksScanned, int& nTransactionsScanned)
{
nBlocksScanned = 0;
nTransactionsScanned = 0;
CTxDB txdb("r");
int64_t nSupply = 0;
for (std::vector<CBlockIndex*>::const_iterator pindexIt = chain.begin(); pindexIt != chain.end(); ++pindexIt)
{
CBlockIndex* pindex = *pindexIt;
if (!pindex)
throw runtime_error("recalculatesupply: null active-chain block index");
if (pindex->nHeight == 0)
{
nBlocksScanned++;
continue;
}
CBlock block;
int64_t nBlockValueIn = 0;
int64_t nBlockValueOut = 0;
if (!block.ReadFromDisk(pindex))
throw runtime_error(strprintf("recalculatesupply: failed reading block at height %d", pindex->nHeight));
for (std::vector<CTransaction>::const_iterator txIt = block.vtx.begin(); txIt != block.vtx.end(); ++txIt)
{
const CTransaction& tx = *txIt;
nTransactionsScanned++;
nBlockValueOut += tx.GetValueOut();
if (!tx.IsCoinBase())
{
for (std::vector<CTxIn>::const_iterator txinIt = tx.vin.begin(); txinIt != tx.vin.end(); ++txinIt)
{
const CTxIn& txin = *txinIt;
CTxIndex txindex;
CTransaction txPrev;
if (!txPrev.ReadFromDisk(txdb, txin.prevout, txindex))
throw runtime_error(strprintf(
"recalculatesupply: failed reading prevout %s:%u while processing height %d",
txin.prevout.hash.ToString().c_str(), txin.prevout.n, pindex->nHeight));
if (txin.prevout.n >= txPrev.vout.size())
throw runtime_error(strprintf(
"recalculatesupply: prevout index %u out of range for tx %s at height %d",
txin.prevout.n, txin.prevout.hash.ToString().c_str(), pindex->nHeight));
nBlockValueIn += txPrev.vout[txin.prevout.n].nValue;
}
}
}
nSupply += (nBlockValueOut - nBlockValueIn);
nBlocksScanned++;
}
return nSupply;
}
Value recalculatesupply(const Array& params, bool fHelp)
{
if (fHelp || params.size() > 1)
throw runtime_error(
"recalculatesupply [apply=false]\n"
"Rebuilds money supply by walking the active chain from genesis and summing (valueOut - valueIn) per block.\n"
"Also returns the current UTXO-set total for comparison.\n"
"If apply=true, rewrites nMoneySupply for every block on the active chain and persists the repaired values.\n"
"\nThis is intended for repairing corrupted money-supply tracking after chain/index incidents.");
bool fApply = false;
if (params.size() == 1)
fApply = params[0].get_bool();
LOCK(cs_main);
if (!pindexBest)
throw runtime_error("recalculatesupply: no best block");
CTxDB txdbRead("r");
int nUtxoCount = 0;
CTxDB txdb;
int64_t nCalculatedSupply = txdb.SumUtxoValues(nUtxoCount);
int64_t nOldSupply = pindexBest->nMoneySupply;
int64_t nDifference = nCalculatedSupply - nOldSupply;
int64_t nUtxoSupply = txdbRead.SumUtxoValues(nUtxoCount);
// Sanity check: difference should be reasonable (not millions of TRI)
// Max supply is 2,222,222 TRI, so any difference > 1M TRI is suspicious
if (abs64(nDifference) > 1000000 * COIN)
throw runtime_error(strprintf(
"recalculatesupply: calculated supply differs by %s TRI - this is abnormal, refusing to update",
FormatMoney(abs64(nDifference)).c_str()));
std::vector<CBlockIndex*> activeChain;
GetActiveChainVector(activeChain);
// Update the chain tip's money supply
pindexBest->nMoneySupply = nCalculatedSupply;
int nBlocksScanned = 0;
int nTransactionsScanned = 0;
int64_t nHistoricalSupply = ComputeActiveChainSupplyFromBlocks(activeChain, nBlocksScanned, nTransactionsScanned);
// Persist to LevelDB
CTxDB txdbWrite;
if (!txdbWrite.WriteBlockIndex(CDiskBlockIndex(pindexBest)))
throw runtime_error("recalculatesupply: failed to write updated block index");
int64_t nOldTipSupply = pindexBest->nMoneySupply;
if (fApply)
{
CTxDB txdbWrite;
int64_t nRunningSupply = 0;
for (std::vector<CBlockIndex*>::const_iterator pindexIt = activeChain.begin(); pindexIt != activeChain.end(); ++pindexIt)
{
CBlockIndex* pindex = *pindexIt;
if (!pindex)
throw runtime_error("recalculatesupply: null active-chain block index during apply");
if (pindex->nHeight == 0)
{
pindex->nMoneySupply = 0;
if (!txdbWrite.WriteBlockIndex(CDiskBlockIndex(pindex)))
throw runtime_error("recalculatesupply: failed to persist genesis block index during apply");
continue;
}
CBlock block;
int64_t nBlockValueIn = 0;
int64_t nBlockValueOut = 0;
if (!block.ReadFromDisk(pindex))
throw runtime_error(strprintf("recalculatesupply: failed reading block at height %d during apply", pindex->nHeight));
for (std::vector<CTransaction>::const_iterator txIt = block.vtx.begin(); txIt != block.vtx.end(); ++txIt)
{
const CTransaction& tx = *txIt;
nBlockValueOut += tx.GetValueOut();
if (!tx.IsCoinBase())
{
for (std::vector<CTxIn>::const_iterator txinIt = tx.vin.begin(); txinIt != tx.vin.end(); ++txinIt)
{
const CTxIn& txin = *txinIt;
CTxIndex txindex;
CTransaction txPrev;
if (!txPrev.ReadFromDisk(txdbWrite, txin.prevout, txindex))
throw runtime_error(strprintf(
"recalculatesupply: failed reading prevout %s:%u during apply at height %d",
txin.prevout.hash.ToString().c_str(), txin.prevout.n, pindex->nHeight));
if (txin.prevout.n >= txPrev.vout.size())
throw runtime_error(strprintf(
"recalculatesupply: prevout index %u out of range during apply for tx %s at height %d",
txin.prevout.n, txin.prevout.hash.ToString().c_str(), pindex->nHeight));
nBlockValueIn += txPrev.vout[txin.prevout.n].nValue;
}
}
}
nRunningSupply += (nBlockValueOut - nBlockValueIn);
pindex->nMoneySupply = nRunningSupply;
if (!txdbWrite.WriteBlockIndex(CDiskBlockIndex(pindex)))
throw runtime_error(strprintf("recalculatesupply: failed to persist block index at height %d", pindex->nHeight));
}
}
Object result;
result.push_back(Pair("height", (int)nBestHeight));
result.push_back(Pair("old_supply", ValueFromAmount(nOldSupply)));
result.push_back(Pair("new_supply", ValueFromAmount(nCalculatedSupply)));
result.push_back(Pair("difference", ValueFromAmount(nDifference)));
result.push_back(Pair("tip_bestblock", hashBestChain.GetHex()));
result.push_back(Pair("old_tip_supply", ValueFromAmount(nOldTipSupply)));
result.push_back(Pair("recalculated_chain_supply", ValueFromAmount(nHistoricalSupply)));
result.push_back(Pair("utxo_supply", ValueFromAmount(nUtxoSupply)));
result.push_back(Pair("tip_vs_recalculated", ValueFromAmount(nHistoricalSupply - nOldTipSupply)));
result.push_back(Pair("utxo_vs_recalculated", ValueFromAmount(nHistoricalSupply - nUtxoSupply)));
result.push_back(Pair("blocks_scanned", nBlocksScanned));
result.push_back(Pair("transactions_scanned", nTransactionsScanned));
result.push_back(Pair("utxo_count", nUtxoCount));
result.push_back(Pair("applied", fApply));
return result;
}
// triangles: get information of sync-checkpoint
Value getcheckpoint(const Array& params, bool fHelp)
{