Commit Graph

9 Commits

Author SHA1 Message Date
Krystie fb07d50235 feat: compact blocks, column families, fork detector, cross-network discovery, SAM v3, configurable peers
BIP152 Compact Blocks (main.cpp, net.cpp, protocol.h):
- SipHash-2-4 short IDs (48-bit) for transaction identification
- Compact block relay with mempool reconstruction
- Merkle root verification before acceptance
- Graceful fallback to full block on any mismatch
- Collision detection for ambiguous short IDs

RocksDB Column Families (txdb-rocksdb.cpp/h):
- 5 CFs: default, blockindex, txindex, utxo, addrindex
- Per-CF tuning: UTXO optimized for point lookups, addrindex for scans
- Backward-compatible: falls back to default CF for pre-migration data
- Prefix-based routing in ReadRaw/WriteRaw/EraseRaw/ExistsRaw

Fork Detector (main.cpp, net.cpp, net.h):
- Background thread checks local tip vs peer median every 60s post-IBD
- Alerts on divergence > forkthreshold (default 5 blocks)
- Optional auto-rebuild trigger on severe divergence

Cross-Network Tor↔I2P Discovery (net.cpp, init.cpp):
- I2P seed addresses loaded into addrman alongside onion seeds
- Address relay bridges .onion and .b32.i2p between networks
- IsI2PAddr/IsOnionAddr helpers for network-type detection

Configurable Outbound Connections (net.cpp, init.cpp):
- -maxoutboundconnections flag (range 4-32, default 8)

Mempool Fee-Priority Boost (miner.cpp):
- 2x fee weight in PoS block assembly for higher staking rewards

SAM v3 Direct Streaming (i2p/i2p_embedded.cpp/h):
- CI2PSamSocket class with full SAM v3 protocol
- SESSION CREATE + STREAM CONNECT handshake
- Factory method on CI2PEmbedded for native I2P connections
- SAM bridge readiness check in bootstrap loop
2026-06-27 19:19:30 -07:00
Krystie b623396186 perf+sec: 15 improvements across consensus, DB, network, sync
CONSENSUS SECURITY (main.cpp):
- Re-enable PoS kernel verification post-IBD (was unconditionally disabled)
- Re-enable coinstake reward validation post-IBD (was commented out)
- Re-enable anti-spam difficulty check (was if(false && ...))

SYNC PERFORMANCE (main.cpp):
- Batch address index writes in ConnectBlock (hundreds of DB ops → one per address)
- Throttle IBD printfs (per-block → per-10K-blocks or fDebug-gated)

DATABASE (txdb-rocksdb.cpp/h, txdb-base.cpp):
- Non-batched WriteRaw: WAL sync=false (was fsync per write)
- UTXO cache: FIFO eviction → true LRU with access-order tracking
- RocksDB memtable: 64MB → 256MB + max_write_buffer_number=4
- pendingBatch: std::map → std::unordered_map (O(log n) → O(1))
- max_open_files: 1000 → unlimited

NETWORK (net.cpp, netbase.cpp):
- TCP_NODELAY on all sockets (disable Nagle's algorithm)
- SO_KEEPALIVE on all sockets (faster dead-peer detection)
- Adaptive MilliSleep: 1ms during IBD, 10ms otherwise
- writev() scatter-gather I/O for send() coalescing (up to 16 msgs/syscall)
- O(1) CountInFlight counter (was O(n) scan of entire header map)
2026-06-27 18:17:59 -07:00
Krystie 0c6a2223cb chaindb_runtime: full test coverage + fixes for hidden bugs
- txdb-factory.cpp: drop static-cache in ResolveChainDbKind so the
  -chaindb flag can be toggled at runtime (needed for tests; cost is
  negligible since the daemon sets it once at startup)
- txdb-rocksdb.cpp: fix ExistsRaw to honor pending-batch delete markers.
  Previously a key erased inside an open batch was still reported as
  existing because the underlying DB hadn't been updated yet. Mirror
  ReadRaw's correct behavior: a delete marker shadows the DB value.
- chaindb_runtime_tests.cpp: per-test fresh handle via close-reopen
  dance so the static g_rocksdb singleton doesn't leak state between
  cases. Tests filter framework keys (length-prefixed 'version' and
  'dbformat') from iterator walks. block_index test fixed to Seek()
  not Seek("blockindex") since the serialized keys start with the
  length byte 0x0a.
- snapshotnet_tests.cpp, chaindb_runtime_tests.cpp: include wallet.h,
  ui_interface.h, uint256.h, checkpoints.h as needed for linker; add
  BOOST_TEST_MODULE decl; define global stubs (pwalletMain,
  uiInterface, fConfChange, etc.) so wallet.cpp link succeeds.

Result: test_snapshotnet + test_chaindb_runtime both pass with zero
errors. Found and fixed a real production bug in ExistsRaw along
the way.
2026-06-25 03:39:04 -07:00
Sami Ahmed 800f508abd init: skip block verification for snapshot-sourced chains
After LoadSnapshot, the daemon has headers + UTXOs but the raw block
bodies haven't been downloaded yet — they'll arrive via P2P as the
node syncs past the snapshot tip. LoadBlockIndex's verification
loop tries to read the last 50 block bodies from disk and fails
with 'OpenBlockFile failed' because the data isn't on disk yet.

Add fLoadedFromSnapshot global, set true at the end of successful
LoadSnapshot. In both txdb-leveldb.cpp and txdb-rocksdb.cpp LoadBlockIndex
verification loops, when ReadFromDisk fails AND fLoadedFromSnapshot is
true, log a warning and continue (the UTXO set itself was already
content-hash verified during LoadSnapshot, so we have strong evidence
the chain state is correct).

For non-snapshot chains (full blk0001.dat downloaded, normal IBD), the
ReadFromDisk failure remains a fatal error as before.

Combined with the prior fix in utxosnapshot.cpp that sets
fSerializeChainTrust=true before writes, the full snapshot path now
works end-to-end on a fresh datadir.
2026-06-19 02:47:30 -07:00
sami7777 e3f397c7e5 Heal pnext links on active chain at LoadBlockIndex
Persisted hashNext can be stale or zeroed by crash-interrupted reorgs, breaking GetKernelStakeModifier()'s forward walk and silently rejecting valid new PoS blocks ('check kernel failed', hashProof=0). On DNS3 every one of 2,201,458 links was zeroed on disk. Root cause of the 2026-04-24 chain halt. Rebuild the in-memory links from pindexBest at every startup.
2026-06-11 01:34:20 -07:00
sami7777 3099371864 Fix three CI breakages on the C++20 baseline
1. rocksdb::WriteBatch::Handler typeinfo missing on Ubuntu's librocksdb-dev.
   Both SecMsgBatchScanner (smessage.cpp) and CRocksBatchScanner
   (txdb-rocksdb.cpp) inherited from Handler to scan an active WriteBatch
   for pending writes/deletes; that subclass-based scan fails to link
   because Ubuntu's package hides the parent's typeinfo. Replaced both
   scanners with a parallel std::map<std::string, std::optional<std::string>>
   maintained alongside each WriteBatch — Put adds a value entry, Delete
   adds a nullopt entry, ScanBatch becomes an O(log n) map lookup. Same
   semantics, no Handler dependency.

2. macOS Homebrew's RocksDB 10.x removed the raw DB** overload of
   DB::Open; only std::unique_ptr<DB>* remains. txdb-rocksdb.cpp called
   the raw form, breaking the macOS build. Added the same SFINAE Open
   wrapper used in smessage.cpp (commit 4265343) that picks whichever
   overload the linked rocksdb actually has.

3. CSignal<>'s SignalState::slots member collided with Qt's `#define slots`
   to empty, stripping the member name in any TU that pulls in <QtCore>
   (e.g. moc-generated files that include util_signal.h transitively).
   Renamed to slot_map.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 19:03:43 -07:00
sami7777 59ee532bf6 Port smessage to RocksDB, make RocksDB a hard dep
The secure-messaging store (smsgDB) used the LevelDB API directly. Mass-
mapped to the equivalent RocksDB types: leveldb::DB/Status/WriteBatch/
Iterator/Slice/ReadOptions/WriteOptions/WriteBatch::Handler -> rocksdb::*.
The RocksDB API surface for our usage is binary-compatible — pure namespace
substitution, no semantic changes. Consumers in rpcsmessage.cpp and
qt/messagemodel.cpp updated to match.

RocksDB now becomes a hard build dependency (was optional behind
BUILD_ROCKSDB). The chain-DB rocksdb backend is consequently always
available; -chaindb=leveldb remains the default until the Phase-4
LevelDB retirement. Removed the BUILD_ROCKSDB cmake option, the
#ifdef BUILD_ROCKSDB guards in txdb*, and the runtime error path
that triggered when the flag was off.

CI updated: librocksdb-dev (Ubuntu), mingw-w64-x86_64-rocksdb (MSYS2),
and rocksdb (Homebrew) added to all build jobs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 18:11:00 -07:00
sami7777 2ba0ecf428 Cleanup: drop boost::filesystem/thread/chrono, retire dead code
Migration from boost to std-library equivalents and removal of unreachable
code paths. Touches infrastructure only — no consensus rule or wallet
serialization changes.

Dead code removed:
- IRC bootstrap (irc.cpp/h, 417 lines): orphan from pre-Tor era, no callers.
- Alert system (alert.cpp/h + sendalert RPC + Qt UI signal, ~500 lines):
  retired post-V5 fork; old peers' alert messages now hit the unknown-cmd
  default branch, logged + ignored.
- Legacy P2P handlers in main.cpp: "checkpoint" (already a no-op stub since
  V5 fork master-key removal), "checkorder"/"reply" (2010-era Receive-by-IP
  feature), plus their unused supporting structures (CRequestTracker,
  PushRequest overloads, mapRequests/cs_mapRequests, mapReuseKey).
- Unreachable RPCs clearwallettransactions and scanforalltxns (~175 lines):
  defined in rpcwallet.cpp but never registered in the dispatch table.
- Stale -alertnotify CLI help text (option was advertised but never wired).

boost::filesystem -> std::filesystem (C++17):
- 30 source files, 5 headers. namespace fs = boost::filesystem swapped to
  namespace fs = std::filesystem; boost::filesystem::ifstream/ofstream
  replaced with std::ifstream/ofstream (path-aware in C++17);
  fs::system_complete -> fs::absolute; boost::filesystem::filesystem_error
  -> std::filesystem::filesystem_error.
- Build system: dropped Boost::filesystem from link libs and Boost
  components; PCH includes updated.
- Added explicit <filesystem> includes where types were previously
  available only transitively (db.h, rpcblockchain.cpp).

boost::thread -> std::thread (12 files):
- sync.h CCriticalSection/CWaitableCriticalSection now alias
  std::recursive_mutex/std::mutex. boost::unique_lock and
  boost::condition_variable / boost::mutex::scoped_lock swapped to std
  equivalents; sync.cpp boost::thread_specific_ptr -> thread_local
  std::unique_ptr.
- init.cpp boost::thread_group rewritten as std::vector<std::thread> with
  manual join loop. boost::thread::hardware_concurrency ->
  std::thread::hardware_concurrency.
- main.cpp/wallet.cpp -blocknotify/-walletnotify shell-out threads now use
  std::thread(...).detach() — fixes a latent bug where modern boost::thread
  destructor would call std::terminate on the joinable thread.
- util.cpp NewThread now catches std::system_error.
- No interruption_point/interrupt usage anywhere — pure mechanical swap.

boost::chrono / boost::posix_time -> std::chrono (3 of 5 files):
- util.h: MilliSleep, GetTimeMillis, GetTimeMicros rewritten on std::chrono
  (system_clock for epoch math, sleep_for for delays).
- snapshotnet.cpp: sleep_for swapped.
- DoS_tests.cpp: timing harness uses steady_clock.
- Skipped: rpcdump.cpp (boost::posix_time::time_input_facet has no clean
  std::get_time equivalent) and qt/qtipcserver.cpp (locked to
  boost::posix_time by boost::interprocess::message_queue::timed_receive).

Other housekeeping:
- Dropped unnecessary "using namespace boost;" from txdb-leveldb.cpp,
  txdb-rocksdb.cpp, walletdb.cpp, db.cpp (verified no unqualified boost
  names in those TUs).
- Removed unused extern declaration for clearwallettransactions.

Build fixes for non-unity builds on MinGW64/GCC 15:
- net.cpp: dropped stale #include "irc.h".
- addrman.cpp + main.cpp: explicit <cmath> include for sqrt/pow (was
  arriving transitively via boost headers).
- rpcblockchain.cpp + init.cpp: defensive #undef STRICT/ADVISORY/PERMISSIVE
  since windows.h macros collide with the Checkpoints:: enum values when
  std headers reorder include flow.
- tor_embed_hooks.cpp: triangles_tor_check_interrupted now polls fShutdown
  instead of boost::this_thread::interruption_requested (we never used
  boost interruption — the hook was always effectively a no-op).
- snapshotnet.cpp: fs::remove error handle uses std::error_code.
- serialize.h: added <ios> for std::ios::badbit/failbit (was relying on
  transitive include via boost).

Note: unity builds currently fail on this branch due to std::byte (C++17)
colliding with COM 'byte' typedef from shlobj.h when 'using namespace std;'
from earlier files in the unity slice leaks into util.cpp's parse of
shlobj.h. Build with -DENABLE_UNITY_BUILD=OFF (the default).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 15:27:14 -07:00
sami7777 569ca99e66 M1.3: RocksDB chain database backend behind BUILD_ROCKSDB flag
Adds CRocksTxDB, the second concrete backend for CTxDBBase. Mirrors
CTxDB (LevelDB) one-for-one with rocksdb:: substitutions: same key
serialization (inherited from CTxDBBase), same active-batch semantics,
same LoadBlockIndex flow including the dbformat v3 chain-trust upgrade.

Build flag BUILD_ROCKSDB defaults OFF, so the existing LevelDB build is
untouched — RocksDB headers are only included when the flag is on, and
the entire .cpp file is wrapped in #ifdef BUILD_ROCKSDB.

Build system:
  * Top-level option(BUILD_ROCKSDB ... OFF)
  * find_package(RocksDB CONFIG) with pkg-config fallback
  * Conditional list(APPEND CORE_SOURCES txdb-rocksdb.cpp)
  * Conditional target_link_libraries(... RocksDB::rocksdb)

Data layout: RocksDB lives under <datadir>/rocksdb/, separate from
<datadir>/txleveldb/, so both backends can coexist for migration and
parity testing.

Acknowledged debt: LoadBlockIndex is duplicated between CTxDB and
CRocksTxDB. Will be extracted into CTxDBBase once the iterator and
batch abstractions are proven across both backends (M1.4 or later).

Validated: default-OFF build still compiles cleanly. The BUILD_ROCKSDB=ON
path is NOT compile-validated yet — RocksDB isn't installed on this dev
machine. The code is straight namespace substitution from the working
LevelDB backend; whoever first enables the flag should report any
header/API drift between rocksdb releases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 03:36:20 -07:00