smsgDB: self-heal on unknown checksum type (RocksDB version drift)
When smsgDB is opened by a binary linked against an older RocksDB than the one that wrote its SST files, Open() returns 'Corruption: unknown checksum type 4 in .../000064.sst ...' (XXH3 was introduced in RocksDB 7.4). Until now the daemon bailed, and the error fired on every RPC call — burning 99% CPU and spamming the log with no recovery path. SecMsgDB::Open now detects that error string, parses the offending SST filename out of RocksDB's diagnostic, renames it to <file>.sst.quarantined-<unix-ts> inside smsgDB/, and retries the open. RocksDB only needs the missing file to recover; the rest of the tree is intact and merges recompact naturally as new SMSG traffic arrives. Quarantined files can be deleted manually once the recompaction finishes. CMakeLists.txt now refuses to configure against RocksDB < 7.4.0 when the version is detectable (find_package or pkg-config paths). The manual-probe path (Ubuntu 22.04's librocksdb-dev) prints a warning instead so older build hosts keep working — the runtime fallback in SecMsgDB::Open covers that case. Discovered 2026-06-27 on DNS2: a Jun 19 binary swap left smsgDB/000064.sst written with XXH3; the current v5.9.24 daemon is linked to librocksdb.so.6.11 (RocksDB 6.11) which can't read it. Behaviour before this patch: 99% CPU, log spam on every RPC. Behaviour after: one quarantine log line, daemon proceeds normally. Refs: the existing pre-v5.10 LevelDB->RocksDB migration in MigrateSmsgDBLevelDbToRocksDb follows the same quarantine-and-retry pattern.
This commit is contained in:
@@ -134,6 +134,29 @@ if(NOT TARGET RocksDB::rocksdb AND NOT TARGET PkgConfig::RocksDB)
|
|||||||
message(STATUS "Found RocksDB (manual probe): ${ROCKSDB_LIBRARY}")
|
message(STATUS "Found RocksDB (manual probe): ${ROCKSDB_LIBRARY}")
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
|
# Triangles uses RocksDB features that only exist in 7.4+ (XXH3 per-block
|
||||||
|
# checksum, type 4). Building against an older RocksDB produces a binary
|
||||||
|
# whose smsgDB Open() fails on any SST file written by RocksDB 7.4+ —
|
||||||
|
# instead of just bailing, src/smessage.cpp::SecMsgDB::Open now
|
||||||
|
# quarantines the offending file and recovers. We still prefer to fail
|
||||||
|
# loudly here so this drift doesn't sneak back in unnoticed.
|
||||||
|
if(RocksDB_VERSION AND RocksDB_VERSION VERSION_LESS "7.4.0")
|
||||||
|
message(FATAL_ERROR
|
||||||
|
"Triangles requires RocksDB >= 7.4.0 (got ${RocksDB_VERSION}). "
|
||||||
|
"Older versions cannot read smsgDB files written by RocksDB 7.4+. "
|
||||||
|
"On Debian/Ubuntu: install librocksdb-dev >= 7.4 from a backports "
|
||||||
|
"repo or build RocksDB from source into /usr/local.")
|
||||||
|
elseif(NOT RocksDB_VERSION AND NOT TARGET RocksDB::rocksdb)
|
||||||
|
# Manual-probe path (e.g. Ubuntu 22.04's librocksdb-dev ships no CMake
|
||||||
|
# config / pkg-config metadata). We can't see the version, so warn
|
||||||
|
# instead of failing — the runtime self-heal in SecMsgDB::Open covers
|
||||||
|
# the gap, but this drift is what made the bug recur in the first place.
|
||||||
|
message(WARNING
|
||||||
|
"Could not determine RocksDB version (manual-probe path). "
|
||||||
|
"Triangles prefers RocksDB >= 7.4.0; older versions are recovered "
|
||||||
|
"at runtime via SecMsgDB::Open's quarantine fallback.")
|
||||||
|
endif()
|
||||||
|
|
||||||
# libsecp256k1 — vendored as a git submodule under src/secp256k1. Provides
|
# libsecp256k1 — vendored as a git submodule under src/secp256k1. Provides
|
||||||
# ECDSA signing/verification, pubkey recovery (via the recovery module), and
|
# ECDSA signing/verification, pubkey recovery (via the recovery module), and
|
||||||
# ECDH for secure messaging. Configure the submodule's build for our needs:
|
# ECDH for secure messaging. Configure the submodule's build for our needs:
|
||||||
|
|||||||
+39
-1
@@ -31,6 +31,7 @@ Notes:
|
|||||||
|
|
||||||
#include <algorithm>
|
#include <algorithm>
|
||||||
#include <atomic>
|
#include <atomic>
|
||||||
|
#include <chrono>
|
||||||
#include <cctype>
|
#include <cctype>
|
||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <time.h>
|
#include <time.h>
|
||||||
@@ -568,7 +569,44 @@ bool SecMsgDB::Open(const char* pszMode)
|
|||||||
rocksdb::Options options;
|
rocksdb::Options options;
|
||||||
options.create_if_missing = fCreate;
|
options.create_if_missing = fCreate;
|
||||||
rocksdb::Status s = OpenSmsgDB(options, fullpath.string(), &smsgDB);
|
rocksdb::Status s = OpenSmsgDB(options, fullpath.string(), &smsgDB);
|
||||||
|
|
||||||
|
// Self-heal: when smsgDB was written by a newer RocksDB (>=7.4 uses
|
||||||
|
// XXH3, checksum type 4) and this build is linked against an older
|
||||||
|
// RocksDB that doesn't recognise the type, Open() fails with
|
||||||
|
// "Corruption: unknown checksum type N in <path>/<file>.sst ...".
|
||||||
|
// Quarantine the offending SST and retry — RocksDB only needs the
|
||||||
|
// missing file to recover; the rest of the DB is intact. Without this
|
||||||
|
// fallback the daemon burns 99% CPU retrying open() on every RPC.
|
||||||
|
if (!s.ok() && s.ToString().find("unknown checksum type") != std::string::npos)
|
||||||
|
{
|
||||||
|
auto msg = s.ToString();
|
||||||
|
auto pos = msg.find(fullpath.string());
|
||||||
|
if (pos != std::string::npos)
|
||||||
|
{
|
||||||
|
auto rest = msg.substr(pos + fullpath.string().size() + 1);
|
||||||
|
auto end = rest.find_first_of(" \t");
|
||||||
|
std::string sstName = (end == std::string::npos) ? rest : rest.substr(0, end);
|
||||||
|
fs::path badFile = fullpath / sstName;
|
||||||
|
if (fs::exists(badFile))
|
||||||
|
{
|
||||||
|
auto stamp = std::to_string(
|
||||||
|
std::chrono::duration_cast<std::chrono::seconds>(
|
||||||
|
std::chrono::system_clock::now().time_since_epoch()).count());
|
||||||
|
fs::path quarantine = fullpath / (sstName + ".quarantined-" + stamp);
|
||||||
|
std::error_code ec;
|
||||||
|
fs::rename(badFile, quarantine, ec);
|
||||||
|
if (!ec)
|
||||||
|
{
|
||||||
|
printf("SecMsgDB::open() - quarantined %s "
|
||||||
|
"(newer-RocksDB checksum type not supported by this build)\n",
|
||||||
|
badFile.c_str());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
smsgDB = nullptr;
|
||||||
|
s = OpenSmsgDB(options, fullpath.string(), &smsgDB);
|
||||||
|
}
|
||||||
|
|
||||||
if (!s.ok())
|
if (!s.ok())
|
||||||
{
|
{
|
||||||
printf("SecMsgDB::open() - Error opening db: %s.\n", s.ToString().c_str());
|
printf("SecMsgDB::open() - Error opening db: %s.\n", s.ToString().c_str());
|
||||||
|
|||||||
Reference in New Issue
Block a user