sami7777/triangles_v5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

smsgDB: self-heal on unknown checksum type (RocksDB version drift)

Browse Source 
When smsgDB is opened by a binary linked against an older RocksDB than
the one that wrote its SST files, Open() returns
'Corruption: unknown checksum type 4 in .../000064.sst ...' (XXH3 was
introduced in RocksDB 7.4). Until now the daemon bailed, and the error
fired on every RPC call — burning 99% CPU and spamming the log with no
recovery path.

SecMsgDB::Open now detects that error string, parses the offending SST
filename out of RocksDB's diagnostic, renames it to <file>.sst.quarantined-<unix-ts>
inside smsgDB/, and retries the open. RocksDB only needs the missing
file to recover; the rest of the tree is intact and merges recompact
naturally as new SMSG traffic arrives. Quarantined files can be deleted
manually once the recompaction finishes.

CMakeLists.txt now refuses to configure against RocksDB < 7.4.0 when
the version is detectable (find_package or pkg-config paths). The
manual-probe path (Ubuntu 22.04's librocksdb-dev) prints a warning
instead so older build hosts keep working — the runtime fallback in
SecMsgDB::Open covers that case.

Discovered 2026-06-27 on DNS2: a Jun 19 binary swap left
smsgDB/000064.sst written with XXH3; the current v5.9.24 daemon is
linked to librocksdb.so.6.11 (RocksDB 6.11) which can't read it.
Behaviour before this patch: 99% CPU, log spam on every RPC.
Behaviour after: one quarantine log line, daemon proceeds normally.

Refs: the existing pre-v5.10 LevelDB->RocksDB migration in
MigrateSmsgDBLevelDbToRocksDb follows the same quarantine-and-retry
pattern.
This commit is contained in:
Krystie
2026-06-27 15:04:04 -07:00
parent 53c9654caf
commit dca34a02bb
2 changed files with 62 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/smessage.cpp
+39 -1
View File
@@ -31,6 +31,7 @@ Notes:
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cctype>
#include <stdint.h>
#include <time.h>
@@ -568,7 +569,44 @@ bool SecMsgDB::Open(const char* pszMode)
rocksdb::Options options;
options.create_if_missing = fCreate;
rocksdb::Status s = OpenSmsgDB(options, fullpath.string(), &smsgDB);
// Self-heal: when smsgDB was written by a newer RocksDB (>=7.4 uses
// XXH3, checksum type 4) and this build is linked against an older
// RocksDB that doesn't recognise the type, Open() fails with
// "Corruption: unknown checksum type N in <path>/<file>.sst ...".
// Quarantine the offending SST and retry — RocksDB only needs the
// missing file to recover; the rest of the DB is intact. Without this
// fallback the daemon burns 99% CPU retrying open() on every RPC.
if (!s.ok() && s.ToString().find("unknown checksum type") != std::string::npos)
{
auto msg = s.ToString();
auto pos = msg.find(fullpath.string());
if (pos != std::string::npos)
{
auto rest = msg.substr(pos + fullpath.string().size() + 1);
auto end = rest.find_first_of(" \t");
std::string sstName = (end == std::string::npos) ? rest : rest.substr(0, end);
fs::path badFile = fullpath / sstName;
if (fs::exists(badFile))
{
auto stamp = std::to_string(
std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::system_clock::now().time_since_epoch()).count());
fs::path quarantine = fullpath / (sstName + ".quarantined-" + stamp);
std::error_code ec;
fs::rename(badFile, quarantine, ec);
if (!ec)
{
printf("SecMsgDB::open() - quarantined %s "
"(newer-RocksDB checksum type not supported by this build)\n",
badFile.c_str());
}
}
}
smsgDB = nullptr;
s = OpenSmsgDB(options, fullpath.string(), &smsgDB);
}
if (!s.ok())
{
printf("SecMsgDB::open() - Error opening db: %s.\n", s.ToString().c_str());