sami7777/triangles_v5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pharao Release

Browse Source 
Version 4.0.0.4
- refactored code
- migrated vom Qt4 to Qt5
- added secure messaging
This commit is contained in:
TrianglesCommunityProject
2014-10-05 23:37:30 +02:00
parent 215a52f3e6
commit 69aedf2533
544 changed files with 165233 additions and 76602 deletions
Download Patch File Download Diff File Expand all files Collapse all files
COPYING
+3 -3
View File
@@ -1,7 +1,7 @@
Copyright (c) 2014 triangles Developers
Copyright (c) 2013 NovaCoin Developers
Copyright (c) 2014 Triangles Team
Copyright (c) 2013-2014 Triangles Team
Copyright (c) 2011-2012 PPCoin Developers
Copyright (c) 2009-2012 Bitcoin Developers
Copyright (c) 2009-2014 Bitcoin developers
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
INSTALL
+2 -2
View File
@@ -2,8 +2,8 @@ Building triangles
See doc/readme-qt.rst for instructions on building triangles QT,
the intended-for-end-users, nice-graphical-interface, reference
implementation of triangles.
implementation of Triangles.
See doc/build-*.txt for instructions on building trianglesd,
the intended-for-services, no-graphical-interface, reference
implementation of triangles.
implementation of Triangles.
README
+1 -1
View File
@@ -1 +1 @@
README.md
README.md
README.md
+2 -2
View File
@@ -1,6 +1,6 @@
Triangles (TRI) Release
Triangles (TRI) "Pharao" Version 4.0.0.4 Release,
triangles is a cool new crypto currency that will feature a uniquely implemented anonymization feature that uses exchanges on the back end and a decoupled transaction flow architecture.
Triangles is a cool new crypto currency that features TOR implementation and secure messaging.
This wallet supports the staking=0 option in the triangles.conf file to disable the stake miner thread for pool and exchange operators.
doc/Doxyfile
+3 -3
View File
@@ -28,13 +28,13 @@ DOXYFILE_ENCODING = UTF-8
# The PROJECT_NAME tag is a single word (or a sequence of words surrounded
# by quotes) that should identify the project.
PROJECT_NAME = Bitcoin
PROJECT_NAME = Triangles
# The PROJECT_NUMBER tag can be used to enter a project or revision number.
# This could be handy for archiving the generated documentation or
# if some version control system is used.
PROJECT_NUMBER = 0.5.0
PROJECT_NUMBER = 4.0.0
# Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer
@@ -47,7 +47,7 @@ PROJECT_BRIEF = "P2P Digital Currency"
# exceed 55 pixels and the maximum width should not exceed 200 pixels.
# Doxygen will copy the logo to the output directory.
PROJECT_LOGO = doc/bitcoin_logo_doxygen.png
PROJECT_LOGO = doc/triangles_logo_doxygen.png
# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute)
# base path where the generated documentation will be put.
doc/README
+12 -8
View File
@@ -1,14 +1,18 @@
Triangles 4.0.0.4 BETA
Copyright (c) 2014 TRI Developers
triangles 3.3.3.3 BETA
Copyright (c) 2014 TRI Developers
Copyright (c) 2013 NovaCoin Developers
Copyright (c) 2011-2012 Bitcoin Developers
Copyright (c) 2014 Triangles team
Copyright (c) 2011-2012 PPCoin Developers
Distributed under the MIT/X11 software license, see the accompanying
file license.txt or http://www.opensource.org/licenses/mit-license.php.
This product includes software developed by the OpenSSL Project for use in
the OpenSSL Toolkit (http://www.openssl.org/). This product includes
cryptographic software written by Eric Young (eay@cryptsoft.com).
Intro
-----
Triangles is a free open source project derived from Bitcoin, with
the goal of providing a long-term energy-efficient X13 anon PoS crypto-currency.
Built on the foundation of Shadowcoin, PPCoin and Bitcoin, innovations such as proof-of-stake
help further advance the field of crypto-currency.
doc/README_windows.txt
+18
View File
@@ -0,0 +1,18 @@
Triangles 4.0.0.4 BETA
Copyright (c) 2014 Triangles team
Copyright (c) 2011-2013 PPCoin Developers
Distributed under the MIT/X11 software license, see the accompanying
file license.txt or http://www.opensource.org/licenses/mit-license.php.
This product includes software developed by the OpenSSL Project for use in
the OpenSSL Toolkit (http://www.openssl.org/). This product includes
cryptographic software written by Eric Young (eay@cryptsoft.com).
Intro
-----
Triangles is a free open source project derived from Bitcoin, with
the goal of providing a long-term energy-efficient X13 anon PoS crypto-currency.
Built on the foundation of Shadowcoin, PPCoin and Bitcoin, innovations such as proof-of-stake
help further advance the field of crypto-currency.
doc/assets-attribution.txt
+1 -5
View File
@@ -38,7 +38,7 @@ Designer: http://www.everaldo.com
Icon Pack: Crystal SVG
License: LGPL
Icon: src/qt/res/icons/bitcoin.png, src/qt/res/icons/toolbar.png
Icon: src/qt/res/icons/triangles.png, src/qt/res/icons/toolbar.png
Designer: Bitboy (optimized for 16x16 by Wladimir van der Laan)
License: Public Domain
Site: http://forum.bitcoin.org/?topic=1756.0
@@ -49,8 +49,4 @@ Designer: Everaldo (Everaldo Coelho)
License: GNU/GPL
Site: http://findicons.com/icon/17102/reload?id=17102
Image: src/qt/res/images/splash2.jpg (Wallet image)
Designer: Crobbo (forum)
Site: https://bitcointalk.org/index.php?topic=32273.0
License: Public domain
doc/build-msw.txt
+24 -9
View File
@@ -1,4 +1,4 @@
Copyright (c) 2009-2012 Bitcoin Developers
Copyright (c) 2009-2012 Bitcoin developers
Distributed under the MIT/X11 software license, see the accompanying
file license.txt or http://www.opensource.org/licenses/mit-license.php.
This product includes software developed by the OpenSSL Project for use in
@@ -7,7 +7,7 @@ cryptographic software written by Eric Young (eay@cryptsoft.com) and UPnP
software written by Thomas Bernard.
See readme-qt.rst for instructions on building Bitcoin QT, the
See readme-qt.rst for instructions on building Triangles QT, the
graphical user interface.
WINDOWS BUILD NOTES
@@ -36,10 +36,12 @@ Boost MIT-like license
miniupnpc New (3-clause) BSD license
Versions used in this release:
OpenSSL 1.0.1b
OpenSSL 1.0.1h
Berkeley DB 4.8.30.NC
Boost 1.47.0
miniupnpc 1.6
Boost 1.55.0
miniupnpc 1.9
libevent 2.0.21 stable
leveldb 1.2
OpenSSL
@@ -48,7 +50,7 @@ MSYS shell:
un-tar sources with MSYS 'tar xfz' to avoid issue with symlinks (OpenSSL ticket 2377)
change 'MAKE' env. variable from 'C:\MinGW32\bin\mingw32-make.exe' to '/c/MinGW32/bin/mingw32-make.exe'
cd /c/openssl-1.0.1b-mgw
cd /c/openssl-1.0.1h
./config
make
@@ -76,9 +78,22 @@ make -f Makefile.mingw
mkdir miniupnpc
cp *.h miniupnpc/
Bitcoin
Libevent
--------
MSYS SHELL:
tar xvzf ./libevent-2.0.21-stable.tar
./configure --enable-static --disable-shared
make
LevelDB (sources included in this release)
-------
MSYS SHELL:
cd src/leveldb
TARGET_OS=NATIVE_WINDOWS make libleveldb.a libmemenv.a
triangles-qt.exe
----------------
DOS prompt:
cd \bitcoin\src
cd \triangles-master\src
mingw32-make -f makefile.mingw
strip bitcoind.exe
strip trianglesd.exe
doc/build-osx.txt
+9 -14
View File
@@ -1,4 +1,4 @@
Copyright (c) 2009-2012 Bitcoin Developers
Copyright (c) 2009-2012 Bitcoin developers
Distributed under the MIT/X11 software license, see the accompanying file
license.txt or http://www.opensource.org/licenses/mit-license.php. This
product includes software developed by the OpenSSL Project for use in the
@@ -7,19 +7,14 @@ software written by Eric Young (eay@cryptsoft.com) and UPnP software written by
Thomas Bernard.
Mac OS X bitcoind build instructions
Mac OS X trianglesd build instructions
Laszlo Hanyecz <solar@heliacal.net>
Douglas Huff <dhuff@jrbobdobbs.org>
These instructions are not complete and probably won't work.
They will be ammneded in the future.
Please read this if you are on 10.9:
https://github.com/bitcoin/bitcoin/issues/3228
See readme-qt.rst for instructions on building Bitcoin QT, the
See readme-qt.rst for instructions on building triangles QT, the
graphical user interface.
Tested on 10.5 and 10.6 intel. PPC is not supported because it's big-endian.
@@ -34,7 +29,7 @@ but you can get the current version from http://developer.apple.com
1. Clone the github tree to get the source code:
git clone git@github.com:bitcoin/bitcoin.git bitcoin
git clone https://github.com/TrianglesCommunityProject/triangles triangles
2. Download and install MacPorts from http://www.macports.org/
@@ -48,14 +43,14 @@ sudo port install boost db48 openssl miniupnpc
Optionally install qrencode (and set USE_QRCODE=1):
sudo port install qrencode
4. Now you should be able to build bitcoind:
4. Now you should be able to build trianglesd:
cd bitcoin/src
cd triangles/src
make -f makefile.osx
Run:
./bitcoind --help # for a list of command-line options.
./trianglesd --help # for a list of command-line options.
Run
./bitcoind -daemon # to start the bitcoin daemon.
./trianglesd -daemon # to start the triangles daemon.
Run
./bitcoind help # When the daemon is running, to get a list of RPC commands
./trianglesd help # When the daemon is running, to get a list of RPC commands
doc/build-unix.txt
+30 -23
View File
@@ -1,4 +1,4 @@
Copyright (c) 2009-2012 Bitcoin Developers
Copyright (c) 2009-2012 Bitcoin developers
Distributed under the MIT/X11 software license, see the accompanying
file license.txt or http://www.opensource.org/licenses/mit-license.php.
This product includes software developed by the OpenSSL Project for use in
@@ -14,10 +14,10 @@ To Build
--------
cd src/
make -f makefile.unix # Headless bitcoin
make -f makefile.unix # Headless Triangles
See readme-qt.rst for instructions on building Bitcoin QT,
the graphical bitcoin.
See readme-qt.rst for instructions on building Triangles QT,
the graphical Triangles.
Dependencies
------------
@@ -25,10 +25,11 @@ Dependencies
Library Purpose Description
------- ------- -----------
libssl SSL Support Secure communications
libdb4.8 Berkeley DB Blockchain & wallet storage
libdb Berkeley DB Blockchain & wallet storage
libboost Boost C++ Library
miniupnpc UPnP Support Optional firewall-jumping support
libqrencode QRCode generation Optional QRCode generation
libevent Callbacks Execute a callback function when a specific event occurs
Note that libexecinfo should be installed, if you building under *BSD systems.
This library provides backtrace facility.
@@ -54,19 +55,19 @@ Licenses of statically linked libraries:
Versions used in this release:
GCC 4.3.3
OpenSSL 0.9.8g
Berkeley DB 4.8.30.NC
Boost 1.37
miniupnpc 1.6
OpenSSL 1.0.1h
Berkeley DB 5.1 (4.8.30.NC should work too)
Boost 1.48
miniupnpc 1.9
Dependency Build Instructions: Ubuntu & Debian
----------------------------------------------
sudo apt-get install build-essential
sudo apt-get install libssl-dev
sudo apt-get install libdb4.8-dev
sudo apt-get install libdb4.8++-dev
Boost 1.40+: sudo apt-get install libboost-all-dev
or Boost 1.37: sudo apt-get install libboost1.37-dev
sudo apt-get install libdb-dev
sudo apt-get install libdb++-dev
sudo apt-get install libboost1.48-all-dev
sudo apt-get install libevent-dev
sudo apt-get install libqrencode-dev
If using Boost 1.37, append -mt to the boost libraries in the makefile.
@@ -75,21 +76,21 @@ If using Boost 1.37, append -mt to the boost libraries in the makefile.
Dependency Build Instructions: Gentoo
-------------------------------------
Note: If you just want to install bitcoind on Gentoo, you can add the Bitcoin
Note: If you just want to install trianglesd on Gentoo, you can add the Triangles
overlay and use your package manager:
layman -a bitcoin && emerge bitcoind
layman -a triangles && emerge trianglesd
emerge -av1 --noreplace boost glib openssl sys-libs/db:4.8
emerge -av1 --noreplace boost glib openssl sys-libs/db
Take the following steps to build (no UPnP support):
cd ${BITCOIN_DIR}/src
cd ${TRIANGLES_DIR}/src
make -f makefile.unix USE_UPNP= BDB_INCLUDE_PATH='/usr/include/db4.8'
strip bitcoind
strip trianglesd
Notes
-----
The release is built with GCC and then "strip bitcoind" to strip the debug
The release is built with GCC and then "strip trianglesd" to strip the debug
symbols, which reduces the executable size by about 90%.
@@ -116,10 +117,16 @@ sudo su
./bootstrap.sh
./bjam install
Libevent
--------
If you need to build Libevent yourself
./configure --enable-static --disable-shared
make
Security
--------
To help make your bitcoin installation more secure by making certain attacks impossible to
To help make your Triangles installation more secure by making certain attacks impossible to
exploit even if a vulnerability is found, you can take the following measures:
* Position Independent Executable
@@ -136,7 +143,7 @@ exploit even if a vulnerability is found, you can take the following measures:
make -f makefile.unix ... -e PIE=1
To test that you have built PIE executable, install scanelf, part of paxutils, and use:
scanelf -e ./bitcoin
scanelf -e ./triangles
The output should contain:
TYPE
@@ -144,13 +151,13 @@ exploit even if a vulnerability is found, you can take the following measures:
* Non-executable Stack
If the stack is executable then trivial stack based buffer overflow exploits are possible if
vulnerable buffers are found. By default, bitcoin should be built with a non-executable stack
vulnerable buffers are found. By default, Triangles should be built with a non-executable stack
but if one of the libraries it uses asks for an executable stack or someone makes a mistake
and uses a compiler extension which requires an executable stack, it will silently build an
executable without the non-executable stack protection.
To verify that the stack is non-executable after compiling use:
scanelf -e ./bitcoin
scanelf -e ./triangles
the output should contain:
STK/REL/PTL
doc/coding.txt
+2 -2
View File
@@ -28,7 +28,7 @@ someVariable.
Common types:
n integer number: short, unsigned short, int, unsigned int,
int64, uint64, sometimes char if used as a number
int64_t, uint64_t, sometimes char if used as a number
d double, float
f flag
hash uint256
@@ -90,7 +90,7 @@ in 500ms.
ThreadRPCServer : Remote procedure call handler, listens on port 8332
for connections and services them.
ThreadBitcoinMiner : Generates bitcoins
ThreadStakeMiner : Generates TRIs
ThreadMapPort : Universal plug-and-play startup/shutdown
doc/readme-qt.rst
+17 -21
View File
@@ -1,28 +1,24 @@
Bitcoin-qt: Qt4 GUI for Bitcoin
triangles-qt: Qt5 GUI for triangles
===============================
Note: Not all of these instructions are updated yet, so they probably don't work.
Feel free to update them! ;)
Build instructions
===================
Debian
-------
First, make sure that the required packages for Qt4 development of your
distribution are installed.
On recent Ubuntu releases, the `libdb4.8++-dev` package is not in the default
repositories. However it's in the bitcoin repository, which we can add
with:
apt-add-repository ppa:bitcoin/bitcoin
Then we need to make sure our build environment is up to date. For Debian and Ubuntu:
First, make sure that the required packages for Qt5 development of your
distribution are installed, for Debian and Ubuntu these are:
::
apt-get install qt4-qmake libqt4-dev build-essential libboost-dev libboost-system-dev \
apt-get install qt5-default qt5-qmake qtbase5-dev-tools qttools5-dev-tools \
build-essential libboost-dev libboost-system-dev \
libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev \
libssl-dev libdb4.8++-dev libminiupnpc-dev libminiupnpc8
libssl-dev libdb++-dev
then execute the following:
@@ -31,9 +27,9 @@ then execute the following:
qmake
make
Alternatively, install Qt Creator and open the `bitcoin-qt.pro` file.
Alternatively, install Qt Creator and open the `triangles-qt.pro` file.
An executable named `bitcoin-qt` will be built.
An executable named `triangles-qt` will be built.
Windows
@@ -50,8 +46,8 @@ Windows build instructions:
- Open the .pro file in QT creator and build as normal (ctrl-B)
.. _`QT Windows SDK`: http://qt.nokia.com/downloads/sdk-windows-cpp
.. _`dependencies archive`: https://download.visucore.com/bitcoin/qtgui_deps_1.zip
.. [#] PGP signature: https://download.visucore.com/bitcoin/qtgui_deps_1.zip.sig (signed with RSA key ID `610945D0`_)
.. _`dependencies archive`: https://download.visucore.com/triangles/qtgui_deps_1.zip
.. [#] PGP signature: https://download.visucore.com/triangles/qtgui_deps_1.zip.sig (signed with RSA key ID `610945D0`_)
.. _`610945D0`: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x610945D0
@@ -81,7 +77,7 @@ Build configuration options
UPNnP port forwarding
---------------------
To use UPnP for port forwarding behind a NAT router (recommended, as more connections overall allow for a faster and more stable bitcoin experience), pass the following argument to qmake:
To use UPnP for port forwarding behind a NAT router (recommended, as more connections overall allow for a faster and more stable triangles experience), pass the following argument to qmake:
::
@@ -129,9 +125,9 @@ flag to qmake to control this:
Berkely DB version warning
==========================
A warning for people using the *static binary* version of Bitcoin on a Linux/UNIX-ish system (tl;dr: **Berkely DB databases are not forward compatible**).
A warning for people using the *static binary* version of triangles on a Linux/UNIX-ish system (tl;dr: **Berkely DB databases are not forward compatible**).
The static binary version of Bitcoin is linked against libdb4.8 (see also `this Debian issue`_).
The static binary version of triangles is linked against libdb 5.0 (see also `this Debian issue`_).
Now the nasty thing is that databases from 5.X are not compatible with 4.X.
@@ -146,7 +142,7 @@ Ubuntu 11.10 warning
====================
Ubuntu 11.10 has a package called 'qt-at-spi' installed by default. At the time of writing, having that package
installed causes bitcoin-qt to crash intermittently. The issue has been reported as `launchpad bug 857790`_, but
installed causes triangles-qt to crash intermittently. The issue has been reported as `launchpad bug 857790`_, but
isn't yet fixed.
Until the bug is fixed, you can remove the qt-at-spi package to work around the problem, though this will presumably
doc/release-process.txt
+52 -46
View File
@@ -1,7 +1,10 @@
Note: This is old and outdated and doesn't apply to Triangles.
* update translations (ping tcatm on IRC for now)
* update (commit) version in sources
bitcoin-qt.pro
triangles-qt.pro
src/version.h
share/setup.nsi
doc/README*
@@ -16,7 +19,7 @@
* perform gitian builds
* From a directory containing the bitcoin source, gitian-builder and gitian.sigs
* From a directory containing the triangles source, gitian-builder and gitian.sigs
export SIGNER=(your gitian key, ie bluematt, sipa, etc)
export VERSION=0.5.1
cd ./gitian-builder
@@ -32,61 +35,61 @@
wget 'http://downloads.sourceforge.net/project/boost/boost/1.47.0/boost_1_47_0.tar.bz2'
wget 'http://download.qt.nokia.com/qt/source/qt-everywhere-opensource-src-4.7.4.tar.gz'
cd ..
./bin/gbuild ../bitcoin/contrib/gitian-descriptors/boost-win32.yml
./bin/gbuild ../triangles/contrib/gitian-descriptors/boost-win32.yml
cp build/out/boost-win32-1.47.0-gitian.zip inputs/
./bin/gbuild ../bitcoin/contrib/gitian-descriptors/qt-win32.yml
./bin/gbuild ../triangles/contrib/gitian-descriptors/qt-win32.yml
cp build/out/qt-win32-4.7.4-gitian.zip inputs/
./bin/gbuild ../bitcoin/contrib/gitian-descriptors/deps-win32.yml
cp build/out/bitcoin-deps-0.0.3.zip inputs/
./bin/gbuild ../triangles/contrib/gitian-descriptors/deps-win32.yml
cp build/out/triangles-deps-0.0.3.zip inputs/
* Build bitcoind and bitcoin-qt on Linux32, Linux64, and Win32:
./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian.yml
./bin/gsign --signer $SIGNER --release ${VERSION} --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian.yml
* Build trianglesd and triangles-qt on Linux32, Linux64, and Win32:
./bin/gbuild --commit triangles=v${VERSION} ../triangles/contrib/gitian-descriptors/gitian.yml
./bin/gsign --signer $SIGNER --release ${VERSION} --destination ../gitian.sigs/ ../triangles/contrib/gitian-descriptors/gitian.yml
pushd build/out
zip -r bitcoin-${VERSION}-linux-gitian.zip *
mv bitcoin-${VERSION}-linux-gitian.zip ../../
zip -r triangles-${VERSION}-linux-gitian.zip *
mv triangles-${VERSION}-linux-gitian.zip ../../
popd
./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win32.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-win32 --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win32.yml
./bin/gbuild --commit triangles=v${VERSION} ../triangles/contrib/gitian-descriptors/gitian-win32.yml
./bin/gsign --signer $SIGNER --release ${VERSION}-win32 --destination ../gitian.sigs/ ../triangles/contrib/gitian-descriptors/gitian-win32.yml
pushd build/out
zip -r bitcoin-${VERSION}-win32-gitian.zip *
mv bitcoin-${VERSION}-win32-gitian.zip ../../
zip -r triangles-${VERSION}-win32-gitian.zip *
mv triangles-${VERSION}-win32-gitian.zip ../../
popd
Build output expected:
1. linux 32-bit and 64-bit binaries + source (bitcoin-${VERSION}-linux-gitian.zip)
2. windows 32-bit binary, installer + source (bitcoin-${VERSION}-win32-gitian.zip)
1. linux 32-bit and 64-bit binaries + source (triangles-${VERSION}-linux-gitian.zip)
2. windows 32-bit binary, installer + source (triangles-${VERSION}-win32-gitian.zip)
3. Gitian signatures (in gitian.sigs/${VERSION}[-win32]/(your gitian key)/
* repackage gitian builds for release as stand-alone zip/tar/installer exe
* Linux .tar.gz:
unzip bitcoin-${VERSION}-linux-gitian.zip -d bitcoin-${VERSION}-linux
tar czvf bitcoin-${VERSION}-linux.tar.gz bitcoin-${VERSION}-linux
rm -rf bitcoin-${VERSION}-linux
unzip triangles-${VERSION}-linux-gitian.zip -d triangles-${VERSION}-linux
tar czvf triangles-${VERSION}-linux.tar.gz triangles-${VERSION}-linux
rm -rf triangles-${VERSION}-linux
* Windows .zip and setup.exe:
unzip bitcoin-${VERSION}-win32-gitian.zip -d bitcoin-${VERSION}-win32
mv bitcoin-${VERSION}-win32/bitcoin-*-setup.exe .
zip -r bitcoin-${VERSION}-win32.zip bitcoin-${VERSION}-win32
rm -rf bitcoin-${VERSION}-win32
unzip triangles-${VERSION}-win32-gitian.zip -d triangles-${VERSION}-win32
mv triangles-${VERSION}-win32/triangles-*-setup.exe .
zip -r triangles-${VERSION}-win32.zip triangles-${VERSION}-win32
rm -rf triangles-${VERSION}-win32
* perform Mac build
See this blog post for how Gavin set up his build environment to build the OSX
release; note that a patched version of macdeployqt is not needed anymore, as
the required functionality and fixes are implemented directly in macdeployqtplus:
http://gavintech.blogspot.com/2011/11/deploying-bitcoin-qt-on-osx.html
http://gavintech.blogspot.com/2011/11/deploying-triangles-qt-on-osx.html
Gavin also had trouble with the macports py27-appscript package; he
ended up installing a version that worked with: /usr/bin/easy_install-2.7 appscript
qmake RELEASE=1 USE_UPNP=1 USE_QRCODE=1 bitcoin-qt.pro
qmake RELEASE=1 USE_UPNP=1 USE_QRCODE=1 triangles-qt.pro
make
export QTDIR=/opt/local/share/qt4 # needed to find translations/qt_*.qm files
T=$(contrib/qt_translations.py $QTDIR/translations src/qt/locale)
python2.7 contrib/macdeploy/macdeployqtplus Bitcoin-Qt.app -add-qt-tr $T -dmg -fancy contrib/macdeploy/fancy.plist
python2.7 contrib/macdeploy/macdeployqtplus triangles-Qt.app -add-qt-tr $T -dmg -fancy contrib/macdeploy/fancy.plist
Build output expected:
Bitcoin-Qt.dmg
triangles-Qt.dmg
* upload builds to SourceForge
@@ -98,7 +101,7 @@
* update wiki download links
* update wiki changelog: https://en.bitcoin.it/wiki/Changelog
* update wiki changelog: https://en.triangles.it/wiki/Changelog
* Commit your signature to gitian.sigs:
pushd gitian.sigs
@@ -112,31 +115,34 @@
* After 3 or more people have gitian-built, repackage gitian-signed zips:
* From a directory containing bitcoin source, gitian.sigs and gitian zips
* From a directory containing triangles source, gitian.sigs and gitian zips
export VERSION=0.5.1
mkdir bitcoin-${VERSION}-linux-gitian
pushd bitcoin-${VERSION}-linux-gitian
unzip ../bitcoin-${VERSION}-linux-gitian.zip
mkdir triangles-${VERSION}-linux-gitian
pushd triangles-${VERSION}-linux-gitian
unzip ../triangles-${VERSION}-linux-gitian.zip
mkdir gitian
cp ../bitcoin/contrib/gitian-downloader/*.pgp ./gitian/
cp ../triangles/contrib/gitian-downloader/*.pgp ./gitian/
for signer in $(ls ../gitian.sigs/${VERSION}/); do
cp ../gitian.sigs/${VERSION}/${signer}/bitcoin-build.assert ./gitian/${signer}-build.assert
cp ../gitian.sigs/${VERSION}/${signer}/bitcoin-build.assert.sig ./gitian/${signer}-build.assert.sig
cp ../gitian.sigs/${VERSION}/${signer}/triangles-build.assert ./gitian/${signer}-build.assert
cp ../gitian.sigs/${VERSION}/${signer}/triangles-build.assert.sig ./gitian/${signer}-build.assert.sig
done
zip -r bitcoin-${VERSION}-linux-gitian.zip *
cp bitcoin-${VERSION}-linux-gitian.zip ../
zip -r triangles-${VERSION}-linux-gitian.zip *
cp triangles-${VERSION}-linux-gitian.zip ../
popd
mkdir bitcoin-${VERSION}-win32-gitian
pushd bitcoin-${VERSION}-win32-gitian
unzip ../bitcoin-${VERSION}-win32-gitian.zip
mkdir triangles-${VERSION}-win32-gitian
pushd triangles-${VERSION}-win32-gitian
unzip ../triangles-${VERSION}-win32-gitian.zip
mkdir gitian
cp ../bitcoin/contrib/gitian-downloader/*.pgp ./gitian/
cp ../triangles/contrib/gitian-downloader/*.pgp ./gitian/
for signer in $(ls ../gitian.sigs/${VERSION}-win32/); do
cp ../gitian.sigs/${VERSION}-win32/${signer}/bitcoin-build.assert ./gitian/${signer}-build.assert
cp ../gitian.sigs/${VERSION}-win32/${signer}/bitcoin-build.assert.sig ./gitian/${signer}-build.assert.sig
cp ../gitian.sigs/${VERSION}-win32/${signer}/triangles-build.assert ./gitian/${signer}-build.assert
cp ../gitian.sigs/${VERSION}-win32/${signer}/triangles-build.assert.sig ./gitian/${signer}-build.assert.sig
done
zip -r bitcoin-${VERSION}-win32-gitian.zip *
cp bitcoin-${VERSION}-win32-gitian.zip ../
zip -r triangles-${VERSION}-win32-gitian.zip *
cp triangles-${VERSION}-win32-gitian.zip ../
popd
* Upload gitian zips to SourceForge
ICU
https://qt-project.org/wiki/Compiling-ICU-with-MinGW
doc/translation_process.md
+14 -14
View File
@@ -7,18 +7,18 @@ handle those translations.
Files and Folders
-----------------
### bitcoin-qt.pro
### triangles-qt.pro
This file takes care of generating `.qm` files from `.ts` files. It is mostly
automated.
### src/qt/bitcoin.qrc
### src/qt/triangles.qrc
This file must be updated whenever a new translation is added. Please note that
files must end with `.qm`, not `.ts`.
<qresource prefix="/translations">
<file alias="en">locale/bitcoin_en.qm</file>
<file alias="en">locale/triangles_en.qm</file>
...
</qresource>
@@ -26,11 +26,11 @@ files must end with `.qm`, not `.ts`.
This directory contains all translations. Filenames must adhere to this format:
bitcoin_xx_YY.ts or bitcoin_xx.ts
triangles_xx_YY.ts or triangles_xx.ts
#### Source file
`src/qt/locale/bitcoin_en.ts` is treated in a special way. It is used as the
`src/qt/locale/triangles_en.ts` is treated in a special way. It is used as the
source for all other translations. Whenever a string in the code is changed
this file must be updated to reflect those changes. Usually, this can be
accomplished by running `lupdate` (included in the Qt SDK).
@@ -44,7 +44,7 @@ Syncing with transifex
We are using http://transifex.net as a frontend for translating the client.
https://www.transifex.net/projects/p/bitcoin/resource/tx/
https://www.transifex.net/projects/p/triangles/resource/tx/
The "transifex client" (see: http://help.transifex.net/features/client/)
will help with fetching new translations from transifex. Use the following
@@ -55,9 +55,9 @@ config to be able to connect with the client.
[main]
host = https://www.transifex.net
[bitcoin.tx]
file_filter = src/qt/locale/bitcoin_<lang>.ts
source_file = src/qt/locale/bitcoin_en.ts
[triangles.tx]
file_filter = src/qt/locale/triangles_<lang>.ts
source_file = src/qt/locale/triangles_en.ts
source_lang = en
### .tx/config (for Windows)
@@ -65,9 +65,9 @@ config to be able to connect with the client.
[main]
host = https://www.transifex.net
[bitcoin.tx]
file_filter = src\qt\locale\bitcoin_<lang>.ts
source_file = src\qt\locale\bitcoin_en.ts
[triangles.tx]
file_filter = src\qt\locale\triangles_<lang>.ts
source_file = src\qt\locale\triangles_en.ts
source_lang = en
It is also possible to directly download new translations one by one from transifex.
@@ -75,6 +75,6 @@ It is also possible to directly download new translations one by one from transi
### Fetching new translations
1. `tx pull -a`
2. update `src/qt/bitcoin.qrc` manually or via
`ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/<file alias="\2">locale/\1.qm<\/file>/'`
2. update `src/qt/triangles.qrc` manually or via
`ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(triangles_\(.*\)\).ts/<file alias="\2">locale/\1.qm<\/file>/'`
3. `git add` new translations from `src/qt/locale/`
share/qt/extract_strings_qt.py
+3 -3
View File
@@ -6,7 +6,7 @@ they can be picked up by Qt linguist.
from subprocess import Popen, PIPE
import glob
OUT_CPP="src/qt/bitcoinstrings.cpp"
OUT_CPP="src/qt/trianglesstrings.cpp"
EMPTY=['""']
def parse_po(text):
@@ -62,9 +62,9 @@ f.write("""#include <QtGlobal>
#define UNUSED
#endif
""")
f.write('static const char UNUSED *bitcoin_strings[] = {')
f.write('static const char UNUSED *triangles_strings[] = {')
for (msgid, msgstr) in messages:
if msgid != EMPTY:
f.write('QT_TRANSLATE_NOOP("bitcoin-core", %s),\n' % ('\n'.join(msgid)))
f.write('QT_TRANSLATE_NOOP("Triangles-core", %s),\n' % ('\n'.join(msgid)))
f.write('};')
f.close()
share/qt/make_windows_icon.sh
+1 -1
View File
@@ -2,4 +2,4 @@
# create multiresolution windows icon
ICON_DST=../../src/qt/res/icons/triangles.ico
convert ../../src/qt/res/icons/triangles-16.png ../../src/qt/res/icons/triangles-32.png ../../src/qt/res/icons/triangles-48.png ${ICON_DST}
convert ../../src/qt/res/icons/triangles-32.png ../../src/qt/res/icons/triangles-32.png ../../src/qt/res/icons/triangles-48.png ${ICON_DST}
share/ui.rc
+1 -1
View File
@@ -1,4 +1,4 @@
BottleCaps ICON "pixmaps/BottleCaps.ico"
triangles ICON "pixmaps/triangles.ico"
#include "wx/msw/wx.rc"
src/addrman.cpp
+17 -17
View File
@@ -11,12 +11,12 @@ int CAddrInfo::GetTriedBucket(const std::vector<unsigned char> &nKey) const
CDataStream ss1(SER_GETHASH, 0);
std::vector<unsigned char> vchKey = GetKey();
ss1 << nKey << vchKey;
uint64 hash1 = Hash(ss1.begin(), ss1.end()).Get64();
uint64_t hash1 = Hash(ss1.begin(), ss1.end()).Get64();
CDataStream ss2(SER_GETHASH, 0);
std::vector<unsigned char> vchGroupKey = GetGroup();
ss2 << nKey << vchGroupKey << (hash1 % ADDRMAN_TRIED_BUCKETS_PER_GROUP);
uint64 hash2 = Hash(ss2.begin(), ss2.end()).Get64();
uint64_t hash2 = Hash(ss2.begin(), ss2.end()).Get64();
return hash2 % ADDRMAN_TRIED_BUCKET_COUNT;
}
@@ -26,15 +26,15 @@ int CAddrInfo::GetNewBucket(const std::vector<unsigned char> &nKey, const CNetAd
std::vector<unsigned char> vchGroupKey = GetGroup();
std::vector<unsigned char> vchSourceGroupKey = src.GetGroup();
ss1 << nKey << vchGroupKey << vchSourceGroupKey;
uint64 hash1 = Hash(ss1.begin(), ss1.end()).Get64();
uint64_t hash1 = Hash(ss1.begin(), ss1.end()).Get64();
CDataStream ss2(SER_GETHASH, 0);
ss2 << nKey << vchSourceGroupKey << (hash1 % ADDRMAN_NEW_BUCKETS_PER_SOURCE_GROUP);
uint64 hash2 = Hash(ss2.begin(), ss2.end()).Get64();
uint64_t hash2 = Hash(ss2.begin(), ss2.end()).Get64();
return hash2 % ADDRMAN_NEW_BUCKET_COUNT;
}
bool CAddrInfo::IsTerrible(int64 nNow) const
bool CAddrInfo::IsTerrible(int64_t nNow) const
{
if (nLastTry && nLastTry >= nNow-60) // never remove things tried the last minute
return false;
@@ -54,12 +54,12 @@ bool CAddrInfo::IsTerrible(int64 nNow) const
return false;
}
double CAddrInfo::GetChance(int64 nNow) const
double CAddrInfo::GetChance(int64_t nNow) const
{
double fChance = 1.0;
int64 nSinceLastSeen = nNow - nTime;
int64 nSinceLastTry = nNow - nLastTry;
int64_t nSinceLastSeen = nNow - nTime;
int64_t nSinceLastTry = nNow - nLastTry;
if (nSinceLastSeen < 0) nSinceLastSeen = 0;
if (nSinceLastTry < 0) nSinceLastTry = 0;
@@ -128,7 +128,7 @@ int CAddrMan::SelectTried(int nKBucket)
// random shuffle the first few elements (using the entire list)
// find the least recently tried among them
int64 nOldest = -1;
int64_t nOldest = -1;
int nOldestPos = -1;
for (unsigned int i = 0; i < ADDRMAN_TRIED_ENTRIES_INSPECT_ON_EVICT && i < vTried.size(); i++)
{
@@ -258,7 +258,7 @@ void CAddrMan::MakeTried(CAddrInfo& info, int nId, int nOrigin)
return;
}
void CAddrMan::Good_(const CService &addr, int64 nTime)
void CAddrMan::Good_(const CService &addr, int64_t nTime)
{
// printf("Good: addr=%s\n", addr.ToString().c_str());
@@ -309,7 +309,7 @@ void CAddrMan::Good_(const CService &addr, int64 nTime)
MakeTried(info, nId, nUBucket);
}
bool CAddrMan::Add_(const CAddress &addr, const CNetAddr& source, int64 nTimePenalty)
bool CAddrMan::Add_(const CAddress &addr, const CNetAddr& source, int64_t nTimePenalty)
{
if (!addr.IsRoutable())
return false;
@@ -322,9 +322,9 @@ bool CAddrMan::Add_(const CAddress &addr, const CNetAddr& source, int64 nTimePen
{
// periodically update nTime
bool fCurrentlyOnline = (GetAdjustedTime() - addr.nTime < 24 * 60 * 60);
int64 nUpdateInterval = (fCurrentlyOnline ? 60 * 60 : 24 * 60 * 60);
int64_t nUpdateInterval = (fCurrentlyOnline ? 60 * 60 : 24 * 60 * 60);
if (addr.nTime && (!pinfo->nTime || pinfo->nTime < addr.nTime - nUpdateInterval - nTimePenalty))
pinfo->nTime = max((int64)0, addr.nTime - nTimePenalty);
pinfo->nTime = max((int64_t)0, addr.nTime - nTimePenalty);
// add services
pinfo->nServices |= addr.nServices;
@@ -349,7 +349,7 @@ bool CAddrMan::Add_(const CAddress &addr, const CNetAddr& source, int64 nTimePen
return false;
} else {
pinfo = Create(addr, source, &nId);
pinfo->nTime = max((int64)0, (int64)pinfo->nTime - nTimePenalty);
pinfo->nTime = max((int64_t)0, (int64_t)pinfo->nTime - nTimePenalty);
// printf("Added %s [nTime=%fhr]\n", pinfo->ToString().c_str(), (GetAdjustedTime() - pinfo->nTime) / 3600.0);
nNew++;
fNew = true;
@@ -367,7 +367,7 @@ bool CAddrMan::Add_(const CAddress &addr, const CNetAddr& source, int64 nTimePen
return fNew;
}
void CAddrMan::Attempt_(const CService &addr, int64 nTime)
void CAddrMan::Attempt_(const CService &addr, int64_t nTime)
{
CAddrInfo *pinfo = Find(addr);
@@ -506,7 +506,7 @@ void CAddrMan::GetAddr_(std::vector<CAddress> &vAddr)
}
}
void CAddrMan::Connected_(const CService &addr, int64 nTime)
void CAddrMan::Connected_(const CService &addr, int64_t nTime)
{
CAddrInfo *pinfo = Find(addr);
@@ -521,7 +521,7 @@ void CAddrMan::Connected_(const CService &addr, int64 nTime)
return;
// update info
int64 nUpdateInterval = 20 * 60;
int64_t nUpdateInterval = 20 * 60;
if (nTime - info.nTime > nUpdateInterval)
info.nTime = nTime;
}
src/addrman.h
+15 -86
View File
@@ -1,8 +1,8 @@
// Copyright (c) 2012 Pieter Wuille
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef _BITCOIN_ADDRMAN
#define _BITCOIN_ADDRMAN 1
#ifndef _TRIANGLES_ADDRMAN
#define _TRIANGLES_ADDRMAN 1
#include "netbase.h"
#include "protocol.h"
@@ -24,10 +24,10 @@ private:
CNetAddr source;
// last successful connection by us
int64 nLastSuccess;
int64_t nLastSuccess;
// last try whatsoever by us:
// int64 CAddress::nLastTry
// int64_t CAddress::nLastTry
// connection attempts since last successful attempt
int nAttempts;
@@ -86,10 +86,10 @@ public:
}
// Determine whether the statistics about this entry are bad enough so that it can just be deleted
bool IsTerrible(int64 nNow = GetAdjustedTime()) const;
bool IsTerrible(int64_t nNow = GetAdjustedTime()) const;
// Calculate the relative chance this entry should be given when selecting nodes to connect to
double GetChance(int64 nNow = GetAdjustedTime()) const;
double GetChance(int64_t nNow = GetAdjustedTime()) const;
};
@@ -179,12 +179,6 @@ private:
// find an nId based on its network address
std::map<CNetAddr, int> mapAddr;
// address verification tokens
std::map<CNetAddr, uint64> verificationToken;
// address reconnect tokens
std::map<CNetAddr, uint64> reconnectToken;
// randomly-ordered vector of all nIds
std::vector<int> vRandom;
@@ -226,13 +220,13 @@ protected:
void MakeTried(CAddrInfo& info, int nId, int nOrigin);
// Mark an entry "good", possibly moving it from "new" to "tried".
void Good_(const CService &addr, int64 nTime);
void Good_(const CService &addr, int64_t nTime);
// Add an entry to the "new" table.
bool Add_(const CAddress &addr, const CNetAddr& source, int64 nTimePenalty);
bool Add_(const CAddress &addr, const CNetAddr& source, int64_t nTimePenalty);
// Mark an entry as attempted to connect.
void Attempt_(const CService &addr, int64 nTime);
void Attempt_(const CService &addr, int64_t nTime);
// Select an address to connect to.
// nUnkBias determines how much to favor new addresses over tried ones (min=0, max=100)
@@ -247,7 +241,7 @@ protected:
void GetAddr_(std::vector<CAddress> &vAddr);
// Mark an entry as currently-connected-to.
void Connected_(const CService &addr, int64 nTime);
void Connected_(const CService &addr, int64_t nTime);
public:
@@ -415,7 +409,7 @@ public:
}
// Add a single address.
bool Add(const CAddress &addr, const CNetAddr& source, int64 nTimePenalty = 0)
bool Add(const CAddress &addr, const CNetAddr& source, int64_t nTimePenalty = 0)
{
bool fRet = false;
{
@@ -430,7 +424,7 @@ public:
}
// Add multiple addresses.
bool Add(const std::vector<CAddress> &vAddr, const CNetAddr& source, int64 nTimePenalty = 0)
bool Add(const std::vector<CAddress> &vAddr, const CNetAddr& source, int64_t nTimePenalty = 0)
{
int nAdd = 0;
{
@@ -446,7 +440,7 @@ public:
}
// Mark an entry as accessible.
void Good(const CService &addr, int64 nTime = GetAdjustedTime())
void Good(const CService &addr, int64_t nTime = GetAdjustedTime())
{
{
LOCK(cs);
@@ -457,7 +451,7 @@ public:
}
// Mark an entry as connection attempted to.
void Attempt(const CService &addr, int64 nTime = GetAdjustedTime())
void Attempt(const CService &addr, int64_t nTime = GetAdjustedTime())
{
{
LOCK(cs);
@@ -466,71 +460,6 @@ public:
Check();
}
}
void SetReconnectToken(const CNetAddr &addr, uint64 reconnect_token)
{
{
LOCK(cs);
Check();
reconnectToken[addr] = reconnect_token;
Check();
}
}
bool GetReconnectToken(const CNetAddr &addr, uint64& reconnect_token)
{
bool result = false;
{
LOCK(cs);
Check();
std::map<
CNetAddr,
uint64
>::const_iterator found = reconnectToken.find(
addr
);
if (
reconnectToken.end() != found
) {
reconnect_token = found->second;
result = true;
}
Check();
}
return result;
}
void SetVerificationToken(const CNetAddr &addr, uint64 verification_token)
{
{
LOCK(cs);
Check();
verificationToken[addr] = verification_token;
Check();
}
}
bool CheckVerificationToken(const CNetAddr &addr, uint64 verification_token)
{
bool result = false;
{
LOCK(cs);
Check();
std::map<
CNetAddr,
uint64
>::const_iterator found = verificationToken.find(
addr
);
if (
verificationToken.end() != found
) {
result = verification_token == found->second;
}
Check();
}
return result;
}
// Choose an address to connect to.
// nUnkBias determines how much "new" entries are favored over "tried" ones (0-100).
@@ -560,7 +489,7 @@ public:
}
// Mark an entry as currently-connected-to.
void Connected(const CService &addr, int64 nTime = GetAdjustedTime())
void Connected(const CService &addr, int64_t nTime = GetAdjustedTime())
{
{
LOCK(cs);
src/alert.cpp
+34 -4
View File
@@ -2,6 +2,9 @@
// Alert system
//
#include <algorithm>
#include <boost/algorithm/string/classification.hpp>
#include <boost/algorithm/string/replace.hpp>
#include <boost/foreach.hpp>
#include <map>
@@ -50,8 +53,8 @@ std::string CUnsignedAlert::ToString() const
return strprintf(
"CAlert(\n"
" nVersion = %d\n"
" nRelayUntil = %"PRI64d"\n"
" nExpiration = %"PRI64d"\n"
" nRelayUntil = %"PRId64"\n"
" nExpiration = %"PRId64"\n"
" nID = %d\n"
" nCancel = %d\n"
" setCancel = %s\n"
@@ -167,7 +170,7 @@ CAlert CAlert::getAlertByHash(const uint256 &hash)
return retval;
}
bool CAlert::ProcessAlert()
bool CAlert::ProcessAlert(bool fThread)
{
if (!CheckSignature())
return false;
@@ -231,11 +234,38 @@ bool CAlert::ProcessAlert()
// Add to mapAlerts
mapAlerts.insert(make_pair(GetHash(), *this));
// Notify UI if it applies to me
// Notify UI and -alertnotify if it applies to me
if(AppliesToMe())
{
uiInterface.NotifyAlertChanged(GetHash(), CT_NEW);
std::string strCmd = GetArg("-alertnotify", "");
if (!strCmd.empty())
{
// Alert text should be plain ascii coming from a trusted source, but to
// be safe we first strip anything not in safeChars, then add single quotes around
// the whole string before passing it to the shell:
std::string singleQuote("'");
// safeChars chosen to allow simple messages/URLs/email addresses, but avoid anything
// even possibly remotely dangerous like & or >
std::string safeChars("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890 .,;_/:?@");
std::string safeStatus;
for (std::string::size_type i = 0; i < strStatusBar.size(); i++)
{
if (safeChars.find(strStatusBar[i]) != std::string::npos)
safeStatus.push_back(strStatusBar[i]);
}
safeStatus = singleQuote+safeStatus+singleQuote;
boost::replace_all(strCmd, "%s", safeStatus);
if (fThread)
boost::thread t(runCommand, strCmd); // thread runs free
else
runCommand(strCmd);
}
}
}
printf("accepted alert %d, AppliesToMe()=%d\n", nID, AppliesToMe());
return true;
}
src/alert.h
+7 -5
View File
@@ -3,8 +3,8 @@
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef _BITCOINALERT_H_
#define _BITCOINALERT_H_ 1
#ifndef _TRIANGLESALERT_H_
#define _TRIANGLESALERT_H_ 1
#include <set>
#include <string>
@@ -24,8 +24,8 @@ class CUnsignedAlert
{
public:
int nVersion;
int64 nRelayUntil; // when newer nodes stop relaying to newer nodes
int64 nExpiration;
int64_t nRelayUntil; // when newer nodes stop relaying to newer nodes
int64_t nExpiration;
int nID;
int nCancel;
std::set<int> setCancel;
@@ -91,7 +91,7 @@ public:
bool AppliesToMe() const;
bool RelayTo(CNode* pnode) const;
bool CheckSignature() const;
bool ProcessAlert();
bool ProcessAlert(bool fThread = true);
/*
* Get copy of (active) alert object by hash. Returns a null alert if it is not found.
@@ -100,3 +100,5 @@ public:
};
#endif
src/allocators.h
+2 -2
View File
@@ -2,8 +2,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_ALLOCATORS_H
#define BITCOIN_ALLOCATORS_H
#ifndef TRIANGLES_ALLOCATORS_H
#define TRIANGLES_ALLOCATORS_H
#include <string.h>
#include <string>
src/base58.h
+22 -22
View File
@@ -12,8 +12,8 @@
// - E-mail usually won't line-break if there's no punctuation to break at.
// - Double-clicking selects the whole number as one word if it's all alphanumeric.
//
#ifndef BITCOIN_BASE58_H
#define BITCOIN_BASE58_H
#ifndef TRIANGLES_BASE58_H
#define TRIANGLES_BASE58_H
#include <string>
#include <vector>
@@ -253,25 +253,25 @@ public:
bool operator> (const CBase58Data& b58) const { return CompareTo(b58) > 0; }
};
/** base58-encoded Bitcoin addresses.
/** base58-encoded addresses.
* Public-key-hash-addresses have version 0 (or 111 testnet).
* The data vector contains RIPEMD160(SHA256(pubkey)), where pubkey is the serialized public key.
* Script-hash-addresses have version 5 (or 196 testnet).
* The data vector contains RIPEMD160(SHA256(cscript)), where cscript is the serialized redemption script.
*/
class CBitcoinAddress;
class CBitcoinAddressVisitor : public boost::static_visitor<bool>
class CTrianglesAddress;
class CTrianglesAddressVisitor : public boost::static_visitor<bool>
{
private:
CBitcoinAddress *addr;
CTrianglesAddress *addr;
public:
CBitcoinAddressVisitor(CBitcoinAddress *addrIn) : addr(addrIn) { }
CTrianglesAddressVisitor(CTrianglesAddress *addrIn) : addr(addrIn) { }
bool operator()(const CKeyID &id) const;
bool operator()(const CScriptID &id) const;
bool operator()(const CNoDestination &no) const;
};
class CBitcoinAddress : public CBase58Data
class CTrianglesAddress : public CBase58Data
{
public:
enum
@@ -294,7 +294,7 @@ public:
bool Set(const CTxDestination &dest)
{
return boost::apply_visitor(CBitcoinAddressVisitor(this), dest);
return boost::apply_visitor(CTrianglesAddressVisitor(this), dest);
}
bool IsValid() const
@@ -327,21 +327,21 @@ public:
return fExpectTestNet == fTestNet && vchData.size() == nExpectedSize;
}
CBitcoinAddress()
CTrianglesAddress()
{
}
CBitcoinAddress(const CTxDestination &dest)
CTrianglesAddress(const CTxDestination &dest)
{
Set(dest);
}
CBitcoinAddress(const std::string& strAddress)
CTrianglesAddress(const std::string& strAddress)
{
SetString(strAddress);
}
CBitcoinAddress(const char* pszAddress)
CTrianglesAddress(const char* pszAddress)
{
SetString(pszAddress);
}
@@ -394,18 +394,18 @@ public:
}
};
bool inline CBitcoinAddressVisitor::operator()(const CKeyID &id) const { return addr->Set(id); }
bool inline CBitcoinAddressVisitor::operator()(const CScriptID &id) const { return addr->Set(id); }
bool inline CBitcoinAddressVisitor::operator()(const CNoDestination &id) const { return false; }
bool inline CTrianglesAddressVisitor::operator()(const CKeyID &id) const { return addr->Set(id); }
bool inline CTrianglesAddressVisitor::operator()(const CScriptID &id) const { return addr->Set(id); }
bool inline CTrianglesAddressVisitor::operator()(const CNoDestination &id) const { return false; }
/** A base58-encoded secret key */
class CBitcoinSecret : public CBase58Data
class CTrianglesSecret : public CBase58Data
{
public:
void SetSecret(const CSecret& vchSecret, bool fCompressed)
{
assert(vchSecret.size() == 32);
SetData(128 + (fTestNet ? CBitcoinAddress::PUBKEY_ADDRESS_TEST : CBitcoinAddress::PUBKEY_ADDRESS), &vchSecret[0], vchSecret.size());
SetData(128 + (fTestNet ? CTrianglesAddress::PUBKEY_ADDRESS_TEST : CTrianglesAddress::PUBKEY_ADDRESS), &vchSecret[0], vchSecret.size());
if (fCompressed)
vchData.push_back(1);
}
@@ -424,10 +424,10 @@ public:
bool fExpectTestNet = false;
switch(nVersion)
{
case (128 + CBitcoinAddress::PUBKEY_ADDRESS):
case (128 + CTrianglesAddress::PUBKEY_ADDRESS):
break;
case (128 + CBitcoinAddress::PUBKEY_ADDRESS_TEST):
case (128 + CTrianglesAddress::PUBKEY_ADDRESS_TEST):
fExpectTestNet = true;
break;
@@ -447,12 +447,12 @@ public:
return SetString(strSecret.c_str());
}
CBitcoinSecret(const CSecret& vchSecret, bool fCompressed)
CTrianglesSecret(const CSecret& vchSecret, bool fCompressed)
{
SetSecret(vchSecret, fCompressed);
}
CBitcoinSecret()
CTrianglesSecret()
{
}
};
src/bignum.h
+186 -23
View File
@@ -2,14 +2,20 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_BIGNUM_H
#define BITCOIN_BIGNUM_H
#ifndef TRIANGLES_BIGNUM_H
#define TRIANGLES_BIGNUM_H
#include "serialize.h"
#include "uint256.h"
#include "version.h"
#include <openssl/bn.h>
#include <stdexcept>
#include <vector>
#include <openssl/bn.h>
#include "util.h" // for uint64
#include <stdint.h>
/** Errors thrown by the bignum class */
class bignum_error : public std::runtime_error
@@ -79,17 +85,17 @@ public:
}
//CBigNum(char n) is not portable. Use 'signed char' or 'unsigned char'.
CBigNum(signed char n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(short n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(int n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(long n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(int64 n) { BN_init(this); setint64(n); }
CBigNum(unsigned char n) { BN_init(this); setulong(n); }
CBigNum(unsigned short n) { BN_init(this); setulong(n); }
CBigNum(unsigned int n) { BN_init(this); setulong(n); }
CBigNum(unsigned long n) { BN_init(this); setulong(n); }
CBigNum(uint64 n) { BN_init(this); setuint64(n); }
explicit CBigNum(uint256 n) { BN_init(this); setuint256(n); }
CBigNum(signed char n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(short n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(int n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(long n) { BN_init(this); if (n >= 0) setulong(n); else setint64(n); }
CBigNum(long long n) { BN_init(this); setint64(n); }
CBigNum(unsigned char n) { BN_init(this); setulong(n); }
CBigNum(unsigned short n) { BN_init(this); setulong(n); }
CBigNum(unsigned int n) { BN_init(this); setulong(n); }
CBigNum(unsigned long n) { BN_init(this); setulong(n); }
CBigNum(unsigned long long n) { BN_init(this); setuint64(n); }
explicit CBigNum(uint256 n) { BN_init(this); setuint256(n); }
explicit CBigNum(const std::vector<unsigned char>& vch)
{
@@ -97,6 +103,40 @@ public:
setvch(vch);
}
/** Generates a cryptographically secure random number between zero and range exclusive
* i.e. 0 < returned number < range
* @param range The upper bound on the number.
* @return
*/
static CBigNum randBignum(const CBigNum& range) {
CBigNum ret;
if(!BN_rand_range(&ret, &range)){
throw bignum_error("CBigNum:rand element : BN_rand_range failed");
}
return ret;
}
/** Generates a cryptographically secure random k-bit number
* @param k The bit length of the number.
* @return
*/
static CBigNum RandKBitBigum(const uint32_t k){
CBigNum ret;
if(!BN_rand(&ret, k, -1, 0)){
throw bignum_error("CBigNum:rand element : BN_rand failed");
}
return ret;
}
/**Returns the size in bits of the underlying bignum.
*
* @return the size
*/
int bitSize() const{
return BN_num_bits(this);
}
void setulong(unsigned long n)
{
if (!BN_set_word(this, n))
@@ -122,14 +162,14 @@ public:
return (n > (unsigned long)std::numeric_limits<int>::max() ? std::numeric_limits<int>::min() : -(int)n);
}
void setint64(int64 sn)
void setint64(int64_t sn)
{
unsigned char pch[sizeof(sn) + 6];
unsigned char* p = pch + 4;
bool fNegative;
uint64 n;
uint64_t n;
if (sn < (int64)0)
if (sn < (int64_t)0)
{
// Since the minimum signed integer cannot be represented as positive so long as its type is signed, and it's not well-defined what happens if you make it unsigned before negating it, we instead increment the negative integer by 1, convert it, then increment the (now positive) unsigned integer by 1 to compensate
n = -(sn + 1);
@@ -165,7 +205,7 @@ public:
BN_mpi2bn(pch, p - pch, this);
}
uint64 getuint64()
uint64_t getuint64()
{
unsigned int nSize = BN_bn2mpi(this, NULL);
if (nSize < 4)
@@ -174,13 +214,13 @@ public:
BN_bn2mpi(this, &vch[0]);
if (vch.size() > 4)
vch[4] &= 0x7f;
uint64 n = 0;
uint64_t n = 0;
for (unsigned int i = 0, j = vch.size()-1; i < sizeof(n) && j >= 4; i++, j--)
((unsigned char*)&n)[i] = vch[j];
return n;
}
void setuint64(uint64 n)
void setuint64(uint64_t n)
{
unsigned char pch[sizeof(n) + 6];
unsigned char* p = pch + 4;
@@ -235,7 +275,7 @@ public:
BN_mpi2bn(pch, p - pch, this);
}
uint256 getuint256()
uint256 getuint256() const
{
unsigned int nSize = BN_bn2mpi(this, NULL);
if (nSize < 4)
@@ -383,6 +423,122 @@ public:
setvch(vch);
}
/**
* exponentiation with an int. this^e
* @param e the exponent as an int
* @return
*/
CBigNum pow(const int e) const {
return this->pow(CBigNum(e));
}
/**
* exponentiation this^e
* @param e the exponent
* @return
*/
CBigNum pow(const CBigNum& e) const {
CAutoBN_CTX pctx;
CBigNum ret;
if (!BN_exp(&ret, this, &e, pctx))
throw bignum_error("CBigNum::pow : BN_exp failed");
return ret;
}
/**
* modular multiplication: (this * b) mod m
* @param b operand
* @param m modulus
*/
CBigNum mul_mod(const CBigNum& b, const CBigNum& m) const {
CAutoBN_CTX pctx;
CBigNum ret;
if (!BN_mod_mul(&ret, this, &b, &m, pctx))
throw bignum_error("CBigNum::mul_mod : BN_mod_mul failed");
return ret;
}
/**
* modular exponentiation: this^e mod n
* @param e exponent
* @param m modulus
*/
CBigNum pow_mod(const CBigNum& e, const CBigNum& m) const {
CAutoBN_CTX pctx;
CBigNum ret;
if( e < 0){
// g^-x = (g^-1)^x
CBigNum inv = this->inverse(m);
CBigNum posE = e * -1;
if (!BN_mod_exp(&ret, &inv, &posE, &m, pctx))
throw bignum_error("CBigNum::pow_mod: BN_mod_exp failed on negative exponent");
}else
if (!BN_mod_exp(&ret, this, &e, &m, pctx))
throw bignum_error("CBigNum::pow_mod : BN_mod_exp failed");
return ret;
}
/**
* Calculates the inverse of this element mod m.
* i.e. i such this*i = 1 mod m
* @param m the modu
* @return the inverse
*/
CBigNum inverse(const CBigNum& m) const {
CAutoBN_CTX pctx;
CBigNum ret;
if (!BN_mod_inverse(&ret, this, &m, pctx))
throw bignum_error("CBigNum::inverse*= :BN_mod_inverse");
return ret;
}
/**
* Generates a random (safe) prime of numBits bits
* @param numBits the number of bits
* @param safe true for a safe prime
* @return the prime
*/
static CBigNum generatePrime(const unsigned int numBits, bool safe = false) {
CBigNum ret;
if(!BN_generate_prime_ex(&ret, numBits, (safe == true), NULL, NULL, NULL))
throw bignum_error("CBigNum::generatePrime*= :BN_generate_prime_ex");
return ret;
}
/**
* Calculates the greatest common divisor (GCD) of two numbers.
* @param m the second element
* @return the GCD
*/
CBigNum gcd( const CBigNum& b) const{
CAutoBN_CTX pctx;
CBigNum ret;
if (!BN_gcd(&ret, this, &b, pctx))
throw bignum_error("CBigNum::gcd*= :BN_gcd");
return ret;
}
/**
* Miller-Rabin primality test on this element
* @param checks: optional, the number of Miller-Rabin tests to run
* default causes error rate of 2^-80.
* @return true if prime
*/
bool isPrime(const int checks=BN_prime_checks) const {
CAutoBN_CTX pctx;
int ret = BN_is_prime(this, checks, NULL, pctx, NULL);
if(ret < 0){
throw bignum_error("CBigNum::isPrime :BN_is_prime");
}
return ret;
}
bool isOne() const {
return BN_is_one(this);
}
bool operator!() const
{
@@ -485,6 +641,8 @@ public:
friend inline const CBigNum operator-(const CBigNum& a, const CBigNum& b);
friend inline const CBigNum operator/(const CBigNum& a, const CBigNum& b);
friend inline const CBigNum operator%(const CBigNum& a, const CBigNum& b);
friend inline const CBigNum operator*(const CBigNum& a, const CBigNum& b);
friend inline bool operator<(const CBigNum& a, const CBigNum& b);
};
@@ -534,7 +692,7 @@ inline const CBigNum operator%(const CBigNum& a, const CBigNum& b)
{
CAutoBN_CTX pctx;
CBigNum r;
if (!BN_mod(&r, &a, &b, pctx))
if (!BN_nnmod(&r, &a, &b, pctx))
throw bignum_error("CBigNum::operator% : BN_div failed");
return r;
}
@@ -561,4 +719,9 @@ inline bool operator>=(const CBigNum& a, const CBigNum& b) { return (BN_cmp(&a,
inline bool operator<(const CBigNum& a, const CBigNum& b) { return (BN_cmp(&a, &b) < 0); }
inline bool operator>(const CBigNum& a, const CBigNum& b) { return (BN_cmp(&a, &b) > 0); }
inline std::ostream& operator<<(std::ostream &strm, const CBigNum &b) { return strm << b.ToString(10); }
typedef CBigNum Bignum;
#endif
src/checkpoints.cpp
+13 -29
View File
@@ -7,7 +7,7 @@
#include "checkpoints.h"
#include "db.h"
#include "txdb.h"
#include "main.h"
#include "uint256.h"
@@ -35,6 +35,7 @@ namespace Checkpoints
( 9001, uint256("0x6d5c6c5f201cc9e59659ee0da30d1430dc6bf3b12a8ff4c3864ab8d6286b0007"))
( 9002, uint256("0xa1e20fb1d44688b763690cf74d6aefe859e4cc32981f9e3f2b2ae9702bbcf249"))
( 10881, uint256("0x4b6554c45e1e6764a6f3c309c47baf53c9edd81f624e52b072518cd15da237e6"))
( 17650, uint256("0x224940e1f986a202209b8e762728d1452ab45870c308abf84905674acf326a47"))
;
static MapCheckpoints mapCheckpointsTestnet =
@@ -47,11 +48,12 @@ namespace Checkpoints
( 3935, uint256("0xe16290c9757d1368b8d7c35de07d4f8f70c2c9f9c785b667df0c3bff85086ca6"))
( 5703, uint256("0x587db07bb2172ad7db72c5fabc2518262a1b27f503f99417510b2c6fafa6557b"))
( 9000, uint256("0x00000000019ef6b2f5e7c324c7d083ee94502305aabc7e9cd73a7fb2a57bb8db"))
( 9001, uint256("0x6d5c6c5f201cc9e59659ee0da30d1430dc6bf3b12a8ff4c3864ab8d6286b0007"))
( 9001, uint256("0x6d5c6c5f201cc9e59659ee0da30d1430dc6bf3b12a8ff4c3864ab8d6286b0007"))
( 9002, uint256("0xa1e20fb1d44688b763690cf74d6aefe859e4cc32981f9e3f2b2ae9702bbcf249"))
( 10881, uint256("0x4b6554c45e1e6764a6f3c309c47baf53c9edd81f624e52b072518cd15da237e6"))
( 17650, uint256("0x224940e1f986a202209b8e762728d1452ab45870c308abf84905674acf326a47"))
;
bool CheckHardened(int nHeight, const uint256& hash)
{
MapCheckpoints& checkpoints = (fTestNet ? mapCheckpointsTestnet : mapCheckpoints);
@@ -83,8 +85,8 @@ namespace Checkpoints
}
// triangles: synchronized checkpoint (centrally broadcasted)
uint256 hashSyncCheckpoint = uint256("0x4b6554c45e1e6764a6f3c309c47baf53c9edd81f624e52b072518cd15da237e6");
uint256 hashPendingCheckpoint = uint256("0x4b6554c45e1e6764a6f3c309c47baf53c9edd81f624e52b072518cd15da237e6");
uint256 hashSyncCheckpoint = uint256("0x224940e1f986a202209b8e762728d1452ab45870c308abf84905674acf326a47");
uint256 hashPendingCheckpoint = uint256("0x224940e1f986a202209b8e762728d1452ab45870c308abf84905674acf326a47");
CSyncCheckpoint checkpointMessage;
CSyncCheckpoint checkpointMessagePending;
uint256 hashInvalidCheckpoint = 0;
@@ -120,7 +122,7 @@ namespace Checkpoints
CBlockIndex* pindex = pindexSyncCheckpoint;
while (pindex->nHeight > pindexCheckpointRecv->nHeight)
if (!(pindex = pindex->pprev))
return error("ValidateSyncCheckpoint: pprev1 null - block index structure failure");
return error("ValidateSyncCheckpoint: pprev null - block index structure failure");
if (pindex->GetBlockHash() != hashCheckpoint)
{
hashInvalidCheckpoint = hashCheckpoint;
@@ -155,7 +157,6 @@ namespace Checkpoints
}
if (!txdb.TxnCommit())
return error("WriteSyncCheckpoint(): failed to commit to db sync checkpoint %s", hashCheckpoint.ToString().c_str());
txdb.Close();
Checkpoints::hashSyncCheckpoint = hashCheckpoint;
return true;
@@ -186,7 +187,6 @@ namespace Checkpoints
return error("AcceptPendingSyncCheckpoint: SetBestChain failed for sync checkpoint %s", hashPendingCheckpoint.ToString().c_str());
}
}
txdb.Close();
if (!WriteSyncCheckpoint(hashPendingCheckpoint))
return error("AcceptPendingSyncCheckpoint(): failed to write sync checkpoint %s", hashPendingCheckpoint.ToString().c_str());
@@ -208,14 +208,10 @@ namespace Checkpoints
// Automatically select a suitable sync-checkpoint
uint256 AutoSelectSyncCheckpoint()
{
// Proof-of-work blocks are immediately checkpointed
// to defend against 51% attack which rejects other miners block
// Select the last proof-of-work block
const CBlockIndex *pindex = GetLastBlockIndex(pindexBest, false);
// Search forward for a block within max span and 6 blocks
while (pindex->pnext && (pindex->GetBlockTime() + CHECKPOINT_MAX_SPAN <= pindexBest->GetBlockTime() || pindex->nHeight + 6 <= pindexBest->nHeight))
pindex = pindex->pnext;
const CBlockIndex *pindex = pindexBest;
// Search backward for a block within max span and maturity window
while (pindex->pprev && (pindex->GetBlockTime() + CHECKPOINT_MAX_SPAN > pindexBest->GetBlockTime() || pindex->nHeight + 8 > pindexBest->nHeight))
pindex = pindex->pprev;
return pindex->GetBlockHash();
}
@@ -277,7 +273,6 @@ namespace Checkpoints
{
return error("ResetSyncCheckpoint: SetBestChain failed for hardened checkpoint %s", hash.ToString().c_str());
}
txdb.Close();
}
else if(!mapBlockIndex.count(hash))
{
@@ -313,7 +308,7 @@ namespace Checkpoints
{
// Test signing a sync-checkpoint with genesis block
CSyncCheckpoint checkpoint;
checkpoint.hashCheckpoint = !fTestNet ? hashGenesisBlock : hashGenesisBlockTestNet;
checkpoint.hashCheckpoint = !fTestNet ? hashGenesisBlockOfficial : hashGenesisBlockTestNet;
CDataStream sMsg(SER_NETWORK, PROTOCOL_VERSION);
sMsg << (CUnsignedSyncCheckpoint)checkpoint;
checkpoint.vchMsg = std::vector<unsigned char>(sMsg.begin(), sMsg.end());
@@ -370,16 +365,6 @@ namespace Checkpoints
return (nBestHeight >= pindexSync->nHeight + nCoinbaseMaturity ||
pindexSync->GetBlockTime() + nStakeMinAge < GetAdjustedTime());
}
// Is the sync-checkpoint too old?
bool IsSyncCheckpointTooOld(unsigned int nSeconds)
{
LOCK(cs_hashSyncCheckpoint);
// sync-checkpoint should always be accepted block
assert(mapBlockIndex.count(hashSyncCheckpoint));
const CBlockIndex* pindexSync = mapBlockIndex[hashSyncCheckpoint];
return (pindexSync->GetBlockTime() + nSeconds < GetAdjustedTime());
}
}
// triangles: sync-checkpoint master key
@@ -443,7 +428,6 @@ bool CSyncCheckpoint::ProcessSyncCheckpoint(CNode* pfrom)
return error("ProcessSyncCheckpoint: SetBestChain failed for sync checkpoint %s", hashCheckpoint.ToString().c_str());
}
}
txdb.Close();
if (!Checkpoints::WriteSyncCheckpoint(hashCheckpoint))
return error("ProcessSyncCheckpoint(): failed to write sync checkpoint %s", hashCheckpoint.ToString().c_str());
src/checkpoints.h
+19 -3
View File
@@ -1,8 +1,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_CHECKPOINT_H
#define BITCOIN_CHECKPOINT_H
#ifndef TRIANGLES_CHECKPOINT_H
#define TRIANGLES_CHECKPOINT_H
#include <map>
#include "net.h"
@@ -10,6 +10,12 @@
#define CHECKPOINT_MAX_SPAN (60 * 60 * 2) // max 2 hours before latest block
#ifdef WIN32
#undef STRICT
#undef PERMISSIVE
#undef ADVISORY
#endif
class uint256;
class CBlockIndex;
class CSyncCheckpoint;
@@ -19,6 +25,17 @@ class CSyncCheckpoint;
*/
namespace Checkpoints
{
/** Checkpointing mode */
enum CPMode
{
// Scrict checkpoints policy, perform conflicts verification and resolve conflicts
STRICT = 0,
// Advisory checkpoints policy, perform conflicts verification but don't try to resolve them
ADVISORY = 1,
// Permissive checkpoints policy, don't perform any checking
PERMISSIVE = 2
};
// Returns true if block passes checkpoint checks
bool CheckHardened(int nHeight, const uint256& hash);
@@ -44,7 +61,6 @@ namespace Checkpoints
bool SetCheckpointPrivKey(std::string strPrivKey);
bool SendSyncCheckpoint(uint256 hashCheckpoint);
bool IsMatureSyncCheckpoint();
bool IsSyncCheckpointTooOld(unsigned int nSeconds);
}
// triangles: synchronized checkpoint
src/clientversion.h
+5 -5
View File
@@ -5,11 +5,11 @@
// client versioning
//
// These need to be macros, as version.cpp's and bitcoin-qt.rc's voodoo requires it
#define CLIENT_VERSION_MAJOR 3
#define CLIENT_VERSION_MINOR 4
#define CLIENT_VERSION_REVISION 3
#define CLIENT_VERSION_BUILD 1
// These need to be macros, as version.cpp's and triangles-qt.rc's voodoo requires it
#define CLIENT_VERSION_MAJOR 4
#define CLIENT_VERSION_MINOR 0
#define CLIENT_VERSION_REVISION 0
#define CLIENT_VERSION_BUILD 4
// Converts the parameter X to a string after macro replacement on X has been performed.
// Don't merge these into one macro!
src/compat.h
+5 -3
View File
@@ -2,8 +2,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef _BITCOIN_COMPAT_H
#define _BITCOIN_COMPAT_H 1
#ifndef _TRIANGLES_COMPAT_H
#define _TRIANGLES_COMPAT_H 1
#ifdef WIN32
#define _WIN32_WINNT 0x0501
@@ -23,9 +23,11 @@
#include <net/if.h>
#include <netinet/in.h>
#include <ifaddrs.h>
#endif
typedef u_int SOCKET;
#endif
#ifdef WIN32
#define MSG_NOSIGNAL 0
#define MSG_DONTWAIT 0
src/crypter.cpp
+16 -2
View File
@@ -11,6 +11,7 @@
#endif
#include "crypter.h"
#include "scrypt.h"
bool CCrypter::SetKeyFromPassphrase(const SecureString& strKeyData, const std::vector<unsigned char>& chSalt, const unsigned int nRounds, const unsigned int nDerivationMethod)
{
@@ -19,13 +20,26 @@ bool CCrypter::SetKeyFromPassphrase(const SecureString& strKeyData, const std::v
int i = 0;
if (nDerivationMethod == 0)
{
i = EVP_BytesToKey(EVP_aes_256_cbc(), EVP_sha512(), &chSalt[0],
(unsigned char *)&strKeyData[0], strKeyData.size(), nRounds, chKey, chIV);
}
if (nDerivationMethod == 1)
{
// Passphrase conversion
uint256 scryptHash = scrypt_salted_multiround_hash((const void*)strKeyData.c_str(), strKeyData.size(), &chSalt[0], 8, nRounds);
i = EVP_BytesToKey(EVP_aes_256_cbc(), EVP_sha512(), &chSalt[0],
(unsigned char *)&scryptHash, sizeof scryptHash, nRounds, chKey, chIV);
OPENSSL_cleanse(&scryptHash, sizeof scryptHash);
}
if (i != (int)WALLET_CRYPTO_KEY_SIZE)
{
memset(&chKey, 0, sizeof chKey);
memset(&chIV, 0, sizeof chIV);
OPENSSL_cleanse(&chKey, sizeof chKey);
OPENSSL_cleanse(&chIV, sizeof chIV);
return false;
}
src/crypter.h
+22 -2
View File
@@ -56,6 +56,26 @@ public:
nDerivationMethod = 0;
vchOtherDerivationParameters = std::vector<unsigned char>(0);
}
CMasterKey(unsigned int nDerivationMethodIndex)
{
switch (nDerivationMethodIndex)
{
case 0: // sha512
default:
nDeriveIterations = 25000;
nDerivationMethod = 0;
vchOtherDerivationParameters = std::vector<unsigned char>(0);
break;
case 1: // scrypt+sha512
nDeriveIterations = 10000;
nDerivationMethod = 1;
vchOtherDerivationParameters = std::vector<unsigned char>(0);
break;
}
}
};
typedef std::vector<unsigned char, secure_allocator<unsigned char> > CKeyingMaterial;
@@ -76,8 +96,8 @@ public:
void CleanKey()
{
memset(&chKey, 0, sizeof chKey);
memset(&chIV, 0, sizeof chIV);
OPENSSL_cleanse(&chKey, sizeof chKey);
OPENSSL_cleanse(&chIV, sizeof chIV);
fKeySet = false;
}
src/db.cpp
+30 -411
View File
@@ -5,11 +5,9 @@
#include "db.h"
#include "net.h"
#include "checkpoints.h"
#include "util.h"
#include "main.h"
#include "kernel.h"
#include <boost/version.hpp>
#include "ui_interface.h"
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
@@ -85,12 +83,18 @@ bool CDBEnv::Open(boost::filesystem::path pathEnv_)
dbenv.set_cachesize(nDbCache / 1024, (nDbCache % 1024)*1048576, 1);
dbenv.set_lg_bsize(1048576);
dbenv.set_lg_max(10485760);
dbenv.set_lk_max_locks(10000);
// Bugfix: Bump lk_max_locks default to 537000, to safely handle reorgs with up to 5 blocks reversed
// dbenv.set_lk_max_locks(10000);
dbenv.set_lk_max_locks(537000);
dbenv.set_lk_max_objects(10000);
dbenv.set_errfile(fopen(pathErrorFile.string().c_str(), "a")); /// debug
dbenv.set_flags(DB_AUTO_COMMIT, 1);
dbenv.set_flags(DB_TXN_WRITE_NOSYNC, 1);
// dbenv.log_set_config(DB_LOG_AUTO_REMOVE, 1);
#ifdef DB_LOG_AUTO_REMOVE
dbenv.log_set_config(DB_LOG_AUTO_REMOVE, 1);
#endif
int ret = dbenv.open(strPath.c_str(),
DB_CREATE |
DB_INIT_LOCK |
@@ -106,6 +110,7 @@ bool CDBEnv::Open(boost::filesystem::path pathEnv_)
fDbEnvInit = true;
fMockDb = false;
return true;
}
@@ -125,7 +130,9 @@ void CDBEnv::MakeMock()
dbenv.set_lk_max_locks(10000);
dbenv.set_lk_max_objects(10000);
dbenv.set_flags(DB_AUTO_COMMIT, 1);
// dbenv.log_set_config(DB_LOG_IN_MEMORY, 1);
#ifdef DB_LOG_IN_MEMORY
dbenv.log_set_config(DB_LOG_IN_MEMORY, 1);
#endif
int ret = dbenv.open(NULL,
DB_CREATE |
DB_INIT_LOCK |
@@ -172,9 +179,18 @@ bool CDBEnv::Salvage(std::string strFile, bool fAggressive,
Db db(&dbenv, 0);
int result = db.verify(strFile.c_str(), NULL, &strDump, flags);
if (result != 0)
if (result == DB_VERIFY_BAD)
{
printf("ERROR: db salvage failed\n");
printf("Error: Salvage found errors, all data may not be recoverable.\n");
if (!fAggressive)
{
printf("Error: Rerun with aggressive mode to ignore errors and continue.\n");
return false;
}
}
if (result != 0 && result != DB_VERIFY_BAD)
{
printf("ERROR: db salvage failed: %d\n",result);
return false;
}
@@ -423,7 +439,7 @@ bool CDB::Rewrite(const string& strFile, const char* pszSkip)
return fSuccess;
}
}
Sleep(100);
MilliSleep(100);
}
return false;
}
@@ -431,7 +447,7 @@ bool CDB::Rewrite(const string& strFile, const char* pszSkip)
void CDBEnv::Flush(bool fShutdown)
{
int64 nStart = GetTimeMillis();
int64_t nStart = GetTimeMillis();
// Flush log data to the actual data file
// on all files that are not in use
printf("Flush(%s)%s\n", fShutdown ? "true" : "false", fDbEnvInit ? "" : " db not started");
@@ -462,7 +478,7 @@ void CDBEnv::Flush(bool fShutdown)
else
mi++;
}
printf("DBFlush(%s)%s ended %15"PRI64d"ms\n", fShutdown ? "true" : "false", fDbEnvInit ? "" : " db not started", GetTimeMillis() - nStart);
printf("DBFlush(%s)%s ended %15"PRId64"ms\n", fShutdown ? "true" : "false", fDbEnvInit ? "" : " db not started", GetTimeMillis() - nStart);
if (fShutdown)
{
char** listp;
@@ -476,405 +492,6 @@ void CDBEnv::Flush(bool fShutdown)
}
//
// CTxDB
//
bool CTxDB::ReadTxIndex(uint256 hash, CTxIndex& txindex)
{
assert(!fClient);
txindex.SetNull();
return Read(make_pair(string("tx"), hash), txindex);
}
bool CTxDB::UpdateTxIndex(uint256 hash, const CTxIndex& txindex)
{
assert(!fClient);
return Write(make_pair(string("tx"), hash), txindex);
}
bool CTxDB::AddTxIndex(const CTransaction& tx, const CDiskTxPos& pos, int nHeight)
{
assert(!fClient);
// Add to tx index
uint256 hash = tx.GetHash();
CTxIndex txindex(pos, tx.vout.size());
return Write(make_pair(string("tx"), hash), txindex);
}
bool CTxDB::EraseTxIndex(const CTransaction& tx)
{
assert(!fClient);
uint256 hash = tx.GetHash();
return Erase(make_pair(string("tx"), hash));
}
bool CTxDB::ContainsTx(uint256 hash)
{
assert(!fClient);
return Exists(make_pair(string("tx"), hash));
}
bool CTxDB::ReadDiskTx(uint256 hash, CTransaction& tx, CTxIndex& txindex)
{
assert(!fClient);
tx.SetNull();
if (!ReadTxIndex(hash, txindex))
return false;
return (tx.ReadFromDisk(txindex.pos));
}
bool CTxDB::ReadDiskTx(uint256 hash, CTransaction& tx)
{
CTxIndex txindex;
return ReadDiskTx(hash, tx, txindex);
}
bool CTxDB::ReadDiskTx(COutPoint outpoint, CTransaction& tx, CTxIndex& txindex)
{
return ReadDiskTx(outpoint.hash, tx, txindex);
}
bool CTxDB::ReadDiskTx(COutPoint outpoint, CTransaction& tx)
{
CTxIndex txindex;
return ReadDiskTx(outpoint.hash, tx, txindex);
}
bool CTxDB::WriteBlockIndex(const CDiskBlockIndex& blockindex)
{
return Write(make_pair(string("blockindex"), blockindex.GetBlockHash()), blockindex);
}
bool CTxDB::ReadHashBestChain(uint256& hashBestChain)
{
return Read(string("hashBestChain"), hashBestChain);
}
bool CTxDB::WriteHashBestChain(uint256 hashBestChain)
{
return Write(string("hashBestChain"), hashBestChain);
}
bool CTxDB::ReadBestInvalidTrust(CBigNum& bnBestInvalidTrust)
{
return Read(string("bnBestInvalidTrust"), bnBestInvalidTrust);
}
bool CTxDB::WriteBestInvalidTrust(CBigNum bnBestInvalidTrust)
{
return Write(string("bnBestInvalidTrust"), bnBestInvalidTrust);
}
bool CTxDB::ReadSyncCheckpoint(uint256& hashCheckpoint)
{
return Read(string("hashSyncCheckpoint"), hashCheckpoint);
}
bool CTxDB::WriteSyncCheckpoint(uint256 hashCheckpoint)
{
return Write(string("hashSyncCheckpoint"), hashCheckpoint);
}
bool CTxDB::ReadCheckpointPubKey(string& strPubKey)
{
return Read(string("strCheckpointPubKey"), strPubKey);
}
bool CTxDB::WriteCheckpointPubKey(const string& strPubKey)
{
return Write(string("strCheckpointPubKey"), strPubKey);
}
CBlockIndex static * InsertBlockIndex(uint256 hash)
{
if (hash == 0)
return NULL;
// Return existing
map<uint256, CBlockIndex*>::iterator mi = mapBlockIndex.find(hash);
if (mi != mapBlockIndex.end())
return (*mi).second;
// Create new
CBlockIndex* pindexNew = new CBlockIndex();
if (!pindexNew)
throw runtime_error("LoadBlockIndex() : new CBlockIndex failed");
mi = mapBlockIndex.insert(make_pair(hash, pindexNew)).first;
pindexNew->phashBlock = &((*mi).first);
return pindexNew;
}
bool CTxDB::LoadBlockIndex()
{
if (!LoadBlockIndexGuts())
return false;
if (fRequestShutdown)
return true;
// Calculate bnChainTrust
vector<pair<int, CBlockIndex*> > vSortedByHeight;
vSortedByHeight.reserve(mapBlockIndex.size());
BOOST_FOREACH(const PAIRTYPE(uint256, CBlockIndex*)& item, mapBlockIndex)
{
CBlockIndex* pindex = item.second;
vSortedByHeight.push_back(make_pair(pindex->nHeight, pindex));
}
sort(vSortedByHeight.begin(), vSortedByHeight.end());
BOOST_FOREACH(const PAIRTYPE(int, CBlockIndex*)& item, vSortedByHeight)
{
CBlockIndex* pindex = item.second;
pindex->bnChainTrust = (pindex->pprev ? pindex->pprev->bnChainTrust : 0) + pindex->GetBlockTrust();
// triangles: calculate stake modifier checksum
pindex->nStakeModifierChecksum = GetStakeModifierChecksum(pindex);
if (!CheckStakeModifierCheckpoints(pindex->nHeight, pindex->nStakeModifierChecksum))
return error("CTxDB::LoadBlockIndex() : Failed stake modifier checkpoint height=%d, modifier=0x%016"PRI64x, pindex->nHeight, pindex->nStakeModifier);
}
// Load hashBestChain pointer to end of best chain
if (!ReadHashBestChain(hashBestChain))
{
if (pindexGenesisBlock == NULL)
return true;
return error("CTxDB::LoadBlockIndex() : hashBestChain not loaded");
}
if (!mapBlockIndex.count(hashBestChain))
return error("CTxDB::LoadBlockIndex() : hashBestChain not found in the block index");
pindexBest = mapBlockIndex[hashBestChain];
nBestHeight = pindexBest->nHeight;
bnBestChainTrust = pindexBest->bnChainTrust;
printf("LoadBlockIndex(): hashBestChain=%s height=%d trust=%s date=%s\n",
hashBestChain.ToString().substr(0,20).c_str(), nBestHeight, bnBestChainTrust.ToString().c_str(),
DateTimeStrFormat("%x %H:%M:%S", pindexBest->GetBlockTime()).c_str());
// triangles: load hashSyncCheckpoint
if (!ReadSyncCheckpoint(Checkpoints::hashSyncCheckpoint))
return error("CTxDB::LoadBlockIndex() : hashSyncCheckpoint not loaded");
printf("LoadBlockIndex(): synchronized checkpoint %s\n", Checkpoints::hashSyncCheckpoint.ToString().c_str());
// Load bnBestInvalidTrust, OK if it doesn't exist
ReadBestInvalidTrust(bnBestInvalidTrust);
// Verify blocks in the best chain
int nCheckLevel = GetArg("-checklevel", 1);
int nCheckDepth = GetArg( "-checkblocks", 2500);
if (nCheckDepth == 0)
nCheckDepth = 1000000000; // suffices until the year 19000
if (nCheckDepth > nBestHeight)
nCheckDepth = nBestHeight;
printf("Verifying last %i blocks at level %i\n", nCheckDepth, nCheckLevel);
CBlockIndex* pindexFork = NULL;
map<pair<unsigned int, unsigned int>, CBlockIndex*> mapBlockPos;
for (CBlockIndex* pindex = pindexBest; pindex && pindex->pprev; pindex = pindex->pprev)
{
if (fRequestShutdown || pindex->nHeight < nBestHeight-nCheckDepth)
break;
CBlock block;
if (!block.ReadFromDisk(pindex))
return error("LoadBlockIndex() : block.ReadFromDisk failed");
// check level 1: verify block validity
if (nCheckLevel>0 && !block.CheckBlock())
{
printf("LoadBlockIndex() : *** found bad block at %d, hash=%s\n", pindex->nHeight, pindex->GetBlockHash().ToString().c_str());
pindexFork = pindex->pprev;
}
// check level 2: verify transaction index validity
if (nCheckLevel>1)
{
pair<unsigned int, unsigned int> pos = make_pair(pindex->nFile, pindex->nBlockPos);
mapBlockPos[pos] = pindex;
BOOST_FOREACH(const CTransaction &tx, block.vtx)
{
uint256 hashTx = tx.GetHash();
CTxIndex txindex;
if (ReadTxIndex(hashTx, txindex))
{
// check level 3: checker transaction hashes
if (nCheckLevel>2 || pindex->nFile != txindex.pos.nFile || pindex->nBlockPos != txindex.pos.nBlockPos)
{
// either an error or a duplicate transaction
CTransaction txFound;
if (!txFound.ReadFromDisk(txindex.pos))
{
printf("LoadBlockIndex() : *** cannot read mislocated transaction %s\n", hashTx.ToString().c_str());
pindexFork = pindex->pprev;
}
else
if (txFound.GetHash() != hashTx) // not a duplicate tx
{
printf("LoadBlockIndex(): *** invalid tx position for %s\n", hashTx.ToString().c_str());
pindexFork = pindex->pprev;
}
}
// check level 4: check whether spent txouts were spent within the main chain
unsigned int nOutput = 0;
if (nCheckLevel>3)
{
BOOST_FOREACH(const CDiskTxPos &txpos, txindex.vSpent)
{
if (!txpos.IsNull())
{
pair<unsigned int, unsigned int> posFind = make_pair(txpos.nFile, txpos.nBlockPos);
if (!mapBlockPos.count(posFind))
{
printf("LoadBlockIndex(): *** found bad spend at %d, hashBlock=%s, hashTx=%s\n", pindex->nHeight, pindex->GetBlockHash().ToString().c_str(), hashTx.ToString().c_str());
pindexFork = pindex->pprev;
}
// check level 6: check whether spent txouts were spent by a valid transaction that consume them
if (nCheckLevel>5)
{
CTransaction txSpend;
if (!txSpend.ReadFromDisk(txpos))
{
printf("LoadBlockIndex(): *** cannot read spending transaction of %s:%i from disk\n", hashTx.ToString().c_str(), nOutput);
pindexFork = pindex->pprev;
}
else if (!txSpend.CheckTransaction())
{
printf("LoadBlockIndex(): *** spending transaction of %s:%i is invalid\n", hashTx.ToString().c_str(), nOutput);
pindexFork = pindex->pprev;
}
else
{
bool fFound = false;
BOOST_FOREACH(const CTxIn &txin, txSpend.vin)
if (txin.prevout.hash == hashTx && txin.prevout.n == nOutput)
fFound = true;
if (!fFound)
{
printf("LoadBlockIndex(): *** spending transaction of %s:%i does not spend it\n", hashTx.ToString().c_str(), nOutput);
pindexFork = pindex->pprev;
}
}
}
}
nOutput++;
}
}
}
// check level 5: check whether all prevouts are marked spent
if (nCheckLevel>4)
{
BOOST_FOREACH(const CTxIn &txin, tx.vin)
{
CTxIndex txindex;
if (ReadTxIndex(txin.prevout.hash, txindex))
if (txindex.vSpent.size()-1 < txin.prevout.n || txindex.vSpent[txin.prevout.n].IsNull())
{
printf("LoadBlockIndex(): *** found unspent prevout %s:%i in %s\n", txin.prevout.hash.ToString().c_str(), txin.prevout.n, hashTx.ToString().c_str());
pindexFork = pindex->pprev;
}
}
}
}
}
}
if (pindexFork && !fRequestShutdown)
{
// Reorg back to the fork
printf("LoadBlockIndex() : *** moving best chain pointer back to block %d\n", pindexFork->nHeight);
CBlock block;
if (!block.ReadFromDisk(pindexFork))
return error("LoadBlockIndex() : block.ReadFromDisk failed");
CTxDB txdb;
block.SetBestChain(txdb, pindexFork);
}
return true;
}
bool CTxDB::LoadBlockIndexGuts()
{
// Get database cursor
Dbc* pcursor = GetCursor();
if (!pcursor)
return false;
// Load mapBlockIndex
unsigned int fFlags = DB_SET_RANGE;
loop
{
// Read next record
CDataStream ssKey(SER_DISK, CLIENT_VERSION);
if (fFlags == DB_SET_RANGE)
ssKey << make_pair(string("blockindex"), uint256(0));
CDataStream ssValue(SER_DISK, CLIENT_VERSION);
int ret = ReadAtCursor(pcursor, ssKey, ssValue, fFlags);
fFlags = DB_NEXT;
if (ret == DB_NOTFOUND)
break;
else if (ret != 0)
return false;
// Unserialize
try {
string strType;
ssKey >> strType;
if (strType == "blockindex" && !fRequestShutdown)
{
CDiskBlockIndex diskindex;
ssValue >> diskindex;
// Construct block index object
CBlockIndex* pindexNew = InsertBlockIndex(diskindex.GetBlockHash());
pindexNew->pprev = InsertBlockIndex(diskindex.hashPrev);
pindexNew->pnext = InsertBlockIndex(diskindex.hashNext);
pindexNew->nFile = diskindex.nFile;
pindexNew->nBlockPos = diskindex.nBlockPos;
pindexNew->nHeight = diskindex.nHeight;
pindexNew->nMint = diskindex.nMint;
pindexNew->nMoneySupply = diskindex.nMoneySupply;
pindexNew->nFlags = diskindex.nFlags;
pindexNew->nStakeModifier = diskindex.nStakeModifier;
pindexNew->prevoutStake = diskindex.prevoutStake;
pindexNew->nStakeTime = diskindex.nStakeTime;
pindexNew->hashProofOfStake = diskindex.hashProofOfStake;
pindexNew->nVersion = diskindex.nVersion;
pindexNew->hashMerkleRoot = diskindex.hashMerkleRoot;
pindexNew->nTime = diskindex.nTime;
pindexNew->nBits = diskindex.nBits;
pindexNew->nNonce = diskindex.nNonce;
// Watch for genesis block
if (pindexGenesisBlock == NULL && diskindex.GetBlockHash() == (!fTestNet ? hashGenesisBlock : hashGenesisBlockTestNet))
pindexGenesisBlock = pindexNew;
if (!pindexNew->CheckIndex())
return error("LoadBlockIndex() : CheckIndex failed at %d", pindexNew->nHeight);
// triangles: build setStakeSeen
if (pindexNew->IsProofOfStake())
setStakeSeen.insert(make_pair(pindexNew->prevoutStake, pindexNew->nStakeTime));
}
else
{
break; // if shutdown requested or finished loading block index
}
} // try
catch (std::exception &e) {
return error("%s() : deserialize error", __PRETTY_FUNCTION__);
}
}
pcursor->close();
return true;
}
//
// CAddrDB
//
@@ -932,8 +549,10 @@ bool CAddrDB::Read(CAddrMan& addr)
return error("CAddrman::Read() : open failed");
// use file size to size memory buffer
int fileSize = GetFilesize(filein);
int fileSize = boost::filesystem::file_size(pathAddr);
int dataSize = fileSize - sizeof(uint256);
// Don't try to resize to a negative number if file is small
if ( dataSize < 0 ) dataSize = 0;
vector<unsigned char> vchData;
vchData.resize(dataSize);
uint256 hashIn;
src/db.h
+6 -44
View File
@@ -2,8 +2,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_DB_H
#define BITCOIN_DB_H
#ifndef TRIANGLES_DB_H
#define TRIANGLES_DB_H
#include "main.h"
@@ -11,7 +11,7 @@
#include <string>
#include <vector>
#include <db/db_cxx.h>
#include <db_cxx.h>
class CAddress;
class CAddrMan;
@@ -312,46 +312,6 @@ public:
};
/** Access to the transaction database (blkindex.dat) */
class CTxDB : public CDB
{
public:
CTxDB(const char* pszMode="r+") : CDB("blkindex.dat", pszMode) { }
private:
CTxDB(const CTxDB&);
void operator=(const CTxDB&);
public:
bool ReadTxIndex(uint256 hash, CTxIndex& txindex);
bool UpdateTxIndex(uint256 hash, const CTxIndex& txindex);
bool AddTxIndex(const CTransaction& tx, const CDiskTxPos& pos, int nHeight);
bool EraseTxIndex(const CTransaction& tx);
bool ContainsTx(uint256 hash);
bool ReadDiskTx(uint256 hash, CTransaction& tx, CTxIndex& txindex);
bool ReadDiskTx(uint256 hash, CTransaction& tx);
bool ReadDiskTx(COutPoint outpoint, CTransaction& tx, CTxIndex& txindex);
bool ReadDiskTx(COutPoint outpoint, CTransaction& tx);
bool WriteBlockIndex(const CDiskBlockIndex& blockindex);
bool ReadHashBestChain(uint256& hashBestChain);
bool WriteHashBestChain(uint256 hashBestChain);
bool ReadBestInvalidTrust(CBigNum& bnBestInvalidTrust);
bool WriteBestInvalidTrust(CBigNum bnBestInvalidTrust);
bool ReadSyncCheckpoint(uint256& hashCheckpoint);
bool WriteSyncCheckpoint(uint256 hashCheckpoint);
bool ReadCheckpointPubKey(std::string& strPubKey);
bool WriteCheckpointPubKey(const std::string& strPubKey);
bool LoadBlockIndex();
private:
bool LoadBlockIndexGuts();
};
/** Access to the (IP) address database (peers.dat) */
class CAddrDB
{
@@ -363,4 +323,6 @@ public:
bool Read(CAddrMan& addr);
};
#endif // BITCOIN_DB_H
#endif // TRIANGLES_DB_H
src/init.cpp
+203 -90
View File
@@ -2,14 +2,15 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#include "db.h"
#include "txdb.h"
#include "walletdb.h"
#include "bitcoinrpc.h"
#include "trianglesrpc.h"
#include "net.h"
#include "init.h"
#include "util.h"
#include "ui_interface.h"
#include "checkpoints.h"
#include "smessage.h"
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/filesystem/convenience.hpp>
@@ -21,11 +22,20 @@
#include <signal.h>
#endif
using namespace std;
using namespace boost;
CWallet* pwalletMain;
CClientUIInterface uiInterface;
std::string strWalletFileName;
bool fConfChange;
bool fEnforceCanonical;
unsigned int nNodeLifespan;
unsigned int nDerivationMethodIndex;
//unsigned int nMinerSleep;
bool fUseFastIndex;
enum Checkpoints::CPMode CheckpointsMode;
//////////////////////////////////////////////////////////////////////////////
//
@@ -35,7 +45,7 @@ CClientUIInterface uiInterface;
void ExitTimeout(void* parg)
{
#ifdef WIN32
Sleep(5000);
MilliSleep(5000);
ExitProcess(0);
#endif
}
@@ -43,7 +53,7 @@ void ExitTimeout(void* parg)
void StartShutdown()
{
#ifdef QT_GUI
// ensure we leave the Qt main loop for a clean GUI exit (Shutdown() is called in bitcoin.cpp afterwards)
// ensure we leave the Qt main loop for a clean GUI exit (Shutdown() is called in triangles.cpp afterwards)
uiInterface.QueueShutdown();
#else
// Without UI, Shutdown() can simply be started in a new thread
@@ -57,7 +67,7 @@ void Shutdown(void* parg)
static bool fTaken;
// Make this thread recognisable as the shutdown thread
RenameThread("bitcoin-shutoff");
RenameThread("Triangles-shutoff");
bool fFirstThread = false;
{
@@ -72,7 +82,11 @@ void Shutdown(void* parg)
if (fFirstThread)
{
fShutdown = true;
SecureMsgShutdown();
nTransactionsUpdated++;
// CTxDB().Close();
bitdb.Flush(false);
StopNode();
bitdb.Flush(true);
@@ -80,19 +94,19 @@ void Shutdown(void* parg)
UnregisterWallet(pwalletMain);
delete pwalletMain;
NewThread(ExitTimeout, NULL);
Sleep(50);
printf("triangles exited\n\n");
MilliSleep(50);
printf("Triangles exited\n\n");
fExit = true;
#ifndef QT_GUI
// ensure non-UI client gets exited here, but let Bitcoin-Qt reach 'return 0;' in bitcoin.cpp
// ensure non-UI client gets exited here, but let Triangles-Qt reach 'return 0;' in triangles.cpp
exit(0);
#endif
}
else
{
while (!fExit)
Sleep(500);
Sleep(100);
MilliSleep(500);
MilliSleep(100);
ExitThread(0);
}
}
@@ -124,7 +138,7 @@ bool AppInit(int argc, char* argv[])
//
// Parameters
//
// If Qt is used, parameters/bitcoin.conf are parsed in qt/bitcoin.cpp's main()
// If Qt is used, parameters/triangles.conf are parsed in qt/triangles.cpp's main()
ParseParameters(argc, argv);
if (!boost::filesystem::is_directory(GetDataDir(false)))
{
@@ -135,8 +149,8 @@ bool AppInit(int argc, char* argv[])
if (mapArgs.count("-?") || mapArgs.count("--help"))
{
// First part of help message is specific to bitcoind / RPC client
std::string strUsage = _("triangles version") + " " + FormatFullVersion() + "\n\n" +
// First part of help message is specific to trianglesd / RPC client
std::string strUsage = _("Triangles version") + " " + FormatFullVersion() + "\n\n" +
_("Usage:") + "\n" +
" trianglesd [options] " + "\n" +
" trianglesd [options] <command> [params] " + _("Send command to -server or trianglesd") + "\n" +
@@ -151,7 +165,7 @@ bool AppInit(int argc, char* argv[])
// Command-line RPC
for (int i = 1; i < argc; i++)
if (!IsSwitchChar(argv[i][0]) && !boost::algorithm::istarts_with(argv[i], "triangles:"))
if (!IsSwitchChar(argv[i][0]) && !boost::algorithm::istarts_with(argv[i], "Triangles:"))
fCommandLine = true;
if (fCommandLine)
@@ -177,7 +191,7 @@ int main(int argc, char* argv[])
{
bool fRet = false;
// Connect bitcoind signal handlers
// Connect trianglesd signal handlers
noui_connect();
fRet = AppInit(argc, argv);
@@ -191,13 +205,13 @@ int main(int argc, char* argv[])
bool static InitError(const std::string &str)
{
uiInterface.ThreadSafeMessageBox(str, _("triangles"), CClientUIInterface::OK | CClientUIInterface::MODAL);
uiInterface.ThreadSafeMessageBox(str, _("Triangles"), CClientUIInterface::OK | CClientUIInterface::MODAL);
return false;
}
bool static InitWarning(const std::string &str)
{
uiInterface.ThreadSafeMessageBox(str, _("triangles"), CClientUIInterface::OK | CClientUIInterface::ICON_EXCLAMATION | CClientUIInterface::MODAL);
uiInterface.ThreadSafeMessageBox(str, _("Triangles"), CClientUIInterface::OK | CClientUIInterface::ICON_EXCLAMATION | CClientUIInterface::MODAL);
return true;
}
@@ -221,25 +235,31 @@ std::string HelpMessage()
" -? " + _("This help message") + "\n" +
" -conf=<file> " + _("Specify configuration file (default: triangles.conf)") + "\n" +
" -pid=<file> " + _("Specify pid file (default: trianglesd.pid)") + "\n" +
" -gen " + _("Generate coins") + "\n" +
" -gen=0 " + _("Don't generate coins") + "\n" +
" -stake " + _("Stake coins") + "\n" +
" -stake=0 " + _("Turn off staking") + "\n" +
" -datadir=<dir> " + _("Specify data directory") + "\n" +
" -wallet=<dir> " + _("Specify wallet file (within data directory)") + "\n" +
" -dbcache=<n> " + _("Set database cache size in megabytes (default: 25)") + "\n" +
" -dblogsize=<n> " + _("Set database disk log size in megabytes (default: 100)") + "\n" +
" -timeout=<n> " + _("Specify connection timeout in milliseconds (default: 5000)") + "\n" +
" -socks=<n> " + _("Select the version of socks proxy to use (4-5, default: 5)") + "\n" +
" -tor=<ip:port> " + _("Use proxy to reach tor hidden services") + "\n"
" -dns " + _("Allow DNS lookups for -addnode, -seednode and -connect") + "\n" +
" -port=<port> " + _("Listen for connections on <port> (default: 24242 or testnet: 24495)") + "\n" +
//" -proxy=<ip:port> " + _("Connect through socks proxy") + "\n" +
//" -socks=<n> " + _("Select the version of socks proxy to use (4-5, default: 5)") + "\n" +
" -tor=<ip:port> " + _("Use proxy to reach tor hidden services (default: same as -proxy)") + "\n"
//" -dns " + _("Allow DNS lookups for -addnode, -seednode and -connect") + "\n" +
" -port=<port> " + _("Listen for connections on <port> (default: 24112 or testnet: 24111)") + "\n" +
" -maxconnections=<n> " + _("Maintain at most <n> connections to peers (default: 125)") + "\n" +
" -addnode=<ip> " + _("Add a node to connect to and attempt to keep the connection open") + "\n" +
" -connect=<ip> " + _("Connect only to the specified node(s)") + "\n" +
" -seednode=<ip> " + _("Connect to a node to retrieve peer addresses, and disconnect") + "\n" +
" -externalip=<ip> " + _("Specify your own public address") + "\n" +
//" -onlynet=<net> " + _("Only connect to nodes in network <net> (IPv4, IPv6 or Tor)") + "\n" +
//" -discover " + _("Discover own IP address (default: 1 when listening and no -externalip)") + "\n" +
//" -irc " + _("Find peers using internet relay chat (default: 0)") + "\n" +
//" -listen " + _("Accept connections from outside (default: 1 if no -proxy or -connect)") + "\n" +
//" -bind=<addr> " + _("Bind to given address. Use [host]:port notation for IPv6") + "\n" +
// -dnsseed " + _("Find peers using DNS lookup (default: 1)") + "\n" +
" -staking " + _("Stake your coins to support network and gain reward (default: 1)") + "\n" +
" -synctime " + _("Sync time with other nodes. Disable if time on your system is precise e.g. syncing with NTP (default: 1)") + "\n" +
" -cppolicy " + _("Sync checkpoints policy (default: strict)") + "\n" +
" -onionseed " + _("Find peers using .onion seeds (default: 1 unless -connect)") + "\n" +
" -nosynccheckpoints " + _("Disable sync checkpoints (default: 0)") + "\n" +
" -banscore=<n> " + _("Threshold for disconnecting misbehaving peers (default: 100)") + "\n" +
" -bantime=<n> " + _("Number of seconds to keep misbehaving peers from reconnecting (default: 86400)") + "\n" +
" -maxreceivebuffer=<n> " + _("Maximum per-connection receive buffer, <n>*1000 bytes (default: 5000)") + "\n" +
@@ -253,6 +273,7 @@ std::string HelpMessage()
#endif
" -detachdb " + _("Detach block and address databases. Increases shutdown time (default: 0)") + "\n" +
" -paytxfee=<amt> " + _("Fee per KB to add to transactions you send") + "\n" +
//" -mininput=<amt> " + _("When creating transactions, ignore inputs with value less than this (default: 0.01)") + "\n" +
#ifdef QT_GUI
" -server " + _("Accept command line and JSON-RPC commands") + "\n" +
#endif
@@ -270,11 +291,14 @@ std::string HelpMessage()
#endif
" -rpcuser=<user> " + _("Username for JSON-RPC connections") + "\n" +
" -rpcpassword=<pw> " + _("Password for JSON-RPC connections") + "\n" +
" -rpcport=<port> " + _("Listen for JSON-RPC connections on <port> (default: 24444 or testnet: 24445)") + "\n" +
" -rpcport=<port> " + _("Listen for JSON-RPC connections on <port> (default: 19111 or testnet: 19112)") + "\n" +
" -rpcallowip=<ip> " + _("Allow JSON-RPC connections from specified IP address") + "\n" +
" -rpcconnect=<ip> " + _("Send commands to node running on <ip> (default: 127.0.0.1)") + "\n" +
" -blocknotify=<cmd> " + _("Execute command when the best block changes (%s in cmd is replaced by block hash)") + "\n" +
" -walletnotify=<cmd> " + _("Execute command when a wallet transaction changes (%s in cmd is replaced by TxID)") + "\n" +
" -walletnotify=<cmd> " + _("Execute command when a wallet transaction changes (%s in cmd is replaced by TxID)") + "\n" +
" -confchange " + _("Require a confirmations for change (default: 0)") + "\n" +
" -enforcecanonical " + _("Enforce transaction scripts to use canonical PUSH operators (default: 1)") + "\n" +
" -alertnotify=<cmd> " + _("Execute command when a relevant alert is received (%s in cmd is replaced by message)") + "\n" +
" -upgradewallet " + _("Upgrade wallet to latest format") + "\n" +
" -keypool=<n> " + _("Set key pool size to <n> (default: 100)") + "\n" +
" -rescan " + _("Rescan the block chain for missing wallet transactions") + "\n" +
@@ -288,16 +312,38 @@ std::string HelpMessage()
" -blockmaxsize=<n> " + _("Set maximum block size in bytes (default: 250000)") + "\n" +
" -blockprioritysize=<n> " + _("Set maximum size of high-priority/low-fee transactions in bytes (default: 27000)") + "\n" +
"\n" + _("SSL options: (see the Bitcoin Wiki for SSL setup instructions)") + "\n" +
"\n" + _("SSL options: (see the Triangles Wiki for SSL setup instructions)") + "\n" +
" -rpcssl " + _("Use OpenSSL (https) for JSON-RPC connections") + "\n" +
" -rpcsslcertificatechainfile=<file.cert> " + _("Server certificate file (default: server.cert)") + "\n" +
" -rpcsslprivatekeyfile=<file.pem> " + _("Server private key (default: server.pem)") + "\n" +
" -rpcsslciphers=<ciphers> " + _("Acceptable ciphers (default: TLSv1+HIGH:!SSLv2:!aNULL:!eNULL:!AH:!3DES:@STRENGTH)") + "\n";
" -rpcsslciphers=<ciphers> " + _("Acceptable ciphers (default: TLSv1+HIGH:!SSLv2:!aNULL:!eNULL:!AH:!3DES:@STRENGTH)") + "\n" +
"\n" + _("Secure messaging options:") + "\n" +
" -nosmsg " + _("Disable secure messaging.") + "\n" +
" -debugsmsg " + _("Log extra debug messages.") + "\n" +
" -smsgscanchain " + _("Scan the block chain for public key addresses on startup.") + "\n";
return strUsage;
}
/** Initialize bitcoin.
/** Sanity checks
* Ensure that Triangles is running in a usable environment with all
* necessary library support.
*/
bool InitSanityCheck(void)
{
if(!ECC_InitSanityCheck()) {
InitError("OpenSSL appears to lack support for elliptic curve cryptography. For more "
"information, visit https://en.bitcoin.it/wiki/OpenSSL_and_EC_Libraries");
return false;
}
// TODO: remaining sanity checks, see #4081
return true;
}
/** Initialize Triangles.
* @pre Parameters should be parsed and config file should be read.
*/
bool AppInit2()
@@ -346,16 +392,58 @@ bool AppInit2()
// ********************************************************* Step 2: parameter interactions
nNodeLifespan = GetArg("-addrlifespan", 7);
fUseFastIndex = GetBoolArg("-fastindex", true);
//nMinerSleep = GetArg("-minersleep", 500);
CheckpointsMode = Checkpoints::STRICT;
std::string strCpMode = GetArg("-cppolicy", "strict");
if(strCpMode == "strict")
CheckpointsMode = Checkpoints::STRICT;
if(strCpMode == "advisory")
CheckpointsMode = Checkpoints::ADVISORY;
if(strCpMode == "permissive")
CheckpointsMode = Checkpoints::PERMISSIVE;
nDerivationMethodIndex = 0;
fTestNet = GetBoolArg("-testnet");
if (fTestNet) {
SoftSetBoolArg("-irc", true);
}
if (mapArgs.count("-bind")) {
// when specifying an explicit binding address, you want to listen on it
// even when -connect or -proxy is specified
SoftSetBoolArg("-listen", true);
}
if (mapArgs.count("-connect") && mapMultiArgs["-connect"].size() > 0) {
// when only connecting to trusted nodes, do not seed via .onion, or listen by default
SoftSetBoolArg("-onionseed", false);
SoftSetBoolArg("-listen", false);
}
if (mapArgs.count("-proxy")) {
// to protect privacy, do not listen by default if a proxy server is specified
SoftSetBoolArg("-listen", false);
}
//if (!GetBoolArg("-listen", true)) {
// do not map ports or try to retrieve public IP when not listening (pointless)
//triangles: never listen, always using tor.
//SoftSetBoolArg("-upnp", false);
//SoftSetBoolArg("-discover", false);
//}
//if (mapArgs.count("-externalip")) {
// if an explicit public IP is specified, do not try to find others
// SoftSetBoolArg("-discover", false);
//}
if (GetBoolArg("-salvagewallet")) {
// Rewrite just private keys: rescan to find transactions
SoftSetBoolArg("-rescan", true);
@@ -367,12 +455,16 @@ bool AppInit2()
// -debug implies fDebug*
if (fDebug)
fDebugNet = true;
else
fDebugNet = GetBoolArg("-debugnet");
//fStaking = GetBoolArg("-staking", true);
{
fDebugNet = true;
fDebugSmsg = true;
} else
{
fDebugNet = GetBoolArg("-debugnet");
fDebugSmsg = GetBoolArg("-debugsmsg");
}
fNoSmsg = GetBoolArg("-nosmsg");
bitdb.SetDetach(GetBoolArg("-detachdb", false));
#if !defined(WIN32) && !defined(QT_GUI)
@@ -401,13 +493,6 @@ bool AppInit2()
nConnectTimeout = nNewTimeout;
}
// Continue to put "/P2SH/" in the coinbase to monitor
// BIP16 support.
// This can be removed eventually...
const char* pszP2SH = "/P2SH/";
COINBASE_FLAGS << std::vector<unsigned char>(pszP2SH, pszP2SH+strlen(pszP2SH));
if (mapArgs.count("-paytxfee"))
{
if (!ParseMoney(mapArgs["-paytxfee"], nTransactionFee))
@@ -416,17 +501,34 @@ bool AppInit2()
InitWarning(_("Warning: -paytxfee is set very high! This is the transaction fee you will pay if you send a transaction."));
}
fConfChange = GetBoolArg("-confchange", false);
fEnforceCanonical = GetBoolArg("-enforcecanonical", true);
if (mapArgs.count("-mininput"))
{
if (!ParseMoney(mapArgs["-mininput"], nMinimumInputValue))
return InitError(strprintf(_("Invalid amount for -mininput=<amount>: '%s'"), mapArgs["-mininput"].c_str()));
}
// ********************************************************* Step 4: application initialization: dir lock, daemonize, pidfile, debug log
// Sanity check
if (!InitSanityCheck())
return InitError(_("Initialization sanity check failed. Triangles is shutting down."));
std::string strDataDir = GetDataDir().string();
std::string strWalletFileName = GetArg("-wallet", "wallet.dat");
// Make sure only a single Bitcoin process is using the data directory.
// strWalletFileName must be a plain filename without a directory
if (strWalletFileName != boost::filesystem::basename(strWalletFileName) + boost::filesystem::extension(strWalletFileName))
return InitError(strprintf(_("Wallet %s resides outside data directory %s."), strWalletFileName.c_str(), strDataDir.c_str()));
// Make sure only a single Triangles process is using the data directory.
boost::filesystem::path pathLockFile = GetDataDir() / ".lock";
FILE* file = fopen(pathLockFile.string().c_str(), "a"); // empty lock file; created if it doesn't exist.
if (file) fclose(file);
static boost::interprocess::file_lock lock(pathLockFile.string().c_str());
if (!lock.try_lock())
return InitError(strprintf(_("Cannot obtain a lock on data directory %s. triangles is probably already running."), strDataDir.c_str()));
return InitError(strprintf(_("Cannot obtain a lock on data directory %s. Triangles is probably already running."), strDataDir.c_str()));
#if !defined(WIN32) && !defined(QT_GUI)
if (fDaemon)
@@ -453,7 +555,7 @@ bool AppInit2()
if (GetBoolArg("-shrinkdebugfile", !fDebug))
ShrinkDebugFile();
printf("\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n");
printf("triangles version %s (%s)\n", FormatFullVersion().c_str(), CLIENT_DATE.c_str());
printf("Triangles version %s (%s)\n", FormatFullVersion().c_str(), CLIENT_DATE.c_str());
printf("Using OpenSSL version %s\n", SSLeay_version(SSLEAY_VERSION));
if (!fLogTimestamps)
printf("Startup time: %s\n", DateTimeStrFormat("%x %H:%M:%S", GetTime()).c_str());
@@ -462,9 +564,9 @@ bool AppInit2()
std::ostringstream strErrors;
if (fDaemon)
fprintf(stdout, "triangles server starting\n");
fprintf(stdout, "Triangles server starting\n");
int64 nStart;
int64_t nStart;
// ********************************************************* Step 5: verify database integrity
@@ -481,20 +583,20 @@ bool AppInit2()
if (GetBoolArg("-salvagewallet"))
{
// Recover readable keypairs:
if (!CWalletDB::Recover(bitdb, "wallet.dat", true))
if (!CWalletDB::Recover(bitdb, strWalletFileName, true))
return false;
}
if (filesystem::exists(GetDataDir() / "wallet.dat"))
if (filesystem::exists(GetDataDir() / strWalletFileName))
{
CDBEnv::VerifyResult r = bitdb.Verify("wallet.dat", CWalletDB::Recover);
CDBEnv::VerifyResult r = bitdb.Verify(strWalletFileName, CWalletDB::Recover);
if (r == CDBEnv::RECOVER_OK)
{
string msg = strprintf(_("Warning: wallet.dat corrupt, data salvaged!"
" Original wallet.dat saved as wallet.{timestamp}.bak in %s; if"
" your balance or transactions are incorrect you should"
" restore from a backup."), strDataDir.c_str());
uiInterface.ThreadSafeMessageBox(msg, _("triangles"), CClientUIInterface::OK | CClientUIInterface::ICON_EXCLAMATION | CClientUIInterface::MODAL);
uiInterface.ThreadSafeMessageBox(msg, _("Triangles"), CClientUIInterface::OK | CClientUIInterface::ICON_EXCLAMATION | CClientUIInterface::MODAL);
}
if (r == CDBEnv::RECOVER_FAIL)
return InitError(_("wallet.dat corrupt, salvage failed"));
@@ -502,26 +604,28 @@ bool AppInit2()
// ********************************************************* Step 6: network initialization
int nSocksVersion = GetArg("-socks", 5);
if (nSocksVersion != 4 && nSocksVersion != 5)
return InitError(strprintf(_("Unknown -socks proxy version requested: %i"), nSocksVersion));
//int nSocksVersion = GetArg("-socks", 5);
//
//if (nSocksVersion != 4 && nSocksVersion != 5)
// return InitError(strprintf(_("Unknown -socks proxy version requested: %i"), nSocksVersion));
do {
std::set<enum Network> nets;
nets.insert(
NET_TOR
);
nets.insert(NET_TOR);
for (int n = 0; n < NET_MAX; n++) {
enum Network net = (enum Network)n;
if (!nets.count(net))
SetLimited(net);
}
} while (
false
);
} while (false);
CService addrOnion;
CService addrOnion;
// need to move onion_port to a header
unsigned short const onion_port = 19099;
@@ -539,8 +643,12 @@ bool AppInit2()
}
// see Step 2: parameter interactions for more information about these
fNameLookup = GetBoolArg("-dns", true);
fNoListen = !GetBoolArg("-listen", true);
//fDiscover = GetBoolArg("-discover", true);
//fNameLookup = GetBoolArg("-dns", true);
#ifdef USE_UPNP
fUseUPnP = GetBoolArg("-upnp", USE_UPNP);
#endif
bool fBound = false;
if (true) {
if (true) {
@@ -549,9 +657,7 @@ bool AppInit2()
if (!Lookup("127.0.0.1", addrBind, GetListenPort(), false))
return InitError(strprintf(_("Cannot resolve binding address: '%s'"), "127.0.0.1"));
fBound |= Bind(addrBind);
} while (
false
);
} while (false);
}
if (!fBound)
return InitError(_("Failed to listen on any port."));
@@ -575,7 +681,7 @@ bool AppInit2()
return InitError(strprintf(_("Cannot resolve -externalip address: '%s'"), strAddr.c_str()));
AddLocal(CService(strAddr, GetListenPort(), fNameLookup), LOCAL_MANUAL);
}
} else {
} else {
string automatic_onion;
filesystem::path const hostname_path = GetDataDir(
) / "onion" / "hostname";
@@ -595,13 +701,8 @@ bool AppInit2()
AddLocal(CService(automatic_onion, GetListenPort(), fNameLookup), LOCAL_MANUAL);
}
BOOST_FOREACH(string strDest, mapMultiArgs["-seednode"])
AddOneShot(strDest);
if (mapArgs.count("-reservebalance")) // triangles: reserve balance amount
{
int64 nReserveBalance = 0;
if (!ParseMoney(mapArgs["-reservebalance"], nReserveBalance))
{
InitError(_("Invalid amount for -reservebalance=<amount>"));
@@ -609,14 +710,14 @@ bool AppInit2()
}
}
if (mapArgs.count("-checkpointkey")) // triangles: checkpoint master priv key
if (mapArgs.count("-checkpointkey")) // triangles: checkpoint master priv key
{
if (!Checkpoints::SetCheckpointPrivKey(GetArg("-checkpointkey", "")))
InitError(_("Unable to sign checkpoint, wrong checkpointkey?\n"));
}
//AddOneShot(string(""));
BOOST_FOREACH(string strDest, mapMultiArgs["-seednode"])
AddOneShot(strDest);
// ********************************************************* Step 7: load blockchain
@@ -642,15 +743,16 @@ bool AppInit2()
if (!LoadBlockIndex())
return InitError(_("Error loading blkindex.dat"));
// as LoadBlockIndex can take several minutes, it's possible the user
// requested to kill bitcoin-qt during the last operation. If so, exit.
// requested to kill triangles-qt during the last operation. If so, exit.
// As the program has not fully started yet, Shutdown() is possibly overkill.
if (fRequestShutdown)
{
printf("Shutdown requested. Exiting.\n");
return false;
}
printf(" block index %15"PRI64d"ms\n", GetTimeMillis() - nStart);
printf(" block index %15"PRId64"ms\n", GetTimeMillis() - nStart);
if (GetBoolArg("-printblockindex") || GetBoolArg("-printblocktree"))
{
@@ -681,13 +783,16 @@ bool AppInit2()
return false;
}
// ********************************************************* Testing Zerocoin
// ********************************************************* Step 8: load wallet
uiInterface.InitMessage(_("Loading wallet..."));
printf("Loading wallet...\n");
nStart = GetTimeMillis();
bool fFirstRun = true;
pwalletMain = new CWallet("wallet.dat");
pwalletMain = new CWallet(strWalletFileName);
DBErrors nLoadWalletRet = pwalletMain->LoadWallet(fFirstRun);
if (nLoadWalletRet != DB_LOAD_OK)
{
@@ -697,13 +802,13 @@ bool AppInit2()
{
string msg(_("Warning: error reading wallet.dat! All keys read correctly, but transaction data"
" or address book entries might be missing or incorrect."));
uiInterface.ThreadSafeMessageBox(msg, _("triangles"), CClientUIInterface::OK | CClientUIInterface::ICON_EXCLAMATION | CClientUIInterface::MODAL);
uiInterface.ThreadSafeMessageBox(msg, _("Triangles"), CClientUIInterface::OK | CClientUIInterface::ICON_EXCLAMATION | CClientUIInterface::MODAL);
}
else if (nLoadWalletRet == DB_TOO_NEW)
strErrors << _("Error loading wallet.dat: Wallet requires newer version of triangles") << "\n";
strErrors << _("Error loading wallet.dat: Wallet requires newer version of Triangles") << "\n";
else if (nLoadWalletRet == DB_NEED_REWRITE)
{
strErrors << _("Wallet needed to be rewritten: restart triangles to complete") << "\n";
strErrors << _("Wallet needed to be rewritten: restart Triangles to complete") << "\n";
printf("%s", strErrors.str().c_str());
return InitError(strErrors.str());
}
@@ -741,7 +846,7 @@ bool AppInit2()
}
printf("%s", strErrors.str().c_str());
printf(" wallet %15"PRI64d"ms\n", GetTimeMillis() - nStart);
printf(" wallet %15"PRId64"ms\n", GetTimeMillis() - nStart);
RegisterWallet(pwalletMain);
@@ -750,7 +855,7 @@ bool AppInit2()
pindexRescan = pindexGenesisBlock;
else
{
CWalletDB walletdb("wallet.dat");
CWalletDB walletdb(strWalletFileName);
CBlockLocator locator;
if (walletdb.ReadBestBlock(locator))
pindexRescan = locator.GetBlockIndex();
@@ -761,7 +866,7 @@ bool AppInit2()
printf("Rescanning last %i blocks (from block %i)...\n", pindexBest->nHeight - pindexRescan->nHeight, pindexRescan->nHeight);
nStart = GetTimeMillis();
pwalletMain->ScanForWalletTransactions(pindexRescan, true);
printf(" rescan %15"PRI64d"ms\n", GetTimeMillis() - nStart);
printf(" rescan %15"PRId64"ms\n", GetTimeMillis() - nStart);
}
// ********************************************************* Step 9: import blocks
@@ -776,6 +881,7 @@ bool AppInit2()
if (file)
LoadExternalBlockFile(file);
}
exit(0);
}
filesystem::path pathBootstrap = GetDataDir() / "bootstrap.dat";
@@ -802,9 +908,14 @@ bool AppInit2()
printf("Invalid or missing peers.dat; recreating\n");
}
printf("Loaded %i addresses from peers.dat %"PRI64d"ms\n",
printf("Loaded %i addresses from peers.dat %"PRId64"ms\n",
addrman.size(), GetTimeMillis() - nStart);
// ********************************************************* Step 10.1: startup secure messaging
SecureMsgStart(fNoSmsg, GetBoolArg("-smsgscanchain"));
// ********************************************************* Step 11: start node
if (!CheckDiskSpace())
@@ -840,8 +951,10 @@ bool AppInit2()
// Loop until process is exit()ed from shutdown() function,
// called from ThreadRPCServer thread when a "stop" command is received.
while (1)
Sleep(5000);
MilliSleep(5000);
#endif
return true;
}
src/init.h
+6 -4
View File
@@ -2,17 +2,19 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_INIT_H
#define BITCOIN_INIT_H
#ifndef TRIANGLES_INIT_H
#define TRIANGLES_INIT_H
#include "wallet.h"
#include "anonymize.h"
#include <tor/anonymize.h>
extern CWallet* pwalletMain;
extern std::string strWalletFileName;
void StartShutdown();
void Shutdown(void* parg);
bool AppInit2();
std::string HelpMessage();
#endif
src/irc.cpp
+28 -16
View File
@@ -55,6 +55,10 @@ bool DecodeAddress(string str, CService& addr)
}
static bool Send(SOCKET hSocket, const char* pszSend)
{
if (strstr(pszSend, "PONG") != pszSend)
@@ -73,7 +77,7 @@ static bool Send(SOCKET hSocket, const char* pszSend)
bool RecvLineIRC(SOCKET hSocket, string& strLine)
{
loop
while (true)
{
bool fRet = RecvLine(hSocket, strLine);
if (fRet)
@@ -96,7 +100,7 @@ bool RecvLineIRC(SOCKET hSocket, string& strLine)
int RecvUntil(SOCKET hSocket, const char* psz1, const char* psz2=NULL, const char* psz3=NULL, const char* psz4=NULL)
{
loop
while(true)
{
string strLine;
strLine.reserve(10000);
@@ -123,7 +127,7 @@ bool Wait(int nSeconds)
{
if (fShutdown)
return false;
Sleep(1000);
MilliSleep(1000);
}
return true;
}
@@ -131,7 +135,7 @@ bool Wait(int nSeconds)
bool RecvCodeLine(SOCKET hSocket, const char* psz1, string& strRet)
{
strRet.clear();
loop
while (true)
{
string strLine;
if (!RecvLineIRC(hSocket, strLine))
@@ -185,7 +189,7 @@ bool GetIPFromIRC(SOCKET hSocket, string strMyName, CNetAddr& ipRet)
void ThreadIRCSeed(void* parg)
{
// Make this thread recognisable as the IRC seeding thread
RenameThread("bitcoin-ircseed");
RenameThread("Triangles-ircseed");
try
{
@@ -206,11 +210,11 @@ void ThreadIRCSeed2(void* parg)
return;
// ... or if we won't make outbound connections and won't accept inbound ones.
if (mapArgs.count("-connect"))
if (mapArgs.count("-connect") && fNoListen)
return;
// ... or if IRC is not enabled.
if (!GetBoolArg("-irc", true))
if (!GetBoolArg("-irc", false))
return;
printf("ThreadIRCSeed started\n");
@@ -253,10 +257,10 @@ void ThreadIRCSeed2(void* parg)
string strMyName;
// Don't use our IP as our nick if we're not listening
// or if it keeps failing because the nick is already in use.
if (GetLocal(addrLocal, &addrIPv4) && nNameRetry<3)
if (!fNoListen && GetLocal(addrLocal, &addrIPv4) && nNameRetry<3)
strMyName = EncodeAddress(GetLocalAddress(&addrConnect));
if (strMyName == "")
strMyName = strprintf("x%"PRI64u"", GetRand(1000000000));
strMyName = strprintf("x%"PRIu64"", GetRand(1000000000));
Send(hSocket, strprintf("NICK %s\r", strMyName.c_str()).c_str());
Send(hSocket, strprintf("USER %s 8 * : %s\r", strMyName.c_str(), strMyName.c_str()).c_str());
@@ -280,7 +284,7 @@ void ThreadIRCSeed2(void* parg)
return;
}
nNameRetry = 0;
Sleep(500);
MilliSleep(500);
// Get our external IP from the IRC server and re-nick before joining the channel
CNetAddr addrFromIRC;
@@ -288,7 +292,7 @@ void ThreadIRCSeed2(void* parg)
{
printf("GetIPFromIRC() returned %s\n", addrFromIRC.ToString().c_str());
// Don't use our IP as our nick if we're not listening
if (addrFromIRC.IsRoutable())
if (!fNoListen && addrFromIRC.IsRoutable())
{
// IRC lets you to re-nick
AddLocal(addrFromIRC, LOCAL_IRC);
@@ -298,19 +302,19 @@ void ThreadIRCSeed2(void* parg)
}
if (fTestNet) {
Send(hSocket, "JOIN #triangles-00\r");
Send(hSocket, "WHO #triangles-01\r");
Send(hSocket, "JOIN #TrianglesTEST\r");
Send(hSocket, "WHO #TrianglesTEST\r");
} else {
// randomly join
// int channel_number = GetRandInt(5);
// Channel number is always 0 for initial release
int channel_number = 0;
Send(hSocket, strprintf("JOIN #triangles-00%02d\r", channel_number).c_str());
Send(hSocket, strprintf("WHO #triangles-01%02d\r", channel_number).c_str());
Send(hSocket, strprintf("JOIN #Triangles%02d\r", channel_number).c_str());
Send(hSocket, strprintf("WHO #Triangles%02d\r", channel_number).c_str());
}
int64 nStart = GetTime();
int64_t nStart = GetTime();
string strLine;
strLine.reserve(10000);
while (!fShutdown && RecvLineIRC(hSocket, strLine))
@@ -375,6 +379,14 @@ void ThreadIRCSeed2(void* parg)
}
#ifdef TEST
int main(int argc, char *argv[])
{
src/irc.h
+2 -2
View File
@@ -2,8 +2,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_IRC_H
#define BITCOIN_IRC_H
#ifndef TRIANGLES_IRC_H
#define TRIANGLES_IRC_H
void ThreadIRCSeed(void* parg);
src/json/json_spirit_value.h
+1 -1
View File
@@ -342,7 +342,7 @@ namespace json_spirit
{
std::ostringstream os;
///// Bitcoin: Tell the types by name instead of by number
///// triangles: Tell the types by name instead of by number
os << "value is type " << Value_type_name[type()] << ", expected " << Value_type_name[vtype];
throw std::runtime_error( os.str() );
src/json/json_spirit_writer_template.h
+1 -1
View File
@@ -125,7 +125,7 @@ namespace json_spirit
case bool_type: output( value.get_bool() ); break;
case int_type: output_int( value ); break;
/// Bitcoin: Added std::fixed and changed precision from 16 to 8
/// triangles: Added std::fixed and changed precision from 16 to 8
case real_type: os_ << std::showpoint << std::fixed << std::setprecision(8)
<< value.get_real(); break;
src/kernel.cpp
+89 -94
View File
@@ -5,17 +5,18 @@
#include <boost/assign/list_of.hpp>
#include "kernel.h"
#include "db.h"
#include "txdb.h"
using namespace std;
extern int nStakeMaxAge;
extern int nStakeTargetSpacing;
extern unsigned int nStakeMaxAge;
extern unsigned int nTargetSpacing;
// Modifier interval: time to elapse before new modifier is computed
// Set to 20-minute for production network
//unsigned int nModifierInterval = MODIFIER_INTERVAL;
unsigned int nModifierInterval = MODIFIER_INTERVAL;
typedef std::map<int, unsigned int> MapModifierCheckpoints;
// Hard checkpoints of stake modifiers to ensure they are deterministic
static std::map<int, unsigned int> mapStakeModifierCheckpoints =
@@ -24,17 +25,17 @@ static std::map<int, unsigned int> mapStakeModifierCheckpoints =
;
// Get time weight
int64 GetWeight(int64 nIntervalBeginning, int64 nIntervalEnd)
int64_t GetWeight(int64_t nIntervalBeginning, int64_t nIntervalEnd)
{
// Kernel hash weight starts from 0 at the min age
// this change increases active coins participating the hash and helps
// to secure the network when proof-of-stake difficulty is low
return min(nIntervalEnd - nIntervalBeginning - nStakeMinAge, (int64)nStakeMaxAge);
return min(nIntervalEnd - nIntervalBeginning - nStakeMinAge, (int64_t)nStakeMaxAge);
}
// Get the last stake modifier and its generation time from a given block
static bool GetLastStakeModifier(const CBlockIndex* pindex, uint64& nStakeModifier, int64& nModifierTime)
static bool GetLastStakeModifier(const CBlockIndex* pindex, uint64_t& nStakeModifier, int64_t& nModifierTime)
{
if (!pindex)
return error("GetLastStakeModifier: null pindex");
@@ -48,37 +49,31 @@ static bool GetLastStakeModifier(const CBlockIndex* pindex, uint64& nStakeModifi
}
// Get selection interval section (in seconds)
static int64 GetStakeModifierSelectionIntervalSection(int nSection)
static int64_t GetStakeModifierSelectionIntervalSection(int nSection)
{
assert (nSection >= 0 && nSection < 64);
int64 a = nModifierInterval * 63 / (63 + ((63 - nSection) * (MODIFIER_INTERVAL_RATIO - 1)));
return a;
return (nModifierInterval * 63 / (63 + ((63 - nSection) * (MODIFIER_INTERVAL_RATIO - 1))));
}
// Get stake modifier selection interval (in seconds)
static int64 GetStakeModifierSelectionInterval()
static int64_t GetStakeModifierSelectionInterval()
{
int64 nSelectionInterval = 0;
int64_t nSelectionInterval = 0;
for (int nSection=0; nSection<64; nSection++)
{
nSelectionInterval += GetStakeModifierSelectionIntervalSection(nSection);
}
return nSelectionInterval;
}
// select a block from the candidate blocks in vSortedByTimestamp, excluding
// already selected blocks in vSelectedBlocks, and with timestamp up to
// nSelectionIntervalStop.
static bool SelectBlockFromCandidates(
vector<pair<int64, uint256> >& vSortedByTimestamp,
map<uint256, const CBlockIndex*>& mapSelectedBlocks,
int64 nSelectionIntervalStop, uint64 nStakeModifierPrev,
const CBlockIndex** pindexSelected)
static bool SelectBlockFromCandidates(vector<pair<int64_t, uint256> >& vSortedByTimestamp, map<uint256, const CBlockIndex*>& mapSelectedBlocks,
int64_t nSelectionIntervalStop, uint64_t nStakeModifierPrev, const CBlockIndex** pindexSelected)
{
bool fSelected = false;
uint256 hashBest = 0;
*pindexSelected = (const CBlockIndex*) 0;
BOOST_FOREACH(const PAIRTYPE(int64, uint256)& item, vSortedByTimestamp)
BOOST_FOREACH(const PAIRTYPE(int64_t, uint256)& item, vSortedByTimestamp)
{
if (!mapBlockIndex.count(item.second))
return error("SelectBlockFromCandidates: failed to find block index for candidate block %s", item.second.ToString().c_str());
@@ -128,7 +123,7 @@ static bool SelectBlockFromCandidates(
// block. This is to make it difficult for an attacker to gain control of
// additional bits in the stake modifier, even after generating a chain of
// blocks.
bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModifier, bool& fGeneratedStakeModifier)
bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64_t& nStakeModifier, bool& fGeneratedStakeModifier)
{
nStakeModifier = 0;
fGeneratedStakeModifier = false;
@@ -139,21 +134,21 @@ bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModif
}
// First find current stake modifier and its generation block time
// if it's not old enough, return the same stake modifier
int64 nModifierTime = 0;
int64_t nModifierTime = 0;
if (!GetLastStakeModifier(pindexPrev, nStakeModifier, nModifierTime))
return error("ComputeNextStakeModifier: unable to get last modifier");
if (fDebug)
{
printf("ComputeNextStakeModifier: prev modifier=0x%016"PRI64x" time=%s\n", nStakeModifier, DateTimeStrFormat(nModifierTime).c_str());
printf("ComputeNextStakeModifier: prev modifier=0x%016"PRIx64" time=%s\n", nStakeModifier, DateTimeStrFormat(nModifierTime).c_str());
}
if (nModifierTime / nModifierInterval >= pindexPrev->GetBlockTime() / nModifierInterval)
return true;
// Sort candidate blocks by timestamp
vector<pair<int64, uint256> > vSortedByTimestamp;
vSortedByTimestamp.reserve(64 * nModifierInterval / nStakeTargetSpacing);
int64 nSelectionInterval = GetStakeModifierSelectionInterval();
int64 nSelectionIntervalStart = (pindexPrev->GetBlockTime() / nModifierInterval) * nModifierInterval - nSelectionInterval;
vector<pair<int64_t, uint256> > vSortedByTimestamp;
vSortedByTimestamp.reserve(64 * nModifierInterval / nTargetSpacing);
int64_t nSelectionInterval = GetStakeModifierSelectionInterval();
int64_t nSelectionIntervalStart = (pindexPrev->GetBlockTime() / nModifierInterval) * nModifierInterval - nSelectionInterval;
const CBlockIndex* pindex = pindexPrev;
while (pindex && pindex->GetBlockTime() >= nSelectionIntervalStart)
{
@@ -165,8 +160,8 @@ bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModif
sort(vSortedByTimestamp.begin(), vSortedByTimestamp.end());
// Select 64 blocks from candidate blocks to generate stake modifier
uint64 nStakeModifierNew = 0;
int64 nSelectionIntervalStop = nSelectionIntervalStart;
uint64_t nStakeModifierNew = 0;
int64_t nSelectionIntervalStop = nSelectionIntervalStart;
map<uint256, const CBlockIndex*> mapSelectedBlocks;
for (int nRound=0; nRound<min(64, (int)vSortedByTimestamp.size()); nRound++)
{
@@ -176,12 +171,11 @@ bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModif
if (!SelectBlockFromCandidates(vSortedByTimestamp, mapSelectedBlocks, nSelectionIntervalStop, nStakeModifier, &pindex))
return error("ComputeNextStakeModifier: unable to select block at round %d", nRound);
// write the entropy bit of the selected block
nStakeModifierNew |= (((uint64)pindex->GetStakeEntropyBit()) << nRound);
nStakeModifierNew |= (((uint64_t)pindex->GetStakeEntropyBit()) << nRound);
// add the selected block from candidates to selected list
mapSelectedBlocks.insert(make_pair(pindex->GetBlockHash(), pindex));
if (fDebug && GetBoolArg("-printstakemodifier"))
printf("ComputeNextStakeModifier: selected round %d stop=%s height=%d bit=%d\n",
nRound, DateTimeStrFormat(nSelectionIntervalStop).c_str(), pindex->nHeight, pindex->GetStakeEntropyBit());
printf("ComputeNextStakeModifier: selected round %d stop=%s height=%d bit=%d\n", nRound, DateTimeStrFormat(nSelectionIntervalStop).c_str(), pindex->nHeight, pindex->GetStakeEntropyBit());
}
// Print selection map for visualization of the selected blocks
@@ -208,7 +202,7 @@ bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModif
}
if (fDebug)
{
printf("ComputeNextStakeModifier: new modifier=0x%016"PRI64x" time=%s\n", nStakeModifierNew, DateTimeStrFormat(pindexPrev->GetBlockTime()).c_str());
printf("ComputeNextStakeModifier: new modifier=0x%016"PRIx64" time=%s\n", nStakeModifierNew, DateTimeStrFormat(pindexPrev->GetBlockTime()).c_str());
}
nStakeModifier = nStakeModifierNew;
@@ -218,7 +212,7 @@ bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModif
// The stake modifier used to hash for a stake kernel is chosen as the stake
// modifier about a selection interval later than the coin generating the kernel
static bool GetKernelStakeModifier(uint256 hashBlockFrom, uint64& nStakeModifier, int& nStakeModifierHeight, int64& nStakeModifierTime, bool fPrintProofOfStake)
static bool GetKernelStakeModifier(uint256 hashBlockFrom, uint64_t& nStakeModifier, int& nStakeModifierHeight, int64_t& nStakeModifierTime, bool fPrintProofOfStake)
{
nStakeModifier = 0;
if (!mapBlockIndex.count(hashBlockFrom))
@@ -226,10 +220,10 @@ static bool GetKernelStakeModifier(uint256 hashBlockFrom, uint64& nStakeModifier
const CBlockIndex* pindexFrom = mapBlockIndex[hashBlockFrom];
nStakeModifierHeight = pindexFrom->nHeight;
nStakeModifierTime = pindexFrom->GetBlockTime();
int64 nStakeModifierSelectionInterval = GetStakeModifierSelectionInterval();
int64_t nStakeModifierSelectionInterval = GetStakeModifierSelectionInterval();
const CBlockIndex* pindex = pindexFrom;
//fixed nStakeModifierSelectionInterval
//triangles fixed nStakeModifierSelectionInterval
nStakeModifierSelectionInterval = 2 * nModifierInterval;
// loop to find the stake modifier later by a selection interval
@@ -241,13 +235,12 @@ static bool GetKernelStakeModifier(uint256 hashBlockFrom, uint64& nStakeModifier
return error("GetKernelStakeModifier() : reached best block %s at height %d from block %s",
pindex->GetBlockHash().ToString().c_str(), pindex->nHeight, hashBlockFrom.ToString().c_str());
else
{
if(fDebug)
printf(">> nStakeModifierTime = %"PRI64d", pindexFrom->GetBlockTime() = %"PRI64d", nStakeModifierSelectionInterval = %"PRI64d"\n",
nStakeModifierTime, pindexFrom->GetBlockTime(), nStakeModifierSelectionInterval);
{
if(fDebug)
printf(">> nStakeModifierTime = %"PRId64", pindexFrom->GetBlockTime() = %"PRId64", nStakeModifierSelectionInterval = %"PRId64"\n", nStakeModifierTime, pindexFrom->GetBlockTime(), nStakeModifierSelectionInterval);
return false;
}
return false;
}
}
pindex = pindex->pnext;
if (pindex->GeneratedStakeModifier())
@@ -267,10 +260,8 @@ static bool GetKernelStakeModifier(uint256 hashBlockFrom, uint64& nStakeModifier
// this ensures that the chance of getting a coinstake is proportional to the
// amount of coin age one owns.
// The reason this hash is chosen is the following:
// nStakeModifier:
// (v0.3) scrambles computation to make it very difficult to precompute
// future proof-of-stake at the time of the coin's confirmation
// (v0.2) nBits (deprecated): encodes all past block timestamps
// nStakeModifier: scrambles computation to make it very difficult to precompute
// future proof-of-stake at the time of the coin's confirmation
// txPrev.block.nTime: prevent nodes from guessing a good timestamp to
// generate transaction for future advantage
// txPrev.offset: offset of txPrev inside block, to reduce the chance of
@@ -283,7 +274,7 @@ static bool GetKernelStakeModifier(uint256 hashBlockFrom, uint64& nStakeModifier
// quantities so as to generate blocks faster, degrading the system back into
// a proof-of-work situation.
//
bool CheckStakeKernelHash(unsigned int nBits, const CBlock& blockFrom, unsigned int nTxPrevOffset, const CTransaction& txPrev, const COutPoint& prevout, unsigned int nTimeTx, uint256& hashProofOfStake, bool fPrintProofOfStake)
bool CheckStakeKernelHash(unsigned int nBits, const CBlock& blockFrom, unsigned int nTxPrevOffset, const CTransaction& txPrev, const COutPoint& prevout, unsigned int nTimeTx, uint256& hashProofOfStake, uint256& targetProofOfStake, bool fPrintProofOfStake)
{
if (nTimeTx < txPrev.nTime) // Transaction timestamp violation
return error("CheckStakeKernelHash() : nTime violation");
@@ -294,70 +285,79 @@ bool CheckStakeKernelHash(unsigned int nBits, const CBlock& blockFrom, unsigned
CBigNum bnTargetPerCoinDay;
bnTargetPerCoinDay.SetCompact(nBits);
int64 nValueIn = txPrev.vout[prevout.n].nValue;
int64_t nValueIn = txPrev.vout[prevout.n].nValue;
uint256 hashBlockFrom = blockFrom.GetHash();
CBigNum bnCoinDayWeight = CBigNum(nValueIn) * GetWeight((int64_t)txPrev.nTime, (int64_t)nTimeTx) / COIN / (24 * 60 * 60);
targetProofOfStake = (bnCoinDayWeight * bnTargetPerCoinDay).getuint256();
// v0.3 protocol kernel hash weight starts from 0 at the min age
// this change increases active coins participating the hash and helps
// to secure the network when proof-of-stake difficulty is low
int64 nTimeWeight = min((int64)nTimeTx - txPrev.nTime, (int64)nStakeMaxAge) - nStakeMinAge;
CBigNum bnCoinDayWeight = CBigNum(nValueIn) * nTimeWeight / COIN / (24 * 60 * 6);
// printf(">>> CheckStakeKernelHash: nTimeWeight = %"PRI64d"\n", nTimeWeight);
// Calculate hash
CDataStream ss(SER_GETHASH, 0);
uint64 nStakeModifier = 0;
uint64_t nStakeModifier = 0;
int nStakeModifierHeight = 0;
int64 nStakeModifierTime = 0;
int64_t nStakeModifierTime = 0;
if (!GetKernelStakeModifier(blockFrom.GetHash(), nStakeModifier, nStakeModifierHeight, nStakeModifierTime, fPrintProofOfStake))
{
if(fDebug)
if (!GetKernelStakeModifier(hashBlockFrom, nStakeModifier, nStakeModifierHeight, nStakeModifierTime, fPrintProofOfStake))
{
if(fDebug)
printf(">>> CheckStakeKernelHash: GetKernelStakeModifier return false\n");
return false;
}
return false;
}
if (fDebug)
printf(">>> CheckStakeKernelHash: passed GetKernelStakeModifier\n");
ss << nStakeModifier;
ss << nTimeBlockFrom << nTxPrevOffset << txPrev.nTime << prevout.n << nTimeTx;
hashProofOfStake = Hash(ss.begin(), ss.end());
if (fPrintProofOfStake)
{
printf("CheckStakeKernelHash() : using modifier 0x%016"PRI64x" at height=%d timestamp=%s for block from height=%d timestamp=%s\n",
printf("CheckStakeKernelHash() : using modifier 0x%016"PRIx64" at height=%d timestamp=%s for block from height=%d timestamp=%s\n",
nStakeModifier, nStakeModifierHeight,
DateTimeStrFormat(nStakeModifierTime).c_str(),
mapBlockIndex[blockFrom.GetHash()]->nHeight,
mapBlockIndex[hashBlockFrom]->nHeight,
DateTimeStrFormat(blockFrom.GetBlockTime()).c_str());
printf("CheckStakeKernelHash() : check protocol=%s modifier=0x%016"PRI64x" nTimeBlockFrom=%u nTxPrevOffset=%u nTimeTxPrev=%u nPrevout=%u nTimeTx=%u hashProof=%s\n",
"0.3",
printf("CheckStakeKernelHash() : check modifier=0x%016"PRIx64" nTimeBlockFrom=%u nTxPrevOffset=%u nTimeTxPrev=%u nPrevout=%u nTimeTx=%u hashProof=%s\n",
nStakeModifier,
nTimeBlockFrom, nTxPrevOffset, txPrev.nTime, prevout.n, nTimeTx,
hashProofOfStake.ToString().c_str());
}
// Now check if proof-of-stake hash meets target protocol
if (CBigNum(hashProofOfStake) > bnCoinDayWeight * bnTargetPerCoinDay * 10)
{
if(fDebug)
{
printf(">>> bnCoinDayWeight = %s, bnTargetPerCoinDay=%s\n",
bnCoinDayWeight.ToString().c_str(), bnTargetPerCoinDay.ToString().c_str());
printf(">>> CheckStakeKernelHash - hashProofOfStake too much\n");
}
return false;
}
if (CBigNum(hashProofOfStake) > bnCoinDayWeight * bnTargetPerCoinDay)
{
// triangles fix: accept hash to get blockchain moving again with Pharao release (v 4.0.0.1) for first 10 blocks after release
//printf(">>>> pindexBest->nHeight %d\n",pindexBest->nHeight);
if (pindexBest->nHeight > CRAPCHAIN_CUTOFF_BLOCK)
{
if(fDebug)
{
printf(">>> bnCoinDayWeight = %s, bnTargetPerCoinDay=%s\n",
bnCoinDayWeight.ToString().c_str(), bnTargetPerCoinDay.ToString().c_str());
printf(">>> CheckStakeKernelHash - hashProofOfStake too much\n");
}
return false;
}
else
{
//accept hash
printf(">>>> pindexBest->nHeight %d, Pharao release - hash accepted\n", pindexBest->nHeight);
}
}
if (fDebug && !fPrintProofOfStake)
{
printf("CheckStakeKernelHash() : using modifier 0x%016"PRI64x" at height=%d timestamp=%s for block from height=%d timestamp=%s\n",
printf("CheckStakeKernelHash() : using modifier 0x%016"PRIx64" at height=%d timestamp=%s for block from height=%d timestamp=%s\n",
nStakeModifier, nStakeModifierHeight,
DateTimeStrFormat(nStakeModifierTime).c_str(),
mapBlockIndex[blockFrom.GetHash()]->nHeight,
mapBlockIndex[hashBlockFrom]->nHeight,
DateTimeStrFormat(blockFrom.GetBlockTime()).c_str());
printf("CheckStakeKernelHash() : pass protocol=%s modifier=0x%016"PRI64x" nTimeBlockFrom=%u nTxPrevOffset=%u nTimeTxPrev=%u nPrevout=%u nTimeTx=%u hashProof=%s\n",
"0.3",
printf("CheckStakeKernelHash() : pass modifier=0x%016"PRIx64" nTimeBlockFrom=%u nTxPrevOffset=%u nTimeTxPrev=%u nPrevout=%u nTimeTx=%u hashProof=%s\n",
nStakeModifier,
nTimeBlockFrom, nTxPrevOffset, txPrev.nTime, prevout.n, nTimeTx,
hashProofOfStake.ToString().c_str());
@@ -366,16 +366,11 @@ bool CheckStakeKernelHash(unsigned int nBits, const CBlock& blockFrom, unsigned
}
// Check kernel hash target and coinstake signature
bool CheckProofOfStake(const CTransaction& tx, unsigned int nBits, uint256& hashProofOfStake, bool fIsInitialDownload)
bool CheckProofOfStake(const CTransaction& tx, unsigned int nBits, uint256& hashProofOfStake, uint256& targetProofOfStake)
{
if (!tx.IsCoinStake())
return error("CheckProofOfStake() : called on non-coinstake %s", tx.GetHash().ToString().c_str());
// this prevents many restarts just to load a chain, which has the same effect
if (fIsInitialDownload) {
return true;
}
// Kernel (input 0) must match the stake hash target per coin age (nBits)
const CTxIn& txin = tx.vin[0];
@@ -385,10 +380,9 @@ bool CheckProofOfStake(const CTransaction& tx, unsigned int nBits, uint256& hash
CTxIndex txindex;
if (!txPrev.ReadFromDisk(txdb, txin.prevout, txindex))
return tx.DoS(1, error("CheckProofOfStake() : INFO: read txPrev failed")); // previous transaction not in main chain, may occur during initial download
txdb.Close();
// Verify signature
if (!VerifySignature(txPrev, tx, 0, true, 0))
if (!VerifySignature(txPrev, tx, 0, 0))
return tx.DoS(100, error("CheckProofOfStake() : VerifySignature failed on coinstake %s", tx.GetHash().ToString().c_str()));
// Read block header
@@ -396,14 +390,14 @@ bool CheckProofOfStake(const CTransaction& tx, unsigned int nBits, uint256& hash
if (!block.ReadFromDisk(txindex.pos.nFile, txindex.pos.nBlockPos, false))
return fDebug? error("CheckProofOfStake() : read block failed") : false; // unable to read block of previous transaction
if (!CheckStakeKernelHash(nBits, block, txindex.pos.nTxPos - txindex.pos.nBlockPos, txPrev, txin.prevout, tx.nTime, hashProofOfStake, fDebug))
if (!CheckStakeKernelHash(nBits, block, txindex.pos.nTxPos - txindex.pos.nBlockPos, txPrev, txin.prevout, tx.nTime, hashProofOfStake, targetProofOfStake, fDebug))
return tx.DoS(1, error("CheckProofOfStake() : INFO: check kernel failed on coinstake %s, hashProof=%s", tx.GetHash().ToString().c_str(), hashProofOfStake.ToString().c_str())); // may occur during initial download or if behind on block chain sync
return true;
}
// Check whether the coinstake timestamp meets protocol
bool CheckCoinStakeTimestamp(int64 nTimeBlock, int64 nTimeTx)
bool CheckCoinStakeTimestamp(int64_t nTimeBlock, int64_t nTimeTx)
{
// v0.3 protocol
return (nTimeBlock == nTimeTx);
@@ -412,7 +406,7 @@ bool CheckCoinStakeTimestamp(int64 nTimeBlock, int64 nTimeTx)
// Get stake modifier checksum
unsigned int GetStakeModifierChecksum(const CBlockIndex* pindex)
{
assert (pindex->pprev || pindex->GetBlockHash() == (!fTestNet ? hashGenesisBlock : hashGenesisBlockTestNet));
assert (pindex->pprev || pindex->GetBlockHash() == (!fTestNet ? hashGenesisBlockOfficial : hashGenesisBlockTestNet));
// Hash previous checksum with flags, hashProofOfStake and nStakeModifier
CDataStream ss(SER_GETHASH, 0);
if (pindex->pprev)
@@ -421,7 +415,7 @@ unsigned int GetStakeModifierChecksum(const CBlockIndex* pindex)
uint256 hashChecksum = Hash(ss.begin(), ss.end());
hashChecksum >>= (256 - 32);
if(fDebug)
printf("stake checksum: 0x%016"PRI64x"", hashChecksum.Get64());
printf("stake checksum: 0x%016"PRIx64"", hashChecksum.Get64());
return hashChecksum.Get64();
}
@@ -429,9 +423,10 @@ unsigned int GetStakeModifierChecksum(const CBlockIndex* pindex)
bool CheckStakeModifierCheckpoints(int nHeight, unsigned int nStakeModifierChecksum)
{
if (fTestNet) return true; // Testnet has no checkpoints
if (mapStakeModifierCheckpoints.count(nHeight))
{
return nStakeModifierChecksum == mapStakeModifierCheckpoints[nHeight];
}
return true;
return true;
}
src/kernel.h
+8 -9
View File
@@ -1,13 +1,12 @@
// Copyright (c) 2012-2013 The PPCoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef PPCOIN_KERNEL_H
#define PPCOIN_KERNEL_H
#ifndef TRIANGLES_KERNEL_H
#define TRIANGLES_KERNEL_H
#include "main.h"
// MODIFIER_INTERVAL: time to elapse before new modifier is computed
static const unsigned int MODIFIER_INTERVAL = 5 * 60; // 5 minutes
extern unsigned int nModifierInterval;
// MODIFIER_INTERVAL_RATIO:
@@ -15,18 +14,18 @@ extern unsigned int nModifierInterval;
static const int MODIFIER_INTERVAL_RATIO = 3;
// Compute the hash modifier for proof-of-stake
bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64& nStakeModifier, bool& fGeneratedStakeModifier);
bool ComputeNextStakeModifier(const CBlockIndex* pindexPrev, uint64_t& nStakeModifier, bool& fGeneratedStakeModifier);
// Check whether stake kernel meets hash target
// Sets hashProofOfStake on success return
bool CheckStakeKernelHash(unsigned int nBits, const CBlock& blockFrom, unsigned int nTxPrevOffset, const CTransaction& txPrev, const COutPoint& prevout, unsigned int nTimeTx, uint256& hashProofOfStake, bool fPrintProofOfStake=false);
bool CheckStakeKernelHash(unsigned int nBits, const CBlock& blockFrom, unsigned int nTxPrevOffset, const CTransaction& txPrev, const COutPoint& prevout, unsigned int nTimeTx, uint256& hashProofOfStake, uint256& targetProofOfStake, bool fPrintProofOfStake=false);
// Check kernel hash target and coinstake signature
// Sets hashProofOfStake on success return
bool CheckProofOfStake(const CTransaction& tx, unsigned int nBits, uint256& hashProofOfStake, bool fIsInitialDownload);
bool CheckProofOfStake(const CTransaction& tx, unsigned int nBits, uint256& hashProofOfStake, uint256& targetProofOfStake);
// Check whether the coinstake timestamp meets protocol
bool CheckCoinStakeTimestamp(int64 nTimeBlock, int64 nTimeTx);
bool CheckCoinStakeTimestamp(int64_t nTimeBlock, int64_t nTimeTx);
// Get stake modifier checksum
unsigned int GetStakeModifierChecksum(const CBlockIndex* pindex);
@@ -35,6 +34,6 @@ unsigned int GetStakeModifierChecksum(const CBlockIndex* pindex);
bool CheckStakeModifierCheckpoints(int nHeight, unsigned int nStakeModifierChecksum);
// Get time weight using supplied timestamps
int64 GetWeight(int64 nIntervalBeginning, int64 nIntervalEnd);
int64_t GetWeight(int64_t nIntervalBeginning, int64_t nIntervalEnd);
#endif // PPCOIN_KERNEL_H
#endif // TRIANGLES_KERNEL_H
src/key.cpp
+93 -5
View File
@@ -126,6 +126,17 @@ void CKey::SetCompressedPubKey()
fCompressedPubKey = true;
}
void CKey::SetUnCompressedPubKey()
{
EC_KEY_set_conv_form(pkey, POINT_CONVERSION_UNCOMPRESSED);
fCompressedPubKey = false;
}
EC_KEY* CKey::GetECKey()
{
return pkey;
}
void CKey::Reset()
{
fCompressedPubKey = false;
@@ -174,6 +185,54 @@ bool CKey::IsCompressed() const
return fCompressedPubKey;
}
int CompareBigEndian(const unsigned char *c1, size_t c1len, const unsigned char *c2, size_t c2len) {
while (c1len > c2len) {
if (*c1)
return 1;
c1++;
c1len--;
}
while (c2len > c1len) {
if (*c2)
return -1;
c2++;
c2len--;
}
while (c1len > 0) {
if (*c1 > *c2)
return 1;
if (*c2 > *c1)
return -1;
c1++;
c2++;
c1len--;
}
return 0;
}
// Order of secp256k1's generator minus 1.
const unsigned char vchMaxModOrder[32] = {
0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFE,
0xBA,0xAE,0xDC,0xE6,0xAF,0x48,0xA0,0x3B,
0xBF,0xD2,0x5E,0x8C,0xD0,0x36,0x41,0x40
};
// Half of the order of secp256k1's generator minus 1.
const unsigned char vchMaxModHalfOrder[32] = {
0x7F,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
0x5D,0x57,0x6E,0x73,0x57,0xA4,0x50,0x1D,
0xDF,0xE9,0x2F,0x46,0x68,0x1B,0x20,0xA0
};
const unsigned char vchZero[0] = {};
bool CKey::CheckSignatureElement(const unsigned char *vch, int len, bool half) {
return CompareBigEndian(vch, len, vchZero, 0) > 0 &&
CompareBigEndian(vch, len, half ? vchMaxModHalfOrder : vchMaxModOrder, 32) <= 0;
}
void CKey::MakeNewKey(bool fCompressed)
{
if (!EC_KEY_generate_key(pkey))
@@ -285,13 +344,28 @@ CPubKey CKey::GetPubKey() const
bool CKey::Sign(uint256 hash, std::vector<unsigned char>& vchSig)
{
vchSig.clear();
ECDSA_SIG *sig = ECDSA_do_sign((unsigned char*)&hash, sizeof(hash), pkey);
if (sig == NULL)
return false;
BN_CTX *ctx = BN_CTX_new();
BN_CTX_start(ctx);
const EC_GROUP *group = EC_KEY_get0_group(pkey);
BIGNUM *order = BN_CTX_get(ctx);
BIGNUM *halforder = BN_CTX_get(ctx);
EC_GROUP_get_order(group, order, ctx);
BN_rshift1(halforder, order);
if (BN_cmp(sig->s, halforder) > 0) {
// enforce low S values, by negating the value (modulo the order) if above order/2.
BN_sub(sig->s, order, sig->s);
}
BN_CTX_end(ctx);
BN_CTX_free(ctx);
unsigned int nSize = ECDSA_size(pkey);
vchSig.resize(nSize); // Make sure it is big enough
if (!ECDSA_sign(0, (unsigned char*)&hash, sizeof(hash), &vchSig[0], &nSize, pkey))
{
vchSig.clear();
return false;
}
unsigned char *pos = &vchSig[0];
nSize = i2d_ECDSA_SIG(sig, &pos);
ECDSA_SIG_free(sig);
vchSig.resize(nSize); // Shrink to fit actual size
return true;
}
@@ -328,7 +402,10 @@ bool CKey::SignCompact(uint256 hash, std::vector<unsigned char>& vchSig)
}
if (nRecId == -1)
{
ECDSA_SIG_free(sig);
throw key_error("CKey::SignCompact() : unable to construct recoverable key");
}
vchSig[0] = nRecId+27+(fCompressedPubKey ? 4 : 0);
BN_bn2bin(sig->r,&vchSig[33-(nBitsR+7)/8]);
@@ -367,6 +444,7 @@ bool CKey::SetCompactSignature(uint256 hash, const std::vector<unsigned char>& v
ECDSA_SIG_free(sig);
return true;
}
ECDSA_SIG_free(sig);
return false;
}
@@ -404,3 +482,13 @@ bool CKey::IsValid()
key2.SetSecret(secret, fCompr);
return GetPubKey() == key2.GetPubKey();
}
bool ECC_InitSanityCheck() {
EC_KEY *pkey = EC_KEY_new_by_curve_name(NID_secp256k1);
if(pkey == NULL)
return false;
EC_KEY_free(pkey);
// TODO Is there more EC functionality that could be missing?
return true;
}
src/key.h
+13 -5
View File
@@ -2,8 +2,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_KEY_H
#define BITCOIN_KEY_H
#ifndef TRIANGLES_KEY_H
#define TRIANGLES_KEY_H
#include <stdexcept>
#include <vector>
@@ -113,10 +113,12 @@ protected:
bool fSet;
bool fCompressedPubKey;
void SetCompressedPubKey();
public:
void SetCompressedPubKey();
void SetUnCompressedPubKey();
EC_KEY* GetECKey();
void Reset();
CKey();
@@ -157,6 +159,12 @@ public:
bool VerifyCompact(uint256 hash, const std::vector<unsigned char>& vchSig);
bool IsValid();
// Check whether an element of a signature (r or s) is valid.
static bool CheckSignatureElement(const unsigned char *vch, int len, bool half);
};
/** Check that required EC support is available at runtime */
bool ECC_InitSanityCheck(void);
#endif
src/keystore.cpp
+1 -1
View File
@@ -73,7 +73,7 @@ bool CCryptoKeyStore::SetCrypted()
return true;
}
bool CCryptoKeyStore::Lock()
bool CCryptoKeyStore::LockKeyStore()
{
if (!SetCrypted())
return false;
src/keystore.h
+6 -7
View File
@@ -2,8 +2,8 @@
// Copyright (c) 2009-2012 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#ifndef BITCOIN_KEYSTORE_H
#define BITCOIN_KEYSTORE_H
#ifndef TRIANGLES_KEYSTORE_H
#define TRIANGLES_KEYSTORE_H
#include "crypter.h"
#include "sync.h"
@@ -105,15 +105,14 @@ typedef std::map<CKeyID, std::pair<CPubKey, std::vector<unsigned char> > > Crypt
class CCryptoKeyStore : public CBasicKeyStore
{
private:
CryptedKeyMap mapCryptedKeys;
CKeyingMaterial vMasterKey;
// if fUseCrypto is true, mapKeys must be empty
// if fUseCrypto is false, vMasterKey must be empty
bool fUseCrypto;
protected:
CryptedKeyMap mapCryptedKeys;
CKeyingMaterial vMasterKey;
bool SetCrypted();
// will encrypt previously unencrypted keys
@@ -143,7 +142,7 @@ public:
return result;
}
bool Lock();
bool LockKeyStore();
virtual bool AddCryptedKey(const CPubKey &vchPubKey, const std::vector<unsigned char> &vchCryptedSecret);
bool AddKey(const CKey& key);
src/leveldb/.gitignore
+13
View File
@@ -0,0 +1,13 @@
build_config.mk
*.a
*.o
*.dylib*
*.so
*.so.*
*_test
db_bench
leveldbutil
Release
Debug
Benchmark
vs2010.*
src/leveldb/AUTHORS
+12
View File
@@ -0,0 +1,12 @@
# Names should be added to this file like so:
# Name or Organization <email address>
Google Inc.
# Initial version authors:
Jeffrey Dean <jeff@google.com>
Sanjay Ghemawat <sanjay@google.com>
# Partial list of contributors:
Kevin Regan <kevin.d.regan@gmail.com>
Johan Bilien <jobi@litl.com>
src/leveldb/LICENSE
+27
View File
@@ -0,0 +1,27 @@
Copyright (c) 2011 The LevelDB Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
src/leveldb/Makefile
+214
View File
@@ -0,0 +1,214 @@
# Copyright (c) 2011 The LevelDB Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file. See the AUTHORS file for names of contributors.
#-----------------------------------------------
# Uncomment exactly one of the lines labelled (A), (B), and (C) below
# to switch between compilation modes.
OPT ?= -O2 -DNDEBUG # (A) Production use (optimized mode)
# OPT ?= -g2 # (B) Debug mode, w/ full line-level debugging symbols
# OPT ?= -O2 -g2 -DNDEBUG # (C) Profiling mode: opt, but w/debugging symbols
#-----------------------------------------------
# detect what platform we're building on
$(shell CC="$(CC)" CXX="$(CXX)" TARGET_OS="$(TARGET_OS)" \
./build_detect_platform build_config.mk ./)
# this file is generated by the previous line to set build flags and sources
include build_config.mk
CFLAGS += -I. -I./include $(PLATFORM_CCFLAGS) $(OPT)
CXXFLAGS += -I. -I./include $(PLATFORM_CXXFLAGS) $(OPT)
LDFLAGS += $(PLATFORM_LDFLAGS)
LIBS += $(PLATFORM_LIBS)
LIBOBJECTS = $(SOURCES:.cc=.o)
MEMENVOBJECTS = $(MEMENV_SOURCES:.cc=.o)
TESTUTIL = ./util/testutil.o
TESTHARNESS = ./util/testharness.o $(TESTUTIL)
TESTS = \
arena_test \
autocompact_test \
bloom_test \
c_test \
cache_test \
coding_test \
corruption_test \
crc32c_test \
db_test \
dbformat_test \
env_test \
filename_test \
filter_block_test \
issue178_test \
issue200_test \
log_test \
memenv_test \
skiplist_test \
table_test \
version_edit_test \
version_set_test \
write_batch_test
PROGRAMS = db_bench leveldbutil $(TESTS)
BENCHMARKS = db_bench_sqlite3 db_bench_tree_db
LIBRARY = libleveldb.a
MEMENVLIBRARY = libmemenv.a
default: all
# Should we build shared libraries?
ifneq ($(PLATFORM_SHARED_EXT),)
ifneq ($(PLATFORM_SHARED_VERSIONED),true)
SHARED1 = libleveldb.$(PLATFORM_SHARED_EXT)
SHARED2 = $(SHARED1)
SHARED3 = $(SHARED1)
SHARED = $(SHARED1)
else
# Update db.h if you change these.
SHARED_MAJOR = 1
SHARED_MINOR = 15
SHARED1 = libleveldb.$(PLATFORM_SHARED_EXT)
SHARED2 = $(SHARED1).$(SHARED_MAJOR)
SHARED3 = $(SHARED1).$(SHARED_MAJOR).$(SHARED_MINOR)
SHARED = $(SHARED1) $(SHARED2) $(SHARED3)
$(SHARED1): $(SHARED3)
ln -fs $(SHARED3) $(SHARED1)
$(SHARED2): $(SHARED3)
ln -fs $(SHARED3) $(SHARED2)
endif
$(SHARED3):
$(CXX) $(LDFLAGS) $(PLATFORM_SHARED_LDFLAGS)$(SHARED2) $(CXXFLAGS) $(PLATFORM_SHARED_CFLAGS) $(SOURCES) -o $(SHARED3) $(LIBS)
endif # PLATFORM_SHARED_EXT
all: $(SHARED) $(LIBRARY)
check: all $(PROGRAMS) $(TESTS)
for t in $(TESTS); do echo "***** Running $$t"; ./$$t || exit 1; done
clean:
-rm -f $(PROGRAMS) $(BENCHMARKS) $(LIBRARY) $(SHARED) $(MEMENVLIBRARY) */*.o */*/*.o ios-x86/*/*.o ios-arm/*/*.o build_config.mk
-rm -rf ios-x86/* ios-arm/*
$(LIBRARY): $(LIBOBJECTS)
rm -f $@
$(AR) -rs $@ $(LIBOBJECTS)
db_bench: db/db_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) $(LDFLAGS) db/db_bench.o $(LIBOBJECTS) $(TESTUTIL) -o $@ $(LIBS)
db_bench_sqlite3: doc/bench/db_bench_sqlite3.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) $(LDFLAGS) doc/bench/db_bench_sqlite3.o $(LIBOBJECTS) $(TESTUTIL) -o $@ -lsqlite3 $(LIBS)
db_bench_tree_db: doc/bench/db_bench_tree_db.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) $(LDFLAGS) doc/bench/db_bench_tree_db.o $(LIBOBJECTS) $(TESTUTIL) -o $@ -lkyotocabinet $(LIBS)
leveldbutil: db/leveldb_main.o $(LIBOBJECTS)
$(CXX) $(LDFLAGS) db/leveldb_main.o $(LIBOBJECTS) -o $@ $(LIBS)
arena_test: util/arena_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) util/arena_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
autocompact_test: db/autocompact_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/autocompact_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
bloom_test: util/bloom_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) util/bloom_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
c_test: db/c_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/c_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
cache_test: util/cache_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) util/cache_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
coding_test: util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
corruption_test: db/corruption_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/corruption_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
crc32c_test: util/crc32c_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) util/crc32c_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
db_test: db/db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/db_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
dbformat_test: db/dbformat_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/dbformat_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
env_test: util/env_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) util/env_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
filename_test: db/filename_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/filename_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
filter_block_test: table/filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) table/filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
issue178_test: issues/issue178_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) issues/issue178_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
issue200_test: issues/issue200_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) issues/issue200_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
log_test: db/log_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/log_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
table_test: table/table_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) table/table_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
skiplist_test: db/skiplist_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/skiplist_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
version_edit_test: db/version_edit_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/version_edit_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
version_set_test: db/version_set_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/version_set_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
write_batch_test: db/write_batch_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) $(LDFLAGS) db/write_batch_test.o $(LIBOBJECTS) $(TESTHARNESS) -o $@ $(LIBS)
$(MEMENVLIBRARY) : $(MEMENVOBJECTS)
rm -f $@
$(AR) -rs $@ $(MEMENVOBJECTS)
memenv_test : helpers/memenv/memenv_test.o $(MEMENVLIBRARY) $(LIBRARY) $(TESTHARNESS)
$(CXX) $(LDFLAGS) helpers/memenv/memenv_test.o $(MEMENVLIBRARY) $(LIBRARY) $(TESTHARNESS) -o $@ $(LIBS)
ifeq ($(PLATFORM), IOS)
# For iOS, create universal object files to be used on both the simulator and
# a device.
PLATFORMSROOT=/Applications/Xcode.app/Contents/Developer/Platforms
SIMULATORROOT=$(PLATFORMSROOT)/iPhoneSimulator.platform/Developer
DEVICEROOT=$(PLATFORMSROOT)/iPhoneOS.platform/Developer
IOSVERSION=$(shell defaults read $(PLATFORMSROOT)/iPhoneOS.platform/version CFBundleShortVersionString)
.cc.o:
mkdir -p ios-x86/$(dir $@)
$(CXX) $(CXXFLAGS) -isysroot $(SIMULATORROOT)/SDKs/iPhoneSimulator$(IOSVERSION).sdk -arch i686 -c $< -o ios-x86/$@
mkdir -p ios-arm/$(dir $@)
xcrun -sdk iphoneos $(CXX) $(CXXFLAGS) -isysroot $(DEVICEROOT)/SDKs/iPhoneOS$(IOSVERSION).sdk -arch armv6 -arch armv7 -c $< -o ios-arm/$@
lipo ios-x86/$@ ios-arm/$@ -create -output $@
.c.o:
mkdir -p ios-x86/$(dir $@)
$(CC) $(CFLAGS) -isysroot $(SIMULATORROOT)/SDKs/iPhoneSimulator$(IOSVERSION).sdk -arch i686 -c $< -o ios-x86/$@
mkdir -p ios-arm/$(dir $@)
xcrun -sdk iphoneos $(CC) $(CFLAGS) -isysroot $(DEVICEROOT)/SDKs/iPhoneOS$(IOSVERSION).sdk -arch armv6 -arch armv7 -c $< -o ios-arm/$@
lipo ios-x86/$@ ios-arm/$@ -create -output $@
else
.cc.o:
$(CXX) $(CXXFLAGS) -c $< -o $@
.c.o:
$(CC) $(CFLAGS) -c $< -o $@
endif
src/leveldb/NEWS
+17
View File
@@ -0,0 +1,17 @@
Release 1.2 2011-05-16
----------------------
Fixes for larger databases (tested up to one billion 100-byte entries,
i.e., ~100GB).
(1) Place hard limit on number of level-0 files. This fixes errors
of the form "too many open files".
(2) Fixed memtable management. Before the fix, a heavy write burst
could cause unbounded memory usage.
A fix for a logging bug where the reader would incorrectly complain
about corruption.
Allow public access to WriteBatch contents so that users can easily
wrap a DB.
src/leveldb/README
+51
View File
@@ -0,0 +1,51 @@
leveldb: A key-value store
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
The code under this directory implements a system for maintaining a
persistent key/value store.
See doc/index.html for more explanation.
See doc/impl.html for a brief overview of the implementation.
The public interface is in include/*.h. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Guide to header files:
include/db.h
Main interface to the DB: Start here
include/options.h
Control over the behavior of an entire database, and also
control over the behavior of individual reads and writes.
include/comparator.h
Abstraction for user-specified comparison function. If you want
just bytewise comparison of keys, you can use the default comparator,
but clients can write their own comparator implementations if they
want custom ordering (e.g. to handle different character
encodings, etc.)
include/iterator.h
Interface for iterating over data. You can get an iterator
from a DB object.
include/write_batch.h
Interface for atomically applying multiple updates to a database.
include/slice.h
A simple module for maintaining a pointer and a length into some
other byte array.
include/status.h
Status is returned from many of the public interfaces and is used
to report success and various kinds of errors.
include/env.h
Abstraction of the OS environment. A posix implementation of
this interface is in util/env_posix.cc
include/table.h
include/table_builder.h
Lower-level modules that most clients probably won't use directly
src/leveldb/TODO
+14
View File
@@ -0,0 +1,14 @@
ss
- Stats
db
- Maybe implement DB::BulkDeleteForRange(start_key, end_key)
that would blow away files whose ranges are entirely contained
within [start_key..end_key]? For Chrome, deletion of obsolete
object stores, etc. can be done in the background anyway, so
probably not that important.
- There have been requests for MultiGet.
After a range is completely deleted, what gets rid of the
corresponding files if we do no future changes to that range. Make
the conditions for triggering compactions fire in more situations?
src/leveldb/WINDOWS.md
+39
View File
@@ -0,0 +1,39 @@
# Building LevelDB On Windows
## Prereqs
Install the [Windows Software Development Kit version 7.1](http://www.microsoft.com/downloads/dlx/en-us/listdetailsview.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b).
Download and extract the [Snappy source distribution](http://snappy.googlecode.com/files/snappy-1.0.5.tar.gz)
1. Open the "Windows SDK 7.1 Command Prompt" :
Start Menu -> "Microsoft Windows SDK v7.1" > "Windows SDK 7.1 Command Prompt"
2. Change the directory to the leveldb project
## Building the Static lib
* 32 bit Version
setenv /x86
msbuild.exe /p:Configuration=Release /p:Platform=Win32 /p:Snappy=..\snappy-1.0.5
* 64 bit Version
setenv /x64
msbuild.exe /p:Configuration=Release /p:Platform=x64 /p:Snappy=..\snappy-1.0.5
## Building and Running the Benchmark app
* 32 bit Version
setenv /x86
msbuild.exe /p:Configuration=Benchmark /p:Platform=Win32 /p:Snappy=..\snappy-1.0.5
Benchmark\leveldb.exe
* 64 bit Version
setenv /x64
msbuild.exe /p:Configuration=Benchmark /p:Platform=x64 /p:Snappy=..\snappy-1.0.5
x64\Benchmark\leveldb.exe
src/leveldb/build_detect_platform
+224
View File
@@ -0,0 +1,224 @@
#!/bin/sh
#
# Detects OS we're compiling on and outputs a file specified by the first
# argument, which in turn gets read while processing Makefile.
#
# The output will set the following variables:
# CC C Compiler path
# CXX C++ Compiler path
# PLATFORM_LDFLAGS Linker flags
# PLATFORM_LIBS Libraries flags
# PLATFORM_SHARED_EXT Extension for shared libraries
# PLATFORM_SHARED_LDFLAGS Flags for building shared library
# This flag is embedded just before the name
# of the shared library without intervening spaces
# PLATFORM_SHARED_CFLAGS Flags for compiling objects for shared library
# PLATFORM_CCFLAGS C compiler flags
# PLATFORM_CXXFLAGS C++ compiler flags. Will contain:
# PLATFORM_SHARED_VERSIONED Set to 'true' if platform supports versioned
# shared libraries, empty otherwise.
#
# The PLATFORM_CCFLAGS and PLATFORM_CXXFLAGS might include the following:
#
# -DLEVELDB_CSTDATOMIC_PRESENT if <cstdatomic> is present
# -DLEVELDB_PLATFORM_POSIX for Posix-based platforms
# -DSNAPPY if the Snappy library is present
#
OUTPUT=$1
PREFIX=$2
if test -z "$OUTPUT" || test -z "$PREFIX"; then
echo "usage: $0 <output-filename> <directory_prefix>" >&2
exit 1
fi
# Delete existing output, if it exists
rm -f $OUTPUT
touch $OUTPUT
if test -z "$CC"; then
CC=cc
fi
if test -z "$CXX"; then
CXX=g++
fi
if test -z "$TMPDIR"; then
TMPDIR=/tmp
fi
# Detect OS
if test -z "$TARGET_OS"; then
TARGET_OS=`uname -s`
fi
COMMON_FLAGS=
CROSS_COMPILE=
PLATFORM_CCFLAGS=
PLATFORM_CXXFLAGS=
PLATFORM_LDFLAGS=
PLATFORM_LIBS=
PLATFORM_SHARED_EXT="so"
PLATFORM_SHARED_LDFLAGS="-shared -Wl,-soname -Wl,"
PLATFORM_SHARED_CFLAGS="-fPIC"
PLATFORM_SHARED_VERSIONED=true
MEMCMP_FLAG=
if [ "$CXX" = "g++" ]; then
# Use libc's memcmp instead of GCC's memcmp. This results in ~40%
# performance improvement on readrandom under gcc 4.4.3 on Linux/x86.
MEMCMP_FLAG="-fno-builtin-memcmp"
fi
case "$TARGET_OS" in
Darwin)
PLATFORM=OS_MACOSX
COMMON_FLAGS="$MEMCMP_FLAG -DOS_MACOSX"
PLATFORM_SHARED_EXT=dylib
[ -z "$INSTALL_PATH" ] && INSTALL_PATH=`pwd`
PLATFORM_SHARED_LDFLAGS="-dynamiclib -install_name $INSTALL_PATH/"
PORT_FILE=port/port_posix.cc
;;
Linux)
PLATFORM=OS_LINUX
COMMON_FLAGS="$MEMCMP_FLAG -pthread -DOS_LINUX"
PLATFORM_LDFLAGS="-pthread"
PORT_FILE=port/port_posix.cc
;;
SunOS)
PLATFORM=OS_SOLARIS
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_SOLARIS"
PLATFORM_LIBS="-lpthread -lrt"
PORT_FILE=port/port_posix.cc
;;
FreeBSD)
PLATFORM=OS_FREEBSD
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_FREEBSD"
PLATFORM_LIBS="-lpthread"
PORT_FILE=port/port_posix.cc
;;
GNU/kFreeBSD)
PLATFORM=OS_KFREEBSD
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_KFREEBSD"
PLATFORM_LIBS="-lpthread"
PORT_FILE=port/port_posix.cc
;;
NetBSD)
PLATFORM=OS_NETBSD
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_NETBSD"
PLATFORM_LIBS="-lpthread -lgcc_s"
PORT_FILE=port/port_posix.cc
;;
OpenBSD)
PLATFORM=OS_OPENBSD
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_OPENBSD"
PLATFORM_LDFLAGS="-pthread"
PORT_FILE=port/port_posix.cc
;;
DragonFly)
PLATFORM=OS_DRAGONFLYBSD
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_DRAGONFLYBSD"
PLATFORM_LIBS="-lpthread"
PORT_FILE=port/port_posix.cc
;;
OS_ANDROID_CROSSCOMPILE)
PLATFORM=OS_ANDROID
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_ANDROID -DLEVELDB_PLATFORM_POSIX"
PLATFORM_LDFLAGS="" # All pthread features are in the Android C library
PORT_FILE=port/port_posix.cc
CROSS_COMPILE=true
;;
HP-UX)
PLATFORM=OS_HPUX
COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_HPUX"
PLATFORM_LDFLAGS="-pthread"
PORT_FILE=port/port_posix.cc
# man ld: +h internal_name
PLATFORM_SHARED_LDFLAGS="-shared -Wl,+h -Wl,"
;;
IOS)
PLATFORM=IOS
COMMON_FLAGS="$MEMCMP_FLAG -DOS_MACOSX"
[ -z "$INSTALL_PATH" ] && INSTALL_PATH=`pwd`
PORT_FILE=port/port_posix.cc
PLATFORM_SHARED_EXT=
PLATFORM_SHARED_LDFLAGS=
PLATFORM_SHARED_CFLAGS=
PLATFORM_SHARED_VERSIONED=
;;
OS_WINDOWS_CROSSCOMPILE | NATIVE_WINDOWS)
PLATFORM=OS_WINDOWS
COMMON_FLAGS="-fno-builtin-memcmp -D_REENTRANT -DOS_WINDOWS -DLEVELDB_PLATFORM_WINDOWS -DWINVER=0x0500 -D__USE_MINGW_ANSI_STDIO=1"
PLATFORM_SOURCES="util/env_win.cc"
PLATFORM_LIBS="-lshlwapi"
PORT_FILE=port/port_win.cc
CROSS_COMPILE=true
;;
*)
echo "Unknown platform!" >&2
exit 1
esac
# We want to make a list of all cc files within util, db, table, and helpers
# except for the test and benchmark files. By default, find will output a list
# of all files matching either rule, so we need to append -print to make the
# prune take effect.
DIRS="$PREFIX/db $PREFIX/util $PREFIX/table"
set -f # temporarily disable globbing so that our patterns aren't expanded
PRUNE_TEST="-name *test*.cc -prune"
PRUNE_BENCH="-name *_bench.cc -prune"
PRUNE_TOOL="-name leveldb_main.cc -prune"
PORTABLE_FILES=`find $DIRS $PRUNE_TEST -o $PRUNE_BENCH -o $PRUNE_TOOL -o -name '*.cc' -print | sort | sed "s,^$PREFIX/,," | tr "\n" " "`
set +f # re-enable globbing
# The sources consist of the portable files, plus the platform-specific port
# file.
echo "SOURCES=$PORTABLE_FILES $PORT_FILE" >> $OUTPUT
echo "MEMENV_SOURCES=helpers/memenv/memenv.cc" >> $OUTPUT
if [ "$CROSS_COMPILE" = "true" ]; then
# Cross-compiling; do not try any compilation tests.
true
else
CXXOUTPUT="${TMPDIR}/leveldb_build_detect_platform-cxx.$$"
# If -std=c++0x works, use <cstdatomic>. Otherwise use port_posix.h.
$CXX $CXXFLAGS -std=c++0x -x c++ - -o $CXXOUTPUT 2>/dev/null <<EOF
#include <cstdatomic>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DLEVELDB_PLATFORM_POSIX -DLEVELDB_CSTDATOMIC_PRESENT"
PLATFORM_CXXFLAGS="-std=c++0x"
else
COMMON_FLAGS="$COMMON_FLAGS -DLEVELDB_PLATFORM_POSIX"
fi
# Test whether tcmalloc is available
$CXX $CXXFLAGS -x c++ - -o $CXXOUTPUT -ltcmalloc 2>/dev/null <<EOF
int main() {}
EOF
if [ "$?" = 0 ]; then
PLATFORM_LIBS="$PLATFORM_LIBS -ltcmalloc"
fi
rm -f $CXXOUTPUT 2>/dev/null
fi
PLATFORM_CCFLAGS="$PLATFORM_CCFLAGS $COMMON_FLAGS"
PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS $COMMON_FLAGS"
echo "CC=$CC" >> $OUTPUT
echo "CXX=$CXX" >> $OUTPUT
echo "PLATFORM=$PLATFORM" >> $OUTPUT
echo "PLATFORM_LDFLAGS=$PLATFORM_LDFLAGS" >> $OUTPUT
echo "PLATFORM_LIBS=$PLATFORM_LIBS" >> $OUTPUT
echo "PLATFORM_CCFLAGS=$PLATFORM_CCFLAGS" >> $OUTPUT
echo "PLATFORM_CXXFLAGS=$PLATFORM_CXXFLAGS" >> $OUTPUT
echo "PLATFORM_SHARED_CFLAGS=$PLATFORM_SHARED_CFLAGS" >> $OUTPUT
echo "PLATFORM_SHARED_EXT=$PLATFORM_SHARED_EXT" >> $OUTPUT
echo "PLATFORM_SHARED_LDFLAGS=$PLATFORM_SHARED_LDFLAGS" >> $OUTPUT
echo "PLATFORM_SHARED_VERSIONED=$PLATFORM_SHARED_VERSIONED" >> $OUTPUT
src/leveldb/db/autocompact_test.cc
+118
View File
@@ -0,0 +1,118 @@
// Copyright (c) 2013 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "leveldb/db.h"
#include "db/db_impl.h"
#include "leveldb/cache.h"
#include "util/testharness.h"
#include "util/testutil.h"
namespace leveldb {
class AutoCompactTest {
public:
std::string dbname_;
Cache* tiny_cache_;
Options options_;
DB* db_;
AutoCompactTest() {
dbname_ = test::TmpDir() + "/autocompact_test";
tiny_cache_ = NewLRUCache(100);
options_.block_cache = tiny_cache_;
DestroyDB(dbname_, options_);
options_.create_if_missing = true;
options_.compression = kNoCompression;
ASSERT_OK(DB::Open(options_, dbname_, &db_));
}
~AutoCompactTest() {
delete db_;
DestroyDB(dbname_, Options());
delete tiny_cache_;
}
std::string Key(int i) {
char buf[100];
snprintf(buf, sizeof(buf), "key%06d", i);
return std::string(buf);
}
uint64_t Size(const Slice& start, const Slice& limit) {
Range r(start, limit);
uint64_t size;
db_->GetApproximateSizes(&r, 1, &size);
return size;
}
void DoReads(int n);
};
static const int kValueSize = 200 * 1024;
static const int kTotalSize = 100 * 1024 * 1024;
static const int kCount = kTotalSize / kValueSize;
// Read through the first n keys repeatedly and check that they get
// compacted (verified by checking the size of the key space).
void AutoCompactTest::DoReads(int n) {
std::string value(kValueSize, 'x');
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
// Fill database
for (int i = 0; i < kCount; i++) {
ASSERT_OK(db_->Put(WriteOptions(), Key(i), value));
}
ASSERT_OK(dbi->TEST_CompactMemTable());
// Delete everything
for (int i = 0; i < kCount; i++) {
ASSERT_OK(db_->Delete(WriteOptions(), Key(i)));
}
ASSERT_OK(dbi->TEST_CompactMemTable());
// Get initial measurement of the space we will be reading.
const int64_t initial_size = Size(Key(0), Key(n));
const int64_t initial_other_size = Size(Key(n), Key(kCount));
// Read until size drops significantly.
std::string limit_key = Key(n);
for (int read = 0; true; read++) {
ASSERT_LT(read, 100) << "Taking too long to compact";
Iterator* iter = db_->NewIterator(ReadOptions());
for (iter->SeekToFirst();
iter->Valid() && iter->key().ToString() < limit_key;
iter->Next()) {
// Drop data
}
delete iter;
// Wait a little bit to allow any triggered compactions to complete.
Env::Default()->SleepForMicroseconds(1000000);
uint64_t size = Size(Key(0), Key(n));
fprintf(stderr, "iter %3d => %7.3f MB [other %7.3f MB]\n",
read+1, size/1048576.0, Size(Key(n), Key(kCount))/1048576.0);
if (size <= initial_size/10) {
break;
}
}
// Verify that the size of the key space not touched by the reads
// is pretty much unchanged.
const int64_t final_other_size = Size(Key(n), Key(kCount));
ASSERT_LE(final_other_size, initial_other_size + 1048576);
ASSERT_GE(final_other_size, initial_other_size/5 - 1048576);
}
TEST(AutoCompactTest, ReadAll) {
DoReads(kCount);
}
TEST(AutoCompactTest, ReadHalf) {
DoReads(kCount/2);
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/builder.cc
+88
View File
@@ -0,0 +1,88 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/builder.h"
#include "db/filename.h"
#include "db/dbformat.h"
#include "db/table_cache.h"
#include "db/version_edit.h"
#include "leveldb/db.h"
#include "leveldb/env.h"
#include "leveldb/iterator.h"
namespace leveldb {
Status BuildTable(const std::string& dbname,
Env* env,
const Options& options,
TableCache* table_cache,
Iterator* iter,
FileMetaData* meta) {
Status s;
meta->file_size = 0;
iter->SeekToFirst();
std::string fname = TableFileName(dbname, meta->number);
if (iter->Valid()) {
WritableFile* file;
s = env->NewWritableFile(fname, &file);
if (!s.ok()) {
return s;
}
TableBuilder* builder = new TableBuilder(options, file);
meta->smallest.DecodeFrom(iter->key());
for (; iter->Valid(); iter->Next()) {
Slice key = iter->key();
meta->largest.DecodeFrom(key);
builder->Add(key, iter->value());
}
// Finish and check for builder errors
if (s.ok()) {
s = builder->Finish();
if (s.ok()) {
meta->file_size = builder->FileSize();
assert(meta->file_size > 0);
}
} else {
builder->Abandon();
}
delete builder;
// Finish and check for file errors
if (s.ok()) {
s = file->Sync();
}
if (s.ok()) {
s = file->Close();
}
delete file;
file = NULL;
if (s.ok()) {
// Verify that the table is usable
Iterator* it = table_cache->NewIterator(ReadOptions(),
meta->number,
meta->file_size);
s = it->status();
delete it;
}
}
// Check for input iterator errors
if (!iter->status().ok()) {
s = iter->status();
}
if (s.ok() && meta->file_size > 0) {
// Keep it
} else {
env->DeleteFile(fname);
}
return s;
}
} // namespace leveldb
src/leveldb/db/builder.h
+34
View File
@@ -0,0 +1,34 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_BUILDER_H_
#define STORAGE_LEVELDB_DB_BUILDER_H_
#include "leveldb/status.h"
namespace leveldb {
struct Options;
struct FileMetaData;
class Env;
class Iterator;
class TableCache;
class VersionEdit;
// Build a Table file from the contents of *iter. The generated file
// will be named according to meta->number. On success, the rest of
// *meta will be filled with metadata about the generated table.
// If no data is present in *iter, meta->file_size will be set to
// zero, and no Table file will be produced.
extern Status BuildTable(const std::string& dbname,
Env* env,
const Options& options,
TableCache* table_cache,
Iterator* iter,
FileMetaData* meta);
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_BUILDER_H_
src/leveldb/db/c.cc
+595
View File
@@ -0,0 +1,595 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "leveldb/c.h"
#include <stdlib.h>
#include <unistd.h>
#include "leveldb/cache.h"
#include "leveldb/comparator.h"
#include "leveldb/db.h"
#include "leveldb/env.h"
#include "leveldb/filter_policy.h"
#include "leveldb/iterator.h"
#include "leveldb/options.h"
#include "leveldb/status.h"
#include "leveldb/write_batch.h"
using leveldb::Cache;
using leveldb::Comparator;
using leveldb::CompressionType;
using leveldb::DB;
using leveldb::Env;
using leveldb::FileLock;
using leveldb::FilterPolicy;
using leveldb::Iterator;
using leveldb::kMajorVersion;
using leveldb::kMinorVersion;
using leveldb::Logger;
using leveldb::NewBloomFilterPolicy;
using leveldb::NewLRUCache;
using leveldb::Options;
using leveldb::RandomAccessFile;
using leveldb::Range;
using leveldb::ReadOptions;
using leveldb::SequentialFile;
using leveldb::Slice;
using leveldb::Snapshot;
using leveldb::Status;
using leveldb::WritableFile;
using leveldb::WriteBatch;
using leveldb::WriteOptions;
extern "C" {
struct leveldb_t { DB* rep; };
struct leveldb_iterator_t { Iterator* rep; };
struct leveldb_writebatch_t { WriteBatch rep; };
struct leveldb_snapshot_t { const Snapshot* rep; };
struct leveldb_readoptions_t { ReadOptions rep; };
struct leveldb_writeoptions_t { WriteOptions rep; };
struct leveldb_options_t { Options rep; };
struct leveldb_cache_t { Cache* rep; };
struct leveldb_seqfile_t { SequentialFile* rep; };
struct leveldb_randomfile_t { RandomAccessFile* rep; };
struct leveldb_writablefile_t { WritableFile* rep; };
struct leveldb_logger_t { Logger* rep; };
struct leveldb_filelock_t { FileLock* rep; };
struct leveldb_comparator_t : public Comparator {
void* state_;
void (*destructor_)(void*);
int (*compare_)(
void*,
const char* a, size_t alen,
const char* b, size_t blen);
const char* (*name_)(void*);
virtual ~leveldb_comparator_t() {
(*destructor_)(state_);
}
virtual int Compare(const Slice& a, const Slice& b) const {
return (*compare_)(state_, a.data(), a.size(), b.data(), b.size());
}
virtual const char* Name() const {
return (*name_)(state_);
}
// No-ops since the C binding does not support key shortening methods.
virtual void FindShortestSeparator(std::string*, const Slice&) const { }
virtual void FindShortSuccessor(std::string* key) const { }
};
struct leveldb_filterpolicy_t : public FilterPolicy {
void* state_;
void (*destructor_)(void*);
const char* (*name_)(void*);
char* (*create_)(
void*,
const char* const* key_array, const size_t* key_length_array,
int num_keys,
size_t* filter_length);
unsigned char (*key_match_)(
void*,
const char* key, size_t length,
const char* filter, size_t filter_length);
virtual ~leveldb_filterpolicy_t() {
(*destructor_)(state_);
}
virtual const char* Name() const {
return (*name_)(state_);
}
virtual void CreateFilter(const Slice* keys, int n, std::string* dst) const {
std::vector<const char*> key_pointers(n);
std::vector<size_t> key_sizes(n);
for (int i = 0; i < n; i++) {
key_pointers[i] = keys[i].data();
key_sizes[i] = keys[i].size();
}
size_t len;
char* filter = (*create_)(state_, &key_pointers[0], &key_sizes[0], n, &len);
dst->append(filter, len);
free(filter);
}
virtual bool KeyMayMatch(const Slice& key, const Slice& filter) const {
return (*key_match_)(state_, key.data(), key.size(),
filter.data(), filter.size());
}
};
struct leveldb_env_t {
Env* rep;
bool is_default;
};
static bool SaveError(char** errptr, const Status& s) {
assert(errptr != NULL);
if (s.ok()) {
return false;
} else if (*errptr == NULL) {
*errptr = strdup(s.ToString().c_str());
} else {
// TODO(sanjay): Merge with existing error?
free(*errptr);
*errptr = strdup(s.ToString().c_str());
}
return true;
}
static char* CopyString(const std::string& str) {
char* result = reinterpret_cast<char*>(malloc(sizeof(char) * str.size()));
memcpy(result, str.data(), sizeof(char) * str.size());
return result;
}
leveldb_t* leveldb_open(
const leveldb_options_t* options,
const char* name,
char** errptr) {
DB* db;
if (SaveError(errptr, DB::Open(options->rep, std::string(name), &db))) {
return NULL;
}
leveldb_t* result = new leveldb_t;
result->rep = db;
return result;
}
void leveldb_close(leveldb_t* db) {
delete db->rep;
delete db;
}
void leveldb_put(
leveldb_t* db,
const leveldb_writeoptions_t* options,
const char* key, size_t keylen,
const char* val, size_t vallen,
char** errptr) {
SaveError(errptr,
db->rep->Put(options->rep, Slice(key, keylen), Slice(val, vallen)));
}
void leveldb_delete(
leveldb_t* db,
const leveldb_writeoptions_t* options,
const char* key, size_t keylen,
char** errptr) {
SaveError(errptr, db->rep->Delete(options->rep, Slice(key, keylen)));
}
void leveldb_write(
leveldb_t* db,
const leveldb_writeoptions_t* options,
leveldb_writebatch_t* batch,
char** errptr) {
SaveError(errptr, db->rep->Write(options->rep, &batch->rep));
}
char* leveldb_get(
leveldb_t* db,
const leveldb_readoptions_t* options,
const char* key, size_t keylen,
size_t* vallen,
char** errptr) {
char* result = NULL;
std::string tmp;
Status s = db->rep->Get(options->rep, Slice(key, keylen), &tmp);
if (s.ok()) {
*vallen = tmp.size();
result = CopyString(tmp);
} else {
*vallen = 0;
if (!s.IsNotFound()) {
SaveError(errptr, s);
}
}
return result;
}
leveldb_iterator_t* leveldb_create_iterator(
leveldb_t* db,
const leveldb_readoptions_t* options) {
leveldb_iterator_t* result = new leveldb_iterator_t;
result->rep = db->rep->NewIterator(options->rep);
return result;
}
const leveldb_snapshot_t* leveldb_create_snapshot(
leveldb_t* db) {
leveldb_snapshot_t* result = new leveldb_snapshot_t;
result->rep = db->rep->GetSnapshot();
return result;
}
void leveldb_release_snapshot(
leveldb_t* db,
const leveldb_snapshot_t* snapshot) {
db->rep->ReleaseSnapshot(snapshot->rep);
delete snapshot;
}
char* leveldb_property_value(
leveldb_t* db,
const char* propname) {
std::string tmp;
if (db->rep->GetProperty(Slice(propname), &tmp)) {
// We use strdup() since we expect human readable output.
return strdup(tmp.c_str());
} else {
return NULL;
}
}
void leveldb_approximate_sizes(
leveldb_t* db,
int num_ranges,
const char* const* range_start_key, const size_t* range_start_key_len,
const char* const* range_limit_key, const size_t* range_limit_key_len,
uint64_t* sizes) {
Range* ranges = new Range[num_ranges];
for (int i = 0; i < num_ranges; i++) {
ranges[i].start = Slice(range_start_key[i], range_start_key_len[i]);
ranges[i].limit = Slice(range_limit_key[i], range_limit_key_len[i]);
}
db->rep->GetApproximateSizes(ranges, num_ranges, sizes);
delete[] ranges;
}
void leveldb_compact_range(
leveldb_t* db,
const char* start_key, size_t start_key_len,
const char* limit_key, size_t limit_key_len) {
Slice a, b;
db->rep->CompactRange(
// Pass NULL Slice if corresponding "const char*" is NULL
(start_key ? (a = Slice(start_key, start_key_len), &a) : NULL),
(limit_key ? (b = Slice(limit_key, limit_key_len), &b) : NULL));
}
void leveldb_destroy_db(
const leveldb_options_t* options,
const char* name,
char** errptr) {
SaveError(errptr, DestroyDB(name, options->rep));
}
void leveldb_repair_db(
const leveldb_options_t* options,
const char* name,
char** errptr) {
SaveError(errptr, RepairDB(name, options->rep));
}
void leveldb_iter_destroy(leveldb_iterator_t* iter) {
delete iter->rep;
delete iter;
}
unsigned char leveldb_iter_valid(const leveldb_iterator_t* iter) {
return iter->rep->Valid();
}
void leveldb_iter_seek_to_first(leveldb_iterator_t* iter) {
iter->rep->SeekToFirst();
}
void leveldb_iter_seek_to_last(leveldb_iterator_t* iter) {
iter->rep->SeekToLast();
}
void leveldb_iter_seek(leveldb_iterator_t* iter, const char* k, size_t klen) {
iter->rep->Seek(Slice(k, klen));
}
void leveldb_iter_next(leveldb_iterator_t* iter) {
iter->rep->Next();
}
void leveldb_iter_prev(leveldb_iterator_t* iter) {
iter->rep->Prev();
}
const char* leveldb_iter_key(const leveldb_iterator_t* iter, size_t* klen) {
Slice s = iter->rep->key();
*klen = s.size();
return s.data();
}
const char* leveldb_iter_value(const leveldb_iterator_t* iter, size_t* vlen) {
Slice s = iter->rep->value();
*vlen = s.size();
return s.data();
}
void leveldb_iter_get_error(const leveldb_iterator_t* iter, char** errptr) {
SaveError(errptr, iter->rep->status());
}
leveldb_writebatch_t* leveldb_writebatch_create() {
return new leveldb_writebatch_t;
}
void leveldb_writebatch_destroy(leveldb_writebatch_t* b) {
delete b;
}
void leveldb_writebatch_clear(leveldb_writebatch_t* b) {
b->rep.Clear();
}
void leveldb_writebatch_put(
leveldb_writebatch_t* b,
const char* key, size_t klen,
const char* val, size_t vlen) {
b->rep.Put(Slice(key, klen), Slice(val, vlen));
}
void leveldb_writebatch_delete(
leveldb_writebatch_t* b,
const char* key, size_t klen) {
b->rep.Delete(Slice(key, klen));
}
void leveldb_writebatch_iterate(
leveldb_writebatch_t* b,
void* state,
void (*put)(void*, const char* k, size_t klen, const char* v, size_t vlen),
void (*deleted)(void*, const char* k, size_t klen)) {
class H : public WriteBatch::Handler {
public:
void* state_;
void (*put_)(void*, const char* k, size_t klen, const char* v, size_t vlen);
void (*deleted_)(void*, const char* k, size_t klen);
virtual void Put(const Slice& key, const Slice& value) {
(*put_)(state_, key.data(), key.size(), value.data(), value.size());
}
virtual void Delete(const Slice& key) {
(*deleted_)(state_, key.data(), key.size());
}
};
H handler;
handler.state_ = state;
handler.put_ = put;
handler.deleted_ = deleted;
b->rep.Iterate(&handler);
}
leveldb_options_t* leveldb_options_create() {
return new leveldb_options_t;
}
void leveldb_options_destroy(leveldb_options_t* options) {
delete options;
}
void leveldb_options_set_comparator(
leveldb_options_t* opt,
leveldb_comparator_t* cmp) {
opt->rep.comparator = cmp;
}
void leveldb_options_set_filter_policy(
leveldb_options_t* opt,
leveldb_filterpolicy_t* policy) {
opt->rep.filter_policy = policy;
}
void leveldb_options_set_create_if_missing(
leveldb_options_t* opt, unsigned char v) {
opt->rep.create_if_missing = v;
}
void leveldb_options_set_error_if_exists(
leveldb_options_t* opt, unsigned char v) {
opt->rep.error_if_exists = v;
}
void leveldb_options_set_paranoid_checks(
leveldb_options_t* opt, unsigned char v) {
opt->rep.paranoid_checks = v;
}
void leveldb_options_set_env(leveldb_options_t* opt, leveldb_env_t* env) {
opt->rep.env = (env ? env->rep : NULL);
}
void leveldb_options_set_info_log(leveldb_options_t* opt, leveldb_logger_t* l) {
opt->rep.info_log = (l ? l->rep : NULL);
}
void leveldb_options_set_write_buffer_size(leveldb_options_t* opt, size_t s) {
opt->rep.write_buffer_size = s;
}
void leveldb_options_set_max_open_files(leveldb_options_t* opt, int n) {
opt->rep.max_open_files = n;
}
void leveldb_options_set_cache(leveldb_options_t* opt, leveldb_cache_t* c) {
opt->rep.block_cache = c->rep;
}
void leveldb_options_set_block_size(leveldb_options_t* opt, size_t s) {
opt->rep.block_size = s;
}
void leveldb_options_set_block_restart_interval(leveldb_options_t* opt, int n) {
opt->rep.block_restart_interval = n;
}
void leveldb_options_set_compression(leveldb_options_t* opt, int t) {
opt->rep.compression = static_cast<CompressionType>(t);
}
leveldb_comparator_t* leveldb_comparator_create(
void* state,
void (*destructor)(void*),
int (*compare)(
void*,
const char* a, size_t alen,
const char* b, size_t blen),
const char* (*name)(void*)) {
leveldb_comparator_t* result = new leveldb_comparator_t;
result->state_ = state;
result->destructor_ = destructor;
result->compare_ = compare;
result->name_ = name;
return result;
}
void leveldb_comparator_destroy(leveldb_comparator_t* cmp) {
delete cmp;
}
leveldb_filterpolicy_t* leveldb_filterpolicy_create(
void* state,
void (*destructor)(void*),
char* (*create_filter)(
void*,
const char* const* key_array, const size_t* key_length_array,
int num_keys,
size_t* filter_length),
unsigned char (*key_may_match)(
void*,
const char* key, size_t length,
const char* filter, size_t filter_length),
const char* (*name)(void*)) {
leveldb_filterpolicy_t* result = new leveldb_filterpolicy_t;
result->state_ = state;
result->destructor_ = destructor;
result->create_ = create_filter;
result->key_match_ = key_may_match;
result->name_ = name;
return result;
}
void leveldb_filterpolicy_destroy(leveldb_filterpolicy_t* filter) {
delete filter;
}
leveldb_filterpolicy_t* leveldb_filterpolicy_create_bloom(int bits_per_key) {
// Make a leveldb_filterpolicy_t, but override all of its methods so
// they delegate to a NewBloomFilterPolicy() instead of user
// supplied C functions.
struct Wrapper : public leveldb_filterpolicy_t {
const FilterPolicy* rep_;
~Wrapper() { delete rep_; }
const char* Name() const { return rep_->Name(); }
void CreateFilter(const Slice* keys, int n, std::string* dst) const {
return rep_->CreateFilter(keys, n, dst);
}
bool KeyMayMatch(const Slice& key, const Slice& filter) const {
return rep_->KeyMayMatch(key, filter);
}
static void DoNothing(void*) { }
};
Wrapper* wrapper = new Wrapper;
wrapper->rep_ = NewBloomFilterPolicy(bits_per_key);
wrapper->state_ = NULL;
wrapper->destructor_ = &Wrapper::DoNothing;
return wrapper;
}
leveldb_readoptions_t* leveldb_readoptions_create() {
return new leveldb_readoptions_t;
}
void leveldb_readoptions_destroy(leveldb_readoptions_t* opt) {
delete opt;
}
void leveldb_readoptions_set_verify_checksums(
leveldb_readoptions_t* opt,
unsigned char v) {
opt->rep.verify_checksums = v;
}
void leveldb_readoptions_set_fill_cache(
leveldb_readoptions_t* opt, unsigned char v) {
opt->rep.fill_cache = v;
}
void leveldb_readoptions_set_snapshot(
leveldb_readoptions_t* opt,
const leveldb_snapshot_t* snap) {
opt->rep.snapshot = (snap ? snap->rep : NULL);
}
leveldb_writeoptions_t* leveldb_writeoptions_create() {
return new leveldb_writeoptions_t;
}
void leveldb_writeoptions_destroy(leveldb_writeoptions_t* opt) {
delete opt;
}
void leveldb_writeoptions_set_sync(
leveldb_writeoptions_t* opt, unsigned char v) {
opt->rep.sync = v;
}
leveldb_cache_t* leveldb_cache_create_lru(size_t capacity) {
leveldb_cache_t* c = new leveldb_cache_t;
c->rep = NewLRUCache(capacity);
return c;
}
void leveldb_cache_destroy(leveldb_cache_t* cache) {
delete cache->rep;
delete cache;
}
leveldb_env_t* leveldb_create_default_env() {
leveldb_env_t* result = new leveldb_env_t;
result->rep = Env::Default();
result->is_default = true;
return result;
}
void leveldb_env_destroy(leveldb_env_t* env) {
if (!env->is_default) delete env->rep;
delete env;
}
void leveldb_free(void* ptr) {
free(ptr);
}
int leveldb_major_version() {
return kMajorVersion;
}
int leveldb_minor_version() {
return kMinorVersion;
}
} // end extern "C"
src/leveldb/db/c_test.c
+390
View File
@@ -0,0 +1,390 @@
/* Copyright (c) 2011 The LevelDB Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file. See the AUTHORS file for names of contributors. */
#include "leveldb/c.h"
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <unistd.h>
const char* phase = "";
static char dbname[200];
static void StartPhase(const char* name) {
fprintf(stderr, "=== Test %s\n", name);
phase = name;
}
static const char* GetTempDir(void) {
const char* ret = getenv("TEST_TMPDIR");
if (ret == NULL || ret[0] == '\0')
ret = "/tmp";
return ret;
}
#define CheckNoError(err) \
if ((err) != NULL) { \
fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, __LINE__, phase, (err)); \
abort(); \
}
#define CheckCondition(cond) \
if (!(cond)) { \
fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, __LINE__, phase, #cond); \
abort(); \
}
static void CheckEqual(const char* expected, const char* v, size_t n) {
if (expected == NULL && v == NULL) {
// ok
} else if (expected != NULL && v != NULL && n == strlen(expected) &&
memcmp(expected, v, n) == 0) {
// ok
return;
} else {
fprintf(stderr, "%s: expected '%s', got '%s'\n",
phase,
(expected ? expected : "(null)"),
(v ? v : "(null"));
abort();
}
}
static void Free(char** ptr) {
if (*ptr) {
free(*ptr);
*ptr = NULL;
}
}
static void CheckGet(
leveldb_t* db,
const leveldb_readoptions_t* options,
const char* key,
const char* expected) {
char* err = NULL;
size_t val_len;
char* val;
val = leveldb_get(db, options, key, strlen(key), &val_len, &err);
CheckNoError(err);
CheckEqual(expected, val, val_len);
Free(&val);
}
static void CheckIter(leveldb_iterator_t* iter,
const char* key, const char* val) {
size_t len;
const char* str;
str = leveldb_iter_key(iter, &len);
CheckEqual(key, str, len);
str = leveldb_iter_value(iter, &len);
CheckEqual(val, str, len);
}
// Callback from leveldb_writebatch_iterate()
static void CheckPut(void* ptr,
const char* k, size_t klen,
const char* v, size_t vlen) {
int* state = (int*) ptr;
CheckCondition(*state < 2);
switch (*state) {
case 0:
CheckEqual("bar", k, klen);
CheckEqual("b", v, vlen);
break;
case 1:
CheckEqual("box", k, klen);
CheckEqual("c", v, vlen);
break;
}
(*state)++;
}
// Callback from leveldb_writebatch_iterate()
static void CheckDel(void* ptr, const char* k, size_t klen) {
int* state = (int*) ptr;
CheckCondition(*state == 2);
CheckEqual("bar", k, klen);
(*state)++;
}
static void CmpDestroy(void* arg) { }
static int CmpCompare(void* arg, const char* a, size_t alen,
const char* b, size_t blen) {
int n = (alen < blen) ? alen : blen;
int r = memcmp(a, b, n);
if (r == 0) {
if (alen < blen) r = -1;
else if (alen > blen) r = +1;
}
return r;
}
static const char* CmpName(void* arg) {
return "foo";
}
// Custom filter policy
static unsigned char fake_filter_result = 1;
static void FilterDestroy(void* arg) { }
static const char* FilterName(void* arg) {
return "TestFilter";
}
static char* FilterCreate(
void* arg,
const char* const* key_array, const size_t* key_length_array,
int num_keys,
size_t* filter_length) {
*filter_length = 4;
char* result = malloc(4);
memcpy(result, "fake", 4);
return result;
}
unsigned char FilterKeyMatch(
void* arg,
const char* key, size_t length,
const char* filter, size_t filter_length) {
CheckCondition(filter_length == 4);
CheckCondition(memcmp(filter, "fake", 4) == 0);
return fake_filter_result;
}
int main(int argc, char** argv) {
leveldb_t* db;
leveldb_comparator_t* cmp;
leveldb_cache_t* cache;
leveldb_env_t* env;
leveldb_options_t* options;
leveldb_readoptions_t* roptions;
leveldb_writeoptions_t* woptions;
char* err = NULL;
int run = -1;
CheckCondition(leveldb_major_version() >= 1);
CheckCondition(leveldb_minor_version() >= 1);
snprintf(dbname, sizeof(dbname),
"%s/leveldb_c_test-%d",
GetTempDir(),
((int) geteuid()));
StartPhase("create_objects");
cmp = leveldb_comparator_create(NULL, CmpDestroy, CmpCompare, CmpName);
env = leveldb_create_default_env();
cache = leveldb_cache_create_lru(100000);
options = leveldb_options_create();
leveldb_options_set_comparator(options, cmp);
leveldb_options_set_error_if_exists(options, 1);
leveldb_options_set_cache(options, cache);
leveldb_options_set_env(options, env);
leveldb_options_set_info_log(options, NULL);
leveldb_options_set_write_buffer_size(options, 100000);
leveldb_options_set_paranoid_checks(options, 1);
leveldb_options_set_max_open_files(options, 10);
leveldb_options_set_block_size(options, 1024);
leveldb_options_set_block_restart_interval(options, 8);
leveldb_options_set_compression(options, leveldb_no_compression);
roptions = leveldb_readoptions_create();
leveldb_readoptions_set_verify_checksums(roptions, 1);
leveldb_readoptions_set_fill_cache(roptions, 0);
woptions = leveldb_writeoptions_create();
leveldb_writeoptions_set_sync(woptions, 1);
StartPhase("destroy");
leveldb_destroy_db(options, dbname, &err);
Free(&err);
StartPhase("open_error");
db = leveldb_open(options, dbname, &err);
CheckCondition(err != NULL);
Free(&err);
StartPhase("leveldb_free");
db = leveldb_open(options, dbname, &err);
CheckCondition(err != NULL);
leveldb_free(err);
err = NULL;
StartPhase("open");
leveldb_options_set_create_if_missing(options, 1);
db = leveldb_open(options, dbname, &err);
CheckNoError(err);
CheckGet(db, roptions, "foo", NULL);
StartPhase("put");
leveldb_put(db, woptions, "foo", 3, "hello", 5, &err);
CheckNoError(err);
CheckGet(db, roptions, "foo", "hello");
StartPhase("compactall");
leveldb_compact_range(db, NULL, 0, NULL, 0);
CheckGet(db, roptions, "foo", "hello");
StartPhase("compactrange");
leveldb_compact_range(db, "a", 1, "z", 1);
CheckGet(db, roptions, "foo", "hello");
StartPhase("writebatch");
{
leveldb_writebatch_t* wb = leveldb_writebatch_create();
leveldb_writebatch_put(wb, "foo", 3, "a", 1);
leveldb_writebatch_clear(wb);
leveldb_writebatch_put(wb, "bar", 3, "b", 1);
leveldb_writebatch_put(wb, "box", 3, "c", 1);
leveldb_writebatch_delete(wb, "bar", 3);
leveldb_write(db, woptions, wb, &err);
CheckNoError(err);
CheckGet(db, roptions, "foo", "hello");
CheckGet(db, roptions, "bar", NULL);
CheckGet(db, roptions, "box", "c");
int pos = 0;
leveldb_writebatch_iterate(wb, &pos, CheckPut, CheckDel);
CheckCondition(pos == 3);
leveldb_writebatch_destroy(wb);
}
StartPhase("iter");
{
leveldb_iterator_t* iter = leveldb_create_iterator(db, roptions);
CheckCondition(!leveldb_iter_valid(iter));
leveldb_iter_seek_to_first(iter);
CheckCondition(leveldb_iter_valid(iter));
CheckIter(iter, "box", "c");
leveldb_iter_next(iter);
CheckIter(iter, "foo", "hello");
leveldb_iter_prev(iter);
CheckIter(iter, "box", "c");
leveldb_iter_prev(iter);
CheckCondition(!leveldb_iter_valid(iter));
leveldb_iter_seek_to_last(iter);
CheckIter(iter, "foo", "hello");
leveldb_iter_seek(iter, "b", 1);
CheckIter(iter, "box", "c");
leveldb_iter_get_error(iter, &err);
CheckNoError(err);
leveldb_iter_destroy(iter);
}
StartPhase("approximate_sizes");
{
int i;
int n = 20000;
char keybuf[100];
char valbuf[100];
uint64_t sizes[2];
const char* start[2] = { "a", "k00000000000000010000" };
size_t start_len[2] = { 1, 21 };
const char* limit[2] = { "k00000000000000010000", "z" };
size_t limit_len[2] = { 21, 1 };
leveldb_writeoptions_set_sync(woptions, 0);
for (i = 0; i < n; i++) {
snprintf(keybuf, sizeof(keybuf), "k%020d", i);
snprintf(valbuf, sizeof(valbuf), "v%020d", i);
leveldb_put(db, woptions, keybuf, strlen(keybuf), valbuf, strlen(valbuf),
&err);
CheckNoError(err);
}
leveldb_approximate_sizes(db, 2, start, start_len, limit, limit_len, sizes);
CheckCondition(sizes[0] > 0);
CheckCondition(sizes[1] > 0);
}
StartPhase("property");
{
char* prop = leveldb_property_value(db, "nosuchprop");
CheckCondition(prop == NULL);
prop = leveldb_property_value(db, "leveldb.stats");
CheckCondition(prop != NULL);
Free(&prop);
}
StartPhase("snapshot");
{
const leveldb_snapshot_t* snap;
snap = leveldb_create_snapshot(db);
leveldb_delete(db, woptions, "foo", 3, &err);
CheckNoError(err);
leveldb_readoptions_set_snapshot(roptions, snap);
CheckGet(db, roptions, "foo", "hello");
leveldb_readoptions_set_snapshot(roptions, NULL);
CheckGet(db, roptions, "foo", NULL);
leveldb_release_snapshot(db, snap);
}
StartPhase("repair");
{
leveldb_close(db);
leveldb_options_set_create_if_missing(options, 0);
leveldb_options_set_error_if_exists(options, 0);
leveldb_repair_db(options, dbname, &err);
CheckNoError(err);
db = leveldb_open(options, dbname, &err);
CheckNoError(err);
CheckGet(db, roptions, "foo", NULL);
CheckGet(db, roptions, "bar", NULL);
CheckGet(db, roptions, "box", "c");
leveldb_options_set_create_if_missing(options, 1);
leveldb_options_set_error_if_exists(options, 1);
}
StartPhase("filter");
for (run = 0; run < 2; run++) {
// First run uses custom filter, second run uses bloom filter
CheckNoError(err);
leveldb_filterpolicy_t* policy;
if (run == 0) {
policy = leveldb_filterpolicy_create(
NULL, FilterDestroy, FilterCreate, FilterKeyMatch, FilterName);
} else {
policy = leveldb_filterpolicy_create_bloom(10);
}
// Create new database
leveldb_close(db);
leveldb_destroy_db(options, dbname, &err);
leveldb_options_set_filter_policy(options, policy);
db = leveldb_open(options, dbname, &err);
CheckNoError(err);
leveldb_put(db, woptions, "foo", 3, "foovalue", 8, &err);
CheckNoError(err);
leveldb_put(db, woptions, "bar", 3, "barvalue", 8, &err);
CheckNoError(err);
leveldb_compact_range(db, NULL, 0, NULL, 0);
fake_filter_result = 1;
CheckGet(db, roptions, "foo", "foovalue");
CheckGet(db, roptions, "bar", "barvalue");
if (phase == 0) {
// Must not find value when custom filter returns false
fake_filter_result = 0;
CheckGet(db, roptions, "foo", NULL);
CheckGet(db, roptions, "bar", NULL);
fake_filter_result = 1;
CheckGet(db, roptions, "foo", "foovalue");
CheckGet(db, roptions, "bar", "barvalue");
}
leveldb_options_set_filter_policy(options, NULL);
leveldb_filterpolicy_destroy(policy);
}
StartPhase("cleanup");
leveldb_close(db);
leveldb_options_destroy(options);
leveldb_readoptions_destroy(roptions);
leveldb_writeoptions_destroy(woptions);
leveldb_cache_destroy(cache);
leveldb_comparator_destroy(cmp);
leveldb_env_destroy(env);
fprintf(stderr, "PASS\n");
return 0;
}
src/leveldb/db/corruption_test.cc
+374
View File
@@ -0,0 +1,374 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "leveldb/db.h"
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include "leveldb/cache.h"
#include "leveldb/env.h"
#include "leveldb/table.h"
#include "leveldb/write_batch.h"
#include "db/db_impl.h"
#include "db/filename.h"
#include "db/log_format.h"
#include "db/version_set.h"
#include "util/logging.h"
#include "util/testharness.h"
#include "util/testutil.h"
namespace leveldb {
static const int kValueSize = 1000;
class CorruptionTest {
public:
test::ErrorEnv env_;
std::string dbname_;
Cache* tiny_cache_;
Options options_;
DB* db_;
CorruptionTest() {
tiny_cache_ = NewLRUCache(100);
options_.env = &env_;
options_.block_cache = tiny_cache_;
dbname_ = test::TmpDir() + "/db_test";
DestroyDB(dbname_, options_);
db_ = NULL;
options_.create_if_missing = true;
Reopen();
options_.create_if_missing = false;
}
~CorruptionTest() {
delete db_;
DestroyDB(dbname_, Options());
delete tiny_cache_;
}
Status TryReopen() {
delete db_;
db_ = NULL;
return DB::Open(options_, dbname_, &db_);
}
void Reopen() {
ASSERT_OK(TryReopen());
}
void RepairDB() {
delete db_;
db_ = NULL;
ASSERT_OK(::leveldb::RepairDB(dbname_, options_));
}
void Build(int n) {
std::string key_space, value_space;
WriteBatch batch;
for (int i = 0; i < n; i++) {
//if ((i % 100) == 0) fprintf(stderr, "@ %d of %d\n", i, n);
Slice key = Key(i, &key_space);
batch.Clear();
batch.Put(key, Value(i, &value_space));
WriteOptions options;
// Corrupt() doesn't work without this sync on windows; stat reports 0 for
// the file size.
if (i == n - 1) {
options.sync = true;
}
ASSERT_OK(db_->Write(options, &batch));
}
}
void Check(int min_expected, int max_expected) {
int next_expected = 0;
int missed = 0;
int bad_keys = 0;
int bad_values = 0;
int correct = 0;
std::string value_space;
Iterator* iter = db_->NewIterator(ReadOptions());
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
uint64_t key;
Slice in(iter->key());
if (in == "" || in == "~") {
// Ignore boundary keys.
continue;
}
if (!ConsumeDecimalNumber(&in, &key) ||
!in.empty() ||
key < next_expected) {
bad_keys++;
continue;
}
missed += (key - next_expected);
next_expected = key + 1;
if (iter->value() != Value(key, &value_space)) {
bad_values++;
} else {
correct++;
}
}
delete iter;
fprintf(stderr,
"expected=%d..%d; got=%d; bad_keys=%d; bad_values=%d; missed=%d\n",
min_expected, max_expected, correct, bad_keys, bad_values, missed);
ASSERT_LE(min_expected, correct);
ASSERT_GE(max_expected, correct);
}
void Corrupt(FileType filetype, int offset, int bytes_to_corrupt) {
// Pick file to corrupt
std::vector<std::string> filenames;
ASSERT_OK(env_.GetChildren(dbname_, &filenames));
uint64_t number;
FileType type;
std::string fname;
int picked_number = -1;
for (size_t i = 0; i < filenames.size(); i++) {
if (ParseFileName(filenames[i], &number, &type) &&
type == filetype &&
int(number) > picked_number) { // Pick latest file
fname = dbname_ + "/" + filenames[i];
picked_number = number;
}
}
ASSERT_TRUE(!fname.empty()) << filetype;
struct stat sbuf;
if (stat(fname.c_str(), &sbuf) != 0) {
const char* msg = strerror(errno);
ASSERT_TRUE(false) << fname << ": " << msg;
}
if (offset < 0) {
// Relative to end of file; make it absolute
if (-offset > sbuf.st_size) {
offset = 0;
} else {
offset = sbuf.st_size + offset;
}
}
if (offset > sbuf.st_size) {
offset = sbuf.st_size;
}
if (offset + bytes_to_corrupt > sbuf.st_size) {
bytes_to_corrupt = sbuf.st_size - offset;
}
// Do it
std::string contents;
Status s = ReadFileToString(Env::Default(), fname, &contents);
ASSERT_TRUE(s.ok()) << s.ToString();
for (int i = 0; i < bytes_to_corrupt; i++) {
contents[i + offset] ^= 0x80;
}
s = WriteStringToFile(Env::Default(), contents, fname);
ASSERT_TRUE(s.ok()) << s.ToString();
}
int Property(const std::string& name) {
std::string property;
int result;
if (db_->GetProperty(name, &property) &&
sscanf(property.c_str(), "%d", &result) == 1) {
return result;
} else {
return -1;
}
}
// Return the ith key
Slice Key(int i, std::string* storage) {
char buf[100];
snprintf(buf, sizeof(buf), "%016d", i);
storage->assign(buf, strlen(buf));
return Slice(*storage);
}
// Return the value to associate with the specified key
Slice Value(int k, std::string* storage) {
Random r(k);
return test::RandomString(&r, kValueSize, storage);
}
};
TEST(CorruptionTest, Recovery) {
Build(100);
Check(100, 100);
Corrupt(kLogFile, 19, 1); // WriteBatch tag for first record
Corrupt(kLogFile, log::kBlockSize + 1000, 1); // Somewhere in second block
Reopen();
// The 64 records in the first two log blocks are completely lost.
Check(36, 36);
}
TEST(CorruptionTest, RecoverWriteError) {
env_.writable_file_error_ = true;
Status s = TryReopen();
ASSERT_TRUE(!s.ok());
}
TEST(CorruptionTest, NewFileErrorDuringWrite) {
// Do enough writing to force minor compaction
env_.writable_file_error_ = true;
const int num = 3 + (Options().write_buffer_size / kValueSize);
std::string value_storage;
Status s;
for (int i = 0; s.ok() && i < num; i++) {
WriteBatch batch;
batch.Put("a", Value(100, &value_storage));
s = db_->Write(WriteOptions(), &batch);
}
ASSERT_TRUE(!s.ok());
ASSERT_GE(env_.num_writable_file_errors_, 1);
env_.writable_file_error_ = false;
Reopen();
}
TEST(CorruptionTest, TableFile) {
Build(100);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_CompactMemTable();
dbi->TEST_CompactRange(0, NULL, NULL);
dbi->TEST_CompactRange(1, NULL, NULL);
Corrupt(kTableFile, 100, 1);
Check(90, 99);
}
TEST(CorruptionTest, TableFileRepair) {
options_.block_size = 2 * kValueSize; // Limit scope of corruption
options_.paranoid_checks = true;
Reopen();
Build(100);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_CompactMemTable();
dbi->TEST_CompactRange(0, NULL, NULL);
dbi->TEST_CompactRange(1, NULL, NULL);
Corrupt(kTableFile, 100, 1);
RepairDB();
Reopen();
Check(95, 99);
}
TEST(CorruptionTest, TableFileIndexData) {
Build(10000); // Enough to build multiple Tables
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_CompactMemTable();
Corrupt(kTableFile, -2000, 500);
Reopen();
Check(5000, 9999);
}
TEST(CorruptionTest, MissingDescriptor) {
Build(1000);
RepairDB();
Reopen();
Check(1000, 1000);
}
TEST(CorruptionTest, SequenceNumberRecovery) {
ASSERT_OK(db_->Put(WriteOptions(), "foo", "v1"));
ASSERT_OK(db_->Put(WriteOptions(), "foo", "v2"));
ASSERT_OK(db_->Put(WriteOptions(), "foo", "v3"));
ASSERT_OK(db_->Put(WriteOptions(), "foo", "v4"));
ASSERT_OK(db_->Put(WriteOptions(), "foo", "v5"));
RepairDB();
Reopen();
std::string v;
ASSERT_OK(db_->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("v5", v);
// Write something. If sequence number was not recovered properly,
// it will be hidden by an earlier write.
ASSERT_OK(db_->Put(WriteOptions(), "foo", "v6"));
ASSERT_OK(db_->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("v6", v);
Reopen();
ASSERT_OK(db_->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("v6", v);
}
TEST(CorruptionTest, CorruptedDescriptor) {
ASSERT_OK(db_->Put(WriteOptions(), "foo", "hello"));
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_CompactMemTable();
dbi->TEST_CompactRange(0, NULL, NULL);
Corrupt(kDescriptorFile, 0, 1000);
Status s = TryReopen();
ASSERT_TRUE(!s.ok());
RepairDB();
Reopen();
std::string v;
ASSERT_OK(db_->Get(ReadOptions(), "foo", &v));
ASSERT_EQ("hello", v);
}
TEST(CorruptionTest, CompactionInputError) {
Build(10);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_CompactMemTable();
const int last = config::kMaxMemCompactLevel;
ASSERT_EQ(1, Property("leveldb.num-files-at-level" + NumberToString(last)));
Corrupt(kTableFile, 100, 1);
Check(5, 9);
// Force compactions by writing lots of values
Build(10000);
Check(10000, 10000);
}
TEST(CorruptionTest, CompactionInputErrorParanoid) {
options_.paranoid_checks = true;
options_.write_buffer_size = 512 << 10;
Reopen();
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
// Make multiple inputs so we need to compact.
for (int i = 0; i < 2; i++) {
Build(10);
dbi->TEST_CompactMemTable();
Corrupt(kTableFile, 100, 1);
env_.SleepForMicroseconds(100000);
}
dbi->CompactRange(NULL, NULL);
// Write must fail because of corrupted table
std::string tmp1, tmp2;
Status s = db_->Put(WriteOptions(), Key(5, &tmp1), Value(5, &tmp2));
ASSERT_TRUE(!s.ok()) << "write did not fail in corrupted paranoid db";
}
TEST(CorruptionTest, UnrelatedKeys) {
Build(10);
DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
dbi->TEST_CompactMemTable();
Corrupt(kTableFile, 100, 1);
std::string tmp1, tmp2;
ASSERT_OK(db_->Put(WriteOptions(), Key(1000, &tmp1), Value(1000, &tmp2)));
std::string v;
ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
dbi->TEST_CompactMemTable();
ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/db_bench.cc
+979
View File
@@ -0,0 +1,979 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include "db/db_impl.h"
#include "db/version_set.h"
#include "leveldb/cache.h"
#include "leveldb/db.h"
#include "leveldb/env.h"
#include "leveldb/write_batch.h"
#include "port/port.h"
#include "util/crc32c.h"
#include "util/histogram.h"
#include "util/mutexlock.h"
#include "util/random.h"
#include "util/testutil.h"
// Comma-separated list of operations to run in the specified order
// Actual benchmarks:
// fillseq -- write N values in sequential key order in async mode
// fillrandom -- write N values in random key order in async mode
// overwrite -- overwrite N values in random key order in async mode
// fillsync -- write N/100 values in random key order in sync mode
// fill100K -- write N/1000 100K values in random order in async mode
// deleteseq -- delete N keys in sequential order
// deleterandom -- delete N keys in random order
// readseq -- read N times sequentially
// readreverse -- read N times in reverse order
// readrandom -- read N times in random order
// readmissing -- read N missing keys in random order
// readhot -- read N times in random order from 1% section of DB
// seekrandom -- N random seeks
// crc32c -- repeated crc32c of 4K of data
// acquireload -- load N*1000 times
// Meta operations:
// compact -- Compact the entire DB
// stats -- Print DB stats
// sstables -- Print sstable info
// heapprofile -- Dump a heap profile (if supported by this port)
static const char* FLAGS_benchmarks =
"fillseq,"
"fillsync,"
"fillrandom,"
"overwrite,"
"readrandom,"
"readrandom," // Extra run to allow previous compactions to quiesce
"readseq,"
"readreverse,"
"compact,"
"readrandom,"
"readseq,"
"readreverse,"
"fill100K,"
"crc32c,"
"snappycomp,"
"snappyuncomp,"
"acquireload,"
;
// Number of key/values to place in database
static int FLAGS_num = 1000000;
// Number of read operations to do. If negative, do FLAGS_num reads.
static int FLAGS_reads = -1;
// Number of concurrent threads to run.
static int FLAGS_threads = 1;
// Size of each value
static int FLAGS_value_size = 100;
// Arrange to generate values that shrink to this fraction of
// their original size after compression
static double FLAGS_compression_ratio = 0.5;
// Print histogram of operation timings
static bool FLAGS_histogram = false;
// Number of bytes to buffer in memtable before compacting
// (initialized to default value by "main")
static int FLAGS_write_buffer_size = 0;
// Number of bytes to use as a cache of uncompressed data.
// Negative means use default settings.
static int FLAGS_cache_size = -1;
// Maximum number of files to keep open at the same time (use default if == 0)
static int FLAGS_open_files = 0;
// Bloom filter bits per key.
// Negative means use default settings.
static int FLAGS_bloom_bits = -1;
// If true, do not destroy the existing database. If you set this
// flag and also specify a benchmark that wants a fresh database, that
// benchmark will fail.
static bool FLAGS_use_existing_db = false;
// Use the db with the following name.
static const char* FLAGS_db = NULL;
namespace leveldb {
namespace {
// Helper for quickly generating random data.
class RandomGenerator {
private:
std::string data_;
int pos_;
public:
RandomGenerator() {
// We use a limited amount of data over and over again and ensure
// that it is larger than the compression window (32KB), and also
// large enough to serve all typical value sizes we want to write.
Random rnd(301);
std::string piece;
while (data_.size() < 1048576) {
// Add a short fragment that is as compressible as specified
// by FLAGS_compression_ratio.
test::CompressibleString(&rnd, FLAGS_compression_ratio, 100, &piece);
data_.append(piece);
}
pos_ = 0;
}
Slice Generate(size_t len) {
if (pos_ + len > data_.size()) {
pos_ = 0;
assert(len < data_.size());
}
pos_ += len;
return Slice(data_.data() + pos_ - len, len);
}
};
static Slice TrimSpace(Slice s) {
size_t start = 0;
while (start < s.size() && isspace(s[start])) {
start++;
}
size_t limit = s.size();
while (limit > start && isspace(s[limit-1])) {
limit--;
}
return Slice(s.data() + start, limit - start);
}
static void AppendWithSpace(std::string* str, Slice msg) {
if (msg.empty()) return;
if (!str->empty()) {
str->push_back(' ');
}
str->append(msg.data(), msg.size());
}
class Stats {
private:
double start_;
double finish_;
double seconds_;
int done_;
int next_report_;
int64_t bytes_;
double last_op_finish_;
Histogram hist_;
std::string message_;
public:
Stats() { Start(); }
void Start() {
next_report_ = 100;
last_op_finish_ = start_;
hist_.Clear();
done_ = 0;
bytes_ = 0;
seconds_ = 0;
start_ = Env::Default()->NowMicros();
finish_ = start_;
message_.clear();
}
void Merge(const Stats& other) {
hist_.Merge(other.hist_);
done_ += other.done_;
bytes_ += other.bytes_;
seconds_ += other.seconds_;
if (other.start_ < start_) start_ = other.start_;
if (other.finish_ > finish_) finish_ = other.finish_;
// Just keep the messages from one thread
if (message_.empty()) message_ = other.message_;
}
void Stop() {
finish_ = Env::Default()->NowMicros();
seconds_ = (finish_ - start_) * 1e-6;
}
void AddMessage(Slice msg) {
AppendWithSpace(&message_, msg);
}
void FinishedSingleOp() {
if (FLAGS_histogram) {
double now = Env::Default()->NowMicros();
double micros = now - last_op_finish_;
hist_.Add(micros);
if (micros > 20000) {
fprintf(stderr, "long op: %.1f micros%30s\r", micros, "");
fflush(stderr);
}
last_op_finish_ = now;
}
done_++;
if (done_ >= next_report_) {
if (next_report_ < 1000) next_report_ += 100;
else if (next_report_ < 5000) next_report_ += 500;
else if (next_report_ < 10000) next_report_ += 1000;
else if (next_report_ < 50000) next_report_ += 5000;
else if (next_report_ < 100000) next_report_ += 10000;
else if (next_report_ < 500000) next_report_ += 50000;
else next_report_ += 100000;
fprintf(stderr, "... finished %d ops%30s\r", done_, "");
fflush(stderr);
}
}
void AddBytes(int64_t n) {
bytes_ += n;
}
void Report(const Slice& name) {
// Pretend at least one op was done in case we are running a benchmark
// that does not call FinishedSingleOp().
if (done_ < 1) done_ = 1;
std::string extra;
if (bytes_ > 0) {
// Rate is computed on actual elapsed time, not the sum of per-thread
// elapsed times.
double elapsed = (finish_ - start_) * 1e-6;
char rate[100];
snprintf(rate, sizeof(rate), "%6.1f MB/s",
(bytes_ / 1048576.0) / elapsed);
extra = rate;
}
AppendWithSpace(&extra, message_);
fprintf(stdout, "%-12s : %11.3f micros/op;%s%s\n",
name.ToString().c_str(),
seconds_ * 1e6 / done_,
(extra.empty() ? "" : " "),
extra.c_str());
if (FLAGS_histogram) {
fprintf(stdout, "Microseconds per op:\n%s\n", hist_.ToString().c_str());
}
fflush(stdout);
}
};
// State shared by all concurrent executions of the same benchmark.
struct SharedState {
port::Mutex mu;
port::CondVar cv;
int total;
// Each thread goes through the following states:
// (1) initializing
// (2) waiting for others to be initialized
// (3) running
// (4) done
int num_initialized;
int num_done;
bool start;
SharedState() : cv(&mu) { }
};
// Per-thread state for concurrent executions of the same benchmark.
struct ThreadState {
int tid; // 0..n-1 when running in n threads
Random rand; // Has different seeds for different threads
Stats stats;
SharedState* shared;
ThreadState(int index)
: tid(index),
rand(1000 + index) {
}
};
} // namespace
class Benchmark {
private:
Cache* cache_;
const FilterPolicy* filter_policy_;
DB* db_;
int num_;
int value_size_;
int entries_per_batch_;
WriteOptions write_options_;
int reads_;
int heap_counter_;
void PrintHeader() {
const int kKeySize = 16;
PrintEnvironment();
fprintf(stdout, "Keys: %d bytes each\n", kKeySize);
fprintf(stdout, "Values: %d bytes each (%d bytes after compression)\n",
FLAGS_value_size,
static_cast<int>(FLAGS_value_size * FLAGS_compression_ratio + 0.5));
fprintf(stdout, "Entries: %d\n", num_);
fprintf(stdout, "RawSize: %.1f MB (estimated)\n",
((static_cast<int64_t>(kKeySize + FLAGS_value_size) * num_)
/ 1048576.0));
fprintf(stdout, "FileSize: %.1f MB (estimated)\n",
(((kKeySize + FLAGS_value_size * FLAGS_compression_ratio) * num_)
/ 1048576.0));
PrintWarnings();
fprintf(stdout, "------------------------------------------------\n");
}
void PrintWarnings() {
#if defined(__GNUC__) && !defined(__OPTIMIZE__)
fprintf(stdout,
"WARNING: Optimization is disabled: benchmarks unnecessarily slow\n"
);
#endif
#ifndef NDEBUG
fprintf(stdout,
"WARNING: Assertions are enabled; benchmarks unnecessarily slow\n");
#endif
// See if snappy is working by attempting to compress a compressible string
const char text[] = "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy";
std::string compressed;
if (!port::Snappy_Compress(text, sizeof(text), &compressed)) {
fprintf(stdout, "WARNING: Snappy compression is not enabled\n");
} else if (compressed.size() >= sizeof(text)) {
fprintf(stdout, "WARNING: Snappy compression is not effective\n");
}
}
void PrintEnvironment() {
fprintf(stderr, "LevelDB: version %d.%d\n",
kMajorVersion, kMinorVersion);
#if defined(__linux)
time_t now = time(NULL);
fprintf(stderr, "Date: %s", ctime(&now)); // ctime() adds newline
FILE* cpuinfo = fopen("/proc/cpuinfo", "r");
if (cpuinfo != NULL) {
char line[1000];
int num_cpus = 0;
std::string cpu_type;
std::string cache_size;
while (fgets(line, sizeof(line), cpuinfo) != NULL) {
const char* sep = strchr(line, ':');
if (sep == NULL) {
continue;
}
Slice key = TrimSpace(Slice(line, sep - 1 - line));
Slice val = TrimSpace(Slice(sep + 1));
if (key == "model name") {
++num_cpus;
cpu_type = val.ToString();
} else if (key == "cache size") {
cache_size = val.ToString();
}
}
fclose(cpuinfo);
fprintf(stderr, "CPU: %d * %s\n", num_cpus, cpu_type.c_str());
fprintf(stderr, "CPUCache: %s\n", cache_size.c_str());
}
#endif
}
public:
Benchmark()
: cache_(FLAGS_cache_size >= 0 ? NewLRUCache(FLAGS_cache_size) : NULL),
filter_policy_(FLAGS_bloom_bits >= 0
? NewBloomFilterPolicy(FLAGS_bloom_bits)
: NULL),
db_(NULL),
num_(FLAGS_num),
value_size_(FLAGS_value_size),
entries_per_batch_(1),
reads_(FLAGS_reads < 0 ? FLAGS_num : FLAGS_reads),
heap_counter_(0) {
std::vector<std::string> files;
Env::Default()->GetChildren(FLAGS_db, &files);
for (size_t i = 0; i < files.size(); i++) {
if (Slice(files[i]).starts_with("heap-")) {
Env::Default()->DeleteFile(std::string(FLAGS_db) + "/" + files[i]);
}
}
if (!FLAGS_use_existing_db) {
DestroyDB(FLAGS_db, Options());
}
}
~Benchmark() {
delete db_;
delete cache_;
delete filter_policy_;
}
void Run() {
PrintHeader();
Open();
const char* benchmarks = FLAGS_benchmarks;
while (benchmarks != NULL) {
const char* sep = strchr(benchmarks, ',');
Slice name;
if (sep == NULL) {
name = benchmarks;
benchmarks = NULL;
} else {
name = Slice(benchmarks, sep - benchmarks);
benchmarks = sep + 1;
}
// Reset parameters that may be overriddden bwlow
num_ = FLAGS_num;
reads_ = (FLAGS_reads < 0 ? FLAGS_num : FLAGS_reads);
value_size_ = FLAGS_value_size;
entries_per_batch_ = 1;
write_options_ = WriteOptions();
void (Benchmark::*method)(ThreadState*) = NULL;
bool fresh_db = false;
int num_threads = FLAGS_threads;
if (name == Slice("fillseq")) {
fresh_db = true;
method = &Benchmark::WriteSeq;
} else if (name == Slice("fillbatch")) {
fresh_db = true;
entries_per_batch_ = 1000;
method = &Benchmark::WriteSeq;
} else if (name == Slice("fillrandom")) {
fresh_db = true;
method = &Benchmark::WriteRandom;
} else if (name == Slice("overwrite")) {
fresh_db = false;
method = &Benchmark::WriteRandom;
} else if (name == Slice("fillsync")) {
fresh_db = true;
num_ /= 1000;
write_options_.sync = true;
method = &Benchmark::WriteRandom;
} else if (name == Slice("fill100K")) {
fresh_db = true;
num_ /= 1000;
value_size_ = 100 * 1000;
method = &Benchmark::WriteRandom;
} else if (name == Slice("readseq")) {
method = &Benchmark::ReadSequential;
} else if (name == Slice("readreverse")) {
method = &Benchmark::ReadReverse;
} else if (name == Slice("readrandom")) {
method = &Benchmark::ReadRandom;
} else if (name == Slice("readmissing")) {
method = &Benchmark::ReadMissing;
} else if (name == Slice("seekrandom")) {
method = &Benchmark::SeekRandom;
} else if (name == Slice("readhot")) {
method = &Benchmark::ReadHot;
} else if (name == Slice("readrandomsmall")) {
reads_ /= 1000;
method = &Benchmark::ReadRandom;
} else if (name == Slice("deleteseq")) {
method = &Benchmark::DeleteSeq;
} else if (name == Slice("deleterandom")) {
method = &Benchmark::DeleteRandom;
} else if (name == Slice("readwhilewriting")) {
num_threads++; // Add extra thread for writing
method = &Benchmark::ReadWhileWriting;
} else if (name == Slice("compact")) {
method = &Benchmark::Compact;
} else if (name == Slice("crc32c")) {
method = &Benchmark::Crc32c;
} else if (name == Slice("acquireload")) {
method = &Benchmark::AcquireLoad;
} else if (name == Slice("snappycomp")) {
method = &Benchmark::SnappyCompress;
} else if (name == Slice("snappyuncomp")) {
method = &Benchmark::SnappyUncompress;
} else if (name == Slice("heapprofile")) {
HeapProfile();
} else if (name == Slice("stats")) {
PrintStats("leveldb.stats");
} else if (name == Slice("sstables")) {
PrintStats("leveldb.sstables");
} else {
if (name != Slice()) { // No error message for empty name
fprintf(stderr, "unknown benchmark '%s'\n", name.ToString().c_str());
}
}
if (fresh_db) {
if (FLAGS_use_existing_db) {
fprintf(stdout, "%-12s : skipped (--use_existing_db is true)\n",
name.ToString().c_str());
method = NULL;
} else {
delete db_;
db_ = NULL;
DestroyDB(FLAGS_db, Options());
Open();
}
}
if (method != NULL) {
RunBenchmark(num_threads, name, method);
}
}
}
private:
struct ThreadArg {
Benchmark* bm;
SharedState* shared;
ThreadState* thread;
void (Benchmark::*method)(ThreadState*);
};
static void ThreadBody(void* v) {
ThreadArg* arg = reinterpret_cast<ThreadArg*>(v);
SharedState* shared = arg->shared;
ThreadState* thread = arg->thread;
{
MutexLock l(&shared->mu);
shared->num_initialized++;
if (shared->num_initialized >= shared->total) {
shared->cv.SignalAll();
}
while (!shared->start) {
shared->cv.Wait();
}
}
thread->stats.Start();
(arg->bm->*(arg->method))(thread);
thread->stats.Stop();
{
MutexLock l(&shared->mu);
shared->num_done++;
if (shared->num_done >= shared->total) {
shared->cv.SignalAll();
}
}
}
void RunBenchmark(int n, Slice name,
void (Benchmark::*method)(ThreadState*)) {
SharedState shared;
shared.total = n;
shared.num_initialized = 0;
shared.num_done = 0;
shared.start = false;
ThreadArg* arg = new ThreadArg[n];
for (int i = 0; i < n; i++) {
arg[i].bm = this;
arg[i].method = method;
arg[i].shared = &shared;
arg[i].thread = new ThreadState(i);
arg[i].thread->shared = &shared;
Env::Default()->StartThread(ThreadBody, &arg[i]);
}
shared.mu.Lock();
while (shared.num_initialized < n) {
shared.cv.Wait();
}
shared.start = true;
shared.cv.SignalAll();
while (shared.num_done < n) {
shared.cv.Wait();
}
shared.mu.Unlock();
for (int i = 1; i < n; i++) {
arg[0].thread->stats.Merge(arg[i].thread->stats);
}
arg[0].thread->stats.Report(name);
for (int i = 0; i < n; i++) {
delete arg[i].thread;
}
delete[] arg;
}
void Crc32c(ThreadState* thread) {
// Checksum about 500MB of data total
const int size = 4096;
const char* label = "(4K per op)";
std::string data(size, 'x');
int64_t bytes = 0;
uint32_t crc = 0;
while (bytes < 500 * 1048576) {
crc = crc32c::Value(data.data(), size);
thread->stats.FinishedSingleOp();
bytes += size;
}
// Print so result is not dead
fprintf(stderr, "... crc=0x%x\r", static_cast<unsigned int>(crc));
thread->stats.AddBytes(bytes);
thread->stats.AddMessage(label);
}
void AcquireLoad(ThreadState* thread) {
int dummy;
port::AtomicPointer ap(&dummy);
int count = 0;
void *ptr = NULL;
thread->stats.AddMessage("(each op is 1000 loads)");
while (count < 100000) {
for (int i = 0; i < 1000; i++) {
ptr = ap.Acquire_Load();
}
count++;
thread->stats.FinishedSingleOp();
}
if (ptr == NULL) exit(1); // Disable unused variable warning.
}
void SnappyCompress(ThreadState* thread) {
RandomGenerator gen;
Slice input = gen.Generate(Options().block_size);
int64_t bytes = 0;
int64_t produced = 0;
bool ok = true;
std::string compressed;
while (ok && bytes < 1024 * 1048576) { // Compress 1G
ok = port::Snappy_Compress(input.data(), input.size(), &compressed);
produced += compressed.size();
bytes += input.size();
thread->stats.FinishedSingleOp();
}
if (!ok) {
thread->stats.AddMessage("(snappy failure)");
} else {
char buf[100];
snprintf(buf, sizeof(buf), "(output: %.1f%%)",
(produced * 100.0) / bytes);
thread->stats.AddMessage(buf);
thread->stats.AddBytes(bytes);
}
}
void SnappyUncompress(ThreadState* thread) {
RandomGenerator gen;
Slice input = gen.Generate(Options().block_size);
std::string compressed;
bool ok = port::Snappy_Compress(input.data(), input.size(), &compressed);
int64_t bytes = 0;
char* uncompressed = new char[input.size()];
while (ok && bytes < 1024 * 1048576) { // Compress 1G
ok = port::Snappy_Uncompress(compressed.data(), compressed.size(),
uncompressed);
bytes += input.size();
thread->stats.FinishedSingleOp();
}
delete[] uncompressed;
if (!ok) {
thread->stats.AddMessage("(snappy failure)");
} else {
thread->stats.AddBytes(bytes);
}
}
void Open() {
assert(db_ == NULL);
Options options;
options.create_if_missing = !FLAGS_use_existing_db;
options.block_cache = cache_;
options.write_buffer_size = FLAGS_write_buffer_size;
options.max_open_files = FLAGS_open_files;
options.filter_policy = filter_policy_;
Status s = DB::Open(options, FLAGS_db, &db_);
if (!s.ok()) {
fprintf(stderr, "open error: %s\n", s.ToString().c_str());
exit(1);
}
}
void WriteSeq(ThreadState* thread) {
DoWrite(thread, true);
}
void WriteRandom(ThreadState* thread) {
DoWrite(thread, false);
}
void DoWrite(ThreadState* thread, bool seq) {
if (num_ != FLAGS_num) {
char msg[100];
snprintf(msg, sizeof(msg), "(%d ops)", num_);
thread->stats.AddMessage(msg);
}
RandomGenerator gen;
WriteBatch batch;
Status s;
int64_t bytes = 0;
for (int i = 0; i < num_; i += entries_per_batch_) {
batch.Clear();
for (int j = 0; j < entries_per_batch_; j++) {
const int k = seq ? i+j : (thread->rand.Next() % FLAGS_num);
char key[100];
snprintf(key, sizeof(key), "%016d", k);
batch.Put(key, gen.Generate(value_size_));
bytes += value_size_ + strlen(key);
thread->stats.FinishedSingleOp();
}
s = db_->Write(write_options_, &batch);
if (!s.ok()) {
fprintf(stderr, "put error: %s\n", s.ToString().c_str());
exit(1);
}
}
thread->stats.AddBytes(bytes);
}
void ReadSequential(ThreadState* thread) {
Iterator* iter = db_->NewIterator(ReadOptions());
int i = 0;
int64_t bytes = 0;
for (iter->SeekToFirst(); i < reads_ && iter->Valid(); iter->Next()) {
bytes += iter->key().size() + iter->value().size();
thread->stats.FinishedSingleOp();
++i;
}
delete iter;
thread->stats.AddBytes(bytes);
}
void ReadReverse(ThreadState* thread) {
Iterator* iter = db_->NewIterator(ReadOptions());
int i = 0;
int64_t bytes = 0;
for (iter->SeekToLast(); i < reads_ && iter->Valid(); iter->Prev()) {
bytes += iter->key().size() + iter->value().size();
thread->stats.FinishedSingleOp();
++i;
}
delete iter;
thread->stats.AddBytes(bytes);
}
void ReadRandom(ThreadState* thread) {
ReadOptions options;
std::string value;
int found = 0;
for (int i = 0; i < reads_; i++) {
char key[100];
const int k = thread->rand.Next() % FLAGS_num;
snprintf(key, sizeof(key), "%016d", k);
if (db_->Get(options, key, &value).ok()) {
found++;
}
thread->stats.FinishedSingleOp();
}
char msg[100];
snprintf(msg, sizeof(msg), "(%d of %d found)", found, num_);
thread->stats.AddMessage(msg);
}
void ReadMissing(ThreadState* thread) {
ReadOptions options;
std::string value;
for (int i = 0; i < reads_; i++) {
char key[100];
const int k = thread->rand.Next() % FLAGS_num;
snprintf(key, sizeof(key), "%016d.", k);
db_->Get(options, key, &value);
thread->stats.FinishedSingleOp();
}
}
void ReadHot(ThreadState* thread) {
ReadOptions options;
std::string value;
const int range = (FLAGS_num + 99) / 100;
for (int i = 0; i < reads_; i++) {
char key[100];
const int k = thread->rand.Next() % range;
snprintf(key, sizeof(key), "%016d", k);
db_->Get(options, key, &value);
thread->stats.FinishedSingleOp();
}
}
void SeekRandom(ThreadState* thread) {
ReadOptions options;
std::string value;
int found = 0;
for (int i = 0; i < reads_; i++) {
Iterator* iter = db_->NewIterator(options);
char key[100];
const int k = thread->rand.Next() % FLAGS_num;
snprintf(key, sizeof(key), "%016d", k);
iter->Seek(key);
if (iter->Valid() && iter->key() == key) found++;
delete iter;
thread->stats.FinishedSingleOp();
}
char msg[100];
snprintf(msg, sizeof(msg), "(%d of %d found)", found, num_);
thread->stats.AddMessage(msg);
}
void DoDelete(ThreadState* thread, bool seq) {
RandomGenerator gen;
WriteBatch batch;
Status s;
for (int i = 0; i < num_; i += entries_per_batch_) {
batch.Clear();
for (int j = 0; j < entries_per_batch_; j++) {
const int k = seq ? i+j : (thread->rand.Next() % FLAGS_num);
char key[100];
snprintf(key, sizeof(key), "%016d", k);
batch.Delete(key);
thread->stats.FinishedSingleOp();
}
s = db_->Write(write_options_, &batch);
if (!s.ok()) {
fprintf(stderr, "del error: %s\n", s.ToString().c_str());
exit(1);
}
}
}
void DeleteSeq(ThreadState* thread) {
DoDelete(thread, true);
}
void DeleteRandom(ThreadState* thread) {
DoDelete(thread, false);
}
void ReadWhileWriting(ThreadState* thread) {
if (thread->tid > 0) {
ReadRandom(thread);
} else {
// Special thread that keeps writing until other threads are done.
RandomGenerator gen;
while (true) {
{
MutexLock l(&thread->shared->mu);
if (thread->shared->num_done + 1 >= thread->shared->num_initialized) {
// Other threads have finished
break;
}
}
const int k = thread->rand.Next() % FLAGS_num;
char key[100];
snprintf(key, sizeof(key), "%016d", k);
Status s = db_->Put(write_options_, key, gen.Generate(value_size_));
if (!s.ok()) {
fprintf(stderr, "put error: %s\n", s.ToString().c_str());
exit(1);
}
}
// Do not count any of the preceding work/delay in stats.
thread->stats.Start();
}
}
void Compact(ThreadState* thread) {
db_->CompactRange(NULL, NULL);
}
void PrintStats(const char* key) {
std::string stats;
if (!db_->GetProperty(key, &stats)) {
stats = "(failed)";
}
fprintf(stdout, "\n%s\n", stats.c_str());
}
static void WriteToFile(void* arg, const char* buf, int n) {
reinterpret_cast<WritableFile*>(arg)->Append(Slice(buf, n));
}
void HeapProfile() {
char fname[100];
snprintf(fname, sizeof(fname), "%s/heap-%04d", FLAGS_db, ++heap_counter_);
WritableFile* file;
Status s = Env::Default()->NewWritableFile(fname, &file);
if (!s.ok()) {
fprintf(stderr, "%s\n", s.ToString().c_str());
return;
}
bool ok = port::GetHeapProfile(WriteToFile, file);
delete file;
if (!ok) {
fprintf(stderr, "heap profiling not supported\n");
Env::Default()->DeleteFile(fname);
}
}
};
} // namespace leveldb
int main(int argc, char** argv) {
FLAGS_write_buffer_size = leveldb::Options().write_buffer_size;
FLAGS_open_files = leveldb::Options().max_open_files;
std::string default_db_path;
for (int i = 1; i < argc; i++) {
double d;
int n;
char junk;
if (leveldb::Slice(argv[i]).starts_with("--benchmarks=")) {
FLAGS_benchmarks = argv[i] + strlen("--benchmarks=");
} else if (sscanf(argv[i], "--compression_ratio=%lf%c", &d, &junk) == 1) {
FLAGS_compression_ratio = d;
} else if (sscanf(argv[i], "--histogram=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_histogram = n;
} else if (sscanf(argv[i], "--use_existing_db=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_use_existing_db = n;
} else if (sscanf(argv[i], "--num=%d%c", &n, &junk) == 1) {
FLAGS_num = n;
} else if (sscanf(argv[i], "--reads=%d%c", &n, &junk) == 1) {
FLAGS_reads = n;
} else if (sscanf(argv[i], "--threads=%d%c", &n, &junk) == 1) {
FLAGS_threads = n;
} else if (sscanf(argv[i], "--value_size=%d%c", &n, &junk) == 1) {
FLAGS_value_size = n;
} else if (sscanf(argv[i], "--write_buffer_size=%d%c", &n, &junk) == 1) {
FLAGS_write_buffer_size = n;
} else if (sscanf(argv[i], "--cache_size=%d%c", &n, &junk) == 1) {
FLAGS_cache_size = n;
} else if (sscanf(argv[i], "--bloom_bits=%d%c", &n, &junk) == 1) {
FLAGS_bloom_bits = n;
} else if (sscanf(argv[i], "--open_files=%d%c", &n, &junk) == 1) {
FLAGS_open_files = n;
} else if (strncmp(argv[i], "--db=", 5) == 0) {
FLAGS_db = argv[i] + 5;
} else {
fprintf(stderr, "Invalid flag '%s'\n", argv[i]);
exit(1);
}
}
// Choose a location for the test database if none given with --db=<path>
if (FLAGS_db == NULL) {
leveldb::Env::Default()->GetTestDirectory(&default_db_path);
default_db_path += "/dbbench";
FLAGS_db = default_db_path.c_str();
}
leveldb::Benchmark benchmark;
benchmark.Run();
return 0;
}
src/leveldb/db/db_impl.cc
+1513
View File
File diff suppressed because it is too large Load Diff
src/leveldb/db/db_impl.h
+211
View File
@@ -0,0 +1,211 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_DB_IMPL_H_
#define STORAGE_LEVELDB_DB_DB_IMPL_H_
#include <deque>
#include <set>
#include "db/dbformat.h"
#include "db/log_writer.h"
#include "db/snapshot.h"
#include "leveldb/db.h"
#include "leveldb/env.h"
#include "port/port.h"
#include "port/thread_annotations.h"
namespace leveldb {
class MemTable;
class TableCache;
class Version;
class VersionEdit;
class VersionSet;
class DBImpl : public DB {
public:
DBImpl(const Options& options, const std::string& dbname);
virtual ~DBImpl();
// Implementations of the DB interface
virtual Status Put(const WriteOptions&, const Slice& key, const Slice& value);
virtual Status Delete(const WriteOptions&, const Slice& key);
virtual Status Write(const WriteOptions& options, WriteBatch* updates);
virtual Status Get(const ReadOptions& options,
const Slice& key,
std::string* value);
virtual Iterator* NewIterator(const ReadOptions&);
virtual const Snapshot* GetSnapshot();
virtual void ReleaseSnapshot(const Snapshot* snapshot);
virtual bool GetProperty(const Slice& property, std::string* value);
virtual void GetApproximateSizes(const Range* range, int n, uint64_t* sizes);
virtual void CompactRange(const Slice* begin, const Slice* end);
// Extra methods (for testing) that are not in the public DB interface
// Compact any files in the named level that overlap [*begin,*end]
void TEST_CompactRange(int level, const Slice* begin, const Slice* end);
// Force current memtable contents to be compacted.
Status TEST_CompactMemTable();
// Return an internal iterator over the current state of the database.
// The keys of this iterator are internal keys (see format.h).
// The returned iterator should be deleted when no longer needed.
Iterator* TEST_NewInternalIterator();
// Return the maximum overlapping data (in bytes) at next level for any
// file at a level >= 1.
int64_t TEST_MaxNextLevelOverlappingBytes();
// Record a sample of bytes read at the specified internal key.
// Samples are taken approximately once every config::kReadBytesPeriod
// bytes.
void RecordReadSample(Slice key);
private:
friend class DB;
struct CompactionState;
struct Writer;
Iterator* NewInternalIterator(const ReadOptions&,
SequenceNumber* latest_snapshot,
uint32_t* seed);
Status NewDB();
// Recover the descriptor from persistent storage. May do a significant
// amount of work to recover recently logged updates. Any changes to
// be made to the descriptor are added to *edit.
Status Recover(VersionEdit* edit) EXCLUSIVE_LOCKS_REQUIRED(mutex_);
void MaybeIgnoreError(Status* s) const;
// Delete any unneeded files and stale in-memory entries.
void DeleteObsoleteFiles();
// Compact the in-memory write buffer to disk. Switches to a new
// log-file/memtable and writes a new descriptor iff successful.
// Errors are recorded in bg_error_.
void CompactMemTable() EXCLUSIVE_LOCKS_REQUIRED(mutex_);
Status RecoverLogFile(uint64_t log_number,
VersionEdit* edit,
SequenceNumber* max_sequence)
EXCLUSIVE_LOCKS_REQUIRED(mutex_);
Status WriteLevel0Table(MemTable* mem, VersionEdit* edit, Version* base)
EXCLUSIVE_LOCKS_REQUIRED(mutex_);
Status MakeRoomForWrite(bool force /* compact even if there is room? */)
EXCLUSIVE_LOCKS_REQUIRED(mutex_);
WriteBatch* BuildBatchGroup(Writer** last_writer);
void RecordBackgroundError(const Status& s);
void MaybeScheduleCompaction() EXCLUSIVE_LOCKS_REQUIRED(mutex_);
static void BGWork(void* db);
void BackgroundCall();
void BackgroundCompaction() EXCLUSIVE_LOCKS_REQUIRED(mutex_);
void CleanupCompaction(CompactionState* compact)
EXCLUSIVE_LOCKS_REQUIRED(mutex_);
Status DoCompactionWork(CompactionState* compact)
EXCLUSIVE_LOCKS_REQUIRED(mutex_);
Status OpenCompactionOutputFile(CompactionState* compact);
Status FinishCompactionOutputFile(CompactionState* compact, Iterator* input);
Status InstallCompactionResults(CompactionState* compact)
EXCLUSIVE_LOCKS_REQUIRED(mutex_);
// Constant after construction
Env* const env_;
const InternalKeyComparator internal_comparator_;
const InternalFilterPolicy internal_filter_policy_;
const Options options_; // options_.comparator == &internal_comparator_
bool owns_info_log_;
bool owns_cache_;
const std::string dbname_;
// table_cache_ provides its own synchronization
TableCache* table_cache_;
// Lock over the persistent DB state. Non-NULL iff successfully acquired.
FileLock* db_lock_;
// State below is protected by mutex_
port::Mutex mutex_;
port::AtomicPointer shutting_down_;
port::CondVar bg_cv_; // Signalled when background work finishes
MemTable* mem_;
MemTable* imm_; // Memtable being compacted
port::AtomicPointer has_imm_; // So bg thread can detect non-NULL imm_
WritableFile* logfile_;
uint64_t logfile_number_;
log::Writer* log_;
uint32_t seed_; // For sampling.
// Queue of writers.
std::deque<Writer*> writers_;
WriteBatch* tmp_batch_;
SnapshotList snapshots_;
// Set of table files to protect from deletion because they are
// part of ongoing compactions.
std::set<uint64_t> pending_outputs_;
// Has a background compaction been scheduled or is running?
bool bg_compaction_scheduled_;
// Information for a manual compaction
struct ManualCompaction {
int level;
bool done;
const InternalKey* begin; // NULL means beginning of key range
const InternalKey* end; // NULL means end of key range
InternalKey tmp_storage; // Used to keep track of compaction progress
};
ManualCompaction* manual_compaction_;
VersionSet* versions_;
// Have we encountered a background error in paranoid mode?
Status bg_error_;
// Per level compaction stats. stats_[level] stores the stats for
// compactions that produced data for the specified "level".
struct CompactionStats {
int64_t micros;
int64_t bytes_read;
int64_t bytes_written;
CompactionStats() : micros(0), bytes_read(0), bytes_written(0) { }
void Add(const CompactionStats& c) {
this->micros += c.micros;
this->bytes_read += c.bytes_read;
this->bytes_written += c.bytes_written;
}
};
CompactionStats stats_[config::kNumLevels];
// No copying allowed
DBImpl(const DBImpl&);
void operator=(const DBImpl&);
const Comparator* user_comparator() const {
return internal_comparator_.user_comparator();
}
};
// Sanitize db options. The caller should delete result.info_log if
// it is not equal to src.info_log.
extern Options SanitizeOptions(const std::string& db,
const InternalKeyComparator* icmp,
const InternalFilterPolicy* ipolicy,
const Options& src);
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_DB_IMPL_H_
src/leveldb/db/db_iter.cc
+317
View File
@@ -0,0 +1,317 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/db_iter.h"
#include "db/filename.h"
#include "db/db_impl.h"
#include "db/dbformat.h"
#include "leveldb/env.h"
#include "leveldb/iterator.h"
#include "port/port.h"
#include "util/logging.h"
#include "util/mutexlock.h"
#include "util/random.h"
namespace leveldb {
#if 0
static void DumpInternalIter(Iterator* iter) {
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ParsedInternalKey k;
if (!ParseInternalKey(iter->key(), &k)) {
fprintf(stderr, "Corrupt '%s'\n", EscapeString(iter->key()).c_str());
} else {
fprintf(stderr, "@ '%s'\n", k.DebugString().c_str());
}
}
}
#endif
namespace {
// Memtables and sstables that make the DB representation contain
// (userkey,seq,type) => uservalue entries. DBIter
// combines multiple entries for the same userkey found in the DB
// representation into a single entry while accounting for sequence
// numbers, deletion markers, overwrites, etc.
class DBIter: public Iterator {
public:
// Which direction is the iterator currently moving?
// (1) When moving forward, the internal iterator is positioned at
// the exact entry that yields this->key(), this->value()
// (2) When moving backwards, the internal iterator is positioned
// just before all entries whose user key == this->key().
enum Direction {
kForward,
kReverse
};
DBIter(DBImpl* db, const Comparator* cmp, Iterator* iter, SequenceNumber s,
uint32_t seed)
: db_(db),
user_comparator_(cmp),
iter_(iter),
sequence_(s),
direction_(kForward),
valid_(false),
rnd_(seed),
bytes_counter_(RandomPeriod()) {
}
virtual ~DBIter() {
delete iter_;
}
virtual bool Valid() const { return valid_; }
virtual Slice key() const {
assert(valid_);
return (direction_ == kForward) ? ExtractUserKey(iter_->key()) : saved_key_;
}
virtual Slice value() const {
assert(valid_);
return (direction_ == kForward) ? iter_->value() : saved_value_;
}
virtual Status status() const {
if (status_.ok()) {
return iter_->status();
} else {
return status_;
}
}
virtual void Next();
virtual void Prev();
virtual void Seek(const Slice& target);
virtual void SeekToFirst();
virtual void SeekToLast();
private:
void FindNextUserEntry(bool skipping, std::string* skip);
void FindPrevUserEntry();
bool ParseKey(ParsedInternalKey* key);
inline void SaveKey(const Slice& k, std::string* dst) {
dst->assign(k.data(), k.size());
}
inline void ClearSavedValue() {
if (saved_value_.capacity() > 1048576) {
std::string empty;
swap(empty, saved_value_);
} else {
saved_value_.clear();
}
}
// Pick next gap with average value of config::kReadBytesPeriod.
ssize_t RandomPeriod() {
return rnd_.Uniform(2*config::kReadBytesPeriod);
}
DBImpl* db_;
const Comparator* const user_comparator_;
Iterator* const iter_;
SequenceNumber const sequence_;
Status status_;
std::string saved_key_; // == current key when direction_==kReverse
std::string saved_value_; // == current raw value when direction_==kReverse
Direction direction_;
bool valid_;
Random rnd_;
ssize_t bytes_counter_;
// No copying allowed
DBIter(const DBIter&);
void operator=(const DBIter&);
};
inline bool DBIter::ParseKey(ParsedInternalKey* ikey) {
Slice k = iter_->key();
ssize_t n = k.size() + iter_->value().size();
bytes_counter_ -= n;
while (bytes_counter_ < 0) {
bytes_counter_ += RandomPeriod();
db_->RecordReadSample(k);
}
if (!ParseInternalKey(k, ikey)) {
status_ = Status::Corruption("corrupted internal key in DBIter");
return false;
} else {
return true;
}
}
void DBIter::Next() {
assert(valid_);
if (direction_ == kReverse) { // Switch directions?
direction_ = kForward;
// iter_ is pointing just before the entries for this->key(),
// so advance into the range of entries for this->key() and then
// use the normal skipping code below.
if (!iter_->Valid()) {
iter_->SeekToFirst();
} else {
iter_->Next();
}
if (!iter_->Valid()) {
valid_ = false;
saved_key_.clear();
return;
}
// saved_key_ already contains the key to skip past.
} else {
// Store in saved_key_ the current key so we skip it below.
SaveKey(ExtractUserKey(iter_->key()), &saved_key_);
}
FindNextUserEntry(true, &saved_key_);
}
void DBIter::FindNextUserEntry(bool skipping, std::string* skip) {
// Loop until we hit an acceptable entry to yield
assert(iter_->Valid());
assert(direction_ == kForward);
do {
ParsedInternalKey ikey;
if (ParseKey(&ikey) && ikey.sequence <= sequence_) {
switch (ikey.type) {
case kTypeDeletion:
// Arrange to skip all upcoming entries for this key since
// they are hidden by this deletion.
SaveKey(ikey.user_key, skip);
skipping = true;
break;
case kTypeValue:
if (skipping &&
user_comparator_->Compare(ikey.user_key, *skip) <= 0) {
// Entry hidden
} else {
valid_ = true;
saved_key_.clear();
return;
}
break;
}
}
iter_->Next();
} while (iter_->Valid());
saved_key_.clear();
valid_ = false;
}
void DBIter::Prev() {
assert(valid_);
if (direction_ == kForward) { // Switch directions?
// iter_ is pointing at the current entry. Scan backwards until
// the key changes so we can use the normal reverse scanning code.
assert(iter_->Valid()); // Otherwise valid_ would have been false
SaveKey(ExtractUserKey(iter_->key()), &saved_key_);
while (true) {
iter_->Prev();
if (!iter_->Valid()) {
valid_ = false;
saved_key_.clear();
ClearSavedValue();
return;
}
if (user_comparator_->Compare(ExtractUserKey(iter_->key()),
saved_key_) < 0) {
break;
}
}
direction_ = kReverse;
}
FindPrevUserEntry();
}
void DBIter::FindPrevUserEntry() {
assert(direction_ == kReverse);
ValueType value_type = kTypeDeletion;
if (iter_->Valid()) {
do {
ParsedInternalKey ikey;
if (ParseKey(&ikey) && ikey.sequence <= sequence_) {
if ((value_type != kTypeDeletion) &&
user_comparator_->Compare(ikey.user_key, saved_key_) < 0) {
// We encountered a non-deleted value in entries for previous keys,
break;
}
value_type = ikey.type;
if (value_type == kTypeDeletion) {
saved_key_.clear();
ClearSavedValue();
} else {
Slice raw_value = iter_->value();
if (saved_value_.capacity() > raw_value.size() + 1048576) {
std::string empty;
swap(empty, saved_value_);
}
SaveKey(ExtractUserKey(iter_->key()), &saved_key_);
saved_value_.assign(raw_value.data(), raw_value.size());
}
}
iter_->Prev();
} while (iter_->Valid());
}
if (value_type == kTypeDeletion) {
// End
valid_ = false;
saved_key_.clear();
ClearSavedValue();
direction_ = kForward;
} else {
valid_ = true;
}
}
void DBIter::Seek(const Slice& target) {
direction_ = kForward;
ClearSavedValue();
saved_key_.clear();
AppendInternalKey(
&saved_key_, ParsedInternalKey(target, sequence_, kValueTypeForSeek));
iter_->Seek(saved_key_);
if (iter_->Valid()) {
FindNextUserEntry(false, &saved_key_ /* temporary storage */);
} else {
valid_ = false;
}
}
void DBIter::SeekToFirst() {
direction_ = kForward;
ClearSavedValue();
iter_->SeekToFirst();
if (iter_->Valid()) {
FindNextUserEntry(false, &saved_key_ /* temporary storage */);
} else {
valid_ = false;
}
}
void DBIter::SeekToLast() {
direction_ = kReverse;
ClearSavedValue();
iter_->SeekToLast();
FindPrevUserEntry();
}
} // anonymous namespace
Iterator* NewDBIterator(
DBImpl* db,
const Comparator* user_key_comparator,
Iterator* internal_iter,
SequenceNumber sequence,
uint32_t seed) {
return new DBIter(db, user_key_comparator, internal_iter, sequence, seed);
}
} // namespace leveldb
src/leveldb/db/db_iter.h
+28
View File
@@ -0,0 +1,28 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_DB_ITER_H_
#define STORAGE_LEVELDB_DB_DB_ITER_H_
#include <stdint.h>
#include "leveldb/db.h"
#include "db/dbformat.h"
namespace leveldb {
class DBImpl;
// Return a new iterator that converts internal keys (yielded by
// "*internal_iter") that were live at the specified "sequence" number
// into appropriate user keys.
extern Iterator* NewDBIterator(
DBImpl* db,
const Comparator* user_key_comparator,
Iterator* internal_iter,
SequenceNumber sequence,
uint32_t seed);
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_DB_ITER_H_
src/leveldb/db/db_test.cc
+2128
View File
File diff suppressed because it is too large Load Diff
src/leveldb/db/dbformat.cc
+140
View File
@@ -0,0 +1,140 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <stdio.h>
#include "db/dbformat.h"
#include "port/port.h"
#include "util/coding.h"
namespace leveldb {
static uint64_t PackSequenceAndType(uint64_t seq, ValueType t) {
assert(seq <= kMaxSequenceNumber);
assert(t <= kValueTypeForSeek);
return (seq << 8) | t;
}
void AppendInternalKey(std::string* result, const ParsedInternalKey& key) {
result->append(key.user_key.data(), key.user_key.size());
PutFixed64(result, PackSequenceAndType(key.sequence, key.type));
}
std::string ParsedInternalKey::DebugString() const {
char buf[50];
snprintf(buf, sizeof(buf), "' @ %llu : %d",
(unsigned long long) sequence,
int(type));
std::string result = "'";
result += EscapeString(user_key.ToString());
result += buf;
return result;
}
std::string InternalKey::DebugString() const {
std::string result;
ParsedInternalKey parsed;
if (ParseInternalKey(rep_, &parsed)) {
result = parsed.DebugString();
} else {
result = "(bad)";
result.append(EscapeString(rep_));
}
return result;
}
const char* InternalKeyComparator::Name() const {
return "leveldb.InternalKeyComparator";
}
int InternalKeyComparator::Compare(const Slice& akey, const Slice& bkey) const {
// Order by:
// increasing user key (according to user-supplied comparator)
// decreasing sequence number
// decreasing type (though sequence# should be enough to disambiguate)
int r = user_comparator_->Compare(ExtractUserKey(akey), ExtractUserKey(bkey));
if (r == 0) {
const uint64_t anum = DecodeFixed64(akey.data() + akey.size() - 8);
const uint64_t bnum = DecodeFixed64(bkey.data() + bkey.size() - 8);
if (anum > bnum) {
r = -1;
} else if (anum < bnum) {
r = +1;
}
}
return r;
}
void InternalKeyComparator::FindShortestSeparator(
std::string* start,
const Slice& limit) const {
// Attempt to shorten the user portion of the key
Slice user_start = ExtractUserKey(*start);
Slice user_limit = ExtractUserKey(limit);
std::string tmp(user_start.data(), user_start.size());
user_comparator_->FindShortestSeparator(&tmp, user_limit);
if (tmp.size() < user_start.size() &&
user_comparator_->Compare(user_start, tmp) < 0) {
// User key has become shorter physically, but larger logically.
// Tack on the earliest possible number to the shortened user key.
PutFixed64(&tmp, PackSequenceAndType(kMaxSequenceNumber,kValueTypeForSeek));
assert(this->Compare(*start, tmp) < 0);
assert(this->Compare(tmp, limit) < 0);
start->swap(tmp);
}
}
void InternalKeyComparator::FindShortSuccessor(std::string* key) const {
Slice user_key = ExtractUserKey(*key);
std::string tmp(user_key.data(), user_key.size());
user_comparator_->FindShortSuccessor(&tmp);
if (tmp.size() < user_key.size() &&
user_comparator_->Compare(user_key, tmp) < 0) {
// User key has become shorter physically, but larger logically.
// Tack on the earliest possible number to the shortened user key.
PutFixed64(&tmp, PackSequenceAndType(kMaxSequenceNumber,kValueTypeForSeek));
assert(this->Compare(*key, tmp) < 0);
key->swap(tmp);
}
}
const char* InternalFilterPolicy::Name() const {
return user_policy_->Name();
}
void InternalFilterPolicy::CreateFilter(const Slice* keys, int n,
std::string* dst) const {
// We rely on the fact that the code in table.cc does not mind us
// adjusting keys[].
Slice* mkey = const_cast<Slice*>(keys);
for (int i = 0; i < n; i++) {
mkey[i] = ExtractUserKey(keys[i]);
// TODO(sanjay): Suppress dups?
}
user_policy_->CreateFilter(keys, n, dst);
}
bool InternalFilterPolicy::KeyMayMatch(const Slice& key, const Slice& f) const {
return user_policy_->KeyMayMatch(ExtractUserKey(key), f);
}
LookupKey::LookupKey(const Slice& user_key, SequenceNumber s) {
size_t usize = user_key.size();
size_t needed = usize + 13; // A conservative estimate
char* dst;
if (needed <= sizeof(space_)) {
dst = space_;
} else {
dst = new char[needed];
}
start_ = dst;
dst = EncodeVarint32(dst, usize + 8);
kstart_ = dst;
memcpy(dst, user_key.data(), usize);
dst += usize;
EncodeFixed64(dst, PackSequenceAndType(s, kValueTypeForSeek));
dst += 8;
end_ = dst;
}
} // namespace leveldb
src/leveldb/db/dbformat.h
+230
View File
@@ -0,0 +1,230 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_FORMAT_H_
#define STORAGE_LEVELDB_DB_FORMAT_H_
#include <stdio.h>
#include "leveldb/comparator.h"
#include "leveldb/db.h"
#include "leveldb/filter_policy.h"
#include "leveldb/slice.h"
#include "leveldb/table_builder.h"
#include "util/coding.h"
#include "util/logging.h"
namespace leveldb {
// Grouping of constants. We may want to make some of these
// parameters set via options.
namespace config {
static const int kNumLevels = 7;
// Level-0 compaction is started when we hit this many files.
static const int kL0_CompactionTrigger = 4;
// Soft limit on number of level-0 files. We slow down writes at this point.
static const int kL0_SlowdownWritesTrigger = 8;
// Maximum number of level-0 files. We stop writes at this point.
static const int kL0_StopWritesTrigger = 12;
// Maximum level to which a new compacted memtable is pushed if it
// does not create overlap. We try to push to level 2 to avoid the
// relatively expensive level 0=>1 compactions and to avoid some
// expensive manifest file operations. We do not push all the way to
// the largest level since that can generate a lot of wasted disk
// space if the same key space is being repeatedly overwritten.
static const int kMaxMemCompactLevel = 2;
// Approximate gap in bytes between samples of data read during iteration.
static const int kReadBytesPeriod = 1048576;
} // namespace config
class InternalKey;
// Value types encoded as the last component of internal keys.
// DO NOT CHANGE THESE ENUM VALUES: they are embedded in the on-disk
// data structures.
enum ValueType {
kTypeDeletion = 0x0,
kTypeValue = 0x1
};
// kValueTypeForSeek defines the ValueType that should be passed when
// constructing a ParsedInternalKey object for seeking to a particular
// sequence number (since we sort sequence numbers in decreasing order
// and the value type is embedded as the low 8 bits in the sequence
// number in internal keys, we need to use the highest-numbered
// ValueType, not the lowest).
static const ValueType kValueTypeForSeek = kTypeValue;
typedef uint64_t SequenceNumber;
// We leave eight bits empty at the bottom so a type and sequence#
// can be packed together into 64-bits.
static const SequenceNumber kMaxSequenceNumber =
((0x1ull << 56) - 1);
struct ParsedInternalKey {
Slice user_key;
SequenceNumber sequence;
ValueType type;
ParsedInternalKey() { } // Intentionally left uninitialized (for speed)
ParsedInternalKey(const Slice& u, const SequenceNumber& seq, ValueType t)
: user_key(u), sequence(seq), type(t) { }
std::string DebugString() const;
};
// Return the length of the encoding of "key".
inline size_t InternalKeyEncodingLength(const ParsedInternalKey& key) {
return key.user_key.size() + 8;
}
// Append the serialization of "key" to *result.
extern void AppendInternalKey(std::string* result,
const ParsedInternalKey& key);
// Attempt to parse an internal key from "internal_key". On success,
// stores the parsed data in "*result", and returns true.
//
// On error, returns false, leaves "*result" in an undefined state.
extern bool ParseInternalKey(const Slice& internal_key,
ParsedInternalKey* result);
// Returns the user key portion of an internal key.
inline Slice ExtractUserKey(const Slice& internal_key) {
assert(internal_key.size() >= 8);
return Slice(internal_key.data(), internal_key.size() - 8);
}
inline ValueType ExtractValueType(const Slice& internal_key) {
assert(internal_key.size() >= 8);
const size_t n = internal_key.size();
uint64_t num = DecodeFixed64(internal_key.data() + n - 8);
unsigned char c = num & 0xff;
return static_cast<ValueType>(c);
}
// A comparator for internal keys that uses a specified comparator for
// the user key portion and breaks ties by decreasing sequence number.
class InternalKeyComparator : public Comparator {
private:
const Comparator* user_comparator_;
public:
explicit InternalKeyComparator(const Comparator* c) : user_comparator_(c) { }
virtual const char* Name() const;
virtual int Compare(const Slice& a, const Slice& b) const;
virtual void FindShortestSeparator(
std::string* start,
const Slice& limit) const;
virtual void FindShortSuccessor(std::string* key) const;
const Comparator* user_comparator() const { return user_comparator_; }
int Compare(const InternalKey& a, const InternalKey& b) const;
};
// Filter policy wrapper that converts from internal keys to user keys
class InternalFilterPolicy : public FilterPolicy {
private:
const FilterPolicy* const user_policy_;
public:
explicit InternalFilterPolicy(const FilterPolicy* p) : user_policy_(p) { }
virtual const char* Name() const;
virtual void CreateFilter(const Slice* keys, int n, std::string* dst) const;
virtual bool KeyMayMatch(const Slice& key, const Slice& filter) const;
};
// Modules in this directory should keep internal keys wrapped inside
// the following class instead of plain strings so that we do not
// incorrectly use string comparisons instead of an InternalKeyComparator.
class InternalKey {
private:
std::string rep_;
public:
InternalKey() { } // Leave rep_ as empty to indicate it is invalid
InternalKey(const Slice& user_key, SequenceNumber s, ValueType t) {
AppendInternalKey(&rep_, ParsedInternalKey(user_key, s, t));
}
void DecodeFrom(const Slice& s) { rep_.assign(s.data(), s.size()); }
Slice Encode() const {
assert(!rep_.empty());
return rep_;
}
Slice user_key() const { return ExtractUserKey(rep_); }
void SetFrom(const ParsedInternalKey& p) {
rep_.clear();
AppendInternalKey(&rep_, p);
}
void Clear() { rep_.clear(); }
std::string DebugString() const;
};
inline int InternalKeyComparator::Compare(
const InternalKey& a, const InternalKey& b) const {
return Compare(a.Encode(), b.Encode());
}
inline bool ParseInternalKey(const Slice& internal_key,
ParsedInternalKey* result) {
const size_t n = internal_key.size();
if (n < 8) return false;
uint64_t num = DecodeFixed64(internal_key.data() + n - 8);
unsigned char c = num & 0xff;
result->sequence = num >> 8;
result->type = static_cast<ValueType>(c);
result->user_key = Slice(internal_key.data(), n - 8);
return (c <= static_cast<unsigned char>(kTypeValue));
}
// A helper class useful for DBImpl::Get()
class LookupKey {
public:
// Initialize *this for looking up user_key at a snapshot with
// the specified sequence number.
LookupKey(const Slice& user_key, SequenceNumber sequence);
~LookupKey();
// Return a key suitable for lookup in a MemTable.
Slice memtable_key() const { return Slice(start_, end_ - start_); }
// Return an internal key (suitable for passing to an internal iterator)
Slice internal_key() const { return Slice(kstart_, end_ - kstart_); }
// Return the user key
Slice user_key() const { return Slice(kstart_, end_ - kstart_ - 8); }
private:
// We construct a char array of the form:
// klength varint32 <-- start_
// userkey char[klength] <-- kstart_
// tag uint64
// <-- end_
// The array is a suitable MemTable key.
// The suffix starting with "userkey" can be used as an InternalKey.
const char* start_;
const char* kstart_;
const char* end_;
char space_[200]; // Avoid allocation for short keys
// No copying allowed
LookupKey(const LookupKey&);
void operator=(const LookupKey&);
};
inline LookupKey::~LookupKey() {
if (start_ != space_) delete[] start_;
}
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_FORMAT_H_
src/leveldb/db/dbformat_test.cc
+112
View File
@@ -0,0 +1,112 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/dbformat.h"
#include "util/logging.h"
#include "util/testharness.h"
namespace leveldb {
static std::string IKey(const std::string& user_key,
uint64_t seq,
ValueType vt) {
std::string encoded;
AppendInternalKey(&encoded, ParsedInternalKey(user_key, seq, vt));
return encoded;
}
static std::string Shorten(const std::string& s, const std::string& l) {
std::string result = s;
InternalKeyComparator(BytewiseComparator()).FindShortestSeparator(&result, l);
return result;
}
static std::string ShortSuccessor(const std::string& s) {
std::string result = s;
InternalKeyComparator(BytewiseComparator()).FindShortSuccessor(&result);
return result;
}
static void TestKey(const std::string& key,
uint64_t seq,
ValueType vt) {
std::string encoded = IKey(key, seq, vt);
Slice in(encoded);
ParsedInternalKey decoded("", 0, kTypeValue);
ASSERT_TRUE(ParseInternalKey(in, &decoded));
ASSERT_EQ(key, decoded.user_key.ToString());
ASSERT_EQ(seq, decoded.sequence);
ASSERT_EQ(vt, decoded.type);
ASSERT_TRUE(!ParseInternalKey(Slice("bar"), &decoded));
}
class FormatTest { };
TEST(FormatTest, InternalKey_EncodeDecode) {
const char* keys[] = { "", "k", "hello", "longggggggggggggggggggggg" };
const uint64_t seq[] = {
1, 2, 3,
(1ull << 8) - 1, 1ull << 8, (1ull << 8) + 1,
(1ull << 16) - 1, 1ull << 16, (1ull << 16) + 1,
(1ull << 32) - 1, 1ull << 32, (1ull << 32) + 1
};
for (int k = 0; k < sizeof(keys) / sizeof(keys[0]); k++) {
for (int s = 0; s < sizeof(seq) / sizeof(seq[0]); s++) {
TestKey(keys[k], seq[s], kTypeValue);
TestKey("hello", 1, kTypeDeletion);
}
}
}
TEST(FormatTest, InternalKeyShortSeparator) {
// When user keys are same
ASSERT_EQ(IKey("foo", 100, kTypeValue),
Shorten(IKey("foo", 100, kTypeValue),
IKey("foo", 99, kTypeValue)));
ASSERT_EQ(IKey("foo", 100, kTypeValue),
Shorten(IKey("foo", 100, kTypeValue),
IKey("foo", 101, kTypeValue)));
ASSERT_EQ(IKey("foo", 100, kTypeValue),
Shorten(IKey("foo", 100, kTypeValue),
IKey("foo", 100, kTypeValue)));
ASSERT_EQ(IKey("foo", 100, kTypeValue),
Shorten(IKey("foo", 100, kTypeValue),
IKey("foo", 100, kTypeDeletion)));
// When user keys are misordered
ASSERT_EQ(IKey("foo", 100, kTypeValue),
Shorten(IKey("foo", 100, kTypeValue),
IKey("bar", 99, kTypeValue)));
// When user keys are different, but correctly ordered
ASSERT_EQ(IKey("g", kMaxSequenceNumber, kValueTypeForSeek),
Shorten(IKey("foo", 100, kTypeValue),
IKey("hello", 200, kTypeValue)));
// When start user key is prefix of limit user key
ASSERT_EQ(IKey("foo", 100, kTypeValue),
Shorten(IKey("foo", 100, kTypeValue),
IKey("foobar", 200, kTypeValue)));
// When limit user key is prefix of start user key
ASSERT_EQ(IKey("foobar", 100, kTypeValue),
Shorten(IKey("foobar", 100, kTypeValue),
IKey("foo", 200, kTypeValue)));
}
TEST(FormatTest, InternalKeyShortestSuccessor) {
ASSERT_EQ(IKey("g", kMaxSequenceNumber, kValueTypeForSeek),
ShortSuccessor(IKey("foo", 100, kTypeValue)));
ASSERT_EQ(IKey("\xff\xff", 100, kTypeValue),
ShortSuccessor(IKey("\xff\xff", 100, kTypeValue)));
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/filename.cc
+149
View File
@@ -0,0 +1,149 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <ctype.h>
#include <stdio.h>
#include "db/filename.h"
#include "db/dbformat.h"
#include "leveldb/env.h"
#include "util/logging.h"
namespace leveldb {
// A utility routine: write "data" to the named file and Sync() it.
extern Status WriteStringToFileSync(Env* env, const Slice& data,
const std::string& fname);
static std::string MakeFileName(const std::string& name, uint64_t number,
const char* suffix) {
char buf[100];
snprintf(buf, sizeof(buf), "/%06llu.%s",
static_cast<unsigned long long>(number),
suffix);
return name + buf;
}
std::string LogFileName(const std::string& name, uint64_t number) {
assert(number > 0);
return MakeFileName(name, number, "log");
}
// TableFileName returns the filenames we usually write to, while
// SSTTableFileName returns the alternative filenames we also try to read from
// for backward compatibility. For now, swap them around.
// TODO: when compatibility is no longer necessary, swap them back
// (TableFileName to use "ldb" and SSTTableFileName to use "sst").
std::string TableFileName(const std::string& name, uint64_t number) {
assert(number > 0);
return MakeFileName(name, number, "sst");
}
std::string SSTTableFileName(const std::string& name, uint64_t number) {
assert(number > 0);
return MakeFileName(name, number, "ldb");
}
std::string DescriptorFileName(const std::string& dbname, uint64_t number) {
assert(number > 0);
char buf[100];
snprintf(buf, sizeof(buf), "/MANIFEST-%06llu",
static_cast<unsigned long long>(number));
return dbname + buf;
}
std::string CurrentFileName(const std::string& dbname) {
return dbname + "/CURRENT";
}
std::string LockFileName(const std::string& dbname) {
return dbname + "/LOCK";
}
std::string TempFileName(const std::string& dbname, uint64_t number) {
assert(number > 0);
return MakeFileName(dbname, number, "dbtmp");
}
std::string InfoLogFileName(const std::string& dbname) {
return dbname + "/LOG";
}
// Return the name of the old info log file for "dbname".
std::string OldInfoLogFileName(const std::string& dbname) {
return dbname + "/LOG.old";
}
// Owned filenames have the form:
// dbname/CURRENT
// dbname/LOCK
// dbname/LOG
// dbname/LOG.old
// dbname/MANIFEST-[0-9]+
// dbname/[0-9]+.(log|sst|ldb)
bool ParseFileName(const std::string& fname,
uint64_t* number,
FileType* type) {
Slice rest(fname);
if (rest == "CURRENT") {
*number = 0;
*type = kCurrentFile;
} else if (rest == "LOCK") {
*number = 0;
*type = kDBLockFile;
} else if (rest == "LOG" || rest == "LOG.old") {
*number = 0;
*type = kInfoLogFile;
} else if (rest.starts_with("MANIFEST-")) {
rest.remove_prefix(strlen("MANIFEST-"));
uint64_t num;
if (!ConsumeDecimalNumber(&rest, &num)) {
return false;
}
if (!rest.empty()) {
return false;
}
*type = kDescriptorFile;
*number = num;
} else {
// Avoid strtoull() to keep filename format independent of the
// current locale
uint64_t num;
if (!ConsumeDecimalNumber(&rest, &num)) {
return false;
}
Slice suffix = rest;
if (suffix == Slice(".log")) {
*type = kLogFile;
} else if (suffix == Slice(".sst") || suffix == Slice(".ldb")) {
*type = kTableFile;
} else if (suffix == Slice(".dbtmp")) {
*type = kTempFile;
} else {
return false;
}
*number = num;
}
return true;
}
Status SetCurrentFile(Env* env, const std::string& dbname,
uint64_t descriptor_number) {
// Remove leading "dbname/" and add newline to manifest file name
std::string manifest = DescriptorFileName(dbname, descriptor_number);
Slice contents = manifest;
assert(contents.starts_with(dbname + "/"));
contents.remove_prefix(dbname.size() + 1);
std::string tmp = TempFileName(dbname, descriptor_number);
Status s = WriteStringToFileSync(env, contents.ToString() + "\n", tmp);
if (s.ok()) {
s = env->RenameFile(tmp, CurrentFileName(dbname));
}
if (!s.ok()) {
env->DeleteFile(tmp);
}
return s;
}
} // namespace leveldb
src/leveldb/db/filename.h
+85
View File
@@ -0,0 +1,85 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// File names used by DB code
#ifndef STORAGE_LEVELDB_DB_FILENAME_H_
#define STORAGE_LEVELDB_DB_FILENAME_H_
#include <stdint.h>
#include <string>
#include "leveldb/slice.h"
#include "leveldb/status.h"
#include "port/port.h"
namespace leveldb {
class Env;
enum FileType {
kLogFile,
kDBLockFile,
kTableFile,
kDescriptorFile,
kCurrentFile,
kTempFile,
kInfoLogFile // Either the current one, or an old one
};
// Return the name of the log file with the specified number
// in the db named by "dbname". The result will be prefixed with
// "dbname".
extern std::string LogFileName(const std::string& dbname, uint64_t number);
// Return the name of the sstable with the specified number
// in the db named by "dbname". The result will be prefixed with
// "dbname".
extern std::string TableFileName(const std::string& dbname, uint64_t number);
// Return the legacy file name for an sstable with the specified number
// in the db named by "dbname". The result will be prefixed with
// "dbname".
extern std::string SSTTableFileName(const std::string& dbname, uint64_t number);
// Return the name of the descriptor file for the db named by
// "dbname" and the specified incarnation number. The result will be
// prefixed with "dbname".
extern std::string DescriptorFileName(const std::string& dbname,
uint64_t number);
// Return the name of the current file. This file contains the name
// of the current manifest file. The result will be prefixed with
// "dbname".
extern std::string CurrentFileName(const std::string& dbname);
// Return the name of the lock file for the db named by
// "dbname". The result will be prefixed with "dbname".
extern std::string LockFileName(const std::string& dbname);
// Return the name of a temporary file owned by the db named "dbname".
// The result will be prefixed with "dbname".
extern std::string TempFileName(const std::string& dbname, uint64_t number);
// Return the name of the info log file for "dbname".
extern std::string InfoLogFileName(const std::string& dbname);
// Return the name of the old info log file for "dbname".
extern std::string OldInfoLogFileName(const std::string& dbname);
// If filename is a leveldb file, store the type of the file in *type.
// The number encoded in the filename is stored in *number. If the
// filename was successfully parsed, returns true. Else return false.
extern bool ParseFileName(const std::string& filename,
uint64_t* number,
FileType* type);
// Make the CURRENT file point to the descriptor file with the
// specified number.
extern Status SetCurrentFile(Env* env, const std::string& dbname,
uint64_t descriptor_number);
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_FILENAME_H_
src/leveldb/db/filename_test.cc
+123
View File
@@ -0,0 +1,123 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/filename.h"
#include "db/dbformat.h"
#include "port/port.h"
#include "util/logging.h"
#include "util/testharness.h"
namespace leveldb {
class FileNameTest { };
TEST(FileNameTest, Parse) {
Slice db;
FileType type;
uint64_t number;
// Successful parses
static struct {
const char* fname;
uint64_t number;
FileType type;
} cases[] = {
{ "100.log", 100, kLogFile },
{ "0.log", 0, kLogFile },
{ "0.sst", 0, kTableFile },
{ "0.ldb", 0, kTableFile },
{ "CURRENT", 0, kCurrentFile },
{ "LOCK", 0, kDBLockFile },
{ "MANIFEST-2", 2, kDescriptorFile },
{ "MANIFEST-7", 7, kDescriptorFile },
{ "LOG", 0, kInfoLogFile },
{ "LOG.old", 0, kInfoLogFile },
{ "18446744073709551615.log", 18446744073709551615ull, kLogFile },
};
for (int i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
std::string f = cases[i].fname;
ASSERT_TRUE(ParseFileName(f, &number, &type)) << f;
ASSERT_EQ(cases[i].type, type) << f;
ASSERT_EQ(cases[i].number, number) << f;
}
// Errors
static const char* errors[] = {
"",
"foo",
"foo-dx-100.log",
".log",
"",
"manifest",
"CURREN",
"CURRENTX",
"MANIFES",
"MANIFEST",
"MANIFEST-",
"XMANIFEST-3",
"MANIFEST-3x",
"LOC",
"LOCKx",
"LO",
"LOGx",
"18446744073709551616.log",
"184467440737095516150.log",
"100",
"100.",
"100.lop"
};
for (int i = 0; i < sizeof(errors) / sizeof(errors[0]); i++) {
std::string f = errors[i];
ASSERT_TRUE(!ParseFileName(f, &number, &type)) << f;
}
}
TEST(FileNameTest, Construction) {
uint64_t number;
FileType type;
std::string fname;
fname = CurrentFileName("foo");
ASSERT_EQ("foo/", std::string(fname.data(), 4));
ASSERT_TRUE(ParseFileName(fname.c_str() + 4, &number, &type));
ASSERT_EQ(0, number);
ASSERT_EQ(kCurrentFile, type);
fname = LockFileName("foo");
ASSERT_EQ("foo/", std::string(fname.data(), 4));
ASSERT_TRUE(ParseFileName(fname.c_str() + 4, &number, &type));
ASSERT_EQ(0, number);
ASSERT_EQ(kDBLockFile, type);
fname = LogFileName("foo", 192);
ASSERT_EQ("foo/", std::string(fname.data(), 4));
ASSERT_TRUE(ParseFileName(fname.c_str() + 4, &number, &type));
ASSERT_EQ(192, number);
ASSERT_EQ(kLogFile, type);
fname = TableFileName("bar", 200);
ASSERT_EQ("bar/", std::string(fname.data(), 4));
ASSERT_TRUE(ParseFileName(fname.c_str() + 4, &number, &type));
ASSERT_EQ(200, number);
ASSERT_EQ(kTableFile, type);
fname = DescriptorFileName("bar", 100);
ASSERT_EQ("bar/", std::string(fname.data(), 4));
ASSERT_TRUE(ParseFileName(fname.c_str() + 4, &number, &type));
ASSERT_EQ(100, number);
ASSERT_EQ(kDescriptorFile, type);
fname = TempFileName("tmp", 999);
ASSERT_EQ("tmp/", std::string(fname.data(), 4));
ASSERT_TRUE(ParseFileName(fname.c_str() + 4, &number, &type));
ASSERT_EQ(999, number);
ASSERT_EQ(kTempFile, type);
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/leveldb_main.cc
+238
View File
@@ -0,0 +1,238 @@
// Copyright (c) 2012 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <stdio.h>
#include "db/dbformat.h"
#include "db/filename.h"
#include "db/log_reader.h"
#include "db/version_edit.h"
#include "db/write_batch_internal.h"
#include "leveldb/env.h"
#include "leveldb/iterator.h"
#include "leveldb/options.h"
#include "leveldb/status.h"
#include "leveldb/table.h"
#include "leveldb/write_batch.h"
#include "util/logging.h"
namespace leveldb {
namespace {
bool GuessType(const std::string& fname, FileType* type) {
size_t pos = fname.rfind('/');
std::string basename;
if (pos == std::string::npos) {
basename = fname;
} else {
basename = std::string(fname.data() + pos + 1, fname.size() - pos - 1);
}
uint64_t ignored;
return ParseFileName(basename, &ignored, type);
}
// Notified when log reader encounters corruption.
class CorruptionReporter : public log::Reader::Reporter {
public:
virtual void Corruption(size_t bytes, const Status& status) {
printf("corruption: %d bytes; %s\n",
static_cast<int>(bytes),
status.ToString().c_str());
}
};
// Print contents of a log file. (*func)() is called on every record.
bool PrintLogContents(Env* env, const std::string& fname,
void (*func)(Slice)) {
SequentialFile* file;
Status s = env->NewSequentialFile(fname, &file);
if (!s.ok()) {
fprintf(stderr, "%s\n", s.ToString().c_str());
return false;
}
CorruptionReporter reporter;
log::Reader reader(file, &reporter, true, 0);
Slice record;
std::string scratch;
while (reader.ReadRecord(&record, &scratch)) {
printf("--- offset %llu; ",
static_cast<unsigned long long>(reader.LastRecordOffset()));
(*func)(record);
}
delete file;
return true;
}
// Called on every item found in a WriteBatch.
class WriteBatchItemPrinter : public WriteBatch::Handler {
public:
uint64_t offset_;
uint64_t sequence_;
virtual void Put(const Slice& key, const Slice& value) {
printf(" put '%s' '%s'\n",
EscapeString(key).c_str(),
EscapeString(value).c_str());
}
virtual void Delete(const Slice& key) {
printf(" del '%s'\n",
EscapeString(key).c_str());
}
};
// Called on every log record (each one of which is a WriteBatch)
// found in a kLogFile.
static void WriteBatchPrinter(Slice record) {
if (record.size() < 12) {
printf("log record length %d is too small\n",
static_cast<int>(record.size()));
return;
}
WriteBatch batch;
WriteBatchInternal::SetContents(&batch, record);
printf("sequence %llu\n",
static_cast<unsigned long long>(WriteBatchInternal::Sequence(&batch)));
WriteBatchItemPrinter batch_item_printer;
Status s = batch.Iterate(&batch_item_printer);
if (!s.ok()) {
printf(" error: %s\n", s.ToString().c_str());
}
}
bool DumpLog(Env* env, const std::string& fname) {
return PrintLogContents(env, fname, WriteBatchPrinter);
}
// Called on every log record (each one of which is a WriteBatch)
// found in a kDescriptorFile.
static void VersionEditPrinter(Slice record) {
VersionEdit edit;
Status s = edit.DecodeFrom(record);
if (!s.ok()) {
printf("%s\n", s.ToString().c_str());
return;
}
printf("%s", edit.DebugString().c_str());
}
bool DumpDescriptor(Env* env, const std::string& fname) {
return PrintLogContents(env, fname, VersionEditPrinter);
}
bool DumpTable(Env* env, const std::string& fname) {
uint64_t file_size;
RandomAccessFile* file = NULL;
Table* table = NULL;
Status s = env->GetFileSize(fname, &file_size);
if (s.ok()) {
s = env->NewRandomAccessFile(fname, &file);
}
if (s.ok()) {
// We use the default comparator, which may or may not match the
// comparator used in this database. However this should not cause
// problems since we only use Table operations that do not require
// any comparisons. In particular, we do not call Seek or Prev.
s = Table::Open(Options(), file, file_size, &table);
}
if (!s.ok()) {
fprintf(stderr, "%s\n", s.ToString().c_str());
delete table;
delete file;
return false;
}
ReadOptions ro;
ro.fill_cache = false;
Iterator* iter = table->NewIterator(ro);
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ParsedInternalKey key;
if (!ParseInternalKey(iter->key(), &key)) {
printf("badkey '%s' => '%s'\n",
EscapeString(iter->key()).c_str(),
EscapeString(iter->value()).c_str());
} else {
char kbuf[20];
const char* type;
if (key.type == kTypeDeletion) {
type = "del";
} else if (key.type == kTypeValue) {
type = "val";
} else {
snprintf(kbuf, sizeof(kbuf), "%d", static_cast<int>(key.type));
type = kbuf;
}
printf("'%s' @ %8llu : %s => '%s'\n",
EscapeString(key.user_key).c_str(),
static_cast<unsigned long long>(key.sequence),
type,
EscapeString(iter->value()).c_str());
}
}
s = iter->status();
if (!s.ok()) {
printf("iterator error: %s\n", s.ToString().c_str());
}
delete iter;
delete table;
delete file;
return true;
}
bool DumpFile(Env* env, const std::string& fname) {
FileType ftype;
if (!GuessType(fname, &ftype)) {
fprintf(stderr, "%s: unknown file type\n", fname.c_str());
return false;
}
switch (ftype) {
case kLogFile: return DumpLog(env, fname);
case kDescriptorFile: return DumpDescriptor(env, fname);
case kTableFile: return DumpTable(env, fname);
default: {
fprintf(stderr, "%s: not a dump-able file type\n", fname.c_str());
break;
}
}
return false;
}
bool HandleDumpCommand(Env* env, char** files, int num) {
bool ok = true;
for (int i = 0; i < num; i++) {
ok &= DumpFile(env, files[i]);
}
return ok;
}
}
} // namespace leveldb
static void Usage() {
fprintf(
stderr,
"Usage: leveldbutil command...\n"
" dump files... -- dump contents of specified files\n"
);
}
int main(int argc, char** argv) {
leveldb::Env* env = leveldb::Env::Default();
bool ok = true;
if (argc < 2) {
Usage();
ok = false;
} else {
std::string command = argv[1];
if (command == "dump") {
ok = leveldb::HandleDumpCommand(env, argv+2, argc-2);
} else {
Usage();
ok = false;
}
}
return (ok ? 0 : 1);
}
src/leveldb/db/log_format.h
+35
View File
@@ -0,0 +1,35 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// Log format information shared by reader and writer.
// See ../doc/log_format.txt for more detail.
#ifndef STORAGE_LEVELDB_DB_LOG_FORMAT_H_
#define STORAGE_LEVELDB_DB_LOG_FORMAT_H_
namespace leveldb {
namespace log {
enum RecordType {
// Zero is reserved for preallocated files
kZeroType = 0,
kFullType = 1,
// For fragments
kFirstType = 2,
kMiddleType = 3,
kLastType = 4
};
static const int kMaxRecordType = kLastType;
static const int kBlockSize = 32768;
// Header is checksum (4 bytes), type (1 byte), length (2 bytes).
static const int kHeaderSize = 4 + 1 + 2;
} // namespace log
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_LOG_FORMAT_H_
src/leveldb/db/log_reader.cc
+259
View File
@@ -0,0 +1,259 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/log_reader.h"
#include <stdio.h>
#include "leveldb/env.h"
#include "util/coding.h"
#include "util/crc32c.h"
namespace leveldb {
namespace log {
Reader::Reporter::~Reporter() {
}
Reader::Reader(SequentialFile* file, Reporter* reporter, bool checksum,
uint64_t initial_offset)
: file_(file),
reporter_(reporter),
checksum_(checksum),
backing_store_(new char[kBlockSize]),
buffer_(),
eof_(false),
last_record_offset_(0),
end_of_buffer_offset_(0),
initial_offset_(initial_offset) {
}
Reader::~Reader() {
delete[] backing_store_;
}
bool Reader::SkipToInitialBlock() {
size_t offset_in_block = initial_offset_ % kBlockSize;
uint64_t block_start_location = initial_offset_ - offset_in_block;
// Don't search a block if we'd be in the trailer
if (offset_in_block > kBlockSize - 6) {
offset_in_block = 0;
block_start_location += kBlockSize;
}
end_of_buffer_offset_ = block_start_location;
// Skip to start of first block that can contain the initial record
if (block_start_location > 0) {
Status skip_status = file_->Skip(block_start_location);
if (!skip_status.ok()) {
ReportDrop(block_start_location, skip_status);
return false;
}
}
return true;
}
bool Reader::ReadRecord(Slice* record, std::string* scratch) {
if (last_record_offset_ < initial_offset_) {
if (!SkipToInitialBlock()) {
return false;
}
}
scratch->clear();
record->clear();
bool in_fragmented_record = false;
// Record offset of the logical record that we're reading
// 0 is a dummy value to make compilers happy
uint64_t prospective_record_offset = 0;
Slice fragment;
while (true) {
uint64_t physical_record_offset = end_of_buffer_offset_ - buffer_.size();
const unsigned int record_type = ReadPhysicalRecord(&fragment);
switch (record_type) {
case kFullType:
if (in_fragmented_record) {
// Handle bug in earlier versions of log::Writer where
// it could emit an empty kFirstType record at the tail end
// of a block followed by a kFullType or kFirstType record
// at the beginning of the next block.
if (scratch->empty()) {
in_fragmented_record = false;
} else {
ReportCorruption(scratch->size(), "partial record without end(1)");
}
}
prospective_record_offset = physical_record_offset;
scratch->clear();
*record = fragment;
last_record_offset_ = prospective_record_offset;
return true;
case kFirstType:
if (in_fragmented_record) {
// Handle bug in earlier versions of log::Writer where
// it could emit an empty kFirstType record at the tail end
// of a block followed by a kFullType or kFirstType record
// at the beginning of the next block.
if (scratch->empty()) {
in_fragmented_record = false;
} else {
ReportCorruption(scratch->size(), "partial record without end(2)");
}
}
prospective_record_offset = physical_record_offset;
scratch->assign(fragment.data(), fragment.size());
in_fragmented_record = true;
break;
case kMiddleType:
if (!in_fragmented_record) {
ReportCorruption(fragment.size(),
"missing start of fragmented record(1)");
} else {
scratch->append(fragment.data(), fragment.size());
}
break;
case kLastType:
if (!in_fragmented_record) {
ReportCorruption(fragment.size(),
"missing start of fragmented record(2)");
} else {
scratch->append(fragment.data(), fragment.size());
*record = Slice(*scratch);
last_record_offset_ = prospective_record_offset;
return true;
}
break;
case kEof:
if (in_fragmented_record) {
ReportCorruption(scratch->size(), "partial record without end(3)");
scratch->clear();
}
return false;
case kBadRecord:
if (in_fragmented_record) {
ReportCorruption(scratch->size(), "error in middle of record");
in_fragmented_record = false;
scratch->clear();
}
break;
default: {
char buf[40];
snprintf(buf, sizeof(buf), "unknown record type %u", record_type);
ReportCorruption(
(fragment.size() + (in_fragmented_record ? scratch->size() : 0)),
buf);
in_fragmented_record = false;
scratch->clear();
break;
}
}
}
return false;
}
uint64_t Reader::LastRecordOffset() {
return last_record_offset_;
}
void Reader::ReportCorruption(size_t bytes, const char* reason) {
ReportDrop(bytes, Status::Corruption(reason));
}
void Reader::ReportDrop(size_t bytes, const Status& reason) {
if (reporter_ != NULL &&
end_of_buffer_offset_ - buffer_.size() - bytes >= initial_offset_) {
reporter_->Corruption(bytes, reason);
}
}
unsigned int Reader::ReadPhysicalRecord(Slice* result) {
while (true) {
if (buffer_.size() < kHeaderSize) {
if (!eof_) {
// Last read was a full read, so this is a trailer to skip
buffer_.clear();
Status status = file_->Read(kBlockSize, &buffer_, backing_store_);
end_of_buffer_offset_ += buffer_.size();
if (!status.ok()) {
buffer_.clear();
ReportDrop(kBlockSize, status);
eof_ = true;
return kEof;
} else if (buffer_.size() < kBlockSize) {
eof_ = true;
}
continue;
} else if (buffer_.size() == 0) {
// End of file
return kEof;
} else {
size_t drop_size = buffer_.size();
buffer_.clear();
ReportCorruption(drop_size, "truncated record at end of file");
return kEof;
}
}
// Parse the header
const char* header = buffer_.data();
const uint32_t a = static_cast<uint32_t>(header[4]) & 0xff;
const uint32_t b = static_cast<uint32_t>(header[5]) & 0xff;
const unsigned int type = header[6];
const uint32_t length = a | (b << 8);
if (kHeaderSize + length > buffer_.size()) {
size_t drop_size = buffer_.size();
buffer_.clear();
ReportCorruption(drop_size, "bad record length");
return kBadRecord;
}
if (type == kZeroType && length == 0) {
// Skip zero length record without reporting any drops since
// such records are produced by the mmap based writing code in
// env_posix.cc that preallocates file regions.
buffer_.clear();
return kBadRecord;
}
// Check crc
if (checksum_) {
uint32_t expected_crc = crc32c::Unmask(DecodeFixed32(header));
uint32_t actual_crc = crc32c::Value(header + 6, 1 + length);
if (actual_crc != expected_crc) {
// Drop the rest of the buffer since "length" itself may have
// been corrupted and if we trust it, we could find some
// fragment of a real log record that just happens to look
// like a valid log record.
size_t drop_size = buffer_.size();
buffer_.clear();
ReportCorruption(drop_size, "checksum mismatch");
return kBadRecord;
}
}
buffer_.remove_prefix(kHeaderSize + length);
// Skip physical record that started before initial_offset_
if (end_of_buffer_offset_ - buffer_.size() - kHeaderSize - length <
initial_offset_) {
result->clear();
return kBadRecord;
}
*result = Slice(header + kHeaderSize, length);
return type;
}
}
} // namespace log
} // namespace leveldb
src/leveldb/db/log_reader.h
+108
View File
@@ -0,0 +1,108 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_LOG_READER_H_
#define STORAGE_LEVELDB_DB_LOG_READER_H_
#include <stdint.h>
#include "db/log_format.h"
#include "leveldb/slice.h"
#include "leveldb/status.h"
namespace leveldb {
class SequentialFile;
namespace log {
class Reader {
public:
// Interface for reporting errors.
class Reporter {
public:
virtual ~Reporter();
// Some corruption was detected. "size" is the approximate number
// of bytes dropped due to the corruption.
virtual void Corruption(size_t bytes, const Status& status) = 0;
};
// Create a reader that will return log records from "*file".
// "*file" must remain live while this Reader is in use.
//
// If "reporter" is non-NULL, it is notified whenever some data is
// dropped due to a detected corruption. "*reporter" must remain
// live while this Reader is in use.
//
// If "checksum" is true, verify checksums if available.
//
// The Reader will start reading at the first record located at physical
// position >= initial_offset within the file.
Reader(SequentialFile* file, Reporter* reporter, bool checksum,
uint64_t initial_offset);
~Reader();
// Read the next record into *record. Returns true if read
// successfully, false if we hit end of the input. May use
// "*scratch" as temporary storage. The contents filled in *record
// will only be valid until the next mutating operation on this
// reader or the next mutation to *scratch.
bool ReadRecord(Slice* record, std::string* scratch);
// Returns the physical offset of the last record returned by ReadRecord.
//
// Undefined before the first call to ReadRecord.
uint64_t LastRecordOffset();
private:
SequentialFile* const file_;
Reporter* const reporter_;
bool const checksum_;
char* const backing_store_;
Slice buffer_;
bool eof_; // Last Read() indicated EOF by returning < kBlockSize
// Offset of the last record returned by ReadRecord.
uint64_t last_record_offset_;
// Offset of the first location past the end of buffer_.
uint64_t end_of_buffer_offset_;
// Offset at which to start looking for the first record to return
uint64_t const initial_offset_;
// Extend record types with the following special values
enum {
kEof = kMaxRecordType + 1,
// Returned whenever we find an invalid physical record.
// Currently there are three situations in which this happens:
// * The record has an invalid CRC (ReadPhysicalRecord reports a drop)
// * The record is a 0-length record (No drop is reported)
// * The record is below constructor's initial_offset (No drop is reported)
kBadRecord = kMaxRecordType + 2
};
// Skips all blocks that are completely before "initial_offset_".
//
// Returns true on success. Handles reporting.
bool SkipToInitialBlock();
// Return type, or one of the preceding special values
unsigned int ReadPhysicalRecord(Slice* result);
// Reports dropped bytes to the reporter.
// buffer_ must be updated to remove the dropped bytes prior to invocation.
void ReportCorruption(size_t bytes, const char* reason);
void ReportDrop(size_t bytes, const Status& reason);
// No copying allowed
Reader(const Reader&);
void operator=(const Reader&);
};
} // namespace log
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_LOG_READER_H_
src/leveldb/db/log_test.cc
+500
View File
@@ -0,0 +1,500 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/log_reader.h"
#include "db/log_writer.h"
#include "leveldb/env.h"
#include "util/coding.h"
#include "util/crc32c.h"
#include "util/random.h"
#include "util/testharness.h"
namespace leveldb {
namespace log {
// Construct a string of the specified length made out of the supplied
// partial string.
static std::string BigString(const std::string& partial_string, size_t n) {
std::string result;
while (result.size() < n) {
result.append(partial_string);
}
result.resize(n);
return result;
}
// Construct a string from a number
static std::string NumberString(int n) {
char buf[50];
snprintf(buf, sizeof(buf), "%d.", n);
return std::string(buf);
}
// Return a skewed potentially long string
static std::string RandomSkewedString(int i, Random* rnd) {
return BigString(NumberString(i), rnd->Skewed(17));
}
class LogTest {
private:
class StringDest : public WritableFile {
public:
std::string contents_;
virtual Status Close() { return Status::OK(); }
virtual Status Flush() { return Status::OK(); }
virtual Status Sync() { return Status::OK(); }
virtual Status Append(const Slice& slice) {
contents_.append(slice.data(), slice.size());
return Status::OK();
}
};
class StringSource : public SequentialFile {
public:
Slice contents_;
bool force_error_;
bool returned_partial_;
StringSource() : force_error_(false), returned_partial_(false) { }
virtual Status Read(size_t n, Slice* result, char* scratch) {
ASSERT_TRUE(!returned_partial_) << "must not Read() after eof/error";
if (force_error_) {
force_error_ = false;
returned_partial_ = true;
return Status::Corruption("read error");
}
if (contents_.size() < n) {
n = contents_.size();
returned_partial_ = true;
}
*result = Slice(contents_.data(), n);
contents_.remove_prefix(n);
return Status::OK();
}
virtual Status Skip(uint64_t n) {
if (n > contents_.size()) {
contents_.clear();
return Status::NotFound("in-memory file skipepd past end");
}
contents_.remove_prefix(n);
return Status::OK();
}
};
class ReportCollector : public Reader::Reporter {
public:
size_t dropped_bytes_;
std::string message_;
ReportCollector() : dropped_bytes_(0) { }
virtual void Corruption(size_t bytes, const Status& status) {
dropped_bytes_ += bytes;
message_.append(status.ToString());
}
};
StringDest dest_;
StringSource source_;
ReportCollector report_;
bool reading_;
Writer writer_;
Reader reader_;
// Record metadata for testing initial offset functionality
static size_t initial_offset_record_sizes_[];
static uint64_t initial_offset_last_record_offsets_[];
public:
LogTest() : reading_(false),
writer_(&dest_),
reader_(&source_, &report_, true/*checksum*/,
0/*initial_offset*/) {
}
void Write(const std::string& msg) {
ASSERT_TRUE(!reading_) << "Write() after starting to read";
writer_.AddRecord(Slice(msg));
}
size_t WrittenBytes() const {
return dest_.contents_.size();
}
std::string Read() {
if (!reading_) {
reading_ = true;
source_.contents_ = Slice(dest_.contents_);
}
std::string scratch;
Slice record;
if (reader_.ReadRecord(&record, &scratch)) {
return record.ToString();
} else {
return "EOF";
}
}
void IncrementByte(int offset, int delta) {
dest_.contents_[offset] += delta;
}
void SetByte(int offset, char new_byte) {
dest_.contents_[offset] = new_byte;
}
void ShrinkSize(int bytes) {
dest_.contents_.resize(dest_.contents_.size() - bytes);
}
void FixChecksum(int header_offset, int len) {
// Compute crc of type/len/data
uint32_t crc = crc32c::Value(&dest_.contents_[header_offset+6], 1 + len);
crc = crc32c::Mask(crc);
EncodeFixed32(&dest_.contents_[header_offset], crc);
}
void ForceError() {
source_.force_error_ = true;
}
size_t DroppedBytes() const {
return report_.dropped_bytes_;
}
std::string ReportMessage() const {
return report_.message_;
}
// Returns OK iff recorded error message contains "msg"
std::string MatchError(const std::string& msg) const {
if (report_.message_.find(msg) == std::string::npos) {
return report_.message_;
} else {
return "OK";
}
}
void WriteInitialOffsetLog() {
for (int i = 0; i < 4; i++) {
std::string record(initial_offset_record_sizes_[i],
static_cast<char>('a' + i));
Write(record);
}
}
void CheckOffsetPastEndReturnsNoRecords(uint64_t offset_past_end) {
WriteInitialOffsetLog();
reading_ = true;
source_.contents_ = Slice(dest_.contents_);
Reader* offset_reader = new Reader(&source_, &report_, true/*checksum*/,
WrittenBytes() + offset_past_end);
Slice record;
std::string scratch;
ASSERT_TRUE(!offset_reader->ReadRecord(&record, &scratch));
delete offset_reader;
}
void CheckInitialOffsetRecord(uint64_t initial_offset,
int expected_record_offset) {
WriteInitialOffsetLog();
reading_ = true;
source_.contents_ = Slice(dest_.contents_);
Reader* offset_reader = new Reader(&source_, &report_, true/*checksum*/,
initial_offset);
Slice record;
std::string scratch;
ASSERT_TRUE(offset_reader->ReadRecord(&record, &scratch));
ASSERT_EQ(initial_offset_record_sizes_[expected_record_offset],
record.size());
ASSERT_EQ(initial_offset_last_record_offsets_[expected_record_offset],
offset_reader->LastRecordOffset());
ASSERT_EQ((char)('a' + expected_record_offset), record.data()[0]);
delete offset_reader;
}
};
size_t LogTest::initial_offset_record_sizes_[] =
{10000, // Two sizable records in first block
10000,
2 * log::kBlockSize - 1000, // Span three blocks
1};
uint64_t LogTest::initial_offset_last_record_offsets_[] =
{0,
kHeaderSize + 10000,
2 * (kHeaderSize + 10000),
2 * (kHeaderSize + 10000) +
(2 * log::kBlockSize - 1000) + 3 * kHeaderSize};
TEST(LogTest, Empty) {
ASSERT_EQ("EOF", Read());
}
TEST(LogTest, ReadWrite) {
Write("foo");
Write("bar");
Write("");
Write("xxxx");
ASSERT_EQ("foo", Read());
ASSERT_EQ("bar", Read());
ASSERT_EQ("", Read());
ASSERT_EQ("xxxx", Read());
ASSERT_EQ("EOF", Read());
ASSERT_EQ("EOF", Read()); // Make sure reads at eof work
}
TEST(LogTest, ManyBlocks) {
for (int i = 0; i < 100000; i++) {
Write(NumberString(i));
}
for (int i = 0; i < 100000; i++) {
ASSERT_EQ(NumberString(i), Read());
}
ASSERT_EQ("EOF", Read());
}
TEST(LogTest, Fragmentation) {
Write("small");
Write(BigString("medium", 50000));
Write(BigString("large", 100000));
ASSERT_EQ("small", Read());
ASSERT_EQ(BigString("medium", 50000), Read());
ASSERT_EQ(BigString("large", 100000), Read());
ASSERT_EQ("EOF", Read());
}
TEST(LogTest, MarginalTrailer) {
// Make a trailer that is exactly the same length as an empty record.
const int n = kBlockSize - 2*kHeaderSize;
Write(BigString("foo", n));
ASSERT_EQ(kBlockSize - kHeaderSize, WrittenBytes());
Write("");
Write("bar");
ASSERT_EQ(BigString("foo", n), Read());
ASSERT_EQ("", Read());
ASSERT_EQ("bar", Read());
ASSERT_EQ("EOF", Read());
}
TEST(LogTest, MarginalTrailer2) {
// Make a trailer that is exactly the same length as an empty record.
const int n = kBlockSize - 2*kHeaderSize;
Write(BigString("foo", n));
ASSERT_EQ(kBlockSize - kHeaderSize, WrittenBytes());
Write("bar");
ASSERT_EQ(BigString("foo", n), Read());
ASSERT_EQ("bar", Read());
ASSERT_EQ("EOF", Read());
ASSERT_EQ(0, DroppedBytes());
ASSERT_EQ("", ReportMessage());
}
TEST(LogTest, ShortTrailer) {
const int n = kBlockSize - 2*kHeaderSize + 4;
Write(BigString("foo", n));
ASSERT_EQ(kBlockSize - kHeaderSize + 4, WrittenBytes());
Write("");
Write("bar");
ASSERT_EQ(BigString("foo", n), Read());
ASSERT_EQ("", Read());
ASSERT_EQ("bar", Read());
ASSERT_EQ("EOF", Read());
}
TEST(LogTest, AlignedEof) {
const int n = kBlockSize - 2*kHeaderSize + 4;
Write(BigString("foo", n));
ASSERT_EQ(kBlockSize - kHeaderSize + 4, WrittenBytes());
ASSERT_EQ(BigString("foo", n), Read());
ASSERT_EQ("EOF", Read());
}
TEST(LogTest, RandomRead) {
const int N = 500;
Random write_rnd(301);
for (int i = 0; i < N; i++) {
Write(RandomSkewedString(i, &write_rnd));
}
Random read_rnd(301);
for (int i = 0; i < N; i++) {
ASSERT_EQ(RandomSkewedString(i, &read_rnd), Read());
}
ASSERT_EQ("EOF", Read());
}
// Tests of all the error paths in log_reader.cc follow:
TEST(LogTest, ReadError) {
Write("foo");
ForceError();
ASSERT_EQ("EOF", Read());
ASSERT_EQ(kBlockSize, DroppedBytes());
ASSERT_EQ("OK", MatchError("read error"));
}
TEST(LogTest, BadRecordType) {
Write("foo");
// Type is stored in header[6]
IncrementByte(6, 100);
FixChecksum(0, 3);
ASSERT_EQ("EOF", Read());
ASSERT_EQ(3, DroppedBytes());
ASSERT_EQ("OK", MatchError("unknown record type"));
}
TEST(LogTest, TruncatedTrailingRecord) {
Write("foo");
ShrinkSize(4); // Drop all payload as well as a header byte
ASSERT_EQ("EOF", Read());
ASSERT_EQ(kHeaderSize - 1, DroppedBytes());
ASSERT_EQ("OK", MatchError("truncated record at end of file"));
}
TEST(LogTest, BadLength) {
Write("foo");
ShrinkSize(1);
ASSERT_EQ("EOF", Read());
ASSERT_EQ(kHeaderSize + 2, DroppedBytes());
ASSERT_EQ("OK", MatchError("bad record length"));
}
TEST(LogTest, ChecksumMismatch) {
Write("foo");
IncrementByte(0, 10);
ASSERT_EQ("EOF", Read());
ASSERT_EQ(10, DroppedBytes());
ASSERT_EQ("OK", MatchError("checksum mismatch"));
}
TEST(LogTest, UnexpectedMiddleType) {
Write("foo");
SetByte(6, kMiddleType);
FixChecksum(0, 3);
ASSERT_EQ("EOF", Read());
ASSERT_EQ(3, DroppedBytes());
ASSERT_EQ("OK", MatchError("missing start"));
}
TEST(LogTest, UnexpectedLastType) {
Write("foo");
SetByte(6, kLastType);
FixChecksum(0, 3);
ASSERT_EQ("EOF", Read());
ASSERT_EQ(3, DroppedBytes());
ASSERT_EQ("OK", MatchError("missing start"));
}
TEST(LogTest, UnexpectedFullType) {
Write("foo");
Write("bar");
SetByte(6, kFirstType);
FixChecksum(0, 3);
ASSERT_EQ("bar", Read());
ASSERT_EQ("EOF", Read());
ASSERT_EQ(3, DroppedBytes());
ASSERT_EQ("OK", MatchError("partial record without end"));
}
TEST(LogTest, UnexpectedFirstType) {
Write("foo");
Write(BigString("bar", 100000));
SetByte(6, kFirstType);
FixChecksum(0, 3);
ASSERT_EQ(BigString("bar", 100000), Read());
ASSERT_EQ("EOF", Read());
ASSERT_EQ(3, DroppedBytes());
ASSERT_EQ("OK", MatchError("partial record without end"));
}
TEST(LogTest, ErrorJoinsRecords) {
// Consider two fragmented records:
// first(R1) last(R1) first(R2) last(R2)
// where the middle two fragments disappear. We do not want
// first(R1),last(R2) to get joined and returned as a valid record.
// Write records that span two blocks
Write(BigString("foo", kBlockSize));
Write(BigString("bar", kBlockSize));
Write("correct");
// Wipe the middle block
for (int offset = kBlockSize; offset < 2*kBlockSize; offset++) {
SetByte(offset, 'x');
}
ASSERT_EQ("correct", Read());
ASSERT_EQ("EOF", Read());
const int dropped = DroppedBytes();
ASSERT_LE(dropped, 2*kBlockSize + 100);
ASSERT_GE(dropped, 2*kBlockSize);
}
TEST(LogTest, ReadStart) {
CheckInitialOffsetRecord(0, 0);
}
TEST(LogTest, ReadSecondOneOff) {
CheckInitialOffsetRecord(1, 1);
}
TEST(LogTest, ReadSecondTenThousand) {
CheckInitialOffsetRecord(10000, 1);
}
TEST(LogTest, ReadSecondStart) {
CheckInitialOffsetRecord(10007, 1);
}
TEST(LogTest, ReadThirdOneOff) {
CheckInitialOffsetRecord(10008, 2);
}
TEST(LogTest, ReadThirdStart) {
CheckInitialOffsetRecord(20014, 2);
}
TEST(LogTest, ReadFourthOneOff) {
CheckInitialOffsetRecord(20015, 3);
}
TEST(LogTest, ReadFourthFirstBlockTrailer) {
CheckInitialOffsetRecord(log::kBlockSize - 4, 3);
}
TEST(LogTest, ReadFourthMiddleBlock) {
CheckInitialOffsetRecord(log::kBlockSize + 1, 3);
}
TEST(LogTest, ReadFourthLastBlock) {
CheckInitialOffsetRecord(2 * log::kBlockSize + 1, 3);
}
TEST(LogTest, ReadFourthStart) {
CheckInitialOffsetRecord(
2 * (kHeaderSize + 1000) + (2 * log::kBlockSize - 1000) + 3 * kHeaderSize,
3);
}
TEST(LogTest, ReadEnd) {
CheckOffsetPastEndReturnsNoRecords(0);
}
TEST(LogTest, ReadPastEnd) {
CheckOffsetPastEndReturnsNoRecords(5);
}
} // namespace log
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/log_writer.cc
+103
View File
@@ -0,0 +1,103 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/log_writer.h"
#include <stdint.h>
#include "leveldb/env.h"
#include "util/coding.h"
#include "util/crc32c.h"
namespace leveldb {
namespace log {
Writer::Writer(WritableFile* dest)
: dest_(dest),
block_offset_(0) {
for (int i = 0; i <= kMaxRecordType; i++) {
char t = static_cast<char>(i);
type_crc_[i] = crc32c::Value(&t, 1);
}
}
Writer::~Writer() {
}
Status Writer::AddRecord(const Slice& slice) {
const char* ptr = slice.data();
size_t left = slice.size();
// Fragment the record if necessary and emit it. Note that if slice
// is empty, we still want to iterate once to emit a single
// zero-length record
Status s;
bool begin = true;
do {
const int leftover = kBlockSize - block_offset_;
assert(leftover >= 0);
if (leftover < kHeaderSize) {
// Switch to a new block
if (leftover > 0) {
// Fill the trailer (literal below relies on kHeaderSize being 7)
assert(kHeaderSize == 7);
dest_->Append(Slice("\x00\x00\x00\x00\x00\x00", leftover));
}
block_offset_ = 0;
}
// Invariant: we never leave < kHeaderSize bytes in a block.
assert(kBlockSize - block_offset_ - kHeaderSize >= 0);
const size_t avail = kBlockSize - block_offset_ - kHeaderSize;
const size_t fragment_length = (left < avail) ? left : avail;
RecordType type;
const bool end = (left == fragment_length);
if (begin && end) {
type = kFullType;
} else if (begin) {
type = kFirstType;
} else if (end) {
type = kLastType;
} else {
type = kMiddleType;
}
s = EmitPhysicalRecord(type, ptr, fragment_length);
ptr += fragment_length;
left -= fragment_length;
begin = false;
} while (s.ok() && left > 0);
return s;
}
Status Writer::EmitPhysicalRecord(RecordType t, const char* ptr, size_t n) {
assert(n <= 0xffff); // Must fit in two bytes
assert(block_offset_ + kHeaderSize + n <= kBlockSize);
// Format the header
char buf[kHeaderSize];
buf[4] = static_cast<char>(n & 0xff);
buf[5] = static_cast<char>(n >> 8);
buf[6] = static_cast<char>(t);
// Compute the crc of the record type and the payload.
uint32_t crc = crc32c::Extend(type_crc_[t], ptr, n);
crc = crc32c::Mask(crc); // Adjust for storage
EncodeFixed32(buf, crc);
// Write the header and the payload
Status s = dest_->Append(Slice(buf, kHeaderSize));
if (s.ok()) {
s = dest_->Append(Slice(ptr, n));
if (s.ok()) {
s = dest_->Flush();
}
}
block_offset_ += kHeaderSize + n;
return s;
}
} // namespace log
} // namespace leveldb
src/leveldb/db/log_writer.h
+48
View File
@@ -0,0 +1,48 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_LOG_WRITER_H_
#define STORAGE_LEVELDB_DB_LOG_WRITER_H_
#include <stdint.h>
#include "db/log_format.h"
#include "leveldb/slice.h"
#include "leveldb/status.h"
namespace leveldb {
class WritableFile;
namespace log {
class Writer {
public:
// Create a writer that will append data to "*dest".
// "*dest" must be initially empty.
// "*dest" must remain live while this Writer is in use.
explicit Writer(WritableFile* dest);
~Writer();
Status AddRecord(const Slice& slice);
private:
WritableFile* dest_;
int block_offset_; // Current offset in block
// crc32c values for all supported record types. These are
// pre-computed to reduce the overhead of computing the crc of the
// record type stored in the header.
uint32_t type_crc_[kMaxRecordType + 1];
Status EmitPhysicalRecord(RecordType type, const char* ptr, size_t length);
// No copying allowed
Writer(const Writer&);
void operator=(const Writer&);
};
} // namespace log
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_LOG_WRITER_H_
src/leveldb/db/memtable.cc
+145
View File
@@ -0,0 +1,145 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/memtable.h"
#include "db/dbformat.h"
#include "leveldb/comparator.h"
#include "leveldb/env.h"
#include "leveldb/iterator.h"
#include "util/coding.h"
namespace leveldb {
static Slice GetLengthPrefixedSlice(const char* data) {
uint32_t len;
const char* p = data;
p = GetVarint32Ptr(p, p + 5, &len); // +5: we assume "p" is not corrupted
return Slice(p, len);
}
MemTable::MemTable(const InternalKeyComparator& cmp)
: comparator_(cmp),
refs_(0),
table_(comparator_, &arena_) {
}
MemTable::~MemTable() {
assert(refs_ == 0);
}
size_t MemTable::ApproximateMemoryUsage() { return arena_.MemoryUsage(); }
int MemTable::KeyComparator::operator()(const char* aptr, const char* bptr)
const {
// Internal keys are encoded as length-prefixed strings.
Slice a = GetLengthPrefixedSlice(aptr);
Slice b = GetLengthPrefixedSlice(bptr);
return comparator.Compare(a, b);
}
// Encode a suitable internal key target for "target" and return it.
// Uses *scratch as scratch space, and the returned pointer will point
// into this scratch space.
static const char* EncodeKey(std::string* scratch, const Slice& target) {
scratch->clear();
PutVarint32(scratch, target.size());
scratch->append(target.data(), target.size());
return scratch->data();
}
class MemTableIterator: public Iterator {
public:
explicit MemTableIterator(MemTable::Table* table) : iter_(table) { }
virtual bool Valid() const { return iter_.Valid(); }
virtual void Seek(const Slice& k) { iter_.Seek(EncodeKey(&tmp_, k)); }
virtual void SeekToFirst() { iter_.SeekToFirst(); }
virtual void SeekToLast() { iter_.SeekToLast(); }
virtual void Next() { iter_.Next(); }
virtual void Prev() { iter_.Prev(); }
virtual Slice key() const { return GetLengthPrefixedSlice(iter_.key()); }
virtual Slice value() const {
Slice key_slice = GetLengthPrefixedSlice(iter_.key());
return GetLengthPrefixedSlice(key_slice.data() + key_slice.size());
}
virtual Status status() const { return Status::OK(); }
private:
MemTable::Table::Iterator iter_;
std::string tmp_; // For passing to EncodeKey
// No copying allowed
MemTableIterator(const MemTableIterator&);
void operator=(const MemTableIterator&);
};
Iterator* MemTable::NewIterator() {
return new MemTableIterator(&table_);
}
void MemTable::Add(SequenceNumber s, ValueType type,
const Slice& key,
const Slice& value) {
// Format of an entry is concatenation of:
// key_size : varint32 of internal_key.size()
// key bytes : char[internal_key.size()]
// value_size : varint32 of value.size()
// value bytes : char[value.size()]
size_t key_size = key.size();
size_t val_size = value.size();
size_t internal_key_size = key_size + 8;
const size_t encoded_len =
VarintLength(internal_key_size) + internal_key_size +
VarintLength(val_size) + val_size;
char* buf = arena_.Allocate(encoded_len);
char* p = EncodeVarint32(buf, internal_key_size);
memcpy(p, key.data(), key_size);
p += key_size;
EncodeFixed64(p, (s << 8) | type);
p += 8;
p = EncodeVarint32(p, val_size);
memcpy(p, value.data(), val_size);
assert((p + val_size) - buf == encoded_len);
table_.Insert(buf);
}
bool MemTable::Get(const LookupKey& key, std::string* value, Status* s) {
Slice memkey = key.memtable_key();
Table::Iterator iter(&table_);
iter.Seek(memkey.data());
if (iter.Valid()) {
// entry format is:
// klength varint32
// userkey char[klength]
// tag uint64
// vlength varint32
// value char[vlength]
// Check that it belongs to same user key. We do not check the
// sequence number since the Seek() call above should have skipped
// all entries with overly large sequence numbers.
const char* entry = iter.key();
uint32_t key_length;
const char* key_ptr = GetVarint32Ptr(entry, entry+5, &key_length);
if (comparator_.comparator.user_comparator()->Compare(
Slice(key_ptr, key_length - 8),
key.user_key()) == 0) {
// Correct user key
const uint64_t tag = DecodeFixed64(key_ptr + key_length - 8);
switch (static_cast<ValueType>(tag & 0xff)) {
case kTypeValue: {
Slice v = GetLengthPrefixedSlice(key_ptr + key_length);
value->assign(v.data(), v.size());
return true;
}
case kTypeDeletion:
*s = Status::NotFound(Slice());
return true;
}
}
}
return false;
}
} // namespace leveldb
src/leveldb/db/memtable.h
+91
View File
@@ -0,0 +1,91 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_MEMTABLE_H_
#define STORAGE_LEVELDB_DB_MEMTABLE_H_
#include <string>
#include "leveldb/db.h"
#include "db/dbformat.h"
#include "db/skiplist.h"
#include "util/arena.h"
namespace leveldb {
class InternalKeyComparator;
class Mutex;
class MemTableIterator;
class MemTable {
public:
// MemTables are reference counted. The initial reference count
// is zero and the caller must call Ref() at least once.
explicit MemTable(const InternalKeyComparator& comparator);
// Increase reference count.
void Ref() { ++refs_; }
// Drop reference count. Delete if no more references exist.
void Unref() {
--refs_;
assert(refs_ >= 0);
if (refs_ <= 0) {
delete this;
}
}
// Returns an estimate of the number of bytes of data in use by this
// data structure.
//
// REQUIRES: external synchronization to prevent simultaneous
// operations on the same MemTable.
size_t ApproximateMemoryUsage();
// Return an iterator that yields the contents of the memtable.
//
// The caller must ensure that the underlying MemTable remains live
// while the returned iterator is live. The keys returned by this
// iterator are internal keys encoded by AppendInternalKey in the
// db/format.{h,cc} module.
Iterator* NewIterator();
// Add an entry into memtable that maps key to value at the
// specified sequence number and with the specified type.
// Typically value will be empty if type==kTypeDeletion.
void Add(SequenceNumber seq, ValueType type,
const Slice& key,
const Slice& value);
// If memtable contains a value for key, store it in *value and return true.
// If memtable contains a deletion for key, store a NotFound() error
// in *status and return true.
// Else, return false.
bool Get(const LookupKey& key, std::string* value, Status* s);
private:
~MemTable(); // Private since only Unref() should be used to delete it
struct KeyComparator {
const InternalKeyComparator comparator;
explicit KeyComparator(const InternalKeyComparator& c) : comparator(c) { }
int operator()(const char* a, const char* b) const;
};
friend class MemTableIterator;
friend class MemTableBackwardIterator;
typedef SkipList<const char*, KeyComparator> Table;
KeyComparator comparator_;
int refs_;
Arena arena_;
Table table_;
// No copying allowed
MemTable(const MemTable&);
void operator=(const MemTable&);
};
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_MEMTABLE_H_
src/leveldb/db/repair.cc
+462
View File
@@ -0,0 +1,462 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// We recover the contents of the descriptor from the other files we find.
// (1) Any log files are first converted to tables
// (2) We scan every table to compute
// (a) smallest/largest for the table
// (b) largest sequence number in the table
// (3) We generate descriptor contents:
// - log number is set to zero
// - next-file-number is set to 1 + largest file number we found
// - last-sequence-number is set to largest sequence# found across
// all tables (see 2c)
// - compaction pointers are cleared
// - every table file is added at level 0
//
// Possible optimization 1:
// (a) Compute total size and use to pick appropriate max-level M
// (b) Sort tables by largest sequence# in the table
// (c) For each table: if it overlaps earlier table, place in level-0,
// else place in level-M.
// Possible optimization 2:
// Store per-table metadata (smallest, largest, largest-seq#, ...)
// in the table's meta section to speed up ScanTable.
#include "db/builder.h"
#include "db/db_impl.h"
#include "db/dbformat.h"
#include "db/filename.h"
#include "db/log_reader.h"
#include "db/log_writer.h"
#include "db/memtable.h"
#include "db/table_cache.h"
#include "db/version_edit.h"
#include "db/write_batch_internal.h"
#include "leveldb/comparator.h"
#include "leveldb/db.h"
#include "leveldb/env.h"
namespace leveldb {
namespace {
class Repairer {
public:
Repairer(const std::string& dbname, const Options& options)
: dbname_(dbname),
env_(options.env),
icmp_(options.comparator),
ipolicy_(options.filter_policy),
options_(SanitizeOptions(dbname, &icmp_, &ipolicy_, options)),
owns_info_log_(options_.info_log != options.info_log),
owns_cache_(options_.block_cache != options.block_cache),
next_file_number_(1) {
// TableCache can be small since we expect each table to be opened once.
table_cache_ = new TableCache(dbname_, &options_, 10);
}
~Repairer() {
delete table_cache_;
if (owns_info_log_) {
delete options_.info_log;
}
if (owns_cache_) {
delete options_.block_cache;
}
}
Status Run() {
Status status = FindFiles();
if (status.ok()) {
ConvertLogFilesToTables();
ExtractMetaData();
status = WriteDescriptor();
}
if (status.ok()) {
unsigned long long bytes = 0;
for (size_t i = 0; i < tables_.size(); i++) {
bytes += tables_[i].meta.file_size;
}
Log(options_.info_log,
"**** Repaired leveldb %s; "
"recovered %d files; %llu bytes. "
"Some data may have been lost. "
"****",
dbname_.c_str(),
static_cast<int>(tables_.size()),
bytes);
}
return status;
}
private:
struct TableInfo {
FileMetaData meta;
SequenceNumber max_sequence;
};
std::string const dbname_;
Env* const env_;
InternalKeyComparator const icmp_;
InternalFilterPolicy const ipolicy_;
Options const options_;
bool owns_info_log_;
bool owns_cache_;
TableCache* table_cache_;
VersionEdit edit_;
std::vector<std::string> manifests_;
std::vector<uint64_t> table_numbers_;
std::vector<uint64_t> logs_;
std::vector<TableInfo> tables_;
uint64_t next_file_number_;
Status FindFiles() {
std::vector<std::string> filenames;
Status status = env_->GetChildren(dbname_, &filenames);
if (!status.ok()) {
return status;
}
if (filenames.empty()) {
return Status::IOError(dbname_, "repair found no files");
}
uint64_t number;
FileType type;
for (size_t i = 0; i < filenames.size(); i++) {
if (ParseFileName(filenames[i], &number, &type)) {
if (type == kDescriptorFile) {
manifests_.push_back(filenames[i]);
} else {
if (number + 1 > next_file_number_) {
next_file_number_ = number + 1;
}
if (type == kLogFile) {
logs_.push_back(number);
} else if (type == kTableFile) {
table_numbers_.push_back(number);
} else {
// Ignore other files
}
}
}
}
return status;
}
void ConvertLogFilesToTables() {
for (size_t i = 0; i < logs_.size(); i++) {
std::string logname = LogFileName(dbname_, logs_[i]);
Status status = ConvertLogToTable(logs_[i]);
if (!status.ok()) {
Log(options_.info_log, "Log #%llu: ignoring conversion error: %s",
(unsigned long long) logs_[i],
status.ToString().c_str());
}
ArchiveFile(logname);
}
}
Status ConvertLogToTable(uint64_t log) {
struct LogReporter : public log::Reader::Reporter {
Env* env;
Logger* info_log;
uint64_t lognum;
virtual void Corruption(size_t bytes, const Status& s) {
// We print error messages for corruption, but continue repairing.
Log(info_log, "Log #%llu: dropping %d bytes; %s",
(unsigned long long) lognum,
static_cast<int>(bytes),
s.ToString().c_str());
}
};
// Open the log file
std::string logname = LogFileName(dbname_, log);
SequentialFile* lfile;
Status status = env_->NewSequentialFile(logname, &lfile);
if (!status.ok()) {
return status;
}
// Create the log reader.
LogReporter reporter;
reporter.env = env_;
reporter.info_log = options_.info_log;
reporter.lognum = log;
// We intentially make log::Reader do checksumming so that
// corruptions cause entire commits to be skipped instead of
// propagating bad information (like overly large sequence
// numbers).
log::Reader reader(lfile, &reporter, false/*do not checksum*/,
0/*initial_offset*/);
// Read all the records and add to a memtable
std::string scratch;
Slice record;
WriteBatch batch;
MemTable* mem = new MemTable(icmp_);
mem->Ref();
int counter = 0;
while (reader.ReadRecord(&record, &scratch)) {
if (record.size() < 12) {
reporter.Corruption(
record.size(), Status::Corruption("log record too small"));
continue;
}
WriteBatchInternal::SetContents(&batch, record);
status = WriteBatchInternal::InsertInto(&batch, mem);
if (status.ok()) {
counter += WriteBatchInternal::Count(&batch);
} else {
Log(options_.info_log, "Log #%llu: ignoring %s",
(unsigned long long) log,
status.ToString().c_str());
status = Status::OK(); // Keep going with rest of file
}
}
delete lfile;
// Do not record a version edit for this conversion to a Table
// since ExtractMetaData() will also generate edits.
FileMetaData meta;
meta.number = next_file_number_++;
Iterator* iter = mem->NewIterator();
status = BuildTable(dbname_, env_, options_, table_cache_, iter, &meta);
delete iter;
mem->Unref();
mem = NULL;
if (status.ok()) {
if (meta.file_size > 0) {
table_numbers_.push_back(meta.number);
}
}
Log(options_.info_log, "Log #%llu: %d ops saved to Table #%llu %s",
(unsigned long long) log,
counter,
(unsigned long long) meta.number,
status.ToString().c_str());
return status;
}
void ExtractMetaData() {
std::vector<TableInfo> kept;
for (size_t i = 0; i < table_numbers_.size(); i++) {
ScanTable(table_numbers_[i]);
}
}
Iterator* NewTableIterator(const FileMetaData& meta) {
// Same as compaction iterators: if paranoid_checks are on, turn
// on checksum verification.
ReadOptions r;
r.verify_checksums = options_.paranoid_checks;
return table_cache_->NewIterator(r, meta.number, meta.file_size);
}
void ScanTable(uint64_t number) {
TableInfo t;
t.meta.number = number;
std::string fname = TableFileName(dbname_, number);
Status status = env_->GetFileSize(fname, &t.meta.file_size);
if (!status.ok()) {
// Try alternate file name.
fname = SSTTableFileName(dbname_, number);
Status s2 = env_->GetFileSize(fname, &t.meta.file_size);
if (s2.ok()) {
status = Status::OK();
}
}
if (!status.ok()) {
ArchiveFile(TableFileName(dbname_, number));
ArchiveFile(SSTTableFileName(dbname_, number));
Log(options_.info_log, "Table #%llu: dropped: %s",
(unsigned long long) t.meta.number,
status.ToString().c_str());
return;
}
// Extract metadata by scanning through table.
int counter = 0;
Iterator* iter = NewTableIterator(t.meta);
bool empty = true;
ParsedInternalKey parsed;
t.max_sequence = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
Slice key = iter->key();
if (!ParseInternalKey(key, &parsed)) {
Log(options_.info_log, "Table #%llu: unparsable key %s",
(unsigned long long) t.meta.number,
EscapeString(key).c_str());
continue;
}
counter++;
if (empty) {
empty = false;
t.meta.smallest.DecodeFrom(key);
}
t.meta.largest.DecodeFrom(key);
if (parsed.sequence > t.max_sequence) {
t.max_sequence = parsed.sequence;
}
}
if (!iter->status().ok()) {
status = iter->status();
}
delete iter;
Log(options_.info_log, "Table #%llu: %d entries %s",
(unsigned long long) t.meta.number,
counter,
status.ToString().c_str());
if (status.ok()) {
tables_.push_back(t);
} else {
RepairTable(fname, t); // RepairTable archives input file.
}
}
void RepairTable(const std::string& src, TableInfo t) {
// We will copy src contents to a new table and then rename the
// new table over the source.
// Create builder.
std::string copy = TableFileName(dbname_, next_file_number_++);
WritableFile* file;
Status s = env_->NewWritableFile(copy, &file);
if (!s.ok()) {
return;
}
TableBuilder* builder = new TableBuilder(options_, file);
// Copy data.
Iterator* iter = NewTableIterator(t.meta);
int counter = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
builder->Add(iter->key(), iter->value());
counter++;
}
delete iter;
ArchiveFile(src);
if (counter == 0) {
builder->Abandon(); // Nothing to save
} else {
s = builder->Finish();
if (s.ok()) {
t.meta.file_size = builder->FileSize();
}
}
delete builder;
builder = NULL;
if (s.ok()) {
s = file->Close();
}
delete file;
file = NULL;
if (counter > 0 && s.ok()) {
std::string orig = TableFileName(dbname_, t.meta.number);
s = env_->RenameFile(copy, orig);
if (s.ok()) {
Log(options_.info_log, "Table #%llu: %d entries repaired",
(unsigned long long) t.meta.number, counter);
tables_.push_back(t);
}
}
if (!s.ok()) {
env_->DeleteFile(copy);
}
}
Status WriteDescriptor() {
std::string tmp = TempFileName(dbname_, 1);
WritableFile* file;
Status status = env_->NewWritableFile(tmp, &file);
if (!status.ok()) {
return status;
}
SequenceNumber max_sequence = 0;
for (size_t i = 0; i < tables_.size(); i++) {
if (max_sequence < tables_[i].max_sequence) {
max_sequence = tables_[i].max_sequence;
}
}
edit_.SetComparatorName(icmp_.user_comparator()->Name());
edit_.SetLogNumber(0);
edit_.SetNextFile(next_file_number_);
edit_.SetLastSequence(max_sequence);
for (size_t i = 0; i < tables_.size(); i++) {
// TODO(opt): separate out into multiple levels
const TableInfo& t = tables_[i];
edit_.AddFile(0, t.meta.number, t.meta.file_size,
t.meta.smallest, t.meta.largest);
}
//fprintf(stderr, "NewDescriptor:\n%s\n", edit_.DebugString().c_str());
{
log::Writer log(file);
std::string record;
edit_.EncodeTo(&record);
status = log.AddRecord(record);
}
if (status.ok()) {
status = file->Close();
}
delete file;
file = NULL;
if (!status.ok()) {
env_->DeleteFile(tmp);
} else {
// Discard older manifests
for (size_t i = 0; i < manifests_.size(); i++) {
ArchiveFile(dbname_ + "/" + manifests_[i]);
}
// Install new manifest
status = env_->RenameFile(tmp, DescriptorFileName(dbname_, 1));
if (status.ok()) {
status = SetCurrentFile(env_, dbname_, 1);
} else {
env_->DeleteFile(tmp);
}
}
return status;
}
void ArchiveFile(const std::string& fname) {
// Move into another directory. E.g., for
// dir/foo
// rename to
// dir/lost/foo
const char* slash = strrchr(fname.c_str(), '/');
std::string new_dir;
if (slash != NULL) {
new_dir.assign(fname.data(), slash - fname.data());
}
new_dir.append("/lost");
env_->CreateDir(new_dir); // Ignore error
std::string new_file = new_dir;
new_file.append("/");
new_file.append((slash == NULL) ? fname.c_str() : slash + 1);
Status s = env_->RenameFile(fname, new_file);
Log(options_.info_log, "Archiving %s: %s\n",
fname.c_str(), s.ToString().c_str());
}
};
} // namespace
Status RepairDB(const std::string& dbname, const Options& options) {
Repairer repairer(dbname, options);
return repairer.Run();
}
} // namespace leveldb
src/leveldb/db/skiplist.h
+379
View File
@@ -0,0 +1,379 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// Thread safety
// -------------
//
// Writes require external synchronization, most likely a mutex.
// Reads require a guarantee that the SkipList will not be destroyed
// while the read is in progress. Apart from that, reads progress
// without any internal locking or synchronization.
//
// Invariants:
//
// (1) Allocated nodes are never deleted until the SkipList is
// destroyed. This is trivially guaranteed by the code since we
// never delete any skip list nodes.
//
// (2) The contents of a Node except for the next/prev pointers are
// immutable after the Node has been linked into the SkipList.
// Only Insert() modifies the list, and it is careful to initialize
// a node and use release-stores to publish the nodes in one or
// more lists.
//
// ... prev vs. next pointer ordering ...
#include <assert.h>
#include <stdlib.h>
#include "port/port.h"
#include "util/arena.h"
#include "util/random.h"
namespace leveldb {
class Arena;
template<typename Key, class Comparator>
class SkipList {
private:
struct Node;
public:
// Create a new SkipList object that will use "cmp" for comparing keys,
// and will allocate memory using "*arena". Objects allocated in the arena
// must remain allocated for the lifetime of the skiplist object.
explicit SkipList(Comparator cmp, Arena* arena);
// Insert key into the list.
// REQUIRES: nothing that compares equal to key is currently in the list.
void Insert(const Key& key);
// Returns true iff an entry that compares equal to key is in the list.
bool Contains(const Key& key) const;
// Iteration over the contents of a skip list
class Iterator {
public:
// Initialize an iterator over the specified list.
// The returned iterator is not valid.
explicit Iterator(const SkipList* list);
// Returns true iff the iterator is positioned at a valid node.
bool Valid() const;
// Returns the key at the current position.
// REQUIRES: Valid()
const Key& key() const;
// Advances to the next position.
// REQUIRES: Valid()
void Next();
// Advances to the previous position.
// REQUIRES: Valid()
void Prev();
// Advance to the first entry with a key >= target
void Seek(const Key& target);
// Position at the first entry in list.
// Final state of iterator is Valid() iff list is not empty.
void SeekToFirst();
// Position at the last entry in list.
// Final state of iterator is Valid() iff list is not empty.
void SeekToLast();
private:
const SkipList* list_;
Node* node_;
// Intentionally copyable
};
private:
enum { kMaxHeight = 12 };
// Immutable after construction
Comparator const compare_;
Arena* const arena_; // Arena used for allocations of nodes
Node* const head_;
// Modified only by Insert(). Read racily by readers, but stale
// values are ok.
port::AtomicPointer max_height_; // Height of the entire list
inline int GetMaxHeight() const {
return static_cast<int>(
reinterpret_cast<intptr_t>(max_height_.NoBarrier_Load()));
}
// Read/written only by Insert().
Random rnd_;
Node* NewNode(const Key& key, int height);
int RandomHeight();
bool Equal(const Key& a, const Key& b) const { return (compare_(a, b) == 0); }
// Return true if key is greater than the data stored in "n"
bool KeyIsAfterNode(const Key& key, Node* n) const;
// Return the earliest node that comes at or after key.
// Return NULL if there is no such node.
//
// If prev is non-NULL, fills prev[level] with pointer to previous
// node at "level" for every level in [0..max_height_-1].
Node* FindGreaterOrEqual(const Key& key, Node** prev) const;
// Return the latest node with a key < key.
// Return head_ if there is no such node.
Node* FindLessThan(const Key& key) const;
// Return the last node in the list.
// Return head_ if list is empty.
Node* FindLast() const;
// No copying allowed
SkipList(const SkipList&);
void operator=(const SkipList&);
};
// Implementation details follow
template<typename Key, class Comparator>
struct SkipList<Key,Comparator>::Node {
explicit Node(const Key& k) : key(k) { }
Key const key;
// Accessors/mutators for links. Wrapped in methods so we can
// add the appropriate barriers as necessary.
Node* Next(int n) {
assert(n >= 0);
// Use an 'acquire load' so that we observe a fully initialized
// version of the returned Node.
return reinterpret_cast<Node*>(next_[n].Acquire_Load());
}
void SetNext(int n, Node* x) {
assert(n >= 0);
// Use a 'release store' so that anybody who reads through this
// pointer observes a fully initialized version of the inserted node.
next_[n].Release_Store(x);
}
// No-barrier variants that can be safely used in a few locations.
Node* NoBarrier_Next(int n) {
assert(n >= 0);
return reinterpret_cast<Node*>(next_[n].NoBarrier_Load());
}
void NoBarrier_SetNext(int n, Node* x) {
assert(n >= 0);
next_[n].NoBarrier_Store(x);
}
private:
// Array of length equal to the node height. next_[0] is lowest level link.
port::AtomicPointer next_[1];
};
template<typename Key, class Comparator>
typename SkipList<Key,Comparator>::Node*
SkipList<Key,Comparator>::NewNode(const Key& key, int height) {
char* mem = arena_->AllocateAligned(
sizeof(Node) + sizeof(port::AtomicPointer) * (height - 1));
return new (mem) Node(key);
}
template<typename Key, class Comparator>
inline SkipList<Key,Comparator>::Iterator::Iterator(const SkipList* list) {
list_ = list;
node_ = NULL;
}
template<typename Key, class Comparator>
inline bool SkipList<Key,Comparator>::Iterator::Valid() const {
return node_ != NULL;
}
template<typename Key, class Comparator>
inline const Key& SkipList<Key,Comparator>::Iterator::key() const {
assert(Valid());
return node_->key;
}
template<typename Key, class Comparator>
inline void SkipList<Key,Comparator>::Iterator::Next() {
assert(Valid());
node_ = node_->Next(0);
}
template<typename Key, class Comparator>
inline void SkipList<Key,Comparator>::Iterator::Prev() {
// Instead of using explicit "prev" links, we just search for the
// last node that falls before key.
assert(Valid());
node_ = list_->FindLessThan(node_->key);
if (node_ == list_->head_) {
node_ = NULL;
}
}
template<typename Key, class Comparator>
inline void SkipList<Key,Comparator>::Iterator::Seek(const Key& target) {
node_ = list_->FindGreaterOrEqual(target, NULL);
}
template<typename Key, class Comparator>
inline void SkipList<Key,Comparator>::Iterator::SeekToFirst() {
node_ = list_->head_->Next(0);
}
template<typename Key, class Comparator>
inline void SkipList<Key,Comparator>::Iterator::SeekToLast() {
node_ = list_->FindLast();
if (node_ == list_->head_) {
node_ = NULL;
}
}
template<typename Key, class Comparator>
int SkipList<Key,Comparator>::RandomHeight() {
// Increase height with probability 1 in kBranching
static const unsigned int kBranching = 4;
int height = 1;
while (height < kMaxHeight && ((rnd_.Next() % kBranching) == 0)) {
height++;
}
assert(height > 0);
assert(height <= kMaxHeight);
return height;
}
template<typename Key, class Comparator>
bool SkipList<Key,Comparator>::KeyIsAfterNode(const Key& key, Node* n) const {
// NULL n is considered infinite
return (n != NULL) && (compare_(n->key, key) < 0);
}
template<typename Key, class Comparator>
typename SkipList<Key,Comparator>::Node* SkipList<Key,Comparator>::FindGreaterOrEqual(const Key& key, Node** prev)
const {
Node* x = head_;
int level = GetMaxHeight() - 1;
while (true) {
Node* next = x->Next(level);
if (KeyIsAfterNode(key, next)) {
// Keep searching in this list
x = next;
} else {
if (prev != NULL) prev[level] = x;
if (level == 0) {
return next;
} else {
// Switch to next list
level--;
}
}
}
}
template<typename Key, class Comparator>
typename SkipList<Key,Comparator>::Node*
SkipList<Key,Comparator>::FindLessThan(const Key& key) const {
Node* x = head_;
int level = GetMaxHeight() - 1;
while (true) {
assert(x == head_ || compare_(x->key, key) < 0);
Node* next = x->Next(level);
if (next == NULL || compare_(next->key, key) >= 0) {
if (level == 0) {
return x;
} else {
// Switch to next list
level--;
}
} else {
x = next;
}
}
}
template<typename Key, class Comparator>
typename SkipList<Key,Comparator>::Node* SkipList<Key,Comparator>::FindLast()
const {
Node* x = head_;
int level = GetMaxHeight() - 1;
while (true) {
Node* next = x->Next(level);
if (next == NULL) {
if (level == 0) {
return x;
} else {
// Switch to next list
level--;
}
} else {
x = next;
}
}
}
template<typename Key, class Comparator>
SkipList<Key,Comparator>::SkipList(Comparator cmp, Arena* arena)
: compare_(cmp),
arena_(arena),
head_(NewNode(0 /* any key will do */, kMaxHeight)),
max_height_(reinterpret_cast<void*>(1)),
rnd_(0xdeadbeef) {
for (int i = 0; i < kMaxHeight; i++) {
head_->SetNext(i, NULL);
}
}
template<typename Key, class Comparator>
void SkipList<Key,Comparator>::Insert(const Key& key) {
// TODO(opt): We can use a barrier-free variant of FindGreaterOrEqual()
// here since Insert() is externally synchronized.
Node* prev[kMaxHeight];
Node* x = FindGreaterOrEqual(key, prev);
// Our data structure does not allow duplicate insertion
assert(x == NULL || !Equal(key, x->key));
int height = RandomHeight();
if (height > GetMaxHeight()) {
for (int i = GetMaxHeight(); i < height; i++) {
prev[i] = head_;
}
//fprintf(stderr, "Change height from %d to %d\n", max_height_, height);
// It is ok to mutate max_height_ without any synchronization
// with concurrent readers. A concurrent reader that observes
// the new value of max_height_ will see either the old value of
// new level pointers from head_ (NULL), or a new value set in
// the loop below. In the former case the reader will
// immediately drop to the next level since NULL sorts after all
// keys. In the latter case the reader will use the new node.
max_height_.NoBarrier_Store(reinterpret_cast<void*>(height));
}
x = NewNode(key, height);
for (int i = 0; i < height; i++) {
// NoBarrier_SetNext() suffices since we will add a barrier when
// we publish a pointer to "x" in prev[i].
x->NoBarrier_SetNext(i, prev[i]->NoBarrier_Next(i));
prev[i]->SetNext(i, x);
}
}
template<typename Key, class Comparator>
bool SkipList<Key,Comparator>::Contains(const Key& key) const {
Node* x = FindGreaterOrEqual(key, NULL);
if (x != NULL && Equal(key, x->key)) {
return true;
} else {
return false;
}
}
} // namespace leveldb
src/leveldb/db/skiplist_test.cc
+378
View File
@@ -0,0 +1,378 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/skiplist.h"
#include <set>
#include "leveldb/env.h"
#include "util/arena.h"
#include "util/hash.h"
#include "util/random.h"
#include "util/testharness.h"
namespace leveldb {
typedef uint64_t Key;
struct Comparator {
int operator()(const Key& a, const Key& b) const {
if (a < b) {
return -1;
} else if (a > b) {
return +1;
} else {
return 0;
}
}
};
class SkipTest { };
TEST(SkipTest, Empty) {
Arena arena;
Comparator cmp;
SkipList<Key, Comparator> list(cmp, &arena);
ASSERT_TRUE(!list.Contains(10));
SkipList<Key, Comparator>::Iterator iter(&list);
ASSERT_TRUE(!iter.Valid());
iter.SeekToFirst();
ASSERT_TRUE(!iter.Valid());
iter.Seek(100);
ASSERT_TRUE(!iter.Valid());
iter.SeekToLast();
ASSERT_TRUE(!iter.Valid());
}
TEST(SkipTest, InsertAndLookup) {
const int N = 2000;
const int R = 5000;
Random rnd(1000);
std::set<Key> keys;
Arena arena;
Comparator cmp;
SkipList<Key, Comparator> list(cmp, &arena);
for (int i = 0; i < N; i++) {
Key key = rnd.Next() % R;
if (keys.insert(key).second) {
list.Insert(key);
}
}
for (int i = 0; i < R; i++) {
if (list.Contains(i)) {
ASSERT_EQ(keys.count(i), 1);
} else {
ASSERT_EQ(keys.count(i), 0);
}
}
// Simple iterator tests
{
SkipList<Key, Comparator>::Iterator iter(&list);
ASSERT_TRUE(!iter.Valid());
iter.Seek(0);
ASSERT_TRUE(iter.Valid());
ASSERT_EQ(*(keys.begin()), iter.key());
iter.SeekToFirst();
ASSERT_TRUE(iter.Valid());
ASSERT_EQ(*(keys.begin()), iter.key());
iter.SeekToLast();
ASSERT_TRUE(iter.Valid());
ASSERT_EQ(*(keys.rbegin()), iter.key());
}
// Forward iteration test
for (int i = 0; i < R; i++) {
SkipList<Key, Comparator>::Iterator iter(&list);
iter.Seek(i);
// Compare against model iterator
std::set<Key>::iterator model_iter = keys.lower_bound(i);
for (int j = 0; j < 3; j++) {
if (model_iter == keys.end()) {
ASSERT_TRUE(!iter.Valid());
break;
} else {
ASSERT_TRUE(iter.Valid());
ASSERT_EQ(*model_iter, iter.key());
++model_iter;
iter.Next();
}
}
}
// Backward iteration test
{
SkipList<Key, Comparator>::Iterator iter(&list);
iter.SeekToLast();
// Compare against model iterator
for (std::set<Key>::reverse_iterator model_iter = keys.rbegin();
model_iter != keys.rend();
++model_iter) {
ASSERT_TRUE(iter.Valid());
ASSERT_EQ(*model_iter, iter.key());
iter.Prev();
}
ASSERT_TRUE(!iter.Valid());
}
}
// We want to make sure that with a single writer and multiple
// concurrent readers (with no synchronization other than when a
// reader's iterator is created), the reader always observes all the
// data that was present in the skip list when the iterator was
// constructor. Because insertions are happening concurrently, we may
// also observe new values that were inserted since the iterator was
// constructed, but we should never miss any values that were present
// at iterator construction time.
//
// We generate multi-part keys:
// <key,gen,hash>
// where:
// key is in range [0..K-1]
// gen is a generation number for key
// hash is hash(key,gen)
//
// The insertion code picks a random key, sets gen to be 1 + the last
// generation number inserted for that key, and sets hash to Hash(key,gen).
//
// At the beginning of a read, we snapshot the last inserted
// generation number for each key. We then iterate, including random
// calls to Next() and Seek(). For every key we encounter, we
// check that it is either expected given the initial snapshot or has
// been concurrently added since the iterator started.
class ConcurrentTest {
private:
static const uint32_t K = 4;
static uint64_t key(Key key) { return (key >> 40); }
static uint64_t gen(Key key) { return (key >> 8) & 0xffffffffu; }
static uint64_t hash(Key key) { return key & 0xff; }
static uint64_t HashNumbers(uint64_t k, uint64_t g) {
uint64_t data[2] = { k, g };
return Hash(reinterpret_cast<char*>(data), sizeof(data), 0);
}
static Key MakeKey(uint64_t k, uint64_t g) {
assert(sizeof(Key) == sizeof(uint64_t));
assert(k <= K); // We sometimes pass K to seek to the end of the skiplist
assert(g <= 0xffffffffu);
return ((k << 40) | (g << 8) | (HashNumbers(k, g) & 0xff));
}
static bool IsValidKey(Key k) {
return hash(k) == (HashNumbers(key(k), gen(k)) & 0xff);
}
static Key RandomTarget(Random* rnd) {
switch (rnd->Next() % 10) {
case 0:
// Seek to beginning
return MakeKey(0, 0);
case 1:
// Seek to end
return MakeKey(K, 0);
default:
// Seek to middle
return MakeKey(rnd->Next() % K, 0);
}
}
// Per-key generation
struct State {
port::AtomicPointer generation[K];
void Set(int k, intptr_t v) {
generation[k].Release_Store(reinterpret_cast<void*>(v));
}
intptr_t Get(int k) {
return reinterpret_cast<intptr_t>(generation[k].Acquire_Load());
}
State() {
for (int k = 0; k < K; k++) {
Set(k, 0);
}
}
};
// Current state of the test
State current_;
Arena arena_;
// SkipList is not protected by mu_. We just use a single writer
// thread to modify it.
SkipList<Key, Comparator> list_;
public:
ConcurrentTest() : list_(Comparator(), &arena_) { }
// REQUIRES: External synchronization
void WriteStep(Random* rnd) {
const uint32_t k = rnd->Next() % K;
const intptr_t g = current_.Get(k) + 1;
const Key key = MakeKey(k, g);
list_.Insert(key);
current_.Set(k, g);
}
void ReadStep(Random* rnd) {
// Remember the initial committed state of the skiplist.
State initial_state;
for (int k = 0; k < K; k++) {
initial_state.Set(k, current_.Get(k));
}
Key pos = RandomTarget(rnd);
SkipList<Key, Comparator>::Iterator iter(&list_);
iter.Seek(pos);
while (true) {
Key current;
if (!iter.Valid()) {
current = MakeKey(K, 0);
} else {
current = iter.key();
ASSERT_TRUE(IsValidKey(current)) << current;
}
ASSERT_LE(pos, current) << "should not go backwards";
// Verify that everything in [pos,current) was not present in
// initial_state.
while (pos < current) {
ASSERT_LT(key(pos), K) << pos;
// Note that generation 0 is never inserted, so it is ok if
// <*,0,*> is missing.
ASSERT_TRUE((gen(pos) == 0) ||
(gen(pos) > initial_state.Get(key(pos)))
) << "key: " << key(pos)
<< "; gen: " << gen(pos)
<< "; initgen: "
<< initial_state.Get(key(pos));
// Advance to next key in the valid key space
if (key(pos) < key(current)) {
pos = MakeKey(key(pos) + 1, 0);
} else {
pos = MakeKey(key(pos), gen(pos) + 1);
}
}
if (!iter.Valid()) {
break;
}
if (rnd->Next() % 2) {
iter.Next();
pos = MakeKey(key(pos), gen(pos) + 1);
} else {
Key new_target = RandomTarget(rnd);
if (new_target > pos) {
pos = new_target;
iter.Seek(new_target);
}
}
}
}
};
const uint32_t ConcurrentTest::K;
// Simple test that does single-threaded testing of the ConcurrentTest
// scaffolding.
TEST(SkipTest, ConcurrentWithoutThreads) {
ConcurrentTest test;
Random rnd(test::RandomSeed());
for (int i = 0; i < 10000; i++) {
test.ReadStep(&rnd);
test.WriteStep(&rnd);
}
}
class TestState {
public:
ConcurrentTest t_;
int seed_;
port::AtomicPointer quit_flag_;
enum ReaderState {
STARTING,
RUNNING,
DONE
};
explicit TestState(int s)
: seed_(s),
quit_flag_(NULL),
state_(STARTING),
state_cv_(&mu_) {}
void Wait(ReaderState s) {
mu_.Lock();
while (state_ != s) {
state_cv_.Wait();
}
mu_.Unlock();
}
void Change(ReaderState s) {
mu_.Lock();
state_ = s;
state_cv_.Signal();
mu_.Unlock();
}
private:
port::Mutex mu_;
ReaderState state_;
port::CondVar state_cv_;
};
static void ConcurrentReader(void* arg) {
TestState* state = reinterpret_cast<TestState*>(arg);
Random rnd(state->seed_);
int64_t reads = 0;
state->Change(TestState::RUNNING);
while (!state->quit_flag_.Acquire_Load()) {
state->t_.ReadStep(&rnd);
++reads;
}
state->Change(TestState::DONE);
}
static void RunConcurrent(int run) {
const int seed = test::RandomSeed() + (run * 100);
Random rnd(seed);
const int N = 1000;
const int kSize = 1000;
for (int i = 0; i < N; i++) {
if ((i % 100) == 0) {
fprintf(stderr, "Run %d of %d\n", i, N);
}
TestState state(seed + 1);
Env::Default()->Schedule(ConcurrentReader, &state);
state.Wait(TestState::RUNNING);
for (int i = 0; i < kSize; i++) {
state.t_.WriteStep(&rnd);
}
state.quit_flag_.Release_Store(&state); // Any non-NULL arg will do
state.Wait(TestState::DONE);
}
}
TEST(SkipTest, Concurrent1) { RunConcurrent(1); }
TEST(SkipTest, Concurrent2) { RunConcurrent(2); }
TEST(SkipTest, Concurrent3) { RunConcurrent(3); }
TEST(SkipTest, Concurrent4) { RunConcurrent(4); }
TEST(SkipTest, Concurrent5) { RunConcurrent(5); }
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/snapshot.h
+66
View File
@@ -0,0 +1,66 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_SNAPSHOT_H_
#define STORAGE_LEVELDB_DB_SNAPSHOT_H_
#include "leveldb/db.h"
namespace leveldb {
class SnapshotList;
// Snapshots are kept in a doubly-linked list in the DB.
// Each SnapshotImpl corresponds to a particular sequence number.
class SnapshotImpl : public Snapshot {
public:
SequenceNumber number_; // const after creation
private:
friend class SnapshotList;
// SnapshotImpl is kept in a doubly-linked circular list
SnapshotImpl* prev_;
SnapshotImpl* next_;
SnapshotList* list_; // just for sanity checks
};
class SnapshotList {
public:
SnapshotList() {
list_.prev_ = &list_;
list_.next_ = &list_;
}
bool empty() const { return list_.next_ == &list_; }
SnapshotImpl* oldest() const { assert(!empty()); return list_.next_; }
SnapshotImpl* newest() const { assert(!empty()); return list_.prev_; }
const SnapshotImpl* New(SequenceNumber seq) {
SnapshotImpl* s = new SnapshotImpl;
s->number_ = seq;
s->list_ = this;
s->next_ = &list_;
s->prev_ = list_.prev_;
s->prev_->next_ = s;
s->next_->prev_ = s;
return s;
}
void Delete(const SnapshotImpl* s) {
assert(s->list_ == this);
s->prev_->next_ = s->next_;
s->next_->prev_ = s->prev_;
delete s;
}
private:
// Dummy head of doubly-linked list of snapshots
SnapshotImpl list_;
};
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_SNAPSHOT_H_
src/leveldb/db/table_cache.cc
+127
View File
@@ -0,0 +1,127 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/table_cache.h"
#include "db/filename.h"
#include "leveldb/env.h"
#include "leveldb/table.h"
#include "util/coding.h"
namespace leveldb {
struct TableAndFile {
RandomAccessFile* file;
Table* table;
};
static void DeleteEntry(const Slice& key, void* value) {
TableAndFile* tf = reinterpret_cast<TableAndFile*>(value);
delete tf->table;
delete tf->file;
delete tf;
}
static void UnrefEntry(void* arg1, void* arg2) {
Cache* cache = reinterpret_cast<Cache*>(arg1);
Cache::Handle* h = reinterpret_cast<Cache::Handle*>(arg2);
cache->Release(h);
}
TableCache::TableCache(const std::string& dbname,
const Options* options,
int entries)
: env_(options->env),
dbname_(dbname),
options_(options),
cache_(NewLRUCache(entries)) {
}
TableCache::~TableCache() {
delete cache_;
}
Status TableCache::FindTable(uint64_t file_number, uint64_t file_size,
Cache::Handle** handle) {
Status s;
char buf[sizeof(file_number)];
EncodeFixed64(buf, file_number);
Slice key(buf, sizeof(buf));
*handle = cache_->Lookup(key);
if (*handle == NULL) {
std::string fname = TableFileName(dbname_, file_number);
RandomAccessFile* file = NULL;
Table* table = NULL;
s = env_->NewRandomAccessFile(fname, &file);
if (!s.ok()) {
std::string old_fname = SSTTableFileName(dbname_, file_number);
if (env_->NewRandomAccessFile(old_fname, &file).ok()) {
s = Status::OK();
}
}
if (s.ok()) {
s = Table::Open(*options_, file, file_size, &table);
}
if (!s.ok()) {
assert(table == NULL);
delete file;
// We do not cache error results so that if the error is transient,
// or somebody repairs the file, we recover automatically.
} else {
TableAndFile* tf = new TableAndFile;
tf->file = file;
tf->table = table;
*handle = cache_->Insert(key, tf, 1, &DeleteEntry);
}
}
return s;
}
Iterator* TableCache::NewIterator(const ReadOptions& options,
uint64_t file_number,
uint64_t file_size,
Table** tableptr) {
if (tableptr != NULL) {
*tableptr = NULL;
}
Cache::Handle* handle = NULL;
Status s = FindTable(file_number, file_size, &handle);
if (!s.ok()) {
return NewErrorIterator(s);
}
Table* table = reinterpret_cast<TableAndFile*>(cache_->Value(handle))->table;
Iterator* result = table->NewIterator(options);
result->RegisterCleanup(&UnrefEntry, cache_, handle);
if (tableptr != NULL) {
*tableptr = table;
}
return result;
}
Status TableCache::Get(const ReadOptions& options,
uint64_t file_number,
uint64_t file_size,
const Slice& k,
void* arg,
void (*saver)(void*, const Slice&, const Slice&)) {
Cache::Handle* handle = NULL;
Status s = FindTable(file_number, file_size, &handle);
if (s.ok()) {
Table* t = reinterpret_cast<TableAndFile*>(cache_->Value(handle))->table;
s = t->InternalGet(options, k, arg, saver);
cache_->Release(handle);
}
return s;
}
void TableCache::Evict(uint64_t file_number) {
char buf[sizeof(file_number)];
EncodeFixed64(buf, file_number);
cache_->Erase(Slice(buf, sizeof(buf)));
}
} // namespace leveldb
src/leveldb/db/table_cache.h
+61
View File
@@ -0,0 +1,61 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// Thread-safe (provides internal synchronization)
#ifndef STORAGE_LEVELDB_DB_TABLE_CACHE_H_
#define STORAGE_LEVELDB_DB_TABLE_CACHE_H_
#include <string>
#include <stdint.h>
#include "db/dbformat.h"
#include "leveldb/cache.h"
#include "leveldb/table.h"
#include "port/port.h"
namespace leveldb {
class Env;
class TableCache {
public:
TableCache(const std::string& dbname, const Options* options, int entries);
~TableCache();
// Return an iterator for the specified file number (the corresponding
// file length must be exactly "file_size" bytes). If "tableptr" is
// non-NULL, also sets "*tableptr" to point to the Table object
// underlying the returned iterator, or NULL if no Table object underlies
// the returned iterator. The returned "*tableptr" object is owned by
// the cache and should not be deleted, and is valid for as long as the
// returned iterator is live.
Iterator* NewIterator(const ReadOptions& options,
uint64_t file_number,
uint64_t file_size,
Table** tableptr = NULL);
// If a seek to internal key "k" in specified file finds an entry,
// call (*handle_result)(arg, found_key, found_value).
Status Get(const ReadOptions& options,
uint64_t file_number,
uint64_t file_size,
const Slice& k,
void* arg,
void (*handle_result)(void*, const Slice&, const Slice&));
// Evict any entry for the specified file number
void Evict(uint64_t file_number);
private:
Env* const env_;
const std::string dbname_;
const Options* options_;
Cache* cache_;
Status FindTable(uint64_t file_number, uint64_t file_size, Cache::Handle**);
};
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_TABLE_CACHE_H_
src/leveldb/db/version_edit.cc
+266
View File
@@ -0,0 +1,266 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/version_edit.h"
#include "db/version_set.h"
#include "util/coding.h"
namespace leveldb {
// Tag numbers for serialized VersionEdit. These numbers are written to
// disk and should not be changed.
enum Tag {
kComparator = 1,
kLogNumber = 2,
kNextFileNumber = 3,
kLastSequence = 4,
kCompactPointer = 5,
kDeletedFile = 6,
kNewFile = 7,
// 8 was used for large value refs
kPrevLogNumber = 9
};
void VersionEdit::Clear() {
comparator_.clear();
log_number_ = 0;
prev_log_number_ = 0;
last_sequence_ = 0;
next_file_number_ = 0;
has_comparator_ = false;
has_log_number_ = false;
has_prev_log_number_ = false;
has_next_file_number_ = false;
has_last_sequence_ = false;
deleted_files_.clear();
new_files_.clear();
}
void VersionEdit::EncodeTo(std::string* dst) const {
if (has_comparator_) {
PutVarint32(dst, kComparator);
PutLengthPrefixedSlice(dst, comparator_);
}
if (has_log_number_) {
PutVarint32(dst, kLogNumber);
PutVarint64(dst, log_number_);
}
if (has_prev_log_number_) {
PutVarint32(dst, kPrevLogNumber);
PutVarint64(dst, prev_log_number_);
}
if (has_next_file_number_) {
PutVarint32(dst, kNextFileNumber);
PutVarint64(dst, next_file_number_);
}
if (has_last_sequence_) {
PutVarint32(dst, kLastSequence);
PutVarint64(dst, last_sequence_);
}
for (size_t i = 0; i < compact_pointers_.size(); i++) {
PutVarint32(dst, kCompactPointer);
PutVarint32(dst, compact_pointers_[i].first); // level
PutLengthPrefixedSlice(dst, compact_pointers_[i].second.Encode());
}
for (DeletedFileSet::const_iterator iter = deleted_files_.begin();
iter != deleted_files_.end();
++iter) {
PutVarint32(dst, kDeletedFile);
PutVarint32(dst, iter->first); // level
PutVarint64(dst, iter->second); // file number
}
for (size_t i = 0; i < new_files_.size(); i++) {
const FileMetaData& f = new_files_[i].second;
PutVarint32(dst, kNewFile);
PutVarint32(dst, new_files_[i].first); // level
PutVarint64(dst, f.number);
PutVarint64(dst, f.file_size);
PutLengthPrefixedSlice(dst, f.smallest.Encode());
PutLengthPrefixedSlice(dst, f.largest.Encode());
}
}
static bool GetInternalKey(Slice* input, InternalKey* dst) {
Slice str;
if (GetLengthPrefixedSlice(input, &str)) {
dst->DecodeFrom(str);
return true;
} else {
return false;
}
}
static bool GetLevel(Slice* input, int* level) {
uint32_t v;
if (GetVarint32(input, &v) &&
v < config::kNumLevels) {
*level = v;
return true;
} else {
return false;
}
}
Status VersionEdit::DecodeFrom(const Slice& src) {
Clear();
Slice input = src;
const char* msg = NULL;
uint32_t tag;
// Temporary storage for parsing
int level;
uint64_t number;
FileMetaData f;
Slice str;
InternalKey key;
while (msg == NULL && GetVarint32(&input, &tag)) {
switch (tag) {
case kComparator:
if (GetLengthPrefixedSlice(&input, &str)) {
comparator_ = str.ToString();
has_comparator_ = true;
} else {
msg = "comparator name";
}
break;
case kLogNumber:
if (GetVarint64(&input, &log_number_)) {
has_log_number_ = true;
} else {
msg = "log number";
}
break;
case kPrevLogNumber:
if (GetVarint64(&input, &prev_log_number_)) {
has_prev_log_number_ = true;
} else {
msg = "previous log number";
}
break;
case kNextFileNumber:
if (GetVarint64(&input, &next_file_number_)) {
has_next_file_number_ = true;
} else {
msg = "next file number";
}
break;
case kLastSequence:
if (GetVarint64(&input, &last_sequence_)) {
has_last_sequence_ = true;
} else {
msg = "last sequence number";
}
break;
case kCompactPointer:
if (GetLevel(&input, &level) &&
GetInternalKey(&input, &key)) {
compact_pointers_.push_back(std::make_pair(level, key));
} else {
msg = "compaction pointer";
}
break;
case kDeletedFile:
if (GetLevel(&input, &level) &&
GetVarint64(&input, &number)) {
deleted_files_.insert(std::make_pair(level, number));
} else {
msg = "deleted file";
}
break;
case kNewFile:
if (GetLevel(&input, &level) &&
GetVarint64(&input, &f.number) &&
GetVarint64(&input, &f.file_size) &&
GetInternalKey(&input, &f.smallest) &&
GetInternalKey(&input, &f.largest)) {
new_files_.push_back(std::make_pair(level, f));
} else {
msg = "new-file entry";
}
break;
default:
msg = "unknown tag";
break;
}
}
if (msg == NULL && !input.empty()) {
msg = "invalid tag";
}
Status result;
if (msg != NULL) {
result = Status::Corruption("VersionEdit", msg);
}
return result;
}
std::string VersionEdit::DebugString() const {
std::string r;
r.append("VersionEdit {");
if (has_comparator_) {
r.append("\n Comparator: ");
r.append(comparator_);
}
if (has_log_number_) {
r.append("\n LogNumber: ");
AppendNumberTo(&r, log_number_);
}
if (has_prev_log_number_) {
r.append("\n PrevLogNumber: ");
AppendNumberTo(&r, prev_log_number_);
}
if (has_next_file_number_) {
r.append("\n NextFile: ");
AppendNumberTo(&r, next_file_number_);
}
if (has_last_sequence_) {
r.append("\n LastSeq: ");
AppendNumberTo(&r, last_sequence_);
}
for (size_t i = 0; i < compact_pointers_.size(); i++) {
r.append("\n CompactPointer: ");
AppendNumberTo(&r, compact_pointers_[i].first);
r.append(" ");
r.append(compact_pointers_[i].second.DebugString());
}
for (DeletedFileSet::const_iterator iter = deleted_files_.begin();
iter != deleted_files_.end();
++iter) {
r.append("\n DeleteFile: ");
AppendNumberTo(&r, iter->first);
r.append(" ");
AppendNumberTo(&r, iter->second);
}
for (size_t i = 0; i < new_files_.size(); i++) {
const FileMetaData& f = new_files_[i].second;
r.append("\n AddFile: ");
AppendNumberTo(&r, new_files_[i].first);
r.append(" ");
AppendNumberTo(&r, f.number);
r.append(" ");
AppendNumberTo(&r, f.file_size);
r.append(" ");
r.append(f.smallest.DebugString());
r.append(" .. ");
r.append(f.largest.DebugString());
}
r.append("\n}\n");
return r;
}
} // namespace leveldb
src/leveldb/db/version_edit.h
+107
View File
@@ -0,0 +1,107 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_VERSION_EDIT_H_
#define STORAGE_LEVELDB_DB_VERSION_EDIT_H_
#include <set>
#include <utility>
#include <vector>
#include "db/dbformat.h"
namespace leveldb {
class VersionSet;
struct FileMetaData {
int refs;
int allowed_seeks; // Seeks allowed until compaction
uint64_t number;
uint64_t file_size; // File size in bytes
InternalKey smallest; // Smallest internal key served by table
InternalKey largest; // Largest internal key served by table
FileMetaData() : refs(0), allowed_seeks(1 << 30), file_size(0) { }
};
class VersionEdit {
public:
VersionEdit() { Clear(); }
~VersionEdit() { }
void Clear();
void SetComparatorName(const Slice& name) {
has_comparator_ = true;
comparator_ = name.ToString();
}
void SetLogNumber(uint64_t num) {
has_log_number_ = true;
log_number_ = num;
}
void SetPrevLogNumber(uint64_t num) {
has_prev_log_number_ = true;
prev_log_number_ = num;
}
void SetNextFile(uint64_t num) {
has_next_file_number_ = true;
next_file_number_ = num;
}
void SetLastSequence(SequenceNumber seq) {
has_last_sequence_ = true;
last_sequence_ = seq;
}
void SetCompactPointer(int level, const InternalKey& key) {
compact_pointers_.push_back(std::make_pair(level, key));
}
// Add the specified file at the specified number.
// REQUIRES: This version has not been saved (see VersionSet::SaveTo)
// REQUIRES: "smallest" and "largest" are smallest and largest keys in file
void AddFile(int level, uint64_t file,
uint64_t file_size,
const InternalKey& smallest,
const InternalKey& largest) {
FileMetaData f;
f.number = file;
f.file_size = file_size;
f.smallest = smallest;
f.largest = largest;
new_files_.push_back(std::make_pair(level, f));
}
// Delete the specified "file" from the specified "level".
void DeleteFile(int level, uint64_t file) {
deleted_files_.insert(std::make_pair(level, file));
}
void EncodeTo(std::string* dst) const;
Status DecodeFrom(const Slice& src);
std::string DebugString() const;
private:
friend class VersionSet;
typedef std::set< std::pair<int, uint64_t> > DeletedFileSet;
std::string comparator_;
uint64_t log_number_;
uint64_t prev_log_number_;
uint64_t next_file_number_;
SequenceNumber last_sequence_;
bool has_comparator_;
bool has_log_number_;
bool has_prev_log_number_;
bool has_next_file_number_;
bool has_last_sequence_;
std::vector< std::pair<int, InternalKey> > compact_pointers_;
DeletedFileSet deleted_files_;
std::vector< std::pair<int, FileMetaData> > new_files_;
};
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_VERSION_EDIT_H_
src/leveldb/db/version_edit_test.cc
+46
View File
@@ -0,0 +1,46 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/version_edit.h"
#include "util/testharness.h"
namespace leveldb {
static void TestEncodeDecode(const VersionEdit& edit) {
std::string encoded, encoded2;
edit.EncodeTo(&encoded);
VersionEdit parsed;
Status s = parsed.DecodeFrom(encoded);
ASSERT_TRUE(s.ok()) << s.ToString();
parsed.EncodeTo(&encoded2);
ASSERT_EQ(encoded, encoded2);
}
class VersionEditTest { };
TEST(VersionEditTest, EncodeDecode) {
static const uint64_t kBig = 1ull << 50;
VersionEdit edit;
for (int i = 0; i < 4; i++) {
TestEncodeDecode(edit);
edit.AddFile(3, kBig + 300 + i, kBig + 400 + i,
InternalKey("foo", kBig + 500 + i, kTypeValue),
InternalKey("zoo", kBig + 600 + i, kTypeDeletion));
edit.DeleteFile(4, kBig + 700 + i);
edit.SetCompactPointer(i, InternalKey("x", kBig + 900 + i, kTypeValue));
}
edit.SetComparatorName("foo");
edit.SetLogNumber(kBig + 100);
edit.SetNextFile(kBig + 200);
edit.SetLastSequence(kBig + 1000);
TestEncodeDecode(edit);
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/version_set.cc
+1498
View File
File diff suppressed because it is too large Load Diff
src/leveldb/db/version_set.h
+396
View File
@@ -0,0 +1,396 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// The representation of a DBImpl consists of a set of Versions. The
// newest version is called "current". Older versions may be kept
// around to provide a consistent view to live iterators.
//
// Each Version keeps track of a set of Table files per level. The
// entire set of versions is maintained in a VersionSet.
//
// Version,VersionSet are thread-compatible, but require external
// synchronization on all accesses.
#ifndef STORAGE_LEVELDB_DB_VERSION_SET_H_
#define STORAGE_LEVELDB_DB_VERSION_SET_H_
#include <map>
#include <set>
#include <vector>
#include "db/dbformat.h"
#include "db/version_edit.h"
#include "port/port.h"
#include "port/thread_annotations.h"
namespace leveldb {
namespace log { class Writer; }
class Compaction;
class Iterator;
class MemTable;
class TableBuilder;
class TableCache;
class Version;
class VersionSet;
class WritableFile;
// Return the smallest index i such that files[i]->largest >= key.
// Return files.size() if there is no such file.
// REQUIRES: "files" contains a sorted list of non-overlapping files.
extern int FindFile(const InternalKeyComparator& icmp,
const std::vector<FileMetaData*>& files,
const Slice& key);
// Returns true iff some file in "files" overlaps the user key range
// [*smallest,*largest].
// smallest==NULL represents a key smaller than all keys in the DB.
// largest==NULL represents a key largest than all keys in the DB.
// REQUIRES: If disjoint_sorted_files, files[] contains disjoint ranges
// in sorted order.
extern bool SomeFileOverlapsRange(
const InternalKeyComparator& icmp,
bool disjoint_sorted_files,
const std::vector<FileMetaData*>& files,
const Slice* smallest_user_key,
const Slice* largest_user_key);
class Version {
public:
// Append to *iters a sequence of iterators that will
// yield the contents of this Version when merged together.
// REQUIRES: This version has been saved (see VersionSet::SaveTo)
void AddIterators(const ReadOptions&, std::vector<Iterator*>* iters);
// Lookup the value for key. If found, store it in *val and
// return OK. Else return a non-OK status. Fills *stats.
// REQUIRES: lock is not held
struct GetStats {
FileMetaData* seek_file;
int seek_file_level;
};
Status Get(const ReadOptions&, const LookupKey& key, std::string* val,
GetStats* stats);
// Adds "stats" into the current state. Returns true if a new
// compaction may need to be triggered, false otherwise.
// REQUIRES: lock is held
bool UpdateStats(const GetStats& stats);
// Record a sample of bytes read at the specified internal key.
// Samples are taken approximately once every config::kReadBytesPeriod
// bytes. Returns true if a new compaction may need to be triggered.
// REQUIRES: lock is held
bool RecordReadSample(Slice key);
// Reference count management (so Versions do not disappear out from
// under live iterators)
void Ref();
void Unref();
void GetOverlappingInputs(
int level,
const InternalKey* begin, // NULL means before all keys
const InternalKey* end, // NULL means after all keys
std::vector<FileMetaData*>* inputs);
// Returns true iff some file in the specified level overlaps
// some part of [*smallest_user_key,*largest_user_key].
// smallest_user_key==NULL represents a key smaller than all keys in the DB.
// largest_user_key==NULL represents a key largest than all keys in the DB.
bool OverlapInLevel(int level,
const Slice* smallest_user_key,
const Slice* largest_user_key);
// Return the level at which we should place a new memtable compaction
// result that covers the range [smallest_user_key,largest_user_key].
int PickLevelForMemTableOutput(const Slice& smallest_user_key,
const Slice& largest_user_key);
int NumFiles(int level) const { return files_[level].size(); }
// Return a human readable string that describes this version's contents.
std::string DebugString() const;
private:
friend class Compaction;
friend class VersionSet;
class LevelFileNumIterator;
Iterator* NewConcatenatingIterator(const ReadOptions&, int level) const;
// Call func(arg, level, f) for every file that overlaps user_key in
// order from newest to oldest. If an invocation of func returns
// false, makes no more calls.
//
// REQUIRES: user portion of internal_key == user_key.
void ForEachOverlapping(Slice user_key, Slice internal_key,
void* arg,
bool (*func)(void*, int, FileMetaData*));
VersionSet* vset_; // VersionSet to which this Version belongs
Version* next_; // Next version in linked list
Version* prev_; // Previous version in linked list
int refs_; // Number of live refs to this version
// List of files per level
std::vector<FileMetaData*> files_[config::kNumLevels];
// Next file to compact based on seek stats.
FileMetaData* file_to_compact_;
int file_to_compact_level_;
// Level that should be compacted next and its compaction score.
// Score < 1 means compaction is not strictly needed. These fields
// are initialized by Finalize().
double compaction_score_;
int compaction_level_;
explicit Version(VersionSet* vset)
: vset_(vset), next_(this), prev_(this), refs_(0),
file_to_compact_(NULL),
file_to_compact_level_(-1),
compaction_score_(-1),
compaction_level_(-1) {
}
~Version();
// No copying allowed
Version(const Version&);
void operator=(const Version&);
};
class VersionSet {
public:
VersionSet(const std::string& dbname,
const Options* options,
TableCache* table_cache,
const InternalKeyComparator*);
~VersionSet();
// Apply *edit to the current version to form a new descriptor that
// is both saved to persistent state and installed as the new
// current version. Will release *mu while actually writing to the file.
// REQUIRES: *mu is held on entry.
// REQUIRES: no other thread concurrently calls LogAndApply()
Status LogAndApply(VersionEdit* edit, port::Mutex* mu)
EXCLUSIVE_LOCKS_REQUIRED(mu);
// Recover the last saved descriptor from persistent storage.
Status Recover();
// Return the current version.
Version* current() const { return current_; }
// Return the current manifest file number
uint64_t ManifestFileNumber() const { return manifest_file_number_; }
// Allocate and return a new file number
uint64_t NewFileNumber() { return next_file_number_++; }
// Arrange to reuse "file_number" unless a newer file number has
// already been allocated.
// REQUIRES: "file_number" was returned by a call to NewFileNumber().
void ReuseFileNumber(uint64_t file_number) {
if (next_file_number_ == file_number + 1) {
next_file_number_ = file_number;
}
}
// Return the number of Table files at the specified level.
int NumLevelFiles(int level) const;
// Return the combined file size of all files at the specified level.
int64_t NumLevelBytes(int level) const;
// Return the last sequence number.
uint64_t LastSequence() const { return last_sequence_; }
// Set the last sequence number to s.
void SetLastSequence(uint64_t s) {
assert(s >= last_sequence_);
last_sequence_ = s;
}
// Mark the specified file number as used.
void MarkFileNumberUsed(uint64_t number);
// Return the current log file number.
uint64_t LogNumber() const { return log_number_; }
// Return the log file number for the log file that is currently
// being compacted, or zero if there is no such log file.
uint64_t PrevLogNumber() const { return prev_log_number_; }
// Pick level and inputs for a new compaction.
// Returns NULL if there is no compaction to be done.
// Otherwise returns a pointer to a heap-allocated object that
// describes the compaction. Caller should delete the result.
Compaction* PickCompaction();
// Return a compaction object for compacting the range [begin,end] in
// the specified level. Returns NULL if there is nothing in that
// level that overlaps the specified range. Caller should delete
// the result.
Compaction* CompactRange(
int level,
const InternalKey* begin,
const InternalKey* end);
// Return the maximum overlapping data (in bytes) at next level for any
// file at a level >= 1.
int64_t MaxNextLevelOverlappingBytes();
// Create an iterator that reads over the compaction inputs for "*c".
// The caller should delete the iterator when no longer needed.
Iterator* MakeInputIterator(Compaction* c);
// Returns true iff some level needs a compaction.
bool NeedsCompaction() const {
Version* v = current_;
return (v->compaction_score_ >= 1) || (v->file_to_compact_ != NULL);
}
// Add all files listed in any live version to *live.
// May also mutate some internal state.
void AddLiveFiles(std::set<uint64_t>* live);
// Return the approximate offset in the database of the data for
// "key" as of version "v".
uint64_t ApproximateOffsetOf(Version* v, const InternalKey& key);
// Return a human-readable short (single-line) summary of the number
// of files per level. Uses *scratch as backing store.
struct LevelSummaryStorage {
char buffer[100];
};
const char* LevelSummary(LevelSummaryStorage* scratch) const;
private:
class Builder;
friend class Compaction;
friend class Version;
void Finalize(Version* v);
void GetRange(const std::vector<FileMetaData*>& inputs,
InternalKey* smallest,
InternalKey* largest);
void GetRange2(const std::vector<FileMetaData*>& inputs1,
const std::vector<FileMetaData*>& inputs2,
InternalKey* smallest,
InternalKey* largest);
void SetupOtherInputs(Compaction* c);
// Save current contents to *log
Status WriteSnapshot(log::Writer* log);
void AppendVersion(Version* v);
Env* const env_;
const std::string dbname_;
const Options* const options_;
TableCache* const table_cache_;
const InternalKeyComparator icmp_;
uint64_t next_file_number_;
uint64_t manifest_file_number_;
uint64_t last_sequence_;
uint64_t log_number_;
uint64_t prev_log_number_; // 0 or backing store for memtable being compacted
// Opened lazily
WritableFile* descriptor_file_;
log::Writer* descriptor_log_;
Version dummy_versions_; // Head of circular doubly-linked list of versions.
Version* current_; // == dummy_versions_.prev_
// Per-level key at which the next compaction at that level should start.
// Either an empty string, or a valid InternalKey.
std::string compact_pointer_[config::kNumLevels];
// No copying allowed
VersionSet(const VersionSet&);
void operator=(const VersionSet&);
};
// A Compaction encapsulates information about a compaction.
class Compaction {
public:
~Compaction();
// Return the level that is being compacted. Inputs from "level"
// and "level+1" will be merged to produce a set of "level+1" files.
int level() const { return level_; }
// Return the object that holds the edits to the descriptor done
// by this compaction.
VersionEdit* edit() { return &edit_; }
// "which" must be either 0 or 1
int num_input_files(int which) const { return inputs_[which].size(); }
// Return the ith input file at "level()+which" ("which" must be 0 or 1).
FileMetaData* input(int which, int i) const { return inputs_[which][i]; }
// Maximum size of files to build during this compaction.
uint64_t MaxOutputFileSize() const { return max_output_file_size_; }
// Is this a trivial compaction that can be implemented by just
// moving a single input file to the next level (no merging or splitting)
bool IsTrivialMove() const;
// Add all inputs to this compaction as delete operations to *edit.
void AddInputDeletions(VersionEdit* edit);
// Returns true if the information we have available guarantees that
// the compaction is producing data in "level+1" for which no data exists
// in levels greater than "level+1".
bool IsBaseLevelForKey(const Slice& user_key);
// Returns true iff we should stop building the current output
// before processing "internal_key".
bool ShouldStopBefore(const Slice& internal_key);
// Release the input version for the compaction, once the compaction
// is successful.
void ReleaseInputs();
private:
friend class Version;
friend class VersionSet;
explicit Compaction(int level);
int level_;
uint64_t max_output_file_size_;
Version* input_version_;
VersionEdit edit_;
// Each compaction reads inputs from "level_" and "level_+1"
std::vector<FileMetaData*> inputs_[2]; // The two sets of inputs
// State used to check for number of of overlapping grandparent files
// (parent == level_ + 1, grandparent == level_ + 2)
std::vector<FileMetaData*> grandparents_;
size_t grandparent_index_; // Index in grandparent_starts_
bool seen_key_; // Some output key has been seen
int64_t overlapped_bytes_; // Bytes of overlap between current output
// and grandparent files
// State for implementing IsBaseLevelForKey
// level_ptrs_ holds indices into input_version_->levels_: our state
// is that we are positioned at one of the file ranges for each
// higher level than the ones involved in this compaction (i.e. for
// all L >= level_ + 2).
size_t level_ptrs_[config::kNumLevels];
};
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_VERSION_SET_H_
src/leveldb/db/version_set_test.cc
+179
View File
@@ -0,0 +1,179 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "db/version_set.h"
#include "util/logging.h"
#include "util/testharness.h"
#include "util/testutil.h"
namespace leveldb {
class FindFileTest {
public:
std::vector<FileMetaData*> files_;
bool disjoint_sorted_files_;
FindFileTest() : disjoint_sorted_files_(true) { }
~FindFileTest() {
for (int i = 0; i < files_.size(); i++) {
delete files_[i];
}
}
void Add(const char* smallest, const char* largest,
SequenceNumber smallest_seq = 100,
SequenceNumber largest_seq = 100) {
FileMetaData* f = new FileMetaData;
f->number = files_.size() + 1;
f->smallest = InternalKey(smallest, smallest_seq, kTypeValue);
f->largest = InternalKey(largest, largest_seq, kTypeValue);
files_.push_back(f);
}
int Find(const char* key) {
InternalKey target(key, 100, kTypeValue);
InternalKeyComparator cmp(BytewiseComparator());
return FindFile(cmp, files_, target.Encode());
}
bool Overlaps(const char* smallest, const char* largest) {
InternalKeyComparator cmp(BytewiseComparator());
Slice s(smallest != NULL ? smallest : "");
Slice l(largest != NULL ? largest : "");
return SomeFileOverlapsRange(cmp, disjoint_sorted_files_, files_,
(smallest != NULL ? &s : NULL),
(largest != NULL ? &l : NULL));
}
};
TEST(FindFileTest, Empty) {
ASSERT_EQ(0, Find("foo"));
ASSERT_TRUE(! Overlaps("a", "z"));
ASSERT_TRUE(! Overlaps(NULL, "z"));
ASSERT_TRUE(! Overlaps("a", NULL));
ASSERT_TRUE(! Overlaps(NULL, NULL));
}
TEST(FindFileTest, Single) {
Add("p", "q");
ASSERT_EQ(0, Find("a"));
ASSERT_EQ(0, Find("p"));
ASSERT_EQ(0, Find("p1"));
ASSERT_EQ(0, Find("q"));
ASSERT_EQ(1, Find("q1"));
ASSERT_EQ(1, Find("z"));
ASSERT_TRUE(! Overlaps("a", "b"));
ASSERT_TRUE(! Overlaps("z1", "z2"));
ASSERT_TRUE(Overlaps("a", "p"));
ASSERT_TRUE(Overlaps("a", "q"));
ASSERT_TRUE(Overlaps("a", "z"));
ASSERT_TRUE(Overlaps("p", "p1"));
ASSERT_TRUE(Overlaps("p", "q"));
ASSERT_TRUE(Overlaps("p", "z"));
ASSERT_TRUE(Overlaps("p1", "p2"));
ASSERT_TRUE(Overlaps("p1", "z"));
ASSERT_TRUE(Overlaps("q", "q"));
ASSERT_TRUE(Overlaps("q", "q1"));
ASSERT_TRUE(! Overlaps(NULL, "j"));
ASSERT_TRUE(! Overlaps("r", NULL));
ASSERT_TRUE(Overlaps(NULL, "p"));
ASSERT_TRUE(Overlaps(NULL, "p1"));
ASSERT_TRUE(Overlaps("q", NULL));
ASSERT_TRUE(Overlaps(NULL, NULL));
}
TEST(FindFileTest, Multiple) {
Add("150", "200");
Add("200", "250");
Add("300", "350");
Add("400", "450");
ASSERT_EQ(0, Find("100"));
ASSERT_EQ(0, Find("150"));
ASSERT_EQ(0, Find("151"));
ASSERT_EQ(0, Find("199"));
ASSERT_EQ(0, Find("200"));
ASSERT_EQ(1, Find("201"));
ASSERT_EQ(1, Find("249"));
ASSERT_EQ(1, Find("250"));
ASSERT_EQ(2, Find("251"));
ASSERT_EQ(2, Find("299"));
ASSERT_EQ(2, Find("300"));
ASSERT_EQ(2, Find("349"));
ASSERT_EQ(2, Find("350"));
ASSERT_EQ(3, Find("351"));
ASSERT_EQ(3, Find("400"));
ASSERT_EQ(3, Find("450"));
ASSERT_EQ(4, Find("451"));
ASSERT_TRUE(! Overlaps("100", "149"));
ASSERT_TRUE(! Overlaps("251", "299"));
ASSERT_TRUE(! Overlaps("451", "500"));
ASSERT_TRUE(! Overlaps("351", "399"));
ASSERT_TRUE(Overlaps("100", "150"));
ASSERT_TRUE(Overlaps("100", "200"));
ASSERT_TRUE(Overlaps("100", "300"));
ASSERT_TRUE(Overlaps("100", "400"));
ASSERT_TRUE(Overlaps("100", "500"));
ASSERT_TRUE(Overlaps("375", "400"));
ASSERT_TRUE(Overlaps("450", "450"));
ASSERT_TRUE(Overlaps("450", "500"));
}
TEST(FindFileTest, MultipleNullBoundaries) {
Add("150", "200");
Add("200", "250");
Add("300", "350");
Add("400", "450");
ASSERT_TRUE(! Overlaps(NULL, "149"));
ASSERT_TRUE(! Overlaps("451", NULL));
ASSERT_TRUE(Overlaps(NULL, NULL));
ASSERT_TRUE(Overlaps(NULL, "150"));
ASSERT_TRUE(Overlaps(NULL, "199"));
ASSERT_TRUE(Overlaps(NULL, "200"));
ASSERT_TRUE(Overlaps(NULL, "201"));
ASSERT_TRUE(Overlaps(NULL, "400"));
ASSERT_TRUE(Overlaps(NULL, "800"));
ASSERT_TRUE(Overlaps("100", NULL));
ASSERT_TRUE(Overlaps("200", NULL));
ASSERT_TRUE(Overlaps("449", NULL));
ASSERT_TRUE(Overlaps("450", NULL));
}
TEST(FindFileTest, OverlapSequenceChecks) {
Add("200", "200", 5000, 3000);
ASSERT_TRUE(! Overlaps("199", "199"));
ASSERT_TRUE(! Overlaps("201", "300"));
ASSERT_TRUE(Overlaps("200", "200"));
ASSERT_TRUE(Overlaps("190", "200"));
ASSERT_TRUE(Overlaps("200", "210"));
}
TEST(FindFileTest, OverlappingFiles) {
Add("150", "600");
Add("400", "500");
disjoint_sorted_files_ = false;
ASSERT_TRUE(! Overlaps("100", "149"));
ASSERT_TRUE(! Overlaps("601", "700"));
ASSERT_TRUE(Overlaps("100", "150"));
ASSERT_TRUE(Overlaps("100", "200"));
ASSERT_TRUE(Overlaps("100", "300"));
ASSERT_TRUE(Overlaps("100", "400"));
ASSERT_TRUE(Overlaps("100", "500"));
ASSERT_TRUE(Overlaps("375", "400"));
ASSERT_TRUE(Overlaps("450", "450"));
ASSERT_TRUE(Overlaps("450", "500"));
ASSERT_TRUE(Overlaps("450", "700"));
ASSERT_TRUE(Overlaps("600", "700"));
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/db/write_batch.cc
+147
View File
@@ -0,0 +1,147 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// WriteBatch::rep_ :=
// sequence: fixed64
// count: fixed32
// data: record[count]
// record :=
// kTypeValue varstring varstring |
// kTypeDeletion varstring
// varstring :=
// len: varint32
// data: uint8[len]
#include "leveldb/write_batch.h"
#include "leveldb/db.h"
#include "db/dbformat.h"
#include "db/memtable.h"
#include "db/write_batch_internal.h"
#include "util/coding.h"
namespace leveldb {
// WriteBatch header has an 8-byte sequence number followed by a 4-byte count.
static const size_t kHeader = 12;
WriteBatch::WriteBatch() {
Clear();
}
WriteBatch::~WriteBatch() { }
WriteBatch::Handler::~Handler() { }
void WriteBatch::Clear() {
rep_.clear();
rep_.resize(kHeader);
}
Status WriteBatch::Iterate(Handler* handler) const {
Slice input(rep_);
if (input.size() < kHeader) {
return Status::Corruption("malformed WriteBatch (too small)");
}
input.remove_prefix(kHeader);
Slice key, value;
int found = 0;
while (!input.empty()) {
found++;
char tag = input[0];
input.remove_prefix(1);
switch (tag) {
case kTypeValue:
if (GetLengthPrefixedSlice(&input, &key) &&
GetLengthPrefixedSlice(&input, &value)) {
handler->Put(key, value);
} else {
return Status::Corruption("bad WriteBatch Put");
}
break;
case kTypeDeletion:
if (GetLengthPrefixedSlice(&input, &key)) {
handler->Delete(key);
} else {
return Status::Corruption("bad WriteBatch Delete");
}
break;
default:
return Status::Corruption("unknown WriteBatch tag");
}
}
if (found != WriteBatchInternal::Count(this)) {
return Status::Corruption("WriteBatch has wrong count");
} else {
return Status::OK();
}
}
int WriteBatchInternal::Count(const WriteBatch* b) {
return DecodeFixed32(b->rep_.data() + 8);
}
void WriteBatchInternal::SetCount(WriteBatch* b, int n) {
EncodeFixed32(&b->rep_[8], n);
}
SequenceNumber WriteBatchInternal::Sequence(const WriteBatch* b) {
return SequenceNumber(DecodeFixed64(b->rep_.data()));
}
void WriteBatchInternal::SetSequence(WriteBatch* b, SequenceNumber seq) {
EncodeFixed64(&b->rep_[0], seq);
}
void WriteBatch::Put(const Slice& key, const Slice& value) {
WriteBatchInternal::SetCount(this, WriteBatchInternal::Count(this) + 1);
rep_.push_back(static_cast<char>(kTypeValue));
PutLengthPrefixedSlice(&rep_, key);
PutLengthPrefixedSlice(&rep_, value);
}
void WriteBatch::Delete(const Slice& key) {
WriteBatchInternal::SetCount(this, WriteBatchInternal::Count(this) + 1);
rep_.push_back(static_cast<char>(kTypeDeletion));
PutLengthPrefixedSlice(&rep_, key);
}
namespace {
class MemTableInserter : public WriteBatch::Handler {
public:
SequenceNumber sequence_;
MemTable* mem_;
virtual void Put(const Slice& key, const Slice& value) {
mem_->Add(sequence_, kTypeValue, key, value);
sequence_++;
}
virtual void Delete(const Slice& key) {
mem_->Add(sequence_, kTypeDeletion, key, Slice());
sequence_++;
}
};
} // namespace
Status WriteBatchInternal::InsertInto(const WriteBatch* b,
MemTable* memtable) {
MemTableInserter inserter;
inserter.sequence_ = WriteBatchInternal::Sequence(b);
inserter.mem_ = memtable;
return b->Iterate(&inserter);
}
void WriteBatchInternal::SetContents(WriteBatch* b, const Slice& contents) {
assert(contents.size() >= kHeader);
b->rep_.assign(contents.data(), contents.size());
}
void WriteBatchInternal::Append(WriteBatch* dst, const WriteBatch* src) {
SetCount(dst, Count(dst) + Count(src));
assert(src->rep_.size() >= kHeader);
dst->rep_.append(src->rep_.data() + kHeader, src->rep_.size() - kHeader);
}
} // namespace leveldb
src/leveldb/db/write_batch_internal.h
+49
View File
@@ -0,0 +1,49 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
#define STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
#include "leveldb/write_batch.h"
namespace leveldb {
class MemTable;
// WriteBatchInternal provides static methods for manipulating a
// WriteBatch that we don't want in the public WriteBatch interface.
class WriteBatchInternal {
public:
// Return the number of entries in the batch.
static int Count(const WriteBatch* batch);
// Set the count for the number of entries in the batch.
static void SetCount(WriteBatch* batch, int n);
// Return the seqeunce number for the start of this batch.
static SequenceNumber Sequence(const WriteBatch* batch);
// Store the specified number as the seqeunce number for the start of
// this batch.
static void SetSequence(WriteBatch* batch, SequenceNumber seq);
static Slice Contents(const WriteBatch* batch) {
return Slice(batch->rep_);
}
static size_t ByteSize(const WriteBatch* batch) {
return batch->rep_.size();
}
static void SetContents(WriteBatch* batch, const Slice& contents);
static Status InsertInto(const WriteBatch* batch, MemTable* memtable);
static void Append(WriteBatch* dst, const WriteBatch* src);
};
} // namespace leveldb
#endif // STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
src/leveldb/db/write_batch_test.cc
+120
View File
@@ -0,0 +1,120 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "leveldb/db.h"
#include "db/memtable.h"
#include "db/write_batch_internal.h"
#include "leveldb/env.h"
#include "util/logging.h"
#include "util/testharness.h"
namespace leveldb {
static std::string PrintContents(WriteBatch* b) {
InternalKeyComparator cmp(BytewiseComparator());
MemTable* mem = new MemTable(cmp);
mem->Ref();
std::string state;
Status s = WriteBatchInternal::InsertInto(b, mem);
int count = 0;
Iterator* iter = mem->NewIterator();
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
ParsedInternalKey ikey;
ASSERT_TRUE(ParseInternalKey(iter->key(), &ikey));
switch (ikey.type) {
case kTypeValue:
state.append("Put(");
state.append(ikey.user_key.ToString());
state.append(", ");
state.append(iter->value().ToString());
state.append(")");
count++;
break;
case kTypeDeletion:
state.append("Delete(");
state.append(ikey.user_key.ToString());
state.append(")");
count++;
break;
}
state.append("@");
state.append(NumberToString(ikey.sequence));
}
delete iter;
if (!s.ok()) {
state.append("ParseError()");
} else if (count != WriteBatchInternal::Count(b)) {
state.append("CountMismatch()");
}
mem->Unref();
return state;
}
class WriteBatchTest { };
TEST(WriteBatchTest, Empty) {
WriteBatch batch;
ASSERT_EQ("", PrintContents(&batch));
ASSERT_EQ(0, WriteBatchInternal::Count(&batch));
}
TEST(WriteBatchTest, Multiple) {
WriteBatch batch;
batch.Put(Slice("foo"), Slice("bar"));
batch.Delete(Slice("box"));
batch.Put(Slice("baz"), Slice("boo"));
WriteBatchInternal::SetSequence(&batch, 100);
ASSERT_EQ(100, WriteBatchInternal::Sequence(&batch));
ASSERT_EQ(3, WriteBatchInternal::Count(&batch));
ASSERT_EQ("Put(baz, boo)@102"
"Delete(box)@101"
"Put(foo, bar)@100",
PrintContents(&batch));
}
TEST(WriteBatchTest, Corruption) {
WriteBatch batch;
batch.Put(Slice("foo"), Slice("bar"));
batch.Delete(Slice("box"));
WriteBatchInternal::SetSequence(&batch, 200);
Slice contents = WriteBatchInternal::Contents(&batch);
WriteBatchInternal::SetContents(&batch,
Slice(contents.data(),contents.size()-1));
ASSERT_EQ("Put(foo, bar)@200"
"ParseError()",
PrintContents(&batch));
}
TEST(WriteBatchTest, Append) {
WriteBatch b1, b2;
WriteBatchInternal::SetSequence(&b1, 200);
WriteBatchInternal::SetSequence(&b2, 300);
WriteBatchInternal::Append(&b1, &b2);
ASSERT_EQ("",
PrintContents(&b1));
b2.Put("a", "va");
WriteBatchInternal::Append(&b1, &b2);
ASSERT_EQ("Put(a, va)@200",
PrintContents(&b1));
b2.Clear();
b2.Put("b", "vb");
WriteBatchInternal::Append(&b1, &b2);
ASSERT_EQ("Put(a, va)@200"
"Put(b, vb)@201",
PrintContents(&b1));
b2.Delete("foo");
WriteBatchInternal::Append(&b1, &b2);
ASSERT_EQ("Put(a, va)@200"
"Put(b, vb)@202"
"Put(b, vb)@201"
"Delete(foo)@203",
PrintContents(&b1));
}
} // namespace leveldb
int main(int argc, char** argv) {
return leveldb::test::RunAllTests();
}
src/leveldb/doc/bench/db_bench_sqlite3.cc
+718
View File
@@ -0,0 +1,718 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <stdio.h>
#include <stdlib.h>
#include <sqlite3.h>
#include "util/histogram.h"
#include "util/random.h"
#include "util/testutil.h"
// Comma-separated list of operations to run in the specified order
// Actual benchmarks:
//
// fillseq -- write N values in sequential key order in async mode
// fillseqsync -- write N/100 values in sequential key order in sync mode
// fillseqbatch -- batch write N values in sequential key order in async mode
// fillrandom -- write N values in random key order in async mode
// fillrandsync -- write N/100 values in random key order in sync mode
// fillrandbatch -- batch write N values in sequential key order in async mode
// overwrite -- overwrite N values in random key order in async mode
// fillrand100K -- write N/1000 100K values in random order in async mode
// fillseq100K -- write N/1000 100K values in sequential order in async mode
// readseq -- read N times sequentially
// readrandom -- read N times in random order
// readrand100K -- read N/1000 100K values in sequential order in async mode
static const char* FLAGS_benchmarks =
"fillseq,"
"fillseqsync,"
"fillseqbatch,"
"fillrandom,"
"fillrandsync,"
"fillrandbatch,"
"overwrite,"
"overwritebatch,"
"readrandom,"
"readseq,"
"fillrand100K,"
"fillseq100K,"
"readseq,"
"readrand100K,"
;
// Number of key/values to place in database
static int FLAGS_num = 1000000;
// Number of read operations to do. If negative, do FLAGS_num reads.
static int FLAGS_reads = -1;
// Size of each value
static int FLAGS_value_size = 100;
// Print histogram of operation timings
static bool FLAGS_histogram = false;
// Arrange to generate values that shrink to this fraction of
// their original size after compression
static double FLAGS_compression_ratio = 0.5;
// Page size. Default 1 KB.
static int FLAGS_page_size = 1024;
// Number of pages.
// Default cache size = FLAGS_page_size * FLAGS_num_pages = 4 MB.
static int FLAGS_num_pages = 4096;
// If true, do not destroy the existing database. If you set this
// flag and also specify a benchmark that wants a fresh database, that
// benchmark will fail.
static bool FLAGS_use_existing_db = false;
// If true, we allow batch writes to occur
static bool FLAGS_transaction = true;
// If true, we enable Write-Ahead Logging
static bool FLAGS_WAL_enabled = true;
// Use the db with the following name.
static const char* FLAGS_db = NULL;
inline
static void ExecErrorCheck(int status, char *err_msg) {
if (status != SQLITE_OK) {
fprintf(stderr, "SQL error: %s\n", err_msg);
sqlite3_free(err_msg);
exit(1);
}
}
inline
static void StepErrorCheck(int status) {
if (status != SQLITE_DONE) {
fprintf(stderr, "SQL step error: status = %d\n", status);
exit(1);
}
}
inline
static void ErrorCheck(int status) {
if (status != SQLITE_OK) {
fprintf(stderr, "sqlite3 error: status = %d\n", status);
exit(1);
}
}
inline
static void WalCheckpoint(sqlite3* db_) {
// Flush all writes to disk
if (FLAGS_WAL_enabled) {
sqlite3_wal_checkpoint_v2(db_, NULL, SQLITE_CHECKPOINT_FULL, NULL, NULL);
}
}
namespace leveldb {
// Helper for quickly generating random data.
namespace {
class RandomGenerator {
private:
std::string data_;
int pos_;
public:
RandomGenerator() {
// We use a limited amount of data over and over again and ensure
// that it is larger than the compression window (32KB), and also
// large enough to serve all typical value sizes we want to write.
Random rnd(301);
std::string piece;
while (data_.size() < 1048576) {
// Add a short fragment that is as compressible as specified
// by FLAGS_compression_ratio.
test::CompressibleString(&rnd, FLAGS_compression_ratio, 100, &piece);
data_.append(piece);
}
pos_ = 0;
}
Slice Generate(int len) {
if (pos_ + len > data_.size()) {
pos_ = 0;
assert(len < data_.size());
}
pos_ += len;
return Slice(data_.data() + pos_ - len, len);
}
};
static Slice TrimSpace(Slice s) {
int start = 0;
while (start < s.size() && isspace(s[start])) {
start++;
}
int limit = s.size();
while (limit > start && isspace(s[limit-1])) {
limit--;
}
return Slice(s.data() + start, limit - start);
}
} // namespace
class Benchmark {
private:
sqlite3* db_;
int db_num_;
int num_;
int reads_;
double start_;
double last_op_finish_;
int64_t bytes_;
std::string message_;
Histogram hist_;
RandomGenerator gen_;
Random rand_;
// State kept for progress messages
int done_;
int next_report_; // When to report next
void PrintHeader() {
const int kKeySize = 16;
PrintEnvironment();
fprintf(stdout, "Keys: %d bytes each\n", kKeySize);
fprintf(stdout, "Values: %d bytes each\n", FLAGS_value_size);
fprintf(stdout, "Entries: %d\n", num_);
fprintf(stdout, "RawSize: %.1f MB (estimated)\n",
((static_cast<int64_t>(kKeySize + FLAGS_value_size) * num_)
/ 1048576.0));
PrintWarnings();
fprintf(stdout, "------------------------------------------------\n");
}
void PrintWarnings() {
#if defined(__GNUC__) && !defined(__OPTIMIZE__)
fprintf(stdout,
"WARNING: Optimization is disabled: benchmarks unnecessarily slow\n"
);
#endif
#ifndef NDEBUG
fprintf(stdout,
"WARNING: Assertions are enabled; benchmarks unnecessarily slow\n");
#endif
}
void PrintEnvironment() {
fprintf(stderr, "SQLite: version %s\n", SQLITE_VERSION);
#if defined(__linux)
time_t now = time(NULL);
fprintf(stderr, "Date: %s", ctime(&now)); // ctime() adds newline
FILE* cpuinfo = fopen("/proc/cpuinfo", "r");
if (cpuinfo != NULL) {
char line[1000];
int num_cpus = 0;
std::string cpu_type;
std::string cache_size;
while (fgets(line, sizeof(line), cpuinfo) != NULL) {
const char* sep = strchr(line, ':');
if (sep == NULL) {
continue;
}
Slice key = TrimSpace(Slice(line, sep - 1 - line));
Slice val = TrimSpace(Slice(sep + 1));
if (key == "model name") {
++num_cpus;
cpu_type = val.ToString();
} else if (key == "cache size") {
cache_size = val.ToString();
}
}
fclose(cpuinfo);
fprintf(stderr, "CPU: %d * %s\n", num_cpus, cpu_type.c_str());
fprintf(stderr, "CPUCache: %s\n", cache_size.c_str());
}
#endif
}
void Start() {
start_ = Env::Default()->NowMicros() * 1e-6;
bytes_ = 0;
message_.clear();
last_op_finish_ = start_;
hist_.Clear();
done_ = 0;
next_report_ = 100;
}
void FinishedSingleOp() {
if (FLAGS_histogram) {
double now = Env::Default()->NowMicros() * 1e-6;
double micros = (now - last_op_finish_) * 1e6;
hist_.Add(micros);
if (micros > 20000) {
fprintf(stderr, "long op: %.1f micros%30s\r", micros, "");
fflush(stderr);
}
last_op_finish_ = now;
}
done_++;
if (done_ >= next_report_) {
if (next_report_ < 1000) next_report_ += 100;
else if (next_report_ < 5000) next_report_ += 500;
else if (next_report_ < 10000) next_report_ += 1000;
else if (next_report_ < 50000) next_report_ += 5000;
else if (next_report_ < 100000) next_report_ += 10000;
else if (next_report_ < 500000) next_report_ += 50000;
else next_report_ += 100000;
fprintf(stderr, "... finished %d ops%30s\r", done_, "");
fflush(stderr);
}
}
void Stop(const Slice& name) {
double finish = Env::Default()->NowMicros() * 1e-6;
// Pretend at least one op was done in case we are running a benchmark
// that does not call FinishedSingleOp().
if (done_ < 1) done_ = 1;
if (bytes_ > 0) {
char rate[100];
snprintf(rate, sizeof(rate), "%6.1f MB/s",
(bytes_ / 1048576.0) / (finish - start_));
if (!message_.empty()) {
message_ = std::string(rate) + " " + message_;
} else {
message_ = rate;
}
}
fprintf(stdout, "%-12s : %11.3f micros/op;%s%s\n",
name.ToString().c_str(),
(finish - start_) * 1e6 / done_,
(message_.empty() ? "" : " "),
message_.c_str());
if (FLAGS_histogram) {
fprintf(stdout, "Microseconds per op:\n%s\n", hist_.ToString().c_str());
}
fflush(stdout);
}
public:
enum Order {
SEQUENTIAL,
RANDOM
};
enum DBState {
FRESH,
EXISTING
};
Benchmark()
: db_(NULL),
db_num_(0),
num_(FLAGS_num),
reads_(FLAGS_reads < 0 ? FLAGS_num : FLAGS_reads),
bytes_(0),
rand_(301) {
std::vector<std::string> files;
std::string test_dir;
Env::Default()->GetTestDirectory(&test_dir);
Env::Default()->GetChildren(test_dir, &files);
if (!FLAGS_use_existing_db) {
for (int i = 0; i < files.size(); i++) {
if (Slice(files[i]).starts_with("dbbench_sqlite3")) {
std::string file_name(test_dir);
file_name += "/";
file_name += files[i];
Env::Default()->DeleteFile(file_name.c_str());
}
}
}
}
~Benchmark() {
int status = sqlite3_close(db_);
ErrorCheck(status);
}
void Run() {
PrintHeader();
Open();
const char* benchmarks = FLAGS_benchmarks;
while (benchmarks != NULL) {
const char* sep = strchr(benchmarks, ',');
Slice name;
if (sep == NULL) {
name = benchmarks;
benchmarks = NULL;
} else {
name = Slice(benchmarks, sep - benchmarks);
benchmarks = sep + 1;
}
bytes_ = 0;
Start();
bool known = true;
bool write_sync = false;
if (name == Slice("fillseq")) {
Write(write_sync, SEQUENTIAL, FRESH, num_, FLAGS_value_size, 1);
WalCheckpoint(db_);
} else if (name == Slice("fillseqbatch")) {
Write(write_sync, SEQUENTIAL, FRESH, num_, FLAGS_value_size, 1000);
WalCheckpoint(db_);
} else if (name == Slice("fillrandom")) {
Write(write_sync, RANDOM, FRESH, num_, FLAGS_value_size, 1);
WalCheckpoint(db_);
} else if (name == Slice("fillrandbatch")) {
Write(write_sync, RANDOM, FRESH, num_, FLAGS_value_size, 1000);
WalCheckpoint(db_);
} else if (name == Slice("overwrite")) {
Write(write_sync, RANDOM, EXISTING, num_, FLAGS_value_size, 1);
WalCheckpoint(db_);
} else if (name == Slice("overwritebatch")) {
Write(write_sync, RANDOM, EXISTING, num_, FLAGS_value_size, 1000);
WalCheckpoint(db_);
} else if (name == Slice("fillrandsync")) {
write_sync = true;
Write(write_sync, RANDOM, FRESH, num_ / 100, FLAGS_value_size, 1);
WalCheckpoint(db_);
} else if (name == Slice("fillseqsync")) {
write_sync = true;
Write(write_sync, SEQUENTIAL, FRESH, num_ / 100, FLAGS_value_size, 1);
WalCheckpoint(db_);
} else if (name == Slice("fillrand100K")) {
Write(write_sync, RANDOM, FRESH, num_ / 1000, 100 * 1000, 1);
WalCheckpoint(db_);
} else if (name == Slice("fillseq100K")) {
Write(write_sync, SEQUENTIAL, FRESH, num_ / 1000, 100 * 1000, 1);
WalCheckpoint(db_);
} else if (name == Slice("readseq")) {
ReadSequential();
} else if (name == Slice("readrandom")) {
Read(RANDOM, 1);
} else if (name == Slice("readrand100K")) {
int n = reads_;
reads_ /= 1000;
Read(RANDOM, 1);
reads_ = n;
} else {
known = false;
if (name != Slice()) { // No error message for empty name
fprintf(stderr, "unknown benchmark '%s'\n", name.ToString().c_str());
}
}
if (known) {
Stop(name);
}
}
}
void Open() {
assert(db_ == NULL);
int status;
char file_name[100];
char* err_msg = NULL;
db_num_++;
// Open database
std::string tmp_dir;
Env::Default()->GetTestDirectory(&tmp_dir);
snprintf(file_name, sizeof(file_name),
"%s/dbbench_sqlite3-%d.db",
tmp_dir.c_str(),
db_num_);
status = sqlite3_open(file_name, &db_);
if (status) {
fprintf(stderr, "open error: %s\n", sqlite3_errmsg(db_));
exit(1);
}
// Change SQLite cache size
char cache_size[100];
snprintf(cache_size, sizeof(cache_size), "PRAGMA cache_size = %d",
FLAGS_num_pages);
status = sqlite3_exec(db_, cache_size, NULL, NULL, &err_msg);
ExecErrorCheck(status, err_msg);
// FLAGS_page_size is defaulted to 1024
if (FLAGS_page_size != 1024) {
char page_size[100];
snprintf(page_size, sizeof(page_size), "PRAGMA page_size = %d",
FLAGS_page_size);
status = sqlite3_exec(db_, page_size, NULL, NULL, &err_msg);
ExecErrorCheck(status, err_msg);
}
// Change journal mode to WAL if WAL enabled flag is on
if (FLAGS_WAL_enabled) {
std::string WAL_stmt = "PRAGMA journal_mode = WAL";
// LevelDB's default cache size is a combined 4 MB
std::string WAL_checkpoint = "PRAGMA wal_autocheckpoint = 4096";
status = sqlite3_exec(db_, WAL_stmt.c_str(), NULL, NULL, &err_msg);
ExecErrorCheck(status, err_msg);
status = sqlite3_exec(db_, WAL_checkpoint.c_str(), NULL, NULL, &err_msg);
ExecErrorCheck(status, err_msg);
}
// Change locking mode to exclusive and create tables/index for database
std::string locking_stmt = "PRAGMA locking_mode = EXCLUSIVE";
std::string create_stmt =
"CREATE TABLE test (key blob, value blob, PRIMARY KEY(key))";
std::string stmt_array[] = { locking_stmt, create_stmt };
int stmt_array_length = sizeof(stmt_array) / sizeof(std::string);
for (int i = 0; i < stmt_array_length; i++) {
status = sqlite3_exec(db_, stmt_array[i].c_str(), NULL, NULL, &err_msg);
ExecErrorCheck(status, err_msg);
}
}
void Write(bool write_sync, Order order, DBState state,
int num_entries, int value_size, int entries_per_batch) {
// Create new database if state == FRESH
if (state == FRESH) {
if (FLAGS_use_existing_db) {
message_ = "skipping (--use_existing_db is true)";
return;
}
sqlite3_close(db_);
db_ = NULL;
Open();
Start();
}
if (num_entries != num_) {
char msg[100];
snprintf(msg, sizeof(msg), "(%d ops)", num_entries);
message_ = msg;
}
char* err_msg = NULL;
int status;
sqlite3_stmt *replace_stmt, *begin_trans_stmt, *end_trans_stmt;
std::string replace_str = "REPLACE INTO test (key, value) VALUES (?, ?)";
std::string begin_trans_str = "BEGIN TRANSACTION;";
std::string end_trans_str = "END TRANSACTION;";
// Check for synchronous flag in options
std::string sync_stmt = (write_sync) ? "PRAGMA synchronous = FULL" :
"PRAGMA synchronous = OFF";
status = sqlite3_exec(db_, sync_stmt.c_str(), NULL, NULL, &err_msg);
ExecErrorCheck(status, err_msg);
// Preparing sqlite3 statements
status = sqlite3_prepare_v2(db_, replace_str.c_str(), -1,
&replace_stmt, NULL);
ErrorCheck(status);
status = sqlite3_prepare_v2(db_, begin_trans_str.c_str(), -1,
&begin_trans_stmt, NULL);
ErrorCheck(status);
status = sqlite3_prepare_v2(db_, end_trans_str.c_str(), -1,
&end_trans_stmt, NULL);
ErrorCheck(status);
bool transaction = (entries_per_batch > 1);
for (int i = 0; i < num_entries; i += entries_per_batch) {
// Begin write transaction
if (FLAGS_transaction && transaction) {
status = sqlite3_step(begin_trans_stmt);
StepErrorCheck(status);
status = sqlite3_reset(begin_trans_stmt);
ErrorCheck(status);
}
// Create and execute SQL statements
for (int j = 0; j < entries_per_batch; j++) {
const char* value = gen_.Generate(value_size).data();
// Create values for key-value pair
const int k = (order == SEQUENTIAL) ? i + j :
(rand_.Next() % num_entries);
char key[100];
snprintf(key, sizeof(key), "%016d", k);
// Bind KV values into replace_stmt
status = sqlite3_bind_blob(replace_stmt, 1, key, 16, SQLITE_STATIC);
ErrorCheck(status);
status = sqlite3_bind_blob(replace_stmt, 2, value,
value_size, SQLITE_STATIC);
ErrorCheck(status);
// Execute replace_stmt
bytes_ += value_size + strlen(key);
status = sqlite3_step(replace_stmt);
StepErrorCheck(status);
// Reset SQLite statement for another use
status = sqlite3_clear_bindings(replace_stmt);
ErrorCheck(status);
status = sqlite3_reset(replace_stmt);
ErrorCheck(status);
FinishedSingleOp();
}
// End write transaction
if (FLAGS_transaction && transaction) {
status = sqlite3_step(end_trans_stmt);
StepErrorCheck(status);
status = sqlite3_reset(end_trans_stmt);
ErrorCheck(status);
}
}
status = sqlite3_finalize(replace_stmt);
ErrorCheck(status);
status = sqlite3_finalize(begin_trans_stmt);
ErrorCheck(status);
status = sqlite3_finalize(end_trans_stmt);
ErrorCheck(status);
}
void Read(Order order, int entries_per_batch) {
int status;
sqlite3_stmt *read_stmt, *begin_trans_stmt, *end_trans_stmt;
std::string read_str = "SELECT * FROM test WHERE key = ?";
std::string begin_trans_str = "BEGIN TRANSACTION;";
std::string end_trans_str = "END TRANSACTION;";
// Preparing sqlite3 statements
status = sqlite3_prepare_v2(db_, begin_trans_str.c_str(), -1,
&begin_trans_stmt, NULL);
ErrorCheck(status);
status = sqlite3_prepare_v2(db_, end_trans_str.c_str(), -1,
&end_trans_stmt, NULL);
ErrorCheck(status);
status = sqlite3_prepare_v2(db_, read_str.c_str(), -1, &read_stmt, NULL);
ErrorCheck(status);
bool transaction = (entries_per_batch > 1);
for (int i = 0; i < reads_; i += entries_per_batch) {
// Begin read transaction
if (FLAGS_transaction && transaction) {
status = sqlite3_step(begin_trans_stmt);
StepErrorCheck(status);
status = sqlite3_reset(begin_trans_stmt);
ErrorCheck(status);
}
// Create and execute SQL statements
for (int j = 0; j < entries_per_batch; j++) {
// Create key value
char key[100];
int k = (order == SEQUENTIAL) ? i + j : (rand_.Next() % reads_);
snprintf(key, sizeof(key), "%016d", k);
// Bind key value into read_stmt
status = sqlite3_bind_blob(read_stmt, 1, key, 16, SQLITE_STATIC);
ErrorCheck(status);
// Execute read statement
while ((status = sqlite3_step(read_stmt)) == SQLITE_ROW) {}
StepErrorCheck(status);
// Reset SQLite statement for another use
status = sqlite3_clear_bindings(read_stmt);
ErrorCheck(status);
status = sqlite3_reset(read_stmt);
ErrorCheck(status);
FinishedSingleOp();
}
// End read transaction
if (FLAGS_transaction && transaction) {
status = sqlite3_step(end_trans_stmt);
StepErrorCheck(status);
status = sqlite3_reset(end_trans_stmt);
ErrorCheck(status);
}
}
status = sqlite3_finalize(read_stmt);
ErrorCheck(status);
status = sqlite3_finalize(begin_trans_stmt);
ErrorCheck(status);
status = sqlite3_finalize(end_trans_stmt);
ErrorCheck(status);
}
void ReadSequential() {
int status;
sqlite3_stmt *pStmt;
std::string read_str = "SELECT * FROM test ORDER BY key";
status = sqlite3_prepare_v2(db_, read_str.c_str(), -1, &pStmt, NULL);
ErrorCheck(status);
for (int i = 0; i < reads_ && SQLITE_ROW == sqlite3_step(pStmt); i++) {
bytes_ += sqlite3_column_bytes(pStmt, 1) + sqlite3_column_bytes(pStmt, 2);
FinishedSingleOp();
}
status = sqlite3_finalize(pStmt);
ErrorCheck(status);
}
};
} // namespace leveldb
int main(int argc, char** argv) {
std::string default_db_path;
for (int i = 1; i < argc; i++) {
double d;
int n;
char junk;
if (leveldb::Slice(argv[i]).starts_with("--benchmarks=")) {
FLAGS_benchmarks = argv[i] + strlen("--benchmarks=");
} else if (sscanf(argv[i], "--histogram=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_histogram = n;
} else if (sscanf(argv[i], "--compression_ratio=%lf%c", &d, &junk) == 1) {
FLAGS_compression_ratio = d;
} else if (sscanf(argv[i], "--use_existing_db=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_use_existing_db = n;
} else if (sscanf(argv[i], "--num=%d%c", &n, &junk) == 1) {
FLAGS_num = n;
} else if (sscanf(argv[i], "--reads=%d%c", &n, &junk) == 1) {
FLAGS_reads = n;
} else if (sscanf(argv[i], "--value_size=%d%c", &n, &junk) == 1) {
FLAGS_value_size = n;
} else if (leveldb::Slice(argv[i]) == leveldb::Slice("--no_transaction")) {
FLAGS_transaction = false;
} else if (sscanf(argv[i], "--page_size=%d%c", &n, &junk) == 1) {
FLAGS_page_size = n;
} else if (sscanf(argv[i], "--num_pages=%d%c", &n, &junk) == 1) {
FLAGS_num_pages = n;
} else if (sscanf(argv[i], "--WAL_enabled=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_WAL_enabled = n;
} else if (strncmp(argv[i], "--db=", 5) == 0) {
FLAGS_db = argv[i] + 5;
} else {
fprintf(stderr, "Invalid flag '%s'\n", argv[i]);
exit(1);
}
}
// Choose a location for the test database if none given with --db=<path>
if (FLAGS_db == NULL) {
leveldb::Env::Default()->GetTestDirectory(&default_db_path);
default_db_path += "/dbbench";
FLAGS_db = default_db_path.c_str();
}
leveldb::Benchmark benchmark;
benchmark.Run();
return 0;
}
src/leveldb/doc/bench/db_bench_tree_db.cc
+528
View File
@@ -0,0 +1,528 @@
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include <stdio.h>
#include <stdlib.h>
#include <kcpolydb.h>
#include "util/histogram.h"
#include "util/random.h"
#include "util/testutil.h"
// Comma-separated list of operations to run in the specified order
// Actual benchmarks:
//
// fillseq -- write N values in sequential key order in async mode
// fillrandom -- write N values in random key order in async mode
// overwrite -- overwrite N values in random key order in async mode
// fillseqsync -- write N/100 values in sequential key order in sync mode
// fillrandsync -- write N/100 values in random key order in sync mode
// fillrand100K -- write N/1000 100K values in random order in async mode
// fillseq100K -- write N/1000 100K values in seq order in async mode
// readseq -- read N times sequentially
// readseq100K -- read N/1000 100K values in sequential order in async mode
// readrand100K -- read N/1000 100K values in sequential order in async mode
// readrandom -- read N times in random order
static const char* FLAGS_benchmarks =
"fillseq,"
"fillseqsync,"
"fillrandsync,"
"fillrandom,"
"overwrite,"
"readrandom,"
"readseq,"
"fillrand100K,"
"fillseq100K,"
"readseq100K,"
"readrand100K,"
;
// Number of key/values to place in database
static int FLAGS_num = 1000000;
// Number of read operations to do. If negative, do FLAGS_num reads.
static int FLAGS_reads = -1;
// Size of each value
static int FLAGS_value_size = 100;
// Arrange to generate values that shrink to this fraction of
// their original size after compression
static double FLAGS_compression_ratio = 0.5;
// Print histogram of operation timings
static bool FLAGS_histogram = false;
// Cache size. Default 4 MB
static int FLAGS_cache_size = 4194304;
// Page size. Default 1 KB
static int FLAGS_page_size = 1024;
// If true, do not destroy the existing database. If you set this
// flag and also specify a benchmark that wants a fresh database, that
// benchmark will fail.
static bool FLAGS_use_existing_db = false;
// Compression flag. If true, compression is on. If false, compression
// is off.
static bool FLAGS_compression = true;
// Use the db with the following name.
static const char* FLAGS_db = NULL;
inline
static void DBSynchronize(kyotocabinet::TreeDB* db_)
{
// Synchronize will flush writes to disk
if (!db_->synchronize()) {
fprintf(stderr, "synchronize error: %s\n", db_->error().name());
}
}
namespace leveldb {
// Helper for quickly generating random data.
namespace {
class RandomGenerator {
private:
std::string data_;
int pos_;
public:
RandomGenerator() {
// We use a limited amount of data over and over again and ensure
// that it is larger than the compression window (32KB), and also
// large enough to serve all typical value sizes we want to write.
Random rnd(301);
std::string piece;
while (data_.size() < 1048576) {
// Add a short fragment that is as compressible as specified
// by FLAGS_compression_ratio.
test::CompressibleString(&rnd, FLAGS_compression_ratio, 100, &piece);
data_.append(piece);
}
pos_ = 0;
}
Slice Generate(int len) {
if (pos_ + len > data_.size()) {
pos_ = 0;
assert(len < data_.size());
}
pos_ += len;
return Slice(data_.data() + pos_ - len, len);
}
};
static Slice TrimSpace(Slice s) {
int start = 0;
while (start < s.size() && isspace(s[start])) {
start++;
}
int limit = s.size();
while (limit > start && isspace(s[limit-1])) {
limit--;
}
return Slice(s.data() + start, limit - start);
}
} // namespace
class Benchmark {
private:
kyotocabinet::TreeDB* db_;
int db_num_;
int num_;
int reads_;
double start_;
double last_op_finish_;
int64_t bytes_;
std::string message_;
Histogram hist_;
RandomGenerator gen_;
Random rand_;
kyotocabinet::LZOCompressor<kyotocabinet::LZO::RAW> comp_;
// State kept for progress messages
int done_;
int next_report_; // When to report next
void PrintHeader() {
const int kKeySize = 16;
PrintEnvironment();
fprintf(stdout, "Keys: %d bytes each\n", kKeySize);
fprintf(stdout, "Values: %d bytes each (%d bytes after compression)\n",
FLAGS_value_size,
static_cast<int>(FLAGS_value_size * FLAGS_compression_ratio + 0.5));
fprintf(stdout, "Entries: %d\n", num_);
fprintf(stdout, "RawSize: %.1f MB (estimated)\n",
((static_cast<int64_t>(kKeySize + FLAGS_value_size) * num_)
/ 1048576.0));
fprintf(stdout, "FileSize: %.1f MB (estimated)\n",
(((kKeySize + FLAGS_value_size * FLAGS_compression_ratio) * num_)
/ 1048576.0));
PrintWarnings();
fprintf(stdout, "------------------------------------------------\n");
}
void PrintWarnings() {
#if defined(__GNUC__) && !defined(__OPTIMIZE__)
fprintf(stdout,
"WARNING: Optimization is disabled: benchmarks unnecessarily slow\n"
);
#endif
#ifndef NDEBUG
fprintf(stdout,
"WARNING: Assertions are enabled; benchmarks unnecessarily slow\n");
#endif
}
void PrintEnvironment() {
fprintf(stderr, "Kyoto Cabinet: version %s, lib ver %d, lib rev %d\n",
kyotocabinet::VERSION, kyotocabinet::LIBVER, kyotocabinet::LIBREV);
#if defined(__linux)
time_t now = time(NULL);
fprintf(stderr, "Date: %s", ctime(&now)); // ctime() adds newline
FILE* cpuinfo = fopen("/proc/cpuinfo", "r");
if (cpuinfo != NULL) {
char line[1000];
int num_cpus = 0;
std::string cpu_type;
std::string cache_size;
while (fgets(line, sizeof(line), cpuinfo) != NULL) {
const char* sep = strchr(line, ':');
if (sep == NULL) {
continue;
}
Slice key = TrimSpace(Slice(line, sep - 1 - line));
Slice val = TrimSpace(Slice(sep + 1));
if (key == "model name") {
++num_cpus;
cpu_type = val.ToString();
} else if (key == "cache size") {
cache_size = val.ToString();
}
}
fclose(cpuinfo);
fprintf(stderr, "CPU: %d * %s\n", num_cpus, cpu_type.c_str());
fprintf(stderr, "CPUCache: %s\n", cache_size.c_str());
}
#endif
}
void Start() {
start_ = Env::Default()->NowMicros() * 1e-6;
bytes_ = 0;
message_.clear();
last_op_finish_ = start_;
hist_.Clear();
done_ = 0;
next_report_ = 100;
}
void FinishedSingleOp() {
if (FLAGS_histogram) {
double now = Env::Default()->NowMicros() * 1e-6;
double micros = (now - last_op_finish_) * 1e6;
hist_.Add(micros);
if (micros > 20000) {
fprintf(stderr, "long op: %.1f micros%30s\r", micros, "");
fflush(stderr);
}
last_op_finish_ = now;
}
done_++;
if (done_ >= next_report_) {
if (next_report_ < 1000) next_report_ += 100;
else if (next_report_ < 5000) next_report_ += 500;
else if (next_report_ < 10000) next_report_ += 1000;
else if (next_report_ < 50000) next_report_ += 5000;
else if (next_report_ < 100000) next_report_ += 10000;
else if (next_report_ < 500000) next_report_ += 50000;
else next_report_ += 100000;
fprintf(stderr, "... finished %d ops%30s\r", done_, "");
fflush(stderr);
}
}
void Stop(const Slice& name) {
double finish = Env::Default()->NowMicros() * 1e-6;
// Pretend at least one op was done in case we are running a benchmark
// that does not call FinishedSingleOp().
if (done_ < 1) done_ = 1;
if (bytes_ > 0) {
char rate[100];
snprintf(rate, sizeof(rate), "%6.1f MB/s",
(bytes_ / 1048576.0) / (finish - start_));
if (!message_.empty()) {
message_ = std::string(rate) + " " + message_;
} else {
message_ = rate;
}
}
fprintf(stdout, "%-12s : %11.3f micros/op;%s%s\n",
name.ToString().c_str(),
(finish - start_) * 1e6 / done_,
(message_.empty() ? "" : " "),
message_.c_str());
if (FLAGS_histogram) {
fprintf(stdout, "Microseconds per op:\n%s\n", hist_.ToString().c_str());
}
fflush(stdout);
}
public:
enum Order {
SEQUENTIAL,
RANDOM
};
enum DBState {
FRESH,
EXISTING
};
Benchmark()
: db_(NULL),
num_(FLAGS_num),
reads_(FLAGS_reads < 0 ? FLAGS_num : FLAGS_reads),
bytes_(0),
rand_(301) {
std::vector<std::string> files;
std::string test_dir;
Env::Default()->GetTestDirectory(&test_dir);
Env::Default()->GetChildren(test_dir.c_str(), &files);
if (!FLAGS_use_existing_db) {
for (int i = 0; i < files.size(); i++) {
if (Slice(files[i]).starts_with("dbbench_polyDB")) {
std::string file_name(test_dir);
file_name += "/";
file_name += files[i];
Env::Default()->DeleteFile(file_name.c_str());
}
}
}
}
~Benchmark() {
if (!db_->close()) {
fprintf(stderr, "close error: %s\n", db_->error().name());
}
}
void Run() {
PrintHeader();
Open(false);
const char* benchmarks = FLAGS_benchmarks;
while (benchmarks != NULL) {
const char* sep = strchr(benchmarks, ',');
Slice name;
if (sep == NULL) {
name = benchmarks;
benchmarks = NULL;
} else {
name = Slice(benchmarks, sep - benchmarks);
benchmarks = sep + 1;
}
Start();
bool known = true;
bool write_sync = false;
if (name == Slice("fillseq")) {
Write(write_sync, SEQUENTIAL, FRESH, num_, FLAGS_value_size, 1);
} else if (name == Slice("fillrandom")) {
Write(write_sync, RANDOM, FRESH, num_, FLAGS_value_size, 1);
DBSynchronize(db_);
} else if (name == Slice("overwrite")) {
Write(write_sync, RANDOM, EXISTING, num_, FLAGS_value_size, 1);
DBSynchronize(db_);
} else if (name == Slice("fillrandsync")) {
write_sync = true;
Write(write_sync, RANDOM, FRESH, num_ / 100, FLAGS_value_size, 1);
DBSynchronize(db_);
} else if (name == Slice("fillseqsync")) {
write_sync = true;
Write(write_sync, SEQUENTIAL, FRESH, num_ / 100, FLAGS_value_size, 1);
DBSynchronize(db_);
} else if (name == Slice("fillrand100K")) {
Write(write_sync, RANDOM, FRESH, num_ / 1000, 100 * 1000, 1);
DBSynchronize(db_);
} else if (name == Slice("fillseq100K")) {
Write(write_sync, SEQUENTIAL, FRESH, num_ / 1000, 100 * 1000, 1);
DBSynchronize(db_);
} else if (name == Slice("readseq")) {
ReadSequential();
} else if (name == Slice("readrandom")) {
ReadRandom();
} else if (name == Slice("readrand100K")) {
int n = reads_;
reads_ /= 1000;
ReadRandom();
reads_ = n;
} else if (name == Slice("readseq100K")) {
int n = reads_;
reads_ /= 1000;
ReadSequential();
reads_ = n;
} else {
known = false;
if (name != Slice()) { // No error message for empty name
fprintf(stderr, "unknown benchmark '%s'\n", name.ToString().c_str());
}
}
if (known) {
Stop(name);
}
}
}
private:
void Open(bool sync) {
assert(db_ == NULL);
// Initialize db_
db_ = new kyotocabinet::TreeDB();
char file_name[100];
db_num_++;
std::string test_dir;
Env::Default()->GetTestDirectory(&test_dir);
snprintf(file_name, sizeof(file_name),
"%s/dbbench_polyDB-%d.kct",
test_dir.c_str(),
db_num_);
// Create tuning options and open the database
int open_options = kyotocabinet::PolyDB::OWRITER |
kyotocabinet::PolyDB::OCREATE;
int tune_options = kyotocabinet::TreeDB::TSMALL |
kyotocabinet::TreeDB::TLINEAR;
if (FLAGS_compression) {
tune_options |= kyotocabinet::TreeDB::TCOMPRESS;
db_->tune_compressor(&comp_);
}
db_->tune_options(tune_options);
db_->tune_page_cache(FLAGS_cache_size);
db_->tune_page(FLAGS_page_size);
db_->tune_map(256LL<<20);
if (sync) {
open_options |= kyotocabinet::PolyDB::OAUTOSYNC;
}
if (!db_->open(file_name, open_options)) {
fprintf(stderr, "open error: %s\n", db_->error().name());
}
}
void Write(bool sync, Order order, DBState state,
int num_entries, int value_size, int entries_per_batch) {
// Create new database if state == FRESH
if (state == FRESH) {
if (FLAGS_use_existing_db) {
message_ = "skipping (--use_existing_db is true)";
return;
}
delete db_;
db_ = NULL;
Open(sync);
Start(); // Do not count time taken to destroy/open
}
if (num_entries != num_) {
char msg[100];
snprintf(msg, sizeof(msg), "(%d ops)", num_entries);
message_ = msg;
}
// Write to database
for (int i = 0; i < num_entries; i++)
{
const int k = (order == SEQUENTIAL) ? i : (rand_.Next() % num_entries);
char key[100];
snprintf(key, sizeof(key), "%016d", k);
bytes_ += value_size + strlen(key);
std::string cpp_key = key;
if (!db_->set(cpp_key, gen_.Generate(value_size).ToString())) {
fprintf(stderr, "set error: %s\n", db_->error().name());
}
FinishedSingleOp();
}
}
void ReadSequential() {
kyotocabinet::DB::Cursor* cur = db_->cursor();
cur->jump();
std::string ckey, cvalue;
while (cur->get(&ckey, &cvalue, true)) {
bytes_ += ckey.size() + cvalue.size();
FinishedSingleOp();
}
delete cur;
}
void ReadRandom() {
std::string value;
for (int i = 0; i < reads_; i++) {
char key[100];
const int k = rand_.Next() % reads_;
snprintf(key, sizeof(key), "%016d", k);
db_->get(key, &value);
FinishedSingleOp();
}
}
};
} // namespace leveldb
int main(int argc, char** argv) {
std::string default_db_path;
for (int i = 1; i < argc; i++) {
double d;
int n;
char junk;
if (leveldb::Slice(argv[i]).starts_with("--benchmarks=")) {
FLAGS_benchmarks = argv[i] + strlen("--benchmarks=");
} else if (sscanf(argv[i], "--compression_ratio=%lf%c", &d, &junk) == 1) {
FLAGS_compression_ratio = d;
} else if (sscanf(argv[i], "--histogram=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_histogram = n;
} else if (sscanf(argv[i], "--num=%d%c", &n, &junk) == 1) {
FLAGS_num = n;
} else if (sscanf(argv[i], "--reads=%d%c", &n, &junk) == 1) {
FLAGS_reads = n;
} else if (sscanf(argv[i], "--value_size=%d%c", &n, &junk) == 1) {
FLAGS_value_size = n;
} else if (sscanf(argv[i], "--cache_size=%d%c", &n, &junk) == 1) {
FLAGS_cache_size = n;
} else if (sscanf(argv[i], "--page_size=%d%c", &n, &junk) == 1) {
FLAGS_page_size = n;
} else if (sscanf(argv[i], "--compression=%d%c", &n, &junk) == 1 &&
(n == 0 || n == 1)) {
FLAGS_compression = (n == 1) ? true : false;
} else if (strncmp(argv[i], "--db=", 5) == 0) {
FLAGS_db = argv[i] + 5;
} else {
fprintf(stderr, "Invalid flag '%s'\n", argv[i]);
exit(1);
}
}
// Choose a location for the test database if none given with --db=<path>
if (FLAGS_db == NULL) {
leveldb::Env::Default()->GetTestDirectory(&default_db_path);
default_db_path += "/dbbench";
FLAGS_db = default_db_path.c_str();
}
leveldb::Benchmark benchmark;
benchmark.Run();
return 0;
}
src/leveldb/doc/benchmark.html
+459
View File
@@ -0,0 +1,459 @@
<!DOCTYPE html>
<html>
<head>
<title>LevelDB Benchmarks</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style>
body {
font-family:Helvetica,sans-serif;
padding:20px;
}
h2 {
padding-top:30px;
}
table.bn {
width:800px;
border-collapse:collapse;
border:0;
padding:0;
}
table.bnbase {
width:650px;
}
table.bn td {
padding:2px 0;
}
table.bn td.c1 {
font-weight:bold;
width:150px;
}
table.bn td.c1 div.e {
float:right;
font-weight:normal;
}
table.bn td.c2 {
width:150px;
text-align:right;
padding:2px;
}
table.bn td.c3 {
width:350px;
}
table.bn td.c4 {
width:150px;
font-size:small;
padding-left:4px;
}
/* chart bars */
div.bldb {
background-color:#0255df;
}
div.bkct {
background-color:#df5555;
}
div.bsql {
background-color:#aadf55;
}
.code {
font-family:monospace;
font-size:large;
}
.todo {
color: red;
}
</style>
</head>
<body>
<h1>LevelDB Benchmarks</h1>
<p>Google, July 2011</p>
<hr>
<p>In order to test LevelDB's performance, we benchmark it against other well-established database implementations. We compare LevelDB (revision 39) against <a href="http://www.sqlite.org/">SQLite3</a> (version 3.7.6.3) and <a href="http://fallabs.com/kyotocabinet/spex.html">Kyoto Cabinet's</a> (version 1.2.67) TreeDB (a B+Tree based key-value store). We would like to acknowledge Scott Hess and Mikio Hirabayashi for their suggestions and contributions to the SQLite3 and Kyoto Cabinet benchmarks, respectively.</p>
<p>Benchmarks were all performed on a six-core Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, with 12288 KB of total L3 cache and 12 GB of DDR3 RAM at 1333 MHz. (Note that LevelDB uses at most two CPUs since the benchmarks are single threaded: one to run the benchmark, and one for background compactions.) We ran the benchmarks on two machines (with identical processors), one with an Ext3 file system and one with an Ext4 file system. The machine with the Ext3 file system has a SATA Hitachi HDS721050CLA362 hard drive. The machine with the Ext4 file system has a SATA Samsung HD502HJ hard drive. Both hard drives spin at 7200 RPM and have hard drive write-caching enabled (using `hdparm -W 1 [device]`). The numbers reported below are the median of three measurements.</p>
<h4>Benchmark Source Code</h4>
<p>We wrote benchmark tools for SQLite and Kyoto TreeDB based on LevelDB's <span class="code">db_bench</span>. The code for each of the benchmarks resides here:</p>
<ul>
<li> <b>LevelDB:</b> <a href="http://code.google.com/p/leveldb/source/browse/trunk/db/db_bench.cc">db/db_bench.cc</a>.</li>
<li> <b>SQLite:</b> <a href="http://code.google.com/p/leveldb/source/browse/#svn%2Ftrunk%2Fdoc%2Fbench%2Fdb_bench_sqlite3.cc">doc/bench/db_bench_sqlite3.cc</a>.</li>
<li> <b>Kyoto TreeDB:</b> <a href="http://code.google.com/p/leveldb/source/browse/#svn%2Ftrunk%2Fdoc%2Fbench%2Fdb_bench_tree_db.cc">doc/bench/db_bench_tree_db.cc</a>.</li>
</ul>
<h4>Custom Build Specifications</h4>
<ul>
<li>LevelDB: LevelDB was compiled with the <a href="http://code.google.com/p/google-perftools">tcmalloc</a> library and the <a href="http://code.google.com/p/snappy/">Snappy</a> compression library (revision 33). Assertions were disabled.</li>
<li>TreeDB: TreeDB was compiled using the <a href="http://www.oberhumer.com/opensource/lzo/">LZO</a> compression library (version 2.03). Furthermore, we enabled the TSMALL and TLINEAR options when opening the database in order to reduce the footprint of each record.</li>
<li>SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive. We also enabled SQLite's <a href="http://www.sqlite.org/draft/wal.html">write-ahead logging</a>.</li>
</ul>
<h2>1. Baseline Performance</h2>
<p>This section gives the baseline performance of all the
databases. Following sections show how performance changes as various
parameters are varied. For the baseline:</p>
<ul>
<li> Each database is allowed 4 MB of cache memory.</li>
<li> Databases are opened in <em>asynchronous</em> write mode.
(LevelDB's sync option, TreeDB's OAUTOSYNC option, and
SQLite3's synchronous options are all turned off). I.e.,
every write is pushed to the operating system, but the
benchmark does not wait for the write to reach the disk.</li>
<li> Keys are 16 bytes each.</li>
<li> Value are 100 bytes each (with enough redundancy so that
a simple compressor shrinks them to 50% of their original
size).</li>
<li> Sequential reads/writes traverse the key space in increasing order.</li>
<li> Random reads/writes traverse the key space in random order.</li>
</ul>
<h3>A. Sequential Reads</h3>
<table class="bn bnbase">
<tr><td class="c1">LevelDB</td>
<td class="c2">4,030,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">1,010,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:95px">&nbsp;</div></td>
<tr><td class="c1">SQLite3</td>
<td class="c2">383,000 ops/sec</td>
<td class="c3"><div class="bsql" style="width:33px">&nbsp;</div></td>
</table>
<h3>B. Random Reads</h3>
<table class="bn bnbase">
<tr><td class="c1">LevelDB</td>
<td class="c2">129,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:298px">&nbsp;</div></td>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">151,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:350px">&nbsp;</div></td>
<tr><td class="c1">SQLite3</td>
<td class="c2">134,000 ops/sec</td>
<td class="c3"><div class="bsql" style="width:310px">&nbsp;</div></td>
</table>
<h3>C. Sequential Writes</h3>
<table class="bn bnbase">
<tr><td class="c1">LevelDB</td>
<td class="c2">779,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">342,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:154px">&nbsp;</div></td>
<tr><td class="c1">SQLite3</td>
<td class="c2">48,600 ops/sec</td>
<td class="c3"><div class="bsql" style="width:22px">&nbsp;</div></td>
</table>
<h3>D. Random Writes</h3>
<table class="bn bnbase">
<tr><td class="c1">LevelDB</td>
<td class="c2">164,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">88,500 ops/sec</td>
<td class="c3"><div class="bkct" style="width:188px">&nbsp;</div></td>
<tr><td class="c1">SQLite3</td>
<td class="c2">9,860 ops/sec</td>
<td class="c3"><div class="bsql" style="width:21px">&nbsp;</div></td>
</table>
<p>LevelDB outperforms both SQLite3 and TreeDB in sequential and random write operations and sequential read operations. Kyoto Cabinet has the fastest random read operations.</p>
<h2>2. Write Performance under Different Configurations</h2>
<h3>A. Large Values </h3>
<p>For this benchmark, we start with an empty database, and write 100,000 byte values (~50% compressible). To keep the benchmark running time reasonable, we stop after writing 1000 values.</p>
<h4>Sequential Writes</h4>
<table class="bn bnbase">
<tr><td class="c1">LevelDB</td>
<td class="c2">1,100 ops/sec</td>
<td class="c3"><div class="bldb" style="width:234px">&nbsp;</div></td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">1,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:224px">&nbsp;</div></td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">1,600 ops/sec</td>
<td class="c3"><div class="bsql" style="width:350px">&nbsp;</div></td></tr>
</table>
<h4>Random Writes</h4>
<table class="bn bnbase">
<tr><td class="c1">LevelDB</td>
<td class="c2">480 ops/sec</td>
<td class="c3"><div class="bldb" style="width:105px">&nbsp;</div></td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">1,100 ops/sec</td>
<td class="c3"><div class="bkct" style="width:240px">&nbsp;</div></td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">1,600 ops/sec</td>
<td class="c3"><div class="bsql" style="width:350px">&nbsp;</div></td></tr>
</table>
<p>LevelDB doesn't perform as well with large values of 100,000 bytes each. This is because LevelDB writes keys and values at least twice: first time to the transaction log, and second time (during a compaction) to a sorted file.
With larger values, LevelDB's per-operation efficiency is swamped by the
cost of extra copies of large values.</p>
<h3>B. Batch Writes</h3>
<p>A batch write is a set of writes that are applied atomically to the underlying database. A single batch of N writes may be significantly faster than N individual writes. The following benchmark writes one thousand batches where each batch contains one thousand 100-byte values. TreeDB does not support batch writes and is omitted from this benchmark.</p>
<h4>Sequential Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">840,000 entries/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.08x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">124,000 entries/sec</td>
<td class="c3"><div class="bsql" style="width:52px">&nbsp;</div></td>
<td class="c4">(2.55x baseline)</td></tr>
</table>
<h4>Random Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">221,000 entries/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.35x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">22,000 entries/sec</td>
<td class="c3"><div class="bsql" style="width:34px">&nbsp;</div></td>
<td class="c4">(2.23x baseline)</td></tr>
</table>
<p>Because of the way LevelDB persistent storage is organized, batches of
random writes are not much slower (only a factor of 4x) than batches
of sequential writes.</p>
<h3>C. Synchronous Writes</h3>
<p>In the following benchmark, we enable the synchronous writing modes
of all of the databases. Since this change significantly slows down the
benchmark, we stop after 10,000 writes. For synchronous write tests, we've
disabled hard drive write-caching (using `hdparm -W 0 [device]`).</p>
<ul>
<li>For LevelDB, we set WriteOptions.sync = true.</li>
<li>In TreeDB, we enabled TreeDB's OAUTOSYNC option.</li>
<li>For SQLite3, we set "PRAGMA synchronous = FULL".</li>
</ul>
<h4>Sequential Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">100 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(0.003x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">7 ops/sec</td>
<td class="c3"><div class="bkct" style="width:27px">&nbsp;</div></td>
<td class="c4">(0.0004x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">88 ops/sec</td>
<td class="c3"><div class="bsql" style="width:315px">&nbsp;</div></td>
<td class="c4">(0.002x baseline)</td></tr>
</table>
<h4>Random Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">100 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(0.015x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">8 ops/sec</td>
<td class="c3"><div class="bkct" style="width:29px">&nbsp;</div></td>
<td class="c4">(0.001x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">88 ops/sec</td>
<td class="c3"><div class="bsql" style="width:314px">&nbsp;</div></td>
<td class="c4">(0.009x baseline)</td></tr>
</table>
<p>Also see the <code>ext4</code> performance numbers below
since synchronous writes behave significantly differently
on <code>ext3</code> and <code>ext4</code>.</p>
<h3>D. Turning Compression Off</h3>
<p>In the baseline measurements, LevelDB and TreeDB were using
light-weight compression
(<a href="http://code.google.com/p/snappy/">Snappy</a> for LevelDB,
and <a href="http://www.oberhumer.com/opensource/lzo/">LZO</a> for
TreeDB). SQLite3, by default does not use compression. The
experiments below show what happens when compression is disabled in
all of the databases (the SQLite3 numbers are just a copy of
its baseline measurements):</p>
<h4>Sequential Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">594,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(0.76x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">485,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:239px">&nbsp;</div></td>
<td class="c4">(1.42x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">48,600 ops/sec</td>
<td class="c3"><div class="bsql" style="width:29px">&nbsp;</div></td>
<td class="c4">(1.00x baseline)</td></tr>
</table>
<h4>Random Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">135,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:296px">&nbsp;</div></td>
<td class="c4">(0.82x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">159,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.80x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">9,860 ops/sec</td>
<td class="c3"><div class="bsql" style="width:22px">&nbsp;</div></td>
<td class="c4">(1.00x baseline)</td></tr>
</table>
<p>LevelDB's write performance is better with compression than without
since compression decreases the amount of data that has to be written
to disk. Therefore LevelDB users can leave compression enabled in
most scenarios without having worry about a tradeoff between space
usage and performance. TreeDB's performance on the other hand is
better without compression than with compression. Presumably this is
because TreeDB's compression library (LZO) is more expensive than
LevelDB's compression library (Snappy).<p>
<h3>E. Using More Memory</h3>
<p>We increased the overall cache size for each database to 128 MB. For LevelDB, we partitioned 128 MB into a 120 MB write buffer and 8 MB of cache (up from 2 MB of write buffer and 2 MB of cache). For SQLite3, we kept the page size at 1024 bytes, but increased the number of pages to 131,072 (up from 4096). For TreeDB, we also kept the page size at 1024 bytes, but increased the cache size to 128 MB (up from 4 MB).</p>
<h4>Sequential Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">812,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.04x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">321,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:138px">&nbsp;</div></td>
<td class="c4">(0.94x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">48,500 ops/sec</td>
<td class="c3"><div class="bsql" style="width:21px">&nbsp;</div></td>
<td class="c4">(1.00x baseline)</td></tr>
</table>
<h4>Random Writes</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">355,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(2.16x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">284,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:280px">&nbsp;</div></td>
<td class="c4">(3.21x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">9,670 ops/sec</td>
<td class="c3"><div class="bsql" style="width:10px">&nbsp;</div></td>
<td class="c4">(0.98x baseline)</td></tr>
</table>
<p>SQLite's performance does not change substantially when compared to
the baseline, but the random write performance for both LevelDB and
TreeDB increases significantly. LevelDB's performance improves
because a larger write buffer reduces the need to merge sorted files
(since it creates a smaller number of larger sorted files). TreeDB's
performance goes up because the entire database is available in memory
for fast in-place updates.</p>
<h2>3. Read Performance under Different Configurations</h2>
<h3>A. Larger Caches</h3>
<p>We increased the overall memory usage to 128 MB for each database.
For LevelDB, we allocated 8 MB to LevelDB's write buffer and 120 MB
to LevelDB's cache. The other databases don't differentiate between a
write buffer and a cache, so we simply set their cache size to 128
MB.</p>
<h4>Sequential Reads</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">5,210,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.29x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">1,070,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:72px">&nbsp;</div></td>
<td class="c4">(1.06x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">609,000 ops/sec</td>
<td class="c3"><div class="bsql" style="width:41px">&nbsp;</div></td>
<td class="c4">(1.59x baseline)</td></tr>
</table>
<h4>Random Reads</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">190,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:144px">&nbsp;</div></td>
<td class="c4">(1.47x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">463,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:350px">&nbsp;</div></td>
<td class="c4">(3.07x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">186,000 ops/sec</td>
<td class="c3"><div class="bsql" style="width:141px">&nbsp;</div></td>
<td class="c4">(1.39x baseline)</td></tr>
</table>
<p>As expected, the read performance of all of the databases increases
when the caches are enlarged. In particular, TreeDB seems to make
very effective use of a cache that is large enough to hold the entire
database.</p>
<h3>B. No Compression Reads </h3>
<p>For this benchmark, we populated a database with 1 million entries consisting of 16 byte keys and 100 byte values. We compiled LevelDB and Kyoto Cabinet without compression support, so results that are read out from the database are already uncompressed. We've listed the SQLite3 baseline read performance as a point of comparison.</p>
<h4>Sequential Reads</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">4,880,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.21x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">1,230,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:88px">&nbsp;</div></td>
<td class="c4">(3.60x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">383,000 ops/sec</td>
<td class="c3"><div class="bsql" style="width:27px">&nbsp;</div></td>
<td class="c4">(1.00x baseline)</td></tr>
</table>
<h4>Random Reads</h4>
<table class="bn">
<tr><td class="c1">LevelDB</td>
<td class="c2">149,000 ops/sec</td>
<td class="c3"><div class="bldb" style="width:300px">&nbsp;</div></td>
<td class="c4">(1.16x baseline)</td></tr>
<tr><td class="c1">Kyoto TreeDB</td>
<td class="c2">175,000 ops/sec</td>
<td class="c3"><div class="bkct" style="width:350px">&nbsp;</div></td>
<td class="c4">(1.16x baseline)</td></tr>
<tr><td class="c1">SQLite3</td>
<td class="c2">134,000 ops/sec</td>
<td class="c3"><div class="bsql" style="width:268px">&nbsp;</div></td>
<td class="c4">(1.00x baseline)</td></tr>
</table>
<p>Performance of both LevelDB and TreeDB improves a small amount when
compression is disabled. Note however that under different workloads,
performance may very well be better with compression if it allows more
of the working set to fit in memory.</p>
<h2>Note about Ext4 Filesystems</h2>
<p>The preceding numbers are for an ext3 file system. Synchronous writes are much slower under <a href="http://en.wikipedia.org/wiki/Ext4">ext4</a> (LevelDB drops to ~31 writes / second and TreeDB drops to ~5 writes / second; SQLite3's synchronous writes do not noticeably drop) due to ext4's different handling of <span class="code">fsync</span> / <span class="code">msync</span> calls. Even LevelDB's asynchronous write performance drops somewhat since it spreads its storage across multiple files and issues <span class="code">fsync</span> calls when switching to a new file.</p>
<h2>Acknowledgements</h2>
<p>Jeff Dean and Sanjay Ghemawat wrote LevelDB. Kevin Tseng wrote and compiled these benchmarks. Mikio Hirabayashi, Scott Hess, and Gabor Cselle provided help and advice.</p>
</body>
</html>
src/leveldb/doc/doc.css
+89
View File
@@ -0,0 +1,89 @@
body {
margin-left: 0.5in;
margin-right: 0.5in;
background: white;
color: black;
}
h1 {
margin-left: -0.2in;
font-size: 14pt;
}
h2 {
margin-left: -0in;
font-size: 12pt;
}
h3 {
margin-left: -0in;
}
h4 {
margin-left: -0in;
}
hr {
margin-left: -0in;
}
/* Definition lists: definition term bold */
dt {
font-weight: bold;
}
address {
text-align: center;
}
code,samp,var {
color: blue;
}
kbd {
color: #600000;
}
div.note p {
float: right;
width: 3in;
margin-right: 0%;
padding: 1px;
border: 2px solid #6060a0;
background-color: #fffff0;
}
ul {
margin-top: -0em;
margin-bottom: -0em;
}
ol {
margin-top: -0em;
margin-bottom: -0em;
}
UL.nobullets {
list-style-type: none;
list-style-image: none;
margin-left: -1em;
}
p {
margin: 1em 0 1em 0;
padding: 0 0 0 0;
}
pre {
line-height: 1.3em;
padding: 0.4em 0 0.8em 0;
margin: 0 0 0 0;
border: 0 0 0 0;
color: blue;
}
.datatable {
margin-left: auto;
margin-right: auto;
margin-top: 2em;
margin-bottom: 2em;
border: 1px solid;
}
.datatable td,th {
padding: 0 0.5em 0 0.5em;
text-align: right;
}

Some files were not shown because too many files have changed in this diff Show More