Read scalability
WiredTiger is architected to scale linearly on multi-threaded workloads. This test demonstrates that scaling for a read only workload. It also compares the results to another popular key/value store - LevelDB.
The test executes 10 million reads in each of the threads. So the 32 thread version is doing 32 times as much work as the single threaded version.
Here are those results in a table:
| Thread count | WiredTiger us/op | LevelDB us/op |
|---|---|---|
| 1 | 3.44 | 7.19 |
| 2 | 3.41 | 7.63 |
| 4 | 2.97 | 7.85 |
| 8 | 3.39 | 9.71 |
| 12 | 3.31 | 19.05 |
| 16 | 3.31 | 27.31 |
| 20 | 3.55 | 35.63 |
| 24 | 3.73 | 45.44 |
| 28 | 3.94 | 56.97 |
| 32 | 4.25 | 69.72 |
The same results as operations per second rather than latency.
| Thread count | WiredTiger ops/second | LevelDB ops/second |
|---|---|---|
| 1 | 290,697 | 139,682 |
| 2 | 586,510 | 262,123 |
| 4 | 1,346,801 | 509,554 |
| 8 | 2,359,882 | 823,892 |
| 12 | 3,625,377 | 629,921 |
| 16 | 4,833,836 | 585,865 |
| 20 | 5,633,802 | 561,324 |
| 24 | 6,434,316 | 528,169 |
| 28 | 7,106,598 | 491,486 |
| 32 | 7,529,411 | 458,978 |
The benchmark inserts 10 million items into a WiredTiger table, with 16 byte keys and 100 byte values creating about 800 MB of data.
The read phase of the benchmark uses a 10GB cache, has a variable number of threads doing WT_CURSOR::search operations for random values over the key range. During the read phase each thread completes 10 million operations.
Run on an AWS c3.8xlarge instance type, which has:
- 60 GB RAM
- 2 x 320 GB SSD drives configured with raid 0 and xfs
- 4 physical Xeon E5-2680 CPUs - 32 vCPUs
Benchmark and LevelDB source code at revision: https://github.com/wiredtiger/leveldb_wt/tree/ffd56f7eb55d750bd64ef8e77e1b3f8c1d1996cc
With compression disabled for the WiredTiger version.
WiredTiger source code at revision: https://github.com/wiredtiger/wiredtiger/commit/562892d2c5ed3c2e4679e1650411224324059f58
./configure --enable-snappy && make
LD_LIBRARY_PATH=~/work/wiredtiger/build_posix/.libs:~/work/wiredtiger/build_posix/ext/compressors/snappy/.libs
env LDFLAGS="-L~/work/wiredtiger/build_posix/.libs" CXXFLAGS="-I~/work/wiredtiger/build_posix" make all db_bench_wiredtiger
./db_bench_wiredtiger --use_lsm=0 --cache_size=10000000000 --benchmarks=fillseq --num=10000000 --value_size=100 --db=results --use_existing_db=0 --threads=1
for i in 1 2 4 8 12 16 20 24 28 32 ; do
./db_bench_wiredtiger --use_lsm=0 --cache_size=10000000000 --benchmarks=readrandom --num=10000000 --reads=10000000 --value_size=100 --db=results --use_existing_db=1 --threads=$i
done
To run the equivalent with LevelDB, use the same commands, but run ./db_bench rather than ./db_bench_wiredtiger and remove the --use_lsm=0 option.