support binlog + rocksdb group commit and XA

[hermanlee]

Issue by jonahcohen
Wednesday Jan 07, 2015 at 23:01 GMT
Originally opened as MySQLOnRocksDB#8

---

From @mdcallag:

This requires more discussion but many of us are in favor of it.

It might be time to use the binlog as the source of truth to avoid the complexity and inefficiency of keeping RocksDB and the binlog synchronized via internal XA. There are two modes for this. The first mode is durable in which case fsync is done after writing the binlog and RocksDB WAL. The other mode is non-durable in which case fsync might only be done once per second and we rely on lossless semisync to recover. Binlog as source of truth might have been discussed on a MariaDB mail list many years ago - https://lists.launchpad.net/maria-developers/msg01998.html

Some details are at http://yoshinorimatsunobu.blogspot.com/2014/04/semi-synchronous-replication-at-facebook.html

The new protocol will be:

1. write binlog
2. optionally sync binlog
3. optionally wait for semisync ack
4. commit rocksdb - this also persists the GTID within RocksDB for the most recent commit, this also makes changes from the transaction visible to others
5. optionally sync rocksdb WAL

When lossless semisync is used we skip steps 2 and 4. When lossless semisync is not used we do step 2 and skip 3. Step 4 is optional. Recovery in this case is done by:

1. query RocksDB to determine GTID of last commit it has
2. extract/replay transactions from binlog >= GTID from previous step

When running in non durable mode, then on a crash one of the following is true where the relation describes which one has more commits:

1. rocksdb > binlog
2. binlog > rocksdb
3. rocksdb == binlog
   If you know which state the server is in, then you can reach state 3. If in state 1 then append events to the binlog without running them on innodb. If in state 2 then replay events to innodb without recording to binlog. If in state 3 then do nothing. Both RocksDB and the binlog can tell us the last GTID they contain and we can compare that with the binlog archived via lossless semisync to determine the state.

[yoshinorim]

XA means XA between binlog and MyRocks -- writing xid into MyRocks, and proper roll forward/rollback at crash recovery.

[mdcallag]

This shows the impact (perf loss) from not having group commit in MyRocks.

This is the insert rate from a short run of the insert benchmark -- insert only, 3 secondary indexes/table, 10 clients each loading into a separate table, 10M rows total inserted using FB MySQL 5.6 with MyRocks vs Percona Server 5.6.26 for Toku. The database log (MyRocks, Toku) was not sync'ed on commit.

The database is on a 6 disk array with HW RAID and battery backed write cache.

Summary:

• MyRocks loses perf from not having group commit
• Toku in Percona Server no longer has a fast path when unique checks are disabled

Legend:

• sync=1 - binlog enabled, sync_binlog=1
• sync=0 - binlog enabled, sync_binlog=0
• nobl - binlog disabled
• nobl, nocheck - binlog disabled, unique checks disabled

inserts/second
rocks   toku    config
63694   129870  sync=1
64935   133333  sync=0
151515  136986  nobl
204082  136986  nobl, nocheck

[reidHoruff]

XA and group commit support had been added to MyRocks.