Column Family Support

This is documentation about how RocksDB Storage Engine supports RocksDB's Column Families.

Assigning indexes to Column Families

Each index can be assigned a column family. Column Family is specified in the COMMENT field of the index:

create table t1 (
  col1 int,
  ...
  INDEX (col1) COMMENT 'column_family_name'
  ...
) engine=rocksdb;

If no Column Family is specified for an index, then the index is put into the default Column Family which is always named default.

Per-index column families

If one wants to put each index into its own column family, it is possible to use a special name $per_index_cf :

  INDEX index_name (col1) COMMENT '$per_index_cf'

This will put the index into a column family named db_name.table_name.index_name. There is a limitation: ALTER TABLE and RENAME TABLE are disallowed for tables that use $per_index_cf.

Reverse-ordered column families

If efficient execution of ORDER BY key DESC LIMIT n has the top priority, then one can use a column family that stores data in reverse order. When the data is stored in reverse order, ORDER BY DESC index scan is a forward scan and it is done with rocksdb::Iterator::Next() calls, which is faster than scanning with rocksdb::Iterator::Prev() calls.

A column family is reverse-ordered if its name starts with rev:.

(See also: Issue#25).

Specifying Column Family parameters

One can run RocksDB with different parameters for column families. At the moment, RocksDB-SE allows to specify these per-column family options:

• write_buffer_size
• target_file_size_base

One can specify these parameters by setting their appropriate options in my.cnf:

• rocksdb-write-buffer-size=VALUE
• rocksdb-file-size-base=VALUE

If VALUE is a single number, it is applied to the default column family. It is also possible to specify values for each column family:

rocksdb-write-buffer-size='cf1:value1,cf2:value2,cfN:valueN'

Note that specifying cf1:value1 will not cause Column Family cf to be created. It will be created as soon as there is an index that is using it.

If a Column Family is not listed, its settings will be inherited from the default Column Family.

Checking Column Family settings

At start, RocksDB-SE prints to server stderr messages like this:

  2014-09-10 14:47:38 31079 [Note] RocksDB: Column Families at start:
  2014-09-10 14:47:38 31079 [Note]   cf=default
  2014-09-10 14:47:38 31079 [Note]     write_buffer_size=4194304
  2014-09-10 14:47:38 31079 [Note]     target_file_size_base=2097152
  2014-09-10 14:47:38 31079 [Note]   cf=cf1
  2014-09-10 14:47:38 31079 [Note]     write_buffer_size=10000
  2014-09-10 14:47:38 31079 [Note]     target_file_size_base=2097152
  2014-09-10 14:47:38 31079 [Note]   cf=cf3
  2014-09-10 14:47:38 31079 [Note]     write_buffer_size=3000000
  2014-09-10 14:47:38 31079 [Note]     target_file_size_base=2097152

When a Column Family is created, its parameters are also printed into server stderr:

  2014-09-10 14:47:19 31052 [Note] RocksDB: creating column family cf3
  2014-09-10 14:47:19 31052 [Note]     write_buffer_size=3000000
  2014-09-10 14:47:19 31052 [Note]     target_file_size_base=2097152

Listing column families

There is currently no way. SHOW ENGINE ROCKSDB STATUS lists RocksDB's "Live files" which gives some idea of what column families are there, but it's an indirect observation.

Implementation details

Column families are created on demand. That is, when one creates a table with INDEX(...) COMMENT 'cf_name', RocksDB-SE will check whether column family named cf_name exists, and if not, column family will be created.

At the moment, the only place that stores information about which indexes are in which column families is the COMMENT field in table definitions.

There is currently no way to delete a column family. One may drop all indexes that use the column family, but this will not cause the column family to be dropped.