Per vhost message store #766

hairyhum · 2016-04-21T15:45:55Z

Closes #567.

rabbit_msg_store is a gen_server having directory in it's working state and storing data in Dir/server_name.
To be able to have multiple message stores where should be multiple rabbit_msg_store servers with different names and different directories (or just different names).
rabbit_msg_store_vhost_sup is a supervisor, that can create a message store for specified VHost. This supervisor is started instead of rabbit_msg_store during boot process.
rabbit_variable_queue access message store using message store clients and can request this client from supervisor (supplying VHost) the same way as it does from message store directly.

To move messages from old message store format to the VHost based format there are special boot step.
This boot step is running before queue recovery step so queue index and message store is not yet started. This boot step are performing new scope of upgrades: queues.
move_messages_to_vhost_store is upgrade performing message migration. It starts queue indexes for durable queues, recovers old format message store, reads all messages from index that is stored in message store and writes them to the new store. After migrating all durable queues old message store is being deleted.

Message stores are placed in rabbit_mnesia:dir()/VHOST/msg_store_typeVHOST. VHOST is VHost name encoded with base64.
Separate VHOST directories are used to make it easier to operator to control multitenancy. Queue indexes may be moved to VHOST directories as well.

Tested manually so far.

…ting vhost

…ean message storage (together with message indexes) for a vhost

hairyhum · 2016-10-20T16:59:04Z

Queue indexes was also moved to vhost storage directory, so it's easier to distinguish queues from different vhosts.
New queue index location: mnesia_dir/queues/:queue_id -> mnesia_dir/:vhost/queues/:queue_id
So directory mnesia_dir/:vhost effectively contains all messages data.
If vhost is deleted - all it's messages data is deleted, including directory.
It's possible to delete all vhost data by running rabbit_vhost:purge_messages/1, that will delete vhost message store and all it's queues indexes. It's not safe operation.

Testing

Migration to per-vhost message store mechanism can be tested using make in https://github.com/rabbitmq/rabbitmq-server-release/tree/stable/upgrade.

michaelklishin

It looks promising but vhost-to-dir naming function is not safe.

michaelklishin · 2016-11-11T14:02:11Z

src/rabbit_queue_index.erl


 blank_state_dir(Dir) ->
    blank_state_dir_funs(Dir,
                         fun (_) -> ok end,
                         fun (_) -> ok end).

+queue_dir(#resource{ virtual_host = VHost } = QueueName) ->
+    %% Queue directory is rabbit_mnesia_dir/:vhost/queues/:queue_id


It would be a bit clearer if we used {node_database_dir}/vhosts/{vhost}/queues/{queue}.

michaelklishin · 2016-11-11T14:02:32Z

src/rabbit_queue_index.erl

-    end.
+all_queue_directory_names() ->
+    QueuesBaseDir = queues_base_dir(),
+    filelib:wildcard(filename:join([QueuesBaseDir, "*", "queues", "*"])).


See below. It would be less risky if we used {node_database_dir}/vhosts/{vhost}/queues/{queue} for path naming.

michaelklishin · 2016-11-11T14:06:19Z

src/rabbit_vhost.erl

@@ -185,3 +197,6 @@ info_all(Ref, AggregatorPid)        -> info_all(?INFO_KEYS, Ref, AggregatorPid).
 info_all(Items, Ref, AggregatorPid) ->
    rabbit_control_misc:emitting_map(
       AggregatorPid, Ref, fun(VHost) -> info(VHost, Items) end, list()).
+
+dir(Vhost) ->
+    base64:encode(Vhost).


After a brief internal discussion we decided it's not safe to use Base64 or Base32 for our needs because they both use lower and upper case characters. Queue directory naming uses MD5 hashing, which is safe but annoying for operators as it's not easy to infer what directory corresponds to what queue.

I suspect we can create a text file in each vhost (and possibly queue) directory that contains its actual name.

michaelklishin · 2016-11-11T15:19:56Z

Another change worth considering is keeping track of vhost message store processes in an ETS table as opposed to using registered processes: with very high vhost churn we run the risk of exhausting the atom table.

hairyhum · 2016-11-30T12:28:54Z

Queues are migrated in batches of 100 in parallel by default. The batch size can be configured with queue_migration_batch_size environment variable.

hairyhum · 2016-11-30T12:31:38Z

Per queue migration messages are logged to rabbit_upgrade.log file.

Required to decrease ETS tables usage per vhost

Gsantomaggio · 2016-12-20T14:51:37Z

src/rabbit_queue_index.erl

+            ok = rabbit_file:ensure_dir(NewQueueDir),
+            ok = rabbit_file:rename(OldQueueDir, NewQueueDir);
+        false ->
+            rabbit_log:info("Queue index directoy not found for queue ~p~n",


directoy should be directory

… for each vhost on migration. Empty message stores should be started to avoid rebuilding indexes on actual startup.

…tions

It emits a warning and breaks compilation on OTP 20. Originally introduced in 492e23b, likely unintentionally. Fixes #1272, references #567, #766.

hairyhum force-pushed the rabbitmq-server-567 branch from 31a922b to 99ee963 Compare June 2, 2016 12:40

hairyhum force-pushed the rabbitmq-server-567 branch from 43a239d to 38b51ac Compare October 18, 2016 15:37

Daniil Fedotov added 18 commits October 20, 2016 14:49

message store supervisor

abaa770

Starting and restarting app

80436df

Migration algorithm

ec15e97

Work in progress: Migration to vhost based message store

d1bffd0

wip

b3fc71a

wip: migrating to vhost base messge store

492e23b

New upgrade scope queues started by boot step before queue recovery

325e2fc

Remove duplicate migration. Match no_ack message

d1f4a9d

Moved directory selection to vhost_dir function

93ba24b

Update tests

460abf2

Updated tests to support per-vhost message store

fdd7b87

Queue index per vhost location

e6c76fd

Tests for per-vhost message store. Stop vhost message store when dele…

4f12ebe

…ting vhost

Terminate message store ignoring file save errors

fd4db66

Allow restart of a vhost message store. purge_messages function to cl…

8369950

…ean message storage (together with message indexes) for a vhost

Move queues to vhost storage on upgrade

1c81c50

Migration to per-vhost store message

8a45f40

Purge vhost storage in transaction

36f3e67

hairyhum force-pushed the rabbitmq-server-567 branch from 9a94225 to 36f3e67 Compare October 20, 2016 13:50

hairyhum changed the title ~~DO NOT MERGE Per vhost message store~~ Per vhost message store Oct 20, 2016

michaelklishin self-assigned this Nov 9, 2016

michaelklishin and others added 3 commits November 9, 2016 12:22

Rename an upgrade function

008c4eb

Merge branch 'master' into rabbitmq-server-567

4837af6

Naming and cosmetics

05e9dbe

michaelklishin requested changes Nov 11, 2016

View reviewed changes

Merge branch 'master' into rabbitmq-server-567

0b37027

Daniil Fedotov added 2 commits November 29, 2016 16:35

Merge branch 'master' into rabbitmq-server-567

475e86b

Write per queue upgrade log to a separate file

cfc9a88

hairyhum changed the title ~~DO NOT MERGE Per vhost message store~~ Per vhost message store Nov 30, 2016

Make queue migration batch size configurable

f43990d

hairyhum force-pushed the rabbitmq-server-567 branch from b57483d to f43990d Compare November 30, 2016 12:23

hairyhum mentioned this pull request Dec 2, 2016

Lager sink for upgrade logs rabbitmq/rabbitmq-common#152

Merged

dcorbacho and others added 3 commits December 13, 2016 08:39

Merge branch 'master' into rabbitmq-server-567

321abd4

Replace dying_client_index and dying_clients set with a map

806cd80

Required to decrease ETS tables usage per vhost

Merge branch 'master' into rabbitmq-server-567

8a98bdb

Gsantomaggio reviewed Dec 20, 2016

View reviewed changes

Gabriele Santomaggio and others added 14 commits December 20, 2016 17:08

fix typo

3d56870

Merge branch 'master' into rabbitmq-server-567

5f7553c

Group recovery client refs by vhost

308a2b5

Pass queues and recovery terms to fold

1ec2ac8

Refs can be undefined

eafad5a

change error function with rabbit_log:error

569d114

Fix msg_store test

e5557ad

Force recover client references to be maps. Start empty message store…

1403596

… for each vhost on migration. Empty message stores should be started to avoid rebuilding indexes on actual startup.

Recover vhosts message stores in parallel after a crash

5282534

Recover queues after non-clean shutdown

79685d0

Fix logs. Expect multiple logs during startup

83d3222

Revert to syncronous vhost recovery since there are concurency limita…

12901fb

…tions

Merge branch 'master' into rabbitmq-server-567

bcff954

Don't log #resource records

1d4e939

michaelklishin approved these changes Dec 24, 2016

View reviewed changes

michaelklishin merged commit fdf783f into master Dec 24, 2016

michaelklishin added a commit that referenced this pull request Jun 23, 2017

Don't use export_all

371ebdb

It emits a warning and breaks compilation on OTP 20. Originally introduced in 492e23b, likely unintentionally. Fixes #1272, references #567, #766.

dumbbell deleted the rabbitmq-server-567 branch January 2, 2018 15:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Per vhost message store #766

Per vhost message store #766

hairyhum commented Apr 21, 2016 •

edited by michaelklishin

Loading

hairyhum commented Oct 20, 2016

michaelklishin left a comment

michaelklishin Nov 11, 2016

michaelklishin Nov 11, 2016

michaelklishin Nov 11, 2016

michaelklishin commented Nov 11, 2016

hairyhum commented Nov 30, 2016

hairyhum commented Nov 30, 2016

Gsantomaggio Dec 20, 2016

Per vhost message store #766

Per vhost message store #766

Conversation

hairyhum commented Apr 21, 2016 • edited by michaelklishin Loading

hairyhum commented Oct 20, 2016

Testing

michaelklishin left a comment

Choose a reason for hiding this comment

michaelklishin Nov 11, 2016

Choose a reason for hiding this comment

michaelklishin Nov 11, 2016

Choose a reason for hiding this comment

michaelklishin Nov 11, 2016

Choose a reason for hiding this comment

michaelklishin commented Nov 11, 2016

hairyhum commented Nov 30, 2016

hairyhum commented Nov 30, 2016

Gsantomaggio Dec 20, 2016

Choose a reason for hiding this comment

hairyhum commented Apr 21, 2016 •

edited by michaelklishin

Loading