storage: use sensible zone configs for critical system and time series ranges

[a-robinson]

While it makes sense for small clusters, our current default zone configuration that only creates 3 replicas of all data by default is somewhat risky for large clusters, where it may be preferable to keep more than 3 replicas of critical system ranges.

This can be addressed via documentation for 1.0 (cockroachdb/docs#1280, cockroachdb/docs#1248), but before 1.1 we should do some testing with different configurations and consider setting more replicas of the system ranges by default.

[tbg]

A sensible choice would be to replicate these ranges with factor max(min(nodes_in_cluster, 5), highest_replication_factor_for_any_zone). By that token, you could be assured that if you run a range with, say, x11, you can actually tolerate 5 dead replicas (and not just whatever the critical system ranges top out at).

[tbg]

via @bdarnell:

yeah. we should eventually move to more fine-grained defaults for different system sub-ranges (high replication factor and long ttl for range metadata, high replication and low ttl for liveness, low replication for timeseries)

[bdarnell]

The .system special zone currently includes too much. It has both the liveness span (which wants to have a short TTL) and the system descriptor/namespace tables (which need to have a TTL at least as high as the TTL for any other table. Time travel queries are limited to the shorter of the TTLs of the table itself and the table descriptor tables).

[cuongdo]

@petermattis this seems like a risky change at this point; should this move to 1.2?

[bdarnell]

Yes, we already agreed to move this to 1.2 in #17628

[tbg]

I added a docs-todo for a known limitation about this, see #14990.

[tbg]

Filed #28901 to track cascading zone configs.

[petermattis]

@m-schneider Is there anything left to do here?

[m-schneider]

No, closing.

[tbg]

I'm a bit confused. I assume that #27349 is the PR that closes this issue (though it didn't refer it prior to this commit).

That PR is sparse on description, but from the code it looks like what should happen is that if I set up a new five node cluster, I'll get my critical system ranges 5x replicated.

That doesn't seem to be the case:

root@:26257/> select * from crdb_internal.ranges;
  range_id |     start_key     |         start_pretty          |      end_key      |          end_pretty           | database |      table       | index | replicas | lease_holder
+----------+-------------------+-------------------------------+-------------------+-------------------------------+----------+------------------+-------+----------+--------------+
         1 |                   | /Min                          | \004              | /System/""                    |          |                  |       | {1,4,5}  |            1
         2 | \004              | /System/""                    | \004\000liveness- | /System/NodeLiveness          |          |                  |       | {1,3,2}  |            1
         3 | \004\000liveness- | /System/NodeLiveness          | \004\000liveness. | /System/NodeLivenessMax       |          |                  |       | {1,3,5}  |            3
         4 | \004\000liveness. | /System/NodeLivenessMax       | \004tsd           | /System/tsd                   |          |                  |       | {4,5,3}  |            4
         5 | \004tsd           | /System/tsd                   | \004tse           | /System/"tse"                 |          |                  |       | {4,5,3}  |            5
         6 | \004tse           | /System/"tse"                 | \210              | /Table/SystemConfigSpan/Start |          |                  |       | {1,3,2}  |            1
         7 | \210              | /Table/SystemConfigSpan/Start | \223              | /Table/11                     |          |                  |       | {1,3,2}  |            1
         8 | \223              | /Table/11                     | \224              | /Table/12                     | system   | lease            |       | {1,5,2}  |            2
         9 | \224              | /Table/12                     | \225              | /Table/13                     | system   | eventlog         |       | {4,3,2}  |            2
        10 | \225              | /Table/13                     | \226              | /Table/14                     | system   | rangelog         |       | {5,4,2}  |            4
        11 | \226              | /Table/14                     | \227              | /Table/15                     | system   | ui               |       | {1,2,3}  |            1
        12 | \227              | /Table/15                     | \230              | /Table/16                     | system   | jobs             |       | {5,4,2}  |            5
        13 | \230              | /Table/16                     | \231              | /Table/17                     |          |                  |       | {1,3,4}  |            3
        14 | \231              | /Table/17                     | \232              | /Table/18                     |          |                  |       | {4,5,2}  |            5
        15 | \232              | /Table/18                     | \233              | /Table/19                     |          |                  |       | {1,2,4}  |            2
        16 | \233              | /Table/19                     | \234              | /Table/20                     | system   | web_sessions     |       | {1,5,3}  |            3
        17 | \234              | /Table/20                     | \235              | /Table/21                     | system   | table_statistics |       | {1,5,2}  |            2
        18 | \235              | /Table/21                     | \236              | /Table/22                     | system   | locations        |       | {1,3,2}  |            1
        19 | \236              | /Table/22                     | \237              | /Table/23                     |          |                  |       | {4,5,2}  |            2
        20 | \237              | /Table/23                     | \377\377          | /Max                          | system   | role_members     |       | {4,3,5}  |            5
(20 rows)

Time: 29.585ms

warning: no current database set. Use SET database = <dbname> to change, CREATE DATABASE to make a new database.
root@:26257/> select * from crdb_internal.gossip_liveness;
  node_id | epoch |       expiration       | draining | decommissioning
+---------+-------+------------------------+----------+-----------------+
        1 |     1 | 1538992670.035649687,0 |  false   |      false
        2 |     1 | 1538992670.740451009,0 |  false   |      false
        3 |     1 | 1538992671.575627883,0 |  false   |      false
        4 |     1 | 1538992672.506993000,0 |  false   |      false
        5 |     1 | 1538992673.226969883,0 |  false   |      false
(5 rows)

@m-schneider are my assumptions wrong? What should happen in my situation?

[m-schneider]

Taking a look.

[m-schneider]

How did you start up the cluster? I tried to reproduce on master, but I got the expected behavior:

root@:26257/defaultdb> select * from crdb_internal.ranges;                                                                                                                                                                                                                                                                                                         range_id |     start_key     |         start_pretty          |      end_key      |          end_pretty           | database_name |    table_name    | index_name |  replicas   | lease_holder  
+----------+-------------------+-------------------------------+-------------------+-------------------------------+---------------+------------------+------------+-------------+--------------+
         1 |                   | /Min                          | \004              | /System/""                    |               |                  |            | {1,2,3,4,5} |            1  
         2 | \004              | /System/""                    | \004\000liveness- | /System/NodeLiveness          |               |                  |            | {1,2,3,4,5} |            1  
         3 | \004\000liveness- | /System/NodeLiveness          | \004\000liveness. | /System/NodeLivenessMax       |               |                  |            | {1,2,3,4,5} |            1  
         4 | \004\000liveness. | /System/NodeLivenessMax       | \004tsd           | /System/tsd                   |               |                  |            | {1,2,3,4,5} |            1  
         5 | \004tsd           | /System/tsd                   | \004tse           | /System/"tse"                 |               |                  |            | {1,2,5}     |            1  
         6 | \004tse           | /System/"tse"                 | \210              | /Table/SystemConfigSpan/Start |               |                  |            | {1,2,3,4,5} |            4  
         7 | \210              | /Table/SystemConfigSpan/Start | \223              | /Table/11                     |               |                  |            | {1,2,3,4,5} |            1  
         8 | \223              | /Table/11                     | \224              | /Table/12                     | system        | lease            |            | {1,2,3,4,5} |            3  
         9 | \224              | /Table/12                     | \225              | /Table/13                     | system        | eventlog         |            | {1,2,3,4,5} |            1  
        10 | \225              | /Table/13                     | \226              | /Table/14                     | system        | rangelog         |            | {1,2,3,4,5} |            1  
        11 | \226              | /Table/14                     | \227              | /Table/15                     | system        | ui               |            | {1,2,3,4,5} |            5  
        12 | \227              | /Table/15                     | \230              | /Table/16                     | system        | jobs             |            | {1,2,3,4,5} |            2  
        13 | \230              | /Table/16                     | \231              | /Table/17                     |               |                  |            | {1,2,3,4,5} |            1  
        14 | \231              | /Table/17                     | \232              | /Table/18                     |               |                  |            | {1,2,3,4,5} |            1  
        15 | \232              | /Table/18                     | \233              | /Table/19                     |               |                  |            | {1,2,3,4,5} |            4  
        16 | \233              | /Table/19                     | \234              | /Table/20                     | system        | web_sessions     |            | {1,2,3,4,5} |            2  
        17 | \234              | /Table/20                     | \235              | /Table/21                     | system        | table_statistics |            | {1,2,3,4,5} |            1  
        18 | \235              | /Table/21                     | \236              | /Table/22                     | system        | locations        |            | {1,2,3,4,5} |            1  
        19 | \236              | /Table/22                     | \237              | /Table/23                     |               |                  |            | {1,2,3,4,5} |            1  
        20 | \237              | /Table/23                     | \377\377          | /Max                          | system        | role_members     |            | {1,2,3,4,5} |            1  
(20 rows)

Time: 158.343155ms

root@:26257/defaultdb> select * from crdb_internal.gossip_liveness;
  node_id | epoch |       expiration       | draining | decommissioning |            updated_at             
+---------+-------+------------------------+----------+-----------------+----------------------------------+
        1 |     1 | 1539012150.769984390,0 |  false   |      false      | 2018-10-08 15:22:21.273795+00:00  
        2 |     1 | 1539012150.863685273,0 |  false   |      false      | 2018-10-08 15:22:21.369532+00:00  
        3 |     1 | 1539012150.852281022,0 |  false   |      false      | 2018-10-08 15:22:21.361692+00:00  
        4 |     1 | 1539012150.877309202,0 |  false   |      false      | 2018-10-08 15:22:21.382665+00:00  
        5 |     1 | 1539012150.893886556,0 |  false   |      false      | 2018-10-08 15:22:21.399011+00:00  

[tbg]

I used roachdemo and clicked the add button in the ui five times :-)

[tbg]

Let me try that again.. it'd be puzzling if that made a difference.

[tbg]

Hrm. Unfortunately, it behaves as advertised now. I wonder what the difference was? How's our test coverage for this?

[m-schneider]

We have pretty extensive testing in allocator_test.go for various combinations of available vs alive nodes.

[a-robinson]

That doesn't necessarily mean that it always works end-to-end, though.

@tschottdorf do you know which version of cockroach you were on and whether roachdemo was resuming a preexisting cluster or if it initialized a new one? If the cluster was initialized before #27349 / #30480 then you wouldn't see the new behavior regardless of which version you were using.

[m-schneider]

I haven't been able to reproduce after a couple of attempts and after using roachdemo. Should we close for now? If you see this again can you please run:
select * from crdb_internal.kv_store_status;

[tbg]

@a-robinson I looked after reading your comment two days ago but the buffer had been lost. I also gave this a a few more rounds but it just worked. Still very weird. I make build pretty regularly, though perhaps I had a stray cockroach binary from a release branch that I used by accident.
Sorry about the noise!