Release notes for v1.2-alpha.20171113

[jseldess]

• Add release notes file
• Update _config.yml
• Update _data/releases.yml
• Add product and doc changes

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/2eb773706b131eab9ae1d772e4c1239484ef87c4/

[RaduBerinde]

Of my changes, the only one that needs a note is "sql: prefer hard limit" (which is also in 1.1.x): "improved memory usage for certain queries that use limits at multiple levels".

[jseldess]

Commits I need help with. If any are noteworthy, please include a comment with a brief summary (1-2 sentences) as well as its category.

commit bc8cdcf120b967b978759c35ab4ec23964b1643a
Author: Raphael 'kena' Poss <[email protected]>
Date:   Mon Nov 6 14:07:06 2017 +0100

    sql: ensure that DELETE on the fast path is still a valid data source
    
    Prior to this patch, the DELETE statement without WHERE and RETURNING
    would use the "fast path", that is, all the work being done in Start().
    
    The code for this would however violate the `planNode` interface
    contract: after the fast path is taken, the code would panic if the
    `Next()` method is called. `planNode` specifies that `Next()` is
    always callable after `Start()`, although perhaps it has nothing to
    do.

commit d2b64a3dd93c7144f2047bba1a5aa7da05a74f93
Author: Raphael 'kena' Poss <[email protected]>
Date:   Sat Nov 4 12:55:28 2017 +0100

    sql/parser: fix the grouping of a = ANY/SOME/ALL b <postfixop>
    
    Prior to this patch, an expression of the form `a = ANY b <postfix>`,
    for example `1 = ANY 2::INT` would be parsed as `(a = ANY b)
    <postfix>` (`((1) = ANY (2))::INT`), not the required `a = ANY (b
    <postfix>)` (`(1) = ANY ((2)::INT)`).
    
    This is incorrect on its face, and peeking at the pg grammar indeed
    different from what pg does: the right operand of an ANY/SOME/ALL
    comparisons must be an `a_expr`, not a `d_expr`.
    
    Also reverts 5a729eed13b2abe492561470548f44f6ffd5e9c3.

commit 0c63e662afb54fe654718075c9416d1c4f24c5bd
Author: Nikhil Benesch <[email protected]>
Date:   Sun Nov 5 20:01:44 2017 -0500

    build: ensure libroach expresses dependency on protobufs
    
    libroach currently has an implicit dependency on the generated C++
    protobufs; make it explicit to avoid unnecessary rebuilds when
    timestamps on the generated files are updated after libroach has
    finished building.

commit 0c2ef77059609a1dc14b924e340baac5781532d2
Author: Nathan VanBenschoten <[email protected]>
Date:   Fri Nov 3 18:30:08 2017 -0400

    build: fix compilation on arm64
    
    This change makes two adjustments to fix the build on arm64:
    - Avoid passing the `-sse3` flag to RocksDB when it's not available
    - Add preprocessor directives to avoid sse4.2 on ARM

commit 099e793a1fd4cf41989ae7833cfc13716274b892
Author: Daniel Harrison <[email protected]>
Date:   Thu Nov 2 15:06:09 2017 -0400

    sql,sqlbase: add some additional PARTITION BY validations
    
    It turns out that generating the spans for a range partition that is
    subpartitioned (by either list or range) depends on the data in the
    table, so we disallow it. Tables partitioned by both list and range will
    have to be list, then range.
    
    Additionally, this commit disallows partitioning any index or table that
    is interleaved into another. The root of the interleave hierarchy can be
    partitioned and all children will be partitioned the same way. It's
    expected that this will be sufficient for our users, but if not, this
    restriction can be lifted with additional work.


commit 3272dcff08536416204333a84a4fb28f20d4471f
Author: Daniel Harrison <[email protected]>
Date:   Thu Nov 2 12:07:20 2017 -0400

    sql/parser: move DEFAULT from ctext_expr to a_expr
    
    This appears to have been done originally because DEFAULT can only
    appear in a VALUES list within INSERT or on the right side of a SET.
    Otherwise, these ctext_exprs would have been a_exprs.
    
    Because of the way the final PARTITION BY syntax works, we need DEFAULT
    to be in a_expr, so this commit moves it there and rips out the ctext
    stuff. ctext_expr, ctext_expr_list, and ctext_row are no longer used and so are
    removed.
    
    Background:
    
    The PARTITION syntax in a PARTITION BY looks like `PARTITION name VALUES
    IN (1, 2)`.  List partitionings can be one or multiple columns and can
    also contain a DEFAULT (range partitionings are the same but with the
    new MAXVALUE keyword instead of DEFAULT).
    
    The IN syntax is inspired by WHERE IN and is intended to match. This
    means that (1, 2) is parseable as 2 1-column list partitions or 1
    2-column list partitions. ((1), (2)) is only parseable as 2 1-column
    list partitions. The relevant part of the way in_expr works is
    '(' expr_list ')', where expr_list is just a list of a_expr. Among other
    things, a_expr can be the various datums or '(' a_expr ')' or
    '(' expr_list ',' a_expr ')'. This is quite elegant and accepts any
    arbitrary nesting of tuples and extra matched parens.
    
    I tried to get it to work without adding DEFAULT and MAXVALUE to a_expr
    but failed. Because partitions can be 'VALUES IN (DEFAULT)` or `VALUES
    IN ((1, DEFAULT), (2, 3))` we need a similar nesting of tuples. This
    leads to conflicts with a_expr around '(', ')', and '.'. I tried using
    %prec to resolve this but was unable to get it to work.
    
    So instead, we add DEFAULT (and in a followup MAXVALUE) to a_expr.


commit 0bdda5d055dd105a8696e408ce6d4925a2cd97c5
Author: Daniel Harrison <[email protected]>
Date:   Mon Oct 30 16:40:19 2017 -0400

    sqlccl: add a function from tabledesc -> spans for zone configs
    
    GenerateSubzoneSpans constructs from a TableDescriptor the entries
    mapping zone config spans to subzones for use in the SubzoneSpans field
    of config.ZoneConfig. SubzoneSpans controls which splits are created, so
    only the spans corresponding to entries in subzones are returned.
    
    Zone configs target indexes and partitions via `subzones`, which are
    attached to a table-scoped row in `system.zones`. Each subzone
    represents one index (primary or secondary) or one partition (or
    subpartition) and contains the usual zone config constraints. They are
    saved to `system.zones` sparsely (only when set by a user) and are the
    most specific entry in the normal cluster-default/database/table/subzone
    config heirarchy.
    
    Each non-interleaved index and partition can be mapped to spans in the
    keyspace. Indexes and range partitions each map to one span, while each
    list partition maps to one or more spans. Each partition span is
    contained by some index span and each subpartition span is contained by
    one of its parent partition's spans. The spans for a given level of a
    range partitioning (corresponding to one `PARTITION BY` in sql or one
    `PartitionDescriptor`) are disjoint, but the spans for a given level of
    a list partitioning may overlap if DEFAULT is used. A list partitioning
    which includes both (1, DEFAULT) and (1, 2) will overlap with the latter
    getting precedence in the zone config heirarchy. NB: In a valid
    PartitionDescriptor, no partitions with the same number of DEFAULTs will
    overlap (this property is used by `indexCoveringsForPartitioning`).
    
    These subzone spans are kept denormalized to the relevant `system.zone`
    row for performance. Given a TableDescriptor, the spans for every
    index/partition/subpartition are created, filtered out if they don't
    have a config set for them, and precedence applied (via
    `OverlapCoveringMerge`) to produce a set of non-overlapping spans, which
    each map to a subzone.  There may be "holes" (uncovered spans) in this
    set.
    
    The returned spans are returned in exactly the format required by
    `system.zones`. They must be sorted and non-overlapping. Each contains
    an IndexID, which maps to one of the input `subzones` by indexing into
    the slice. As space optimizations, all `Key`s and `EndKey`s of
    `SubzoneSpan` omit the common prefix (the encoded table ID) and if
    `EndKey` is equal to `Key.PrefixEnd()` it is omitted.

commit e973f50ad68f76e2bb9824b67a984135c6d001eb
Author: Jordan Lewis <[email protected]>
Date:   Thu Nov 2 13:14:34 2017 -0400

    sql: restore placeholders after internal queries
    
    Previously, running internal queries via a planner's query methods would
    permanently overwrite the placeholder map. This is not acceptable once
    the placeholder map is used during eval time and not just plan creation
    time.


commit 983e330f539c9ed2e9d4977ae08859914486bc5e
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 30 18:18:59 2017 -0400

    storage: move PushTxnQueue to txnwait.Queue
    
    This removes one more roadblock for moving command evaluation outside of the
    storage package since `PushTxnQueue` is returned from `ReplicaEvalContext`
    (moving into its own package wasn't necessary, we could've just used an
    interface -- but this seems nicer).
    
    I spent some time trying to move the tests as well, but they're very tightly
    coupled to the execution machinery, to the point where what they are really
    testing is not a concern of the new `txnwait` package specifically.

commit dbe8b44d5fdd644075716f47feefc62e7ecfb794
Author: Daniel Harrison <[email protected]>
Date:   Mon Oct 30 14:46:40 2017 -0400

    sql: clean up TranslateValueEncodingToSpan
    
    It was previously named translateValueEncodingToKey. This cleans up the
    semantics and documents them more precisely. It also resolves a TODO to make
    the returned encoded keys match the way they'd be for an index. This is
    preparation for the next commit.
    
    TranslateValueEncodingToSpan parses columns (which are a prefix of the columns
    of `idxDesc`) encoded with the "value" encoding and returns the parsed datums.
    It also reencodes them into a key as they would be for `idxDesc` (accounting
    for index dirs, interleaves, subpartitioning, etc).
    
    For a list partitioning, this returned key can be used as a prefix scan to
    select all rows that have the given columns as a prefix (this is true even if
    the list partitioning contains DEFAULT).
    
    Examples of the key returned for a list partitioning:
    - (1, 2) -> /table/index/1/2
    - (1, DEFAULT) -> /table/index/1
    - (DEFAULT, DEFAULT) -> /table/index
    
    For a range partitioning, this returned key can be used as a exclusive end key
    to select all rows strictly less than ones with the given columns as a prefix
    (this is true even if the range partitioning contains MAXVALUE).
    
    Examples of the key returned for a range partitioning:
    - (1, 2) -> /table/index/1/3
    - (1, MAXVALUE) -> /table/index/2
    - (DEFAULT, DEFAULT) -> (/table/index).PrefixEnd()
    
    Also make the error messages more consistent while I'm in here.


commit 3f8be639dcdcc613488547bf240476d9c280a9f2
Author: Jordan Lewis <[email protected]>
Date:   Thu Nov 2 11:20:08 2017 -0400

    sql: use queryRows in forEachUser instead of query
    
    `forEachUser` in information_schema.go is the only use case of
    `planner.query` outside of the planner itself. Migrate it to `queryRows`
    so `query` can be changed more easily.


commit 1772f11d02133244fc91ec65d8d21c1ce9f67dbd
Author: David Taylor <[email protected]>
Date:   Wed Nov 1 15:47:23 2017 +0000

    sqlccl: lock temp storage against concurrent IMPORTs
    
    Import can take hours of processing CSV data before its first write to
    its temp store. Ensuring the temp store is writable before starting the
    operation avoids potentially wasting significant time and cluster
    resources only to discover a typo or permissions issue.
    
    This should also prevent accidental reuse of a temp store by multiple
    IMPORT jobs, which would overwrite eachother's hard-won results.

commit aa545ca2e026c5eac6957dfefaa06ae251e93d24
Author: Tamir Duberstein <[email protected]>
Date:   Wed Oct 25 20:47:26 2017 -0400

    storage: allocate the slice just once

commit c2159607bfc339ae7145a2cedd529e72a8903b53
Author: David Taylor <[email protected]>
Date:   Wed Nov 1 13:57:55 2017 +0000

    sqlccl: use checkpoint to lock out concurrent ops
    
    Accidentially starting concurrent backups pointed at the same location
    would result in them clobbering eachother. While we already bail if a
    finished backup has written a BACKUP file, we still have a relatively
    large window while it is in progress.
    
    This switches to check for both the finished BACKUP file and the
    BACKUP-CHECKPOINT file that may exist while the backup is still running,
    and ensures it *does* exist by writting an empty one immediately after
    checking for it.
    
    This cuts the window during which a concurrent op could start to just
    the check-then-set round-trip delay.
    
    Additionally, it verifies that we can _write_ to the designated location
    *before* sending potentially expensive work out that would ultimately
    be wasted if the target isn't writable.
    
    Finally, when a job is cancelled, delete its CHECKPOINT file, allowing
    the target location to be used by another job.

commit 3afb45e76e59a410f38fdac877e4f9136f03362d
Author: Jordan Lewis <[email protected]>
Date:   Wed Oct 25 12:16:58 2017 -0400

    sql: pre-evaluate arguments to set
    
    Run Eval() on all arguments passed to Set before beginning the Set plan.
    This is necessary because the implementations of all of the different
    variable setters don't get passed the proper EvalContext, and therefore
    won't be able to properly resolve Placeholders once they become leaf
    values.
    
    This wouldn't be necessary if the Set implementations were able to get
    the correct EvalContext, but doing that properly would require a larger
    refactor.


commit 038a369c8b79e37321658edf889e47a302d8a551
Author: Joey Pereira <[email protected]>
Date:   Wed Nov 1 11:52:11 2017 -0400

    sql,parser: Add SCRUB TABLE ... WITH OPTIONS INDEX(idx_name) syntax
    
    This adds the ability to specify the indexes to check, as opposed to
    checking them all.


commit 46ea181ee77cde7b27ba483ca02d4d3ed1344c90
Author: Joey Pereira <[email protected]>
Date:   Wed Oct 25 15:49:55 2017 -0400

    sql,parser: Add SCRUB TABLE ... WITH OPTIONS INDEX ALL syntax
    
    `SCRUB TABLE ... WITH OPTIONS INDEX ALL` is placeholder syntax for the
    SCRUB feature of checking secondary indexes. With no indexes specified,
    this checks all indexes.

commit afc82400f81ed3fbc31859897853317edf1da0ff
Author: Joey Pereira <[email protected]>
Date:   Wed Nov 1 11:18:50 2017 -0400

    sql: Plumb RangeDescriptorCache and LeaseHolderCache to distSQLBackfill
    
    This plumbs through caches to the distSQLReceiver initialized during
    schema change backfilling. These caches are updated during distSQL
    execution.
    
    This will prevent a panic if the spans during distSQL execution
    do not match the planned spans. It is not currently known if this was a
    problem, but panics are happening on another code path where a
    distSQLReceiver is initialized during query execution (on #19586).


commit 078921ebd8c02dd7edb7337e3132685be83a0e92
Author: Alex Robinson <[email protected]>
Date:   Tue Oct 31 19:32:42 2017 -0400

    storage: Fail TestNodeLivenessSelf if getting self's liveness fails
    
    Rather than letting the test continue and segfaulting later.
    
    Should help get insight to fix #19509


commit 156487befca3d37a87be85eb8bfb84d1abd2714c
Author: Richard Wu <[email protected]>
Date:   Tue Oct 31 13:39:53 2017 -0400

    storage: initialize default temp storage max budget for disk stores
    after cli start
    
    Also wrap error message with human-readable prefix.


commit 5004fc83da89f05d12aaacee6fd72ba6a35d0e0a
Author: Richard Wu <[email protected]>
Date:   Tue Oct 31 14:09:55 2017 -0400

    sql: fix merge skew with MultiRowFetcher and makeKVFetcher

commit 9696fac82b45707434612d938d53541cd80d0fa1
Author: David Taylor <[email protected]>
Date:   Fri Oct 27 19:25:32 2017 +0000

    sqlccl: fix RESTORE of INCREMENTAL after dropping a table
    
    This adds tests for the DROP and TRUNCATE cases.
    
    Also, extend the handling of dropped tables during restore:
    
    If a table was dropped between t1 and t2, it isn’t actually required that a backup of the state at time t2 include the span for that table.
    
    However our checking of the span coverage was requiring that all spans in the backup match the desired time bounds, even those that were not required by the requested restore. This changes that enforcement to be limited to required spans.
    
    This error would only occur when attempting to RESTORE that backup or in later attempts to create additional incremental backups on top of it : the initial backup, that stops including the span, would succeed, as it validates its _previous_ backups.
    
    NB: for ’revision_history’ backups, this is still broken — unlike a latest backup, which is free to not include the span of the dropped table, a ‘revision_history’ backup *does* need to include that span,
    to allow time travel to before the DROP. Tracked in #19600.

commit 951b5a2b90fa69c7badcb3055cf7034a620c5bd6
Author: Jordan Lewis <[email protected]>
Date:   Mon Oct 23 15:48:21 2017 -0400

    sql: plumb EvalContext into Datum methods
    
    Most Datum methods get an EvalContext in their method signature. This
    makes it possible to implement Placeholder as a Datum that passes its
    method calls through to the Datum that it points to.

commit 40c539c299cd44efe0e5c94edbaf2328d9409a07
Author: Nathan VanBenschoten <[email protected]>
Date:   Mon Oct 30 16:02:16 2017 -0400

    tscache: move impl-independent types/vars to cache.go
    
    This change moves `cacheValue` and a few related vars to `cache.go`,
    because these will be shared between multiple `Cache` implementations.
    
    It also renames `cacheValue.timestamp` to `cacheValue.ts`.

commit a400e96ce0a37548df5094619075216a4272be88
Author: Matt Jibson <[email protected]>
Date:   Mon Oct 30 15:13:08 2017 -0400

    sql: add some logging for distributed import


commit 4086826e22e2c98e543aadd6598f00c9a3665872
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 30 14:59:01 2017 -0400

    storage: interface *Replica to ReplicaEvalContext
    
    Introduce an interface implemented by `*Replica` and used by
    `ReplicaEvalContext` to prepare the way for moving `ReplicaEvalContext` into a
    subpackage of `storage`.
    
    See #18779.

commit 9eb2ba89da5b29f969bc5a79228b8b84592c8616
Author: Tobias Schottdorf <[email protected]>
Date:   Sun Oct 29 17:29:04 2017 -0400

    server: don't close the stopper unless draining is safe
    
    This fixes a panic that would occur if the RocksDB instance was closed
    with moving parts still trying to use it.


commit 6b25dffff1f21e43e9dabdd175ab468cad99f8a7
Author: Tobias Schottdorf <[email protected]>
Date:   Thu Oct 26 18:34:52 2017 -0400

    server: don't drain partially started server
    
    Fixes #19571.

commit edfe09af7024bb3983166e3e866a99f808f0b92c
Author: Tobias Schottdorf <[email protected]>
Date:   Thu Oct 26 17:57:03 2017 -0400

    server: avoid Fatal when draining early
    
    Fixes #19579.

commit 65a4121fbd1d1f2295f739ed5e7e5b42cedbc68c
Author: Nathan VanBenschoten <[email protected]>
Date:   Wed Oct 11 16:47:59 2017 -0400

    sql: compare NaN like all other values
    
    Previously, we defined all comparisons other than `NaN = NaN` as returning
    false. This change switches this so that `NaN` acts like a normal value that
    is less than all other values. This is needed so that comparison with `NaN`
    is consistent with the ordering of `NaN` in indexes.
    
    NaN now sorts after NULL and before -Inf for both Float and Decimal
    values. This is always how the encoding was ordered, but we now enforce
    this encoding in expression evaluation as well.

commit 558a51c4b0ac60cd7eff5746f032214cd7c237b9
Author: Nathan VanBenschoten <[email protected]>
Date:   Mon Oct 9 21:39:21 2017 -0400

    sql: redefine NaN = NaN -> true
    
    Fixes #18860.
    
    This change switches `NaN` equality semantics from defining `NaN` as not-equal
    to itself to defining it as equal to itself. This is a very sad violation of
    IEEE754 in the name of Postgres compatibility.

commit f43c258bf60300a345e2eace944a21094717c942
Author: Jordan Lewis <[email protected]>
Date:   Mon Oct 30 11:07:04 2017 -0400

    sql/logictest: print sql on all failures
    
    A previous change added sql printing to tests with incorrect output, but
    neglected to do so for all of the other cases, such as expected error
    found none, expected no error found one, etc.
    
    This will help with quickly identifying the source of test failures in
    CI and in local development.

commit eeeacd69ea8ab1ccea807ba916ce9c80074b7452
Author: Andrew Couch <[email protected]>
Date:   Thu Oct 26 16:54:38 2017 -0400

    ui build: rebuild when styles are updated

commit a3866648749f06f7239e773efe6ccd236dbd1b1d
Author: Alex Robinson <[email protected]>
Date:   Sun Oct 29 17:43:15 2017 -0400

    storage: Don't ignore quiesceLocked's return value
    
    This doesn't look like it was a real correctness issue, but it
    certainly seems wasteful to send quiesce messages to all followers
    if the leader didn't actually quiesce itself.

commit 789a787b8bfd2140b1602edfcd05c8e6adcd71ca
Author: Tobias Schottdorf <[email protected]>
Date:   Thu Oct 26 17:56:23 2017 -0400

    server: open most HTTP endpoints later
    
    Extracted from https://github.com/cockroachdb/cockroach/pull/19591
    which also adds a regression test. The tests in that PR pass, but
    to make sure I verified manually using
    
    ```
    while true; do curl -O /dev/null http://localhost:8080/_status/vars; done
    ```
    
    and `./cockroach start --insecure` in another window, ten times.
    
    Fixes #19559.

commit a07cb7cb1cc013ba0563274b62098225c01a10c4
Author: Alex Robinson <[email protected]>
Date:   Fri Oct 27 11:34:39 2017 -0400

    storage: Don't estimate replica's BehindCount if it's not initialized
    
    Might fix #19010, but it's a bit of a shot in the dark.

commit bc719ac868e2439b3de167e5c0ab65dcbf85c0b3
Author: Alex Robinson <[email protected]>
Date:   Thu Oct 26 20:55:17 2017 -0400

    storage: log when a store temporarily disables preemptive snapshots
    
    Also do a minor rename to make the variable/log message clearer.

commit f42ff6968a4aa9a6620a8514fa398f9a1a9f86a5
Author: Jordan Lewis <[email protected]>
Date:   Wed Oct 25 17:46:58 2017 -0400

    sql: fresh planners for queries run by virt tbls
    
    forEachUser, crdb_internal.zones and crdb_internal.jobs previously ran
    their `SELECT` statements using the planner that invoked it. This is
    dangerous, since running plans can modify their associated planners.
    
    Switch to creating a fresh internal planner for the queries, which is the
    pattern that we use almost everywhere else for internal queries.


commit 338ea77fe0990ca222e91b3f63b05cadca7b3a14
Author: Andrei Matei <[email protected]>
Date:   Thu Oct 19 15:59:52 2017 -0400

    kv: rationalize the test setupMultipleRanges() interface


commit 9394a49a0a0082f715e8a2dff59168ab72d2d125
Author: Matt Jibson <[email protected]>
Date:   Thu Oct 26 03:24:09 2017 -0400

    ccl/storageccl: retry s3 region fetch
    
    Retry an S3 operation that has been causing transient errors. If a
    503 is returned, we may need to slow down our request rate. Although
    the retry package does wait a bit, it is only some 10s of ms. We
    instead wait 5s on these errors to attempt to slightly decrease the
    request rate. This 5s is purely a guess, but it is also a low enough
    time that it won't matter too much if we are waiting longer than
    minimally needed.
    
    For now put this only in the place we have seen the failure. We can
    optionally retry the other locations if we see problems there. This
    is the same approach we used for Azure and GCE with good results.
    
    Fixes #19502


commit 15c6351082c7ab237b67791a486abf7fbe88f02f
Author: Richard Wu <[email protected]>
Date:   Thu Oct 12 17:38:34 2017 -0400

    sql: introduce MultiRowFetcher for one-pass row fetches for multiple
    (interleaved) tables
    
    This PR introduces `MultiRowFetcher` which generalizees `RowFetcher` to
    respond to and return rows from multiple tables from the same set of
    spans.
    
    `MultiRowFetcher` is initialized with a slice of `MultiRowFetcherArgs`
    which encapsulates the arguments for one table in `RowFetcher`.
    
    One key point is that the `reverse bool` argument that exists in
    `RowFetcher` can only be enabled on a per-fetcher basis: one cannot scan
    in arbitrary directions on a per-table basis (since row kvfetcher works
    by scanning KV pairs sequentially).
    
    `MultiRowFetcher` is necessary in order to fetch interleaved tables
    efficiently in one-pass. See the RFC for interleaved tables here
    https://github.com/cockroachdb/cockroach/pull/19028.


commit 427c90c615b3ce3b7f77e99990acdc51ccb688e0
Author: Nikhil Benesch <[email protected]>
Date:   Thu Oct 26 13:11:08 2017 -0400

    build: don't build snappy's unit tests
    
    Since v1.17, Snappy's default make target will build its unit tests,
    which depend on the Google Test library. Avoid that dependency by
    building only snappy and not its unit tests.


commit 5a729eed13b2abe492561470548f44f6ffd5e9c3
Author: Jordan Lewis <[email protected]>
Date:   Thu Oct 26 02:13:49 2017 -0400

    sql: disambiguate ANY/ALL/SOME exprs in formatting
    
    Previously, ANY/ALL/SOME expressions would be formatted without
    parentheses around their antecedents. This could lead to invalid input.
    For example, the following query doesn't parse properly since the type
    annotation binds to the comparison expression, not the array:
    
    ```
    SELECT 1 = ANY ARRAY[1]:::INT[]
    ```
    
    This patch adds the disambiguating enclosing parenthesis.

commit 99b3492742c85bc193d2b64f54e614e3e1984c07
Author: Arjun Narayan <[email protected]>
Date:   Thu Oct 26 10:45:34 2017 -0400

    sql: remove traceKV from RowFetcher.NextRowDecoded
    
    This is unused in the function, and is non-idiomatic since traceKV is
    set when initializing RowFetcher.

commit b45cce4020934d42160ea5acef9f8b3421795129
Author: David Taylor <[email protected]>
Date:   Thu Oct 26 02:05:12 2017 +0000

    settings: avoid multiple *settings.Values at runtime
    
    This eliminates all but one non-testing call to MakeClusterSettings -- which is critical to correctness.
    Any *settings.Values other than the one maintained by the refresh loop will be stale/lies, so only in
    testing do we want call sites creating settings value containers.
    
    Fixes #19533.

commit c86188aa1c0cc66923f603eb670bcfcf900c92f6
Author: Ben Darnell <[email protected]>
Date:   Wed Oct 25 17:14:51 2017 -0400

    storage: Refactor raft message send loop into a new method

commit 3858f196314c21a28e79abbd8cf80c2197a9ae6f
Author: Nathan VanBenschoten <[email protected]>
Date:   Thu Oct 19 01:19:48 2017 -0400

    util/interval: bucket multiple Ranges in single rangeList node
    
    Previously, the linked list implementation version of `RangeGroup` would
    store a single `Range` in each list `Element`. This was pretty heavy on
    allocations because it required a new list `Element` every time we
    inserted a `Range` that couldn't be merged with existing ranges. Is also
    required us to compare against every `Range` in the group when searching,
    which was a big deal because searching for an overlapping `Range` is the
    first step of almost all `RangeGroup` operations.
    
    The implementation now buckets a constant number of `Ranges` in each
    linked list node. This has two benefits. First, it reduces allocations
    by a factor of `rangeListNodeBucketSize` because each list node can now
    store up to `rangeListNodeBucketSize` ranges. Second, it allows us to
    scan through the `rangeList` faster when searching for a certain range
    by comparing only against the largest `Range` in each node as we scan
    forward, until we find one that is larger than our desired `Range`. This
    technique is similar in spirit to a two-layer skip list.
    
    At the time of testing, a `rangeListNodeBucketSize` of `8` seemed to strike
    the best balance between fewer list nodes and larger list nodes based on
    benchmarking results.
    
    ```
    name         old time/op    new time/op    delta
    RangeList-4    1.30µs ± 3%    0.84µs ± 2%  -35.11%  (p=0.000 n=9+10)
    
    name         old alloc/op   new alloc/op   delta
    RangeList-4      444B ± 0%       12B ± 0%  -97.30%  (p=0.000 n=10+10)
    
    name         old allocs/op  new allocs/op  delta
    RangeList-4      21.0 ± 0%      12.0 ± 0%  -42.86%  (p=0.000 n=10+10)
    ```


commit 8806f7337e8c28b2aa2573dae3aa2544d8fa1219
Author: Nathan VanBenschoten <[email protected]>
Date:   Thu Oct 19 01:06:32 2017 -0400

    util/interval: use subtests for each RangeGroup impl

commit 55e082c6dd3575753e7d8a6d9388a129c1aee9cc
Author: Daniel Harrison <[email protected]>
Date:   Thu Oct 19 14:39:06 2017 -0400

    sql: support DEFAULT and MAXVALUE in PARTITION definitions
    
    DEFAULT is used in a list partitioning as a catchall for values not
    matched by any other partition. To avoid ambiguity in a multi-column
    partitioning, when DEFAULT is specified for a column, it must be
    specified for every column "after" that. So `(1, DEFAULT, DEFAULT)` is
    valid but (1, DEFAULT, 1) is not.
    
    MAXVALUE is used in a range partitioning as a sentinel that sorts larger
    than any data in the column. It has the same ambiguity problem as
    DEFAULT and so the same restriction.

commit 6a49d1cfdc3b3da88f28bf6644508b851f49a214
Author: Joey Pereira <[email protected]>
Date:   Wed Oct 25 11:18:59 2017 -0400

    server,sql: Refactor distSQLPlanner to only be initialized once
    
    This moves distSQLPlanner initialization out from the
    SchemaChangeManager and Executor to instead use a shared instance
    initialized in Server.Start.


commit b37e66b1d13c67e6bd2a3994b779ab684198b61f
Author: Joey Pereira <[email protected]>
Date:   Wed Oct 25 13:18:45 2017 -0400

    sql: Unexport methods on DistSQLPlanner that don't escape pkg/sql
    
    These exported methods are used outside of the file they're defined, but
    not out of the packaged.

commit e3bb332df50dc275e7cc11ede8d5411a678f6f74
Author: Joey Pereira <[email protected]>
Date:   Wed Oct 25 11:53:36 2017 -0400

    sql: Change DistSQLPlanner to be exported
    
    This will let us initialize a single DistSQLPlanner in pkg/server.

commit 1358e64e5e45c542636c6bad0545e89a845e3f1a
Author: Nathan VanBenschoten <[email protected]>
Date:   Mon Oct 23 23:22:47 2017 -0400

    storage: create RangeMatchingPred function, use for tscache accesses
    
    This change specializes the `keys.Range` functions that `Replica` uses
    to determine the influence spans when adding and expanding
    `tscache.Requests`. It restricts the `Range` function to
    `roachpb.Requests` that update the `TimestampCache` when adding a
    request and restricts the `Range` function to `roachpb.Requests` that
    consult the `TimestampCache` for expanding requests. This means that
    fewer unnecessary `tscache.Requests` will be expanded because unrelated
    `roachpb.Requests` in a `BatchRequest` will no longer expand the
    influence spans passed to the `tscache` methods.
    
    This wasn't inspired by any particular workload, but it seemed like the
    right thing to do. I also think the change makes purpose of the
    `RSpan` passed to `AddRequest` and `ExpandRequests` more self-explanatory.


commit 636ec6bac08232327c4a55a02496b18f8113c01e
Author: Tobias Schottdorf <[email protected]>
Date:   Tue Oct 24 17:49:36 2017 +0200

    server: rework Gossip bootstrap sequence
    
    A long time ago, in #5441, we changed the server startup sequence so that we
    would only open the main listener at the very end. The impetus was that by
    opening the listener early, various services could receive requests when they
    weren't actually ready, which resulted in unexpected errors during tests.
    
    What we missed at the time was that this wasn't actually a good idea because it
    also robs the node of the ability to accept incoming Gossip connections. As a
    result, if a node didn't manage to talk to Gossip peers (perhaps it was started
    without a join flag and had nothing reasonable persisted), it would never
    successfully connect to Gossip and would never leave `(*Node).start`.
    
    These deadlocks added to the impact of some of our other insufficiently
    streamlined aspects of the startup sequence, namely the choice of `--join`
    flags, and were generally hard to debug especially since the HTTP endpoints, and
    thus many debugging facilities, weren't available.
    
    This commit reworks the sequence so that
    
    - the HTTP endpoint opens early (so that the debug endpoints are accessible)
    - the main cmux listener opens early
        - but opening the Postgres sublistener is delayed
        - and all non-Gossip and non-Ping gRPC traffic fails retryably.
    
    @bdarnell points out:
    
        Some more important context is that in 1.0, the server didn't open its port
        until it was ready to serve, so TCP-level health checks were sufficient. With
        the introduction of the init command in 1.1, a not-yet-initialized server will
        open its port early (which means that higher-level health checks are required
        for determining readiness). Now that we do that, we might as well also have
        initialized-but-not-connected servers do the same.
    
    Fixes #18027.
    Fixes #13027.
    Fixes #19427.

commit 62d7a2f1c54e2e1ed1cf37057c1bf260465b2272
Author: a6802739 <[email protected]>
Date:   Mon Sep 11 20:15:58 2017 +0800

    storage : disable stats-based rebalancing when we use SCATTER


commit ed89fe344c11b5db8ef9648b2dd1e3b2ab5c651f
Author: Nikhil Benesch <[email protected]>
Date:   Wed Oct 25 10:52:49 2017 -0400

    build: reinstall pkg/cmd/FOO when FOO's source files change
    
    Make "knows" to reinstall Go binaries provided by dependencies by
    rebuilding them whenever Gopkg.lock changes. (This is technically too
    aggressive, but works well enough.) The rule, however, was not smart
    enough to determine when a command in pkg/cmd was updated. Teach it to
    consider every file beneath pkg/cmd/FOO as a prerequisite of FOO so that
    FOO is reinstalled whenever its sources change.


commit 291b7b136bf3df64a6b24147df6d2418babb27d0
Author: Peter Mattis <[email protected]>
Date:   Wed Oct 25 08:46:57 2017 -0400

    storage/engine/enginepb: convert TxnMeta to proto3
    
    This shaves 7% off the encoded size of transactions sent on the wire.

commit 7c26c17f1256ec04019677f3e3b01853c6df6b67
Author: Peter Mattis <[email protected]>
Date:   Fri Sep 29 16:43:16 2017 -0400

    roachpb: convert api.proto and data.proto to proto3
    
    This shaves 6% off the encoded size of `Batch{Request,Response}` for a
    simple scan operation.
    
    See #18919

commit b05b7c64e51ee4e823296dbb4dd33bdcce8f2d5d
Author: Peter Mattis <[email protected]>
Date:   Tue Oct 24 19:08:04 2017 -0400

    util/contextutil: silence overly verbose log message
    
    Logging "canceling context" every time a context is cancelled is overly
    verbose and a measurable performance hit. Only wrap the cancellation
    closure if log.V(1) or higher is specified.

commit 14b34d2a2a655c630c9abf89460dd27a4a21490b
Author: Nikhil Benesch <[email protected]>
Date:   Tue Oct 24 15:34:50 2017 -0400

    build: introduce maintainer-clean and unsafe-clean targets
    
    Teach our build system to better handle cleaning with the introduction
    of three new cleaning targets. The distinctions between `clean`,
    `maintainer-clean`, and `unsafe-clean` are as follows:
    
      * clean: remove all easily-regenerated files, where an
        "easily-regenerated" file only requires the build dependencies
        listed on the user-facing "Build from Source" install instructions.
        At present, that's a C++ compiler, a Go compiler, Bash, CMake, and
        autoconf. The cockroach binary, log files, C/C++ dependencies, Go
        compilation artifacts, and generated protobuf code are all fair
        game.
    
      * maintainer-clean: like clean, but also remove hard-to-regenerate
        files. At present, this means the UI build artifacts, which require
        Node and Yarn to regenerate.
    
      * unsafe-clean: like maintainer-clean, but also scrub the repository
        and its submodules of any untracked or ignored files.


commit c01ce3907ab9cbbea338f005d2fba98ec862a98f
Author: Andrew Couch <[email protected]>
Date:   Mon Oct 23 15:34:15 2017 -0400

    ui: statically define format verbs in SummaryBar

commit d7b794219aac6483148b7ff534a61648e5a54810
Author: Joey Pereira <[email protected]>
Date:   Tue Oct 17 22:23:07 2017 -0400

    sql: Add background renewal for table descriptor leases
    
    This commit changes TableDescriptor lease acquisition to acquire a new
    lease if the current lease is about to expire. This prevents routines
    from blocking when there are no valid leases on a frequently accessed
    table.
    
    If renewal continously fails, the error will become user-facing if no
    lease is acquired yet by the time a lease is expired.
    
    Closes #17227.

commit 11fc68c69c8aa8785260fc1a0e62e19e164017a3
Author: Raphael 'kena' Poss <[email protected]>
Date:   Tue Oct 24 21:13:14 2017 +0200

    cli/sql: bump the go-libedit dependency to fix signal handling on darwin
    
    The details of the story can be found here:
    
    https://github.com/knz/go-libedit/commit/5ffa3044c44237e9fc907fd3fc17e4cf604a2f12
    
    (previous attempt:
    https://github.com/knz/go-libedit/commit/3b0f69cb0f672aad6a49ee224c3a801ae576a6e9)
    
    This patch also tweaks the handling of the prompts, since go-libedit
    1.5 doesn't do Go callbacks any more.


commit 36422ee65f58ac0637b6d8acdf54e3d685b7ba6c
Author: Nikhil Benesch <[email protected]>
Date:   Tue Oct 24 01:44:39 2017 -0400

    config,keys,sqlbase: move MakeZoneKey[Prefix] from sqlbase to config
    
    Logically, encoding zone keys belongs in the config package.
    Practically, partitioning requires that the config package contain the
    logic to encode zone keys to avoid a circular dependency. (Finding split
    keys now requires looking at the zone config to determine whether a
    table partition exists, rather than just looking at the structure of the
    key to determine if it spans a table boundary.)

commit b1fee718266a5764ba92633983dd496504843fd4
Author: Nikhil Benesch <[email protected]>
Date:   Tue Oct 24 01:27:46 2017 -0400

    config: split config.go into multiple files
    
    ZoneConfigs and the SystemConfig are different beasts; give them each
    their own file. Split out key encoding and decoding routines into their
    own file, too, which will soon be responsible for constructing keys in
    system.zones.


commit ac0d0939a6b919cd9846739710c8661bede7087f
Author: Joey Pereira <[email protected]>
Date:   Tue Oct 24 12:02:27 2017 -0400

    sql: Fix schema change event logging
    
    Previously, the schema change completion message had repetition. Also a
    new message for the completion of a schema change rollback has also
    been added, as there was no event indicating when it completed.

commit 1fc1502ddd7d29644b1a9dc6caed7e92d4eb27c6
Author: Nikhil Benesch <[email protected]>
Date:   Tue Oct 24 12:30:39 2017 -0400

    ui: require Yarn 1.0+
    
    99b35b3 accidentally introduced a dependency on Yarn 1.0+ with its usage
    of `yarn --cwd`. Since Yarn 1.0 has been available since early
    September, it seems reasonable to mandate its usage.

commit 2fbfa1777747a123db401a8df4cf57ca6b7fdaff
Author: Alex Robinson <[email protected]>
Date:   Tue Oct 24 10:45:44 2017 -0400

    storage: Switch an occasionally spammy log.Infof to log.VEventf
    
    These were never meant to be log.Infof -- I must have removed the
    log.V(2) wrapper from around them and forgetten to switch them to
    log.VEventf. This can create a bunch of log message when a cluster
    becomes unbalanced, such as when a new node joins.

commit d320e3691aac10f5c88e284f3043c8ce5351694f
Author: Dmitry Saveliev <[email protected]>
Date:   Mon Oct 23 22:04:44 2017 +0400

    sql: add missing columns to information_schema.table_constraints

commit de916e1348e8e979c193e5c6a14d364c4ba19634
Author: Matt Tracy <[email protected]>
Date:   Sun Oct 22 23:45:44 2017 -0400

    ui: Add Query Manager Reducer
    
    Adds the Query Manager Reducer, which is designed to to hold metadata
    state for asynchronous queries. For each query, this state includes:
    + Whether the query is currently running.
    + If the last attempt to run the query encountered an error.
    + The time at which the query was last attempted.
    
    Currently, this metadata is being stored in the Admin UI right alongside
    the data itself, and each query is responsible for managing its own
    execution and metadata. This commit does not change this, but creates a
    reducer that will be used to store it after further refactoring - the
    next step will be to create a Saga that executes queries, using this
    reducer to store query metadata.

commit 928ec15149d529cc1cbbcde3519eb2fae43781d4
Author: Nathan VanBenschoten <[email protected]>
Date:   Fri Oct 20 00:02:45 2017 -0400

    tscache: Add *tscache.Request to sync.Pool
    
    This change reduces GC time (shown using GCTrace on my linux machine)
    down from consistently **3%** to consistently **%2%** on a read-only
    workload (`kv --read-percent 100 --concurrency 16 --splits 100 --write-seq 100000`)
    
    This commit, in-conjunction with the previous one, also has a big impact on `kv`
    performance as a whole. The following are the results from two 1 minute `kv` runs
    before and after the changes:
    
    Before:
    ```
    _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
       60.0s        0        1985392        33089.4      0.5      0.4      0.9      2.6     30.4
    
    BenchmarkLoadgenKV/readPercent=100/splits=100/concurrency=16/duration=1m0s       1985395             30221 ns/op
    
    _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
       60.0s        0        1979390        32989.3      0.5      0.4      0.9      2.8     27.3
    
    BenchmarkLoadgenKV/readPercent=100/splits=100/concurrency=16/duration=1m0s       1979390             30312 ns/op
    ```
    
    After:
    ```
    _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
       60.0s        0        2121080        35350.8      0.5      0.4      0.9      1.8     23.1
    
    BenchmarkLoadgenKV/readPercent=100/splits=100/concurrency=16/duration=1m0s       2121080             28287 ns/op
    
    _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
       60.0s        0        2128153        35468.7      0.5      0.4      0.8      1.8     27.3
    
    BenchmarkLoadgenKV/readPercent=100/splits=100/concurrency=16/duration=1m0s       2128153             28193 ns/op
    ```

commit a30f4f8e6dfe51252b355aa8aee1d94f5f3702c2
Author: Nathan VanBenschoten <[email protected]>
Date:   Thu Oct 19 23:51:47 2017 -0400

    storage: clone all Spans before adding them to tscache.Request
    
    This change reduces GC time (shown using GCTrace on my linux machine)
    down from consistently **7%** to consistently **%3%** on a read-only
    workload (`kv --read-percent 100 --concurrency 16 --splits 100 --write-seq 100000`)
    
    Something strange is going on though. If I individually deep copy the `Spans` by
    adding a `Clone` method to `roachpb.Span`, I don't see the same effect. I
    originally thought that the old GC time was due to scanning the `BatchRequest`
    and `endCmds` structs, which were for some reason being referenced and therefore
    unable to be GCed. This proves that that was not the case.
    
    This leads me to believe that the GC cost reduction is due to the single slab of
    memory being allocated for all keys in a `BatchRequest`. This theory is fairly
    unsubstantiated though. I'm curious if anyone has any other ideas as to why we're
    seeing such a big difference.

commit 8d8a7d58cf5b90940c9e9d18904cc931ad2fb985
Author: Nikhil Benesch <[email protected]>
Date:   Mon Oct 23 23:09:44 2017 -0400

    deps: constrain go-ole to master to fix Windows builds
    
    The last release of go-ole, v1.2, was two years ago. go-sigar, via
    StackExchange/wmi, understandably depends on features added in the last
    two years when compiling for Windows.
    
    Fix the Windows build by forcing `dep` to vendor go-ole's master branch
    instead of v1.2.

commit eb79d4b45d451d9efdee3fe0b8fb9309d8351b51
Author: Nikhil Benesch <[email protected]>
Date:   Mon Oct 23 18:11:11 2017 -0400

    ui: prevent Node from infecting Docker's stdout with non-blocking I/O
    
    When NodeJS and Docker are used together, NodeJS manages to infect the
    container's stdout and stderr streams with non-blocking I/O mode. All
    future processes launched in the same shell inherit non-blocking mode,
    but few can handle it. Notably, `go test` drops output on the floor when
    stdout is non-blocking, and `go-test-teamcity` interprets this as a
    panic.
    
    Work around this by piping all NodeJS commands started by Make through `cat`.
    
    Incidentally closes #18378 by removing usage of `yarn run`.

commit 66baf92f6a94ee02fe44cf9ed117b1af8c3a72c8
Author: Nikhil Benesch <[email protected]>
Date:   Thu Sep 7 23:29:43 2017 -0400

    ui: require proxy to be launched via `make watch`
    
    For speed, the proxy server no longer notices changes that affect the
    vendor bundle. This can lead to hard-to-debug errors if the vendor
    bundle is out-of-date when the proxy server is launched. Solve these
    errors by requiring the proxy server to be launched via a phony Make
    target, `make watch`, which can guarantee the vendor target is built
    before launching the proxy server.

commit 99b35b340a0cd3100ef2bbea7d1a829b3d69f47e
Author: Nikhil Benesch <[email protected]>
Date:   Thu Sep 7 17:16:41 2017 -0400

    ui: remove generated files from version control
    
    Checking in src/js/protos.js and embedded.go causes several major
    problems:
    
      * Every change to the UI, even if the diff is only one line,
        entirely rewrites embedded.go, which is upwards of 12MB in size.
        This adds a lot of weight to the Git repository.
    
      * Every commit that touches the UI or protobuf definitions must be
        serialized, as diffs in protos.js and embedded.go cannot be
        automatically merged.
    
      * Including embedded.go in a pull request breaks GitHub's "request
        review" feature.
    
    Removing these files from version control solves all these problems with
    only two downsides:
    
      * Every developer will need Node and Yarn installed. This seems
        reasonable to expect of our engineers and external contributors, but
        not end users. This commit teaches our source archives to include a
        pre-generated embedded.go so that end users who simply want to build
        from source without Node and Yarn can do so.
    
      * Developers will likely need to rebuild the UI a few times per day
        as they switch between branches that span UI changes. This commit
        brings the UI compilation time down to about 5s when only
        first-party sources have been updated and 30s when UI dependencies
        are bumped (infrequently), so this also seems reasonable.

commit 4ac40a1f600c6218fdc72372c86b12f5c05add8a
Author: Spencer Kimball <[email protected]>
Date:   Mon Oct 23 17:04:04 2017 -0400

    storage: avoid unnecessary clock updates and access to clock mutex
    
    Because `Clock.Update` already has code in it to verify that the clock isn't
    being updated to a time more recent than the current physical wall clock time
    plus the max offset, add a new method, `Clock.UpdateAndCheckMaxOffset` to
    combine the update and check into a single lock.
    
    Also, in `Store.Send`, only update the clock in the event that the
    `BatchResponse.Timestamp` has been forwarded from `BatchRequest.Timestamp`.
    
    For single node requests generated via loadgen/kv and resent to
    Store.Send, this improves performance by ~7.5%.

commit 206042491d9782ae5a234e02459b23d2e71c7d8d
Author: David Taylor <[email protected]>
Date:   Mon Oct 23 20:27:18 2017 +0000

    storageccl: add a little detail to debug logs/errors
    
    trying to track down #17819

commit 88c474f369413381902a51805997dc59410e3b74
Author: Spencer Kimball <[email protected]>
Date:   Mon Oct 23 15:14:24 2017 -0400

    roachpb: provide a fast path for updating observed timestamps
    
    For single node requests generated via `loadgen/kv` and resent to
    `Store.Send`, this improves performance by ~5%.

commit e3e500262289f844a09cef34c3944e8f7f5b1589
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 23 21:42:46 2017 +0200

    log: special-case Wrap(safe, "foo")
    
    Extend the previous special case to allow reporting a wrapped
    error verbatim if it is the only argument.
    
    This should allow the changes in #19447 to *actually* report the wrapped
    error including its type (before the string would be reported, but losing
    the wrapped type).

commit 7990d77ea8314398b307e150d482b22dbd7254be
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 23 21:30:07 2017 +0200

    storage: move StateLoader to own package

commit 575e7e96eb734385dd034465b5b64347ff5f3723
Author: marc <[email protected]>
Date:   Mon Oct 23 09:42:30 2017 -0400

    scripts: change gce_worker project to `cockroach-workers`.
    
    This is to centralize workers a bit better.

commit 2c4f8ea8221ee6b79f4dc3ee23d27add3acc1736
Author: Matt Jibson <[email protected]>
Date:   Mon Oct 23 14:57:16 2017 -0400

    contextutil: add cancellation info log
    
    We are currently attempting to track down various errors with context
    cancelled messages (see #18652, #15332, #19199). In order to assist
    in this debugging, add a contextutil package that wraps calls to
    cancel with an info Log that prints the line that caused the cancel
    to execute.
    
    For now only selectively add this to suspect locations. Adding it to
    all cancel funcs produces a huge amount of log spam.

commit 69ed0584428125375424d0e4b3d1ef536b81e098
Author: Spencer Kimball <[email protected]>
Date:   Wed Oct 11 21:23:23 2017 -0400

    kv: implement dist sender optimizations
    
    - Fast path at start of `divideAndSendBatchToRanges` for case where
      all the key span is contained within a single range.
    - Avoid batch truncation in fast path in `sendPartialBatch`.
    - Fast path in `grpcTransport` for case of a single replica - send
      syncrhonously instead of via goroutine.
    - Remove `DistSender.sendRPC`'s `context.WithCancel` as cancellation should
      properly be done in the transport, which is the component which understands
      if cancellation is necessary. gRPCTransport already has a cancellation
      mechanism, so we were doing this twice unnecessarily. In the case of the
      dist sender fast path (added to this commit) which prevents launching a
      goroutine in the event of a single replica, this means the transport can
      avoid starting any cancelable context.
    - Remove unnecessary transaction clone from grpcTransport for local
      server case because after code audit, determined that there are no
      cases where `ba.Txn` is modified at `Store.Send` and below without
      first doing a similar shallow copy of `ba.Txn`.
    
    I removed the shallow clone of the batch's transaction.  This is no
    longer needed, based on a code audit. There is nowhere in the Store ->
    Replica -> Raft evaluation path where we modify the batch's Txn
    without first making a shallow copy of it.

commit 5a80844e9f67573ed099c0be8232c7f32ecab4ac
Author: Daniel Harrison <[email protected]>
Date:   Mon Sep 18 11:30:38 2017 -0400

    sqlccl: implement RESTORE ... AS OF SYSTEM TIME
    
    This requires two pieces: an option to Export that saves every mvcc
    value instead of just the latest and an option to Import that selects
    the latest version of each key with an mvcc timestamp not greater than
    some EndTime.
    
    BACKUP is given an 'mvcc_all' option that triggers the export of all
    mvcc versions of each key, instead of just the latest. The default is
    left as latest since this is expected to be the common case.
    
    RESTORE ... AS OF SYSTEM TIME then can restore to times in the middle of
    BACKUPs made with the mvcc_all option. Not every BACKUP in the RESTORE
    needs the option, only the one covering the relevant timestamp (e.g. a
    full backup over (t0..t2] without mvcc_all and an incremental backup
    over (t2, t4] with mvcc_all can restore to any time in [t2, t4].
    
    If a node with this change sends an Import or Export request to an old
    node, it will ignore the new field and do the old behavior, so using the
    mvcc_all option to BACKUP or RESTORE ... AS OF SYSTEM TIME will require
    all nodes to be upgraded.
    
    A check to skip export of keys that are deleted before the selected
    timestamp in a full backup is moved from MVCCIncrementalIterator to
    Export where it fits more naturally.

commit 471fc16d20675ec5229084b1559228ccab46a25a
Author: Joey Pereira <[email protected]>
Date:   Tue Oct 17 23:11:43 2017 -0400

    sql: Add benchmark for LeaseManager.AcquireByName with cached leases.
    
    LeaseManager.AcquireByName is a critical path for statements and
    generally when accessing the table descriptor. The case when the lease
    is cached serves the majority of operations.


commit 9a027ae8c0ab51467a61d92281e09aea86dc6c84
Author: Daniel Harrison <[email protected]>
Date:   Mon Sep 18 11:01:14 2017 -0400

    engineccl: implement (*MVCCIncrementalIterator).Next

commit 27491f66ffcd51013b012855598d64a6e2498c47
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 23 17:37:19 2017 +0200

    storage: return error from split/merge lock acquisition
    
    Also report the error to sentry because we suspect it to have caused
    the bug referenced below in versions of CockroachDB not running with
    this commit.
    
    See #19172.

commit 42d01ac41adcf50ce88bd4cc2ec80de8b11fe0ab
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 23 18:00:51 2017 +0200

    log: export SendCrashReport

commit 5f55ed65e76499fa75b0ba3dcecd1f104b7bbf1f
Author: Justin Jaffray <[email protected]>
Date:   Fri Oct 20 13:36:48 2017 -0400

    sql: fix repeated escapes in array parsing
    
    Previously, we wouldn't parse properly arrays containing strings with
    multiple escapes in a row. This commit fixes that.
    
    Also added a random generator, which catches the errors we had before.
    
    Going to cherry-pick for 1.1.2.

commit e1d951080dede3a4a96a919df4d17713064757e0
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 23 17:54:39 2017 +0200

    storage: report failed raft command application
    
    Previously, if `applyRaftCommand` returned an error, it would mark the replica
    as corrupt but then go on and execute the side effects and potentially the
    assertions in `assertState()`. These were then likely to fail and return a
    misleading error, as likely seen in #16004.
    
    Instead, cause a fatal error right when observing the error, and potentially
    capture the root cause on sentry.io. The (perhaps too optimisic) expectation
    is that after accounting these disk corruption/space errors, there will be
    much fewer (possibly no) reports triggered by `assertState()`.
    
    Touches #16004.

commit d0ea531f5f159f65141b8d2eab09493b925f0ea9
Author: Joey Pereira <[email protected]>
Date:   Mon Oct 23 10:47:59 2017 -0400

    util: Replace errors with pgerror for IPAddr parsing functions
    
    Previously, helper functions for parsing IPAddr datums did not generate
    pgerrors.


commit 51dbeb78ba5ef9055914d54087108455c5d50be9
Author: Vivek Menezes <[email protected]>
Date:   Mon Oct 23 07:26:25 2017 -0400

    sql: fix TRUNCATE with self-referential FKs
    
    fixes #19322

commit 1c6c3f1503b6aa2fe924be818ddd2afcb50fc05a
Author: Tobias Schottdorf <[email protected]>
Date:   Mon Oct 23 13:55:30 2017 +0200

    storage: factor out replicaStateLoader
    
    This is one of the self-contained threads to unravel for #18784.
    
    The resulting code allows the `StateLoader` struct to be moved into
    its own package (but since moves rot quickly, I'm going to merge this
    first and then execute the move separately).

commit c7827a50627dd864c9195d7b83b5264593fb655a
Author: Andrei Matei <[email protected]>
Date:   Sun Apr 9 15:34:33 2017 -0700

    rpc: unexport SourceAddr

commit f161feb75cba8f7d54a7cdf6d7b84a960341a010
Author: Tamir Duberstein <[email protected]>
Date:   Tue Aug 22 17:02:34 2017 -0400

    rpc: remove workaround
    
    This issue was fixed in https://github.com/golang/go/commit/bf0f692
    which is in go1.9.

commit 73aff1dcb43bb96bdbb275e28ebd9a1cb7228f77
Author: Tamir Duberstein <[email protected]>
Date:   Sat Oct 21 20:13:41 2017 -0400

    server: use net.Pipe instead of TCP HTTP<->gRPC connections
    
    Fixes #10374.
    Closes #17145.

commit a9d85a85324bc6bc19875bd8574e7b483ab7e1bb
Author: Tamir Duberstein <[email protected]>
Date:   Sat Oct 21 20:44:20 2017 -0400

    server: return err instead of nil
    
    Narrow error scopes while I'm here (that's how I noticed this).

commit b507634b7206c4465017805d0678d2c4436b3499
Author: Tamir Duberstein <[email protected]>
Date:   Tue Sep 19 11:19:44 2017 -0400

    build: move unconvert into megacheck
    
    This also checks tests, which were previously not covered.

commit 681c6fcb574a8e96ef01697f8e08e13f795e5e11
Author: Tamir Duberstein <[email protected]>
Date:   Tue Sep 19 12:05:26 2017 -0400

    build: use more idiomatic Init style

commit b7be72798a4cc28b9c1200a9fbe9053aa734ed8b
Author: Tamir Duberstein <[email protected]>
Date:   Tue Sep 19 09:39:45 2017 -0400

    build: move metacheck into TestStyle/Megacheck

commit 44eae5090c2d8d03911253fd2c2453c4bcc2f254
Author: Joey Pereira <[email protected]>
Date:   Thu Oct 19 12:22:22 2017 -0400

    sql: remove unnecessary call to SetSystemConfig in index truncation
    
    It was originally needed because the ResumeSpan was added to the
    TableDescriptor. With the move of ResumeSpan to the Jobs table, it is
    no longer needed.

commit 73d2743d6270aa7cb215e2946de27debd3d246d9
Author: Tobias Schottdorf <[email protected]>
Date:   Fri Oct 20 17:00:05 2017 +0200

    storage: de-flake TestNodeLivenessGetIsLiveMap
    
    Heartbeat() may be return ErrEpochIncremented.



commit 22661596176605a4644e81e82ef58a895d03c8ea
Author: Alex Robinson <[email protected]>
Date:   Wed Oct 18 13:13:42 2017 -0400

    storage: Add trace event when split queue can't find valid split key
    
    To make it possible to debug issues like #19296 without having to build
    a custom binary.

commit eb82358b6d07eff619926cca46d5af5dfd20c77a
Author: Alex Robinson <[email protected]>
Date:   Wed Oct 18 11:59:54 2017 -0400

    engine: Fix splitting to work on ranges with a very large first row
    
    Previously, if the target split point was contained within the first SQL
    row of a range, MVCCFindSplitKey would return a key within that first
    row, which would prevent a split from happening because we only split on
    row boundaries (and don't want to split on the first key in a range).
    Now, we'll continue iterating until we find the first key that isn't
    part of the first row in such ranges.
    
    Fixes #19296

commit 5690ad1d729be8c0da9b31c57d474c142439d236
Author: Tobias Schottdorf <[email protected]>
Date:   Fri Oct 20 18:05:28 2017 +0200

    storage: improve TestPushTxnQueueCancel
    
    I was unable to reproduce
    
    https://github.com/cockroachdb/cockroach/issues/19333
    
    in >25k iterations, but I suspect an error was returned and the
    test never got to check it. With these changes, the next time it
    flakes we will have more intel.
    
    Closes #19333.

commit 09de30e8b0e06dbb3c2ab46ab29f5d4e21c4c825
Author: Tobias Schottdorf <[email protected]>
Date:   Fri Oct 20 13:35:55 2017 +0200

    storage: factor out and rename abort *span*
    
    This is a prelude to #18784.

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/c02f27a67d9c106a0568c0e87e6563e88c4c8106/

[nvanbenschoten]

build change: Fix compilation on the 64-bit ARM architecture.

sql change/backwards-incompatible change: redefine NaN comparisons to be compatible with PostgreSQL. NaN is now equal to itself and sorts before all other non-NULL values.

[a-robinson]

Bug fix: Fix range splitting to work when the first row of a range is larger than half the configured range size. #19339

Bug fix: Remove a spammy log message that would get repeated a lot when a cluster is unbalanced, such as just after a new node joins. #19491

[benesch]

Build change: NodeJS 6+ and Yarn 1.0+ are now required to build CockroachDB.

(All the rest of my commits do not have release notes.)

[bdarnell]

LGTM (no release notes for my commit in the undocumented list)

---

Review status: 0 of 3 files reviewed at latest revision, all discussions resolved, some commit checks failed.

---

Comments from Reviewable

[knz]

bug fix: Using DELETE without WHERE and RETURNING inside [...] does not panic any more. #19822

bug fix: SQL comparisons using the ANY/SOME/ALL operators work properly again with sub-queries and cast expressions. #19801

bug fix: The SQL CLI shell (cockroach sql) again supports window resizing and suspend-to-background properly on macOS. #19429

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/813107c8b38c7136c26e281d6be4aeaae4ed608e/

[petermattis]

Bug fix: Silence an overly verbose log message. #19504

Performance: Trim the wire size of various RPCs. #18930

---

Review status: 0 of 3 files reviewed at latest revision, all discussions resolved, some commit checks pending.

---

Comments from Reviewable

[couchand]

No release notes for the two commits of mine in the undocumented list.

[richardwu]

Bug fix: large, distributed queries that overflow onto disk do not error out anymore. #19689

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/93c072424f3e86d7975db53e76b67bc851b3b65e/

[jseldess]

Thanks to everyone who's gotten me notes so far.

@danhhz, @jordanlewis, @tschottdorf, @dt, @LEGO, @andreimatei, @mjibson, @arjunravinarayan, @mrtracy, @spencerkimball, @justinj, @vivekmenezes, still need help from you all.

[jordanlewis]

@jseldess no release notes for my commits.

[rjnn]

No release notes for my commits.

[justinj]

Bug fix: an improperly typed subquery used with IN no longer panics.

[danhhz]

nothing for mine (i wrote the new release note annotation in the PR descriptions, but somehow that didn't it to the merge commits, hmm...)

[lgo]

From me:

• Performance: Table leases are now acquired in the background when frequently used, removing a jump in latency when they expire. sql: Add background renewal for table descriptor leases  cockroach#19005

There's also the additions of EXPERIMENTAL SCRUB TABLE statements but I don't think that they should/need to be in the release notes.

[dt]

Enterprise: add a placeholder during long-running BACKUP and IMPORT jobs to protect against accidentally using it by concurrent operations. cockroachdb/cockroach#19713 cockroachdb/cockroach#19738
Bug Fix: Allow RESTORE using an incremental BACKUP taken after a table is dropped cockroachdb/cockroach#19601
Bug Fix: Fix always-disabled crash reporting setting cockroachdb/cockroach#19554

[tbg]

cockroachdb/cockroach#19591: (bug fix) Avoid occasional crashes when the server is shut down during startup.
cockroachdb/cockroach#19493: (bug fix) Avoid potential Gossip deadlock on cluster startup
cockroachdb/cockroach#19448: (bug fix) Improve error handling during splits
cockroachdb/cockroach#19447: (bug fix) Proactively shut down server on I/O errors

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/1a93b33716530ebd17c6252af0d2c642c93af37f/

[maddyblue]

Bug fix: be more resilient to S3 quota limits by retrying some operations during backup/restore/import (cockroachdb/cockroach#19556)

The others don't need a release note.

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/7a0eca809eed886d3b71aed8f27a296ded1141be/

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/ec5659d7e92be946f4041db56978c740bd1b91d6/

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/e9e99f220403403969bad4e3deea87da010a1110/

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/12ad833e488d56bc578593dbab4887ac42ca86de/

[benesch]

Updated to remove the source tarball link from the release note page itself, too. Merging once the build finishes; expecting precisely the failures that are fixed in #2149.