[YSQL] Support nested DDL index backfill #4786
Labels
area/ysql
Yugabyte SQL (YSQL)
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Comments
jaki
added a commit
that referenced
this issue
Jun 29, 2020
Summary: Implement core functionality for the backfill part of YSQL multi-stage create index. Do the following checked items: - [x] Add `BACKFILL INDEX` grammar for postgres - [x] Establish basic communication from tserver to postgres - [x] Use ancient write time for inserting rows for backfill - [x] Use supplied read time for selecting rows to backfill - [ ] Establish connection when `yugabyte` role is password protected - [ ] Handle errors anywhere in the schema migration process - [ ] Handle multiple indexes backfilling at same time (issue #4785) - [ ] Have postgres respect master to tserver RPC deadline - [ ] Support create unique index (issue #4899) - [ ] Support nested DDL create index (issue #4786) - [ ] Work on multi-stage drop index Implement it as follows: 1. Pass database name from master to tserver on `BackfillIndex` request 1. Link libpq to tablet in order to send libpq request from tserver 1. Add `BACKFILL INDEX <index_oids> READ TIME <read_time> PARTITION <partition_key> [ FROM <row_key_start> [ TO <row_key_end> ] ]` grammar 1. Wire it down a similar path as `index_build`, but pass down read time and partition key (don't handle row keys yet) through exec params 1. Pass down hard-coded ancient write time 1. Read from indexed table tablet with specified partition key with specified read time 1. Non-transactionally write to index table with specified write time For now, explicitly error on unique index creation and nested DDL index creation because they are unstable. They can later be enabled and wired to use the fast path (no multi-stage). Eventually, after some work, we want to enable them with backfill (multi-stage). Also, remove support for collecting `reltuples` stats on indexes when using backfill. We don't really use this stat, and we don't even collect it for non-index tables, so it shouldn't be a big deal for now. This is part 4 of the effort of bringing index backfill to YSQL. Keep #2301 open. Depends on D8368 Depends on D8578 Test Plan: `./yb_build.sh --cxx-test pgwrapper_pg_libpq-test --gtest_filter 'PgLibPqTest.Backfill*'` Reviewers: amitanand, neil, mihnea Reviewed By: mihnea Subscribers: yql, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D8487
deeps1991
pushed a commit
to deeps1991/yugabyte-db
that referenced
this issue
Jul 22, 2020
Summary: Implement core functionality for the backfill part of YSQL multi-stage create index. Do the following checked items: - [x] Add `BACKFILL INDEX` grammar for postgres - [x] Establish basic communication from tserver to postgres - [x] Use ancient write time for inserting rows for backfill - [x] Use supplied read time for selecting rows to backfill - [ ] Establish connection when `yugabyte` role is password protected - [ ] Handle errors anywhere in the schema migration process - [ ] Handle multiple indexes backfilling at same time (issue yugabyte#4785) - [ ] Have postgres respect master to tserver RPC deadline - [ ] Support create unique index (issue yugabyte#4899) - [ ] Support nested DDL create index (issue yugabyte#4786) - [ ] Work on multi-stage drop index Implement it as follows: 1. Pass database name from master to tserver on `BackfillIndex` request 1. Link libpq to tablet in order to send libpq request from tserver 1. Add `BACKFILL INDEX <index_oids> READ TIME <read_time> PARTITION <partition_key> [ FROM <row_key_start> [ TO <row_key_end> ] ]` grammar 1. Wire it down a similar path as `index_build`, but pass down read time and partition key (don't handle row keys yet) through exec params 1. Pass down hard-coded ancient write time 1. Read from indexed table tablet with specified partition key with specified read time 1. Non-transactionally write to index table with specified write time For now, explicitly error on unique index creation and nested DDL index creation because they are unstable. They can later be enabled and wired to use the fast path (no multi-stage). Eventually, after some work, we want to enable them with backfill (multi-stage). Also, remove support for collecting `reltuples` stats on indexes when using backfill. We don't really use this stat, and we don't even collect it for non-index tables, so it shouldn't be a big deal for now. This is part 4 of the effort of bringing index backfill to YSQL. Keep yugabyte#2301 open. Depends on D8368 Depends on D8578 Test Plan: `./yb_build.sh --cxx-test pgwrapper_pg_libpq-test --gtest_filter 'PgLibPqTest.Backfill*'` Reviewers: amitanand, neil, mihnea Reviewed By: mihnea Subscribers: yql, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D8487
yugabyte-ci
added
kind/bug
yugabyte-ci
added
kind/enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-1935
YSQL multi-stage create index assumes that it is the only DDL (so DDL nesting level 1). With that assumption, it decrements and reincrements nesting level after each stage in order to trigger the end of DDL and make updates visible to other clients. When the nesting level is not 1, the decrements won't cause the DDL to end, and updates won't be visible until the end of the top-level DDL. For example,
CREATE TABLE t (i int, UNIQUE (i));will have a DDL forCREATE TABLEand DDL forCREATE INDEX, and the updates won't be visible until theCREATE TABLEends, meaning that all the stages ofCREATE INDEXwill be combined and invisible until the end, breaking concurrency guarantees. In fact, if backfill were implemented, it would rely on writing to the index, but the index would not be visible to postgres client requesting backfill, so it would be stuck in a retry loop.Come up with some mechanism that can commit the index stage updates while not committing the higher level DDLs.
The text was updated successfully, but these errors were encountered: