-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[#24732] YSQL: Refresh materialized view in-place during upgrade
Summary: **Behavior before this change:** Non-concurrent refresh of a materialized view involves generating the new data in a new DocDB table, and then swapping the tables by replacing the `relfinenode` info of the matview. This involves 16 RPCs to yb-master (pg catalog) with 73 pgsql_write_batches. All of these are not allowed during the ysql major upgrade. Concurrent refresh of a supported materialized view involves generating the new data in temp table, followed by doing a merge of the new data with the old data already in the matview using another temp table. This involves 24 RPC calls to yb-master (pg catalog) with 241 pgsql_write_batches. However 23 of these RPCs are related to the temp tables (force_catalog_modifications set) which are allowed during ysql major upgrade. The only other RPC is related to modifying the `relispopulated` field. **Behavior with this change:** During ysql major upgrade changes to the `relispopulated` field is blocked. This is not an issue with concurrent refresh, since those are always blocked. The non-concurrent refresh of matview now generates the new data in a temporary table, and use the more expensive method of running `DELETE FROM <matview>` followed by a `INSERT INTO <matview> SELECT * FROM <temp_table>`. The advantage of this method is that all pg catalog modifications are only done to the temp table, which is allowed during the upgrade. The performance tradeoff is acceptable, since we only expect small matviews to be refreshed frequently, hences needed during the upgrade, whereas the bigger ones can wait till after the upgrade to be refreshed. This behavior is enabled by `yb_refresh_matview_in_place` or `yb_major_version_upgrade_compatibility>0`. Concurrent refresh has not been modified, since it only creates temporary tables, and does not involve any other catalog modifications. Fixes #24732 Jira: DB-13813 Test Plan: YsqlMajorUpgradeMatviewTest.TestMatView Reviewers: telgersma, smishra, myang Reviewed By: telgersma, myang Subscribers: svc_phabricator, ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D41504
- Loading branch information
Showing
11 changed files
with
246 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
99 changes: 99 additions & 0 deletions
99
src/yb/integration-tests/upgrade-tests/ysql_major_upgrade_matview-test.cc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| // Copyright (c) YugabyteDB, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except | ||
| // in compliance with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software distributed under the License | ||
| // is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express | ||
| // or implied. See the License for the specific language governing permissions and limitations | ||
| // under the License. | ||
| // | ||
|
|
||
| #include "yb/integration-tests/upgrade-tests/pg15_upgrade_test_base.h" | ||
|
|
||
| #include "yb/yql/pgwrapper/libpq_utils.h" | ||
|
|
||
| namespace yb { | ||
|
|
||
| static constexpr auto kTableName = "tbl1"; | ||
| static constexpr auto kNormalMatviewName = "mv1"; | ||
| static constexpr auto kIndexedMatViewName = "mv2"; | ||
|
|
||
| class YsqlMajorUpgradeMatviewTest : public Pg15UpgradeTestBase { | ||
| public: | ||
| void SetUp() override { | ||
| Pg15UpgradeTestBase::SetUp(); | ||
| if (Test::IsSkipped()) { | ||
| return; | ||
| } | ||
|
|
||
| auto conn = ASSERT_RESULT(CreateConnToTs(std::nullopt)); | ||
| ASSERT_OK(conn.ExecuteFormat("CREATE TABLE $0(a int, b int)", kTableName)); | ||
| ASSERT_OK(conn.ExecuteFormat( | ||
| "CREATE MATERIALIZED VIEW $0 AS SELECT a FROM $1", kNormalMatviewName, kTableName)); | ||
| ASSERT_OK(conn.ExecuteFormat( | ||
| "CREATE MATERIALIZED VIEW $0 AS SELECT a FROM $1", kIndexedMatViewName, kTableName)); | ||
| ASSERT_OK(conn.ExecuteFormat("CREATE UNIQUE INDEX idx1 ON $0(a)", kIndexedMatViewName)); | ||
|
|
||
| ASSERT_OK(InsertDataRefreshAndValidate()); | ||
| } | ||
|
|
||
| Status InsertDataRefreshAndValidate( | ||
| std::optional<size_t> ts_id = std::nullopt, int row_count = 10) { | ||
| auto conn = VERIFY_RESULT(CreateConnToTs(ts_id)); | ||
| RETURN_NOT_OK(InsertData(conn, row_count)); | ||
| return RefreshMatviewsAndValidate(conn); | ||
| } | ||
|
|
||
| Status InsertData(pgwrapper::PGConn& conn, int row_count) { | ||
| for (int i = 0; i < row_count; ++i) { | ||
| RETURN_NOT_OK(conn.ExecuteFormat("INSERT INTO $0 VALUES ($1, $1)", kTableName, ++num_rows_)); | ||
| } | ||
| return Status::OK(); | ||
| } | ||
|
|
||
| Status RefreshMatviewsAndValidate(pgwrapper::PGConn& conn) { | ||
| RETURN_NOT_OK(conn.ExecuteFormat("REFRESH MATERIALIZED VIEW $0", kNormalMatviewName)); | ||
| RETURN_NOT_OK( | ||
| conn.ExecuteFormat("REFRESH MATERIALIZED VIEW CONCURRENTLY $0", kIndexedMatViewName)); | ||
|
|
||
| auto table_data = VERIFY_RESULT(conn.FetchRows<int>("SELECT a FROM tbl1 ORDER BY a")); | ||
| LOG(INFO) << "Table data: " << yb::ToString(table_data); | ||
|
|
||
| auto normal_matview_data = VERIFY_RESULT(conn.FetchRows<int>("SELECT a FROM mv1 ORDER BY a")); | ||
| SCHECK_EQ( | ||
| table_data, normal_matview_data, IllegalState, | ||
| Format("Normal matview data mismatch: $0", yb::ToString(normal_matview_data))); | ||
|
|
||
| auto indexes_matview_data = VERIFY_RESULT(conn.FetchRows<int>("SELECT a FROM mv2 ORDER BY a")); | ||
| SCHECK_EQ( | ||
| table_data, indexes_matview_data, IllegalState, | ||
| Format("Indexed matview data mismatch: $0", yb::ToString(indexes_matview_data))); | ||
|
|
||
| return Status::OK(); | ||
| } | ||
|
|
||
| private: | ||
| uint32 num_rows_ = 0; | ||
| }; | ||
|
|
||
| TEST_F(YsqlMajorUpgradeMatviewTest, TestMatView) { | ||
| ASSERT_OK(UpgradeClusterToMixedMode()); | ||
|
|
||
| ASSERT_NOK(InsertDataRefreshAndValidate(kMixedModeTserverPg11)); | ||
|
|
||
| // TODO: Pending backport | ||
| ASSERT_OK(cluster_->SetFlag( | ||
| cluster_->tablet_server(kMixedModeTserverPg15), "ysql_yb_major_version_upgrade_compatibility", | ||
| "11")); | ||
|
|
||
| ASSERT_OK(InsertDataRefreshAndValidate(kMixedModeTserverPg15)); | ||
|
|
||
| ASSERT_OK(FinalizeUpgradeFromMixedMode()); | ||
|
|
||
| ASSERT_OK(InsertDataRefreshAndValidate()); | ||
| } | ||
|
|
||
| } // namespace yb |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.