Read watermark from materialized data

[fabriziomello]

In 38fcd1b we improved the cagg_watermark performance by storing it into a metadata table and update it during the refresh.

But we made a minor mistake here reading the watermark from the partial view instead of the already materialized data that should be much fast because we're reading already aggregated data.

Fixed this mistake by reading the watermark from the underlying materialization hypertable (already aggregated data).

Disable-check: force-changelog-file

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4f2f658) 87.33% compared to head (0e12772) 87.32%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6405      +/-   ##
==========================================
- Coverage   87.33%   87.32%   -0.02%     
==========================================
  Files         187      187              
  Lines       41820    41768      -52     
  Branches     9313     9291      -22     
==========================================
- Hits        36525    36472      -53     
+ Misses       3623     3619       -4     
- Partials     1672     1677       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

@akuzm, @nikkhils: please review this pull request.

Powered by pull-review

[fabriziomello]

Simple Benchmark

Schema

CREATE TABLE conditions(
    time timestamp with time zone NOT NULL,
    device_id INTEGER,
    temperature FLOAT,
    humidity FLOAT
);

SELECT * FROM create_hypertable('conditions', 'time', chunk_time_interval => '1 hour'::interval);

INSERT INTO conditions
SELECT time, (random()*3 + 1)::int, random()*80 - 40, random()*100
FROM generate_series(now() - INTERVAL '2 years', now(), '1 minute') AS time;

CREATE MATERIALIZED VIEW cagg WITH (timescaledb.continuous) AS 
SELECT time_bucket('1 hour', "time"), device_id, AVG(temperature) AS avg_temp, AVG(humidity) AS avg_hum
FROM conditions
GROUP BY 1, 2
WITH NO DATA;

Refresh on main branch

372898 (leader) fabrizio=# CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
CALL
Time: 522531,804 ms (08:42,532)

Some logs:

2023-12-18 14:45:30.426 -03 [372898] LOG:  deleted 0 row(s) from materialization table "_timescaledb_internal._materialized_hypertable_3"
2023-12-18 14:45:30.426 -03 [372898] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:48:50.027 -03 [372898] LOG:  temporary file: path "base/pgsql_tmp/pgsql_tmp372898.0", size 34390016
2023-12-18 14:48:50.027 -03 [372898] CONTEXT:  SQL statement "INSERT INTO _timescaledb_internal._materialized_hypertable_3 SELECT * FROM _timescaledb_internal._partial_view_3 AS I WHERE I.time_bucket >= '4714-11-23 20:53:32-03:06:28 BC' AND I.time_bucket < '2023-12-18 15:00:00-03' ;"
2023-12-18 14:48:50.027 -03 [372898] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:48:50.217 -03 [372898] LOG:  inserted 70084 row(s) into materialization table "_timescaledb_internal._materialized_hypertable_3"
2023-12-18 14:48:50.217 -03 [372898] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:52:06.832 -03 [372898] LOG:  temporary file: path "base/pgsql_tmp/pgsql_tmp372898.1", size 1548288
2023-12-18 14:52:06.832 -03 [372898] CONTEXT:  SQL statement "SELECT pg_catalog.max(time_bucket) FROM _timescaledb_internal._partial_view_3 AS I WHERE I.time_bucket >= '4714-11-23 20:53:32-03:06:28 BC' AND I.time_bucket < '2023-12-18 15:00:00-03' ;"
2023-12-18 14:52:06.832 -03 [372898] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:52:07.131 -03 [372898] LOG:  duration: 522515.833 ms  statement: CALL refresh_continuous_aggregate ('cagg', NULL, NULL);

Refresh using the current PR

367649 (leader) fabrizio=# CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
CALL
Time: 328990,482 ms (05:28,990)

Some logs:

2023-12-18 14:24:08.131 -03 [367649] LOG:  deleted 0 row(s) from materialization table "_timescaledb_internal._materialized_hypertable_3"
2023-12-18 14:24:08.131 -03 [367649] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:27:41.860 -03 [367649] LOG:  temporary file: path "base/pgsql_tmp/pgsql_tmp367649.3", size 34390016
2023-12-18 14:27:41.860 -03 [367649] CONTEXT:  SQL statement "INSERT INTO _timescaledb_internal._materialized_hypertable_3 SELECT * FROM _timescaledb_internal._partial_view_3 AS I WHERE I.time_bucket >= '4714-11-23 20:53:32-03:06:28 BC' AND I.time_bucket < '2023-12-18 15:00:00-03' ;"
2023-12-18 14:27:41.860 -03 [367649] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:27:42.046 -03 [367649] LOG:  inserted 70084 row(s) into materialization table "_timescaledb_internal._materialized_hypertable_3"
2023-12-18 14:27:42.046 -03 [367649] STATEMENT:  CALL refresh_continuous_aggregate ('cagg', NULL, NULL);
2023-12-18 14:27:42.407 -03 [367649] LOG:  duration: 328976.816 ms  statement: CALL refresh_continuous_aggregate ('cagg', NULL, NULL);

Results

main branch Time: 522531,804 ms (08:42,532)
current PR Time: 328990,482 ms (05:28,990)
Improvement ~37%