Use `compute_ctl` to manage Postgres in tests. #3886

hlinnaka · 2023-03-27T15:48:36Z

This adds test coverage for 'compute_ctl', as it is now used by all
the python tests.

There are a few differences in how 'compute_ctl' is called in the
tests, compared to the real web console:

In the tests, the postgresql.conf file is included as one large
string in the spec file, and it is written out as it is to the data
directory. I added a new field for that to the spec file. The real
web console, however, sets all the necessary settings in the
'settings' field, and 'compute_ctl' creates the postgresql.conf from
those settings.
In the tests, the information needed to connect to the storage, i.e.
tenant_id, timeline_id, connection strings to pageserver and
safekeepers, are now passed as new fields in the spec file. The real
web console includes them as the GUCs in the 'settings' field. (Both
of these are different from what the test control plane used to do:
It used to write the GUCs directly in the postgresql.conf file). The
plan is to change the control plane to use the new method, and
remove the old method, but for now, support both.

Some tests that were sensitive to the amount of WAL generated needed
small changes, to accommodate that compute_ctl runs the background
health monitor which makes a few small updates. Also some tests shut
down the pageserver, and now that the background health check can run
some queries while the pageserver is down, that can produce a few
extra errors in the logs, which needed to be allowlisted.

Other changes:

remove obsolete comments about PostgresNode;
Create standby.signal file for Static compute node;
log output of compute_ctl and postgres is merged into endpoints/compute.log.

ololobus

I haven't read everything thoroughly, but here are some comments. In overall, looks nice!

libs/compute_api/src/spec.rs

compute_tools/src/config.rs

libs/compute_api/src/spec.rs

compute_tools/src/bin/compute_ctl.rs

compute_tools/src/pg_helpers.rs

libs/compute_api/src/spec.rs

compute_tools/src/bin/compute_ctl.rs

control_plane/src/endpoint.rs

test_runner/regress/test_wal_acceptor_async.py

control_plane/src/endpoint.rs

If the port is hard-coded, you cannot have more than one compute_ctl running at a time. (Or not at all, if you're unlucky and the port is in use)

'compute_ctl' merely prints them to the log. The next commit will make the local test control plane to also use 'compute_ctl', and it doesn't have the concept of a project, so it cannot fill them with any sensible values.

This adds test coverage for 'compute_ctl', as it is now used by all the python tests. There are a few differences in how 'compute_ctl' is called in the tests, compared to the real web console: - In the tests, the postgresql.conf file is included as one large string in the spec file, and it is written out as it is to the data directory. I added a new field for that to the spec file. The real web console, however, sets all the necessary settings in the 'settings' field, and 'compute_ctl' creates the postgresql.conf from those settings. - In the tests, the information needed to connect to the storage, i.e. tenant_id, timeline_id, connection strings to pageserver and safekeepers, are now passed as new fields in the spec file. The real web console includes them as the GUCs in the 'settings' field. (Both of these are different from what the test control plane used to do: It used to write the GUCs directly in the postgresql.conf file). The plan is to change the control plane to use the new method, and remove the old method, but for now, support both. Some tests that were sensitive to the amount of WAL generated needed small changes, to accommodate that compute_ctl runs the background health monitor which makes a few small updates. Also some tests shut down the pageserver, and now that the background health check can run some queries while the pageserver is down, that can produce a few extra errors in the logs, which needed to be allowlisted.

It was renamed to Endpoint and refactored to reduce code duplication.

…_metadata

hlinnaka · 2023-06-06T14:08:10Z

LGTM, thanks!

This adds test coverage for 'compute_ctl', as it is now used by all the python tests. There are a few differences in how 'compute_ctl' is called in the tests, compared to the real web console: - In the tests, the postgresql.conf file is included as one large string in the spec file, and it is written out as it is to the data directory. I added a new field for that to the spec file. The real web console, however, sets all the necessary settings in the 'settings' field, and 'compute_ctl' creates the postgresql.conf from those settings. - In the tests, the information needed to connect to the storage, i.e. tenant_id, timeline_id, connection strings to pageserver and safekeepers, are now passed as new fields in the spec file. The real web console includes them as the GUCs in the 'settings' field. (Both of these are different from what the test control plane used to do: It used to write the GUCs directly in the postgresql.conf file). The plan is to change the control plane to use the new method, and remove the old method, but for now, support both. Some tests that were sensitive to the amount of WAL generated needed small changes, to accommodate that compute_ctl runs the background health monitor which makes a few small updates. Also some tests shut down the pageserver, and now that the background health check can run some queries while the pageserver is down, that can produce a few extra errors in the logs, which needed to be allowlisted. Other changes: - remove obsolete comments about PostgresNode; - create standby.signal file for Static compute node; - log output of `compute_ctl` and `postgres` is merged into `endpoints/compute.log`. --------- Co-authored-by: Anastasia Lubennikova <[email protected]>

hlinnaka requested review from MMeent, ololobus and kelvich March 27, 2023 15:48

ololobus reviewed Mar 27, 2023

View reviewed changes

hlinnaka force-pushed the test-compute_ctl branch from fa4e96d to 78568fc Compare March 28, 2023 09:00

hlinnaka mentioned this pull request Mar 28, 2023

Remove obsolete hack to rename neon-specific GUCs. #3897

Merged

hlinnaka marked this pull request as ready for review March 30, 2023 09:35

hlinnaka requested review from a team as code owners March 30, 2023 09:35

hlinnaka requested review from shanyp and removed request for a team March 30, 2023 09:35

hlinnaka force-pushed the test-compute_ctl branch 2 times, most recently from 11946ab to 58d39fa Compare April 3, 2023 07:17

This was referenced Apr 3, 2023

Move compute_ctl structs used in HTTP API and spec file to separate crate. #3937

Merged

Rename "Postgres nodes" in control_plane to endpoints. #3938

Merged

hlinnaka force-pushed the test-compute_ctl branch from 58d39fa to 53a56fd Compare April 3, 2023 14:30

hlinnaka changed the base branch from main to neon_local-endpoints April 3, 2023 14:31

hlinnaka force-pushed the test-compute_ctl branch from 53a56fd to 069f73d Compare April 3, 2023 14:58

MMeent reviewed Apr 3, 2023

View reviewed changes

libs/compute_api/src/spec.rs Outdated Show resolved Hide resolved

libs/compute_api/src/spec.rs Outdated Show resolved Hide resolved

libs/compute_api/src/spec.rs Outdated Show resolved Hide resolved

libs/compute_api/src/spec.rs Outdated Show resolved Hide resolved

hlinnaka force-pushed the neon_local-endpoints branch 3 times, most recently from 9083d1e to 56bf5b4 Compare April 3, 2023 17:32

hlinnaka force-pushed the test-compute_ctl branch from 069f73d to 532d880 Compare April 3, 2023 17:34

hlinnaka force-pushed the neon_local-endpoints branch 4 times, most recently from 1326ccc to d823c15 Compare April 9, 2023 19:01

hlinnaka mentioned this pull request Apr 10, 2023

Minor cleanups to spec handling in compute_ctl #3990

Merged

hlinnaka force-pushed the neon_local-endpoints branch from d823c15 to 50e38f5 Compare April 12, 2023 07:01

hlinnaka force-pushed the test-compute_ctl branch 3 times, most recently from aed055e to 13d2254 Compare May 5, 2023 10:50

hlinnaka requested a review from MMeent May 5, 2023 10:53

MMeent reviewed May 5, 2023

View reviewed changes

compute_tools/src/bin/compute_ctl.rs Show resolved Hide resolved

control_plane/src/endpoint.rs Show resolved Hide resolved

control_plane/src/endpoint.rs Outdated Show resolved Hide resolved

test_runner/regress/test_wal_acceptor_async.py Show resolved Hide resolved

hlinnaka force-pushed the test-compute_ctl branch 2 times, most recently from 4b7f5b5 to 75227e8 Compare May 9, 2023 17:25

lubennikovaav force-pushed the test-compute_ctl branch from 75227e8 to e2cd2ad Compare June 1, 2023 12:47

lubennikovaav reviewed Jun 1, 2023

View reviewed changes

control_plane/src/endpoint.rs Outdated Show resolved Hide resolved

hlinnaka and others added 10 commits June 5, 2023 18:46

Add --http-port option to compute_ctl.

01ee0d5

If the port is hard-coded, you cannot have more than one compute_ctl running at a time. (Or not at all, if you're unlucky and the port is in use)

Make cluster_id and name optional in the compute spec file.

227da02

'compute_ctl' merely prints them to the log. The next commit will make the local test control plane to also use 'compute_ctl', and it doesn't have the concept of a project, so it cannot fill them with any sensible values.

Remove obsolete comment about PostgresNode.

cae9a69

It was renamed to Endpoint and refactored to reduce code duplication.

WIP bring back replication settings

44c2c79

WIP only set safekeeper_connstrings for Primary

28336cd

WIP Create standby.signal file for Static compute node

a8f4b5d

Fix err parsing in test_pageserver_lsn_wait_error_safekeeper_stop

ef9557e

fix for only set safekeeper_connstring

6475c7c

Fix allowed error message in test_ignored_tenant_stays_broken_without…

99d27fa

…_metadata

lubennikovaav force-pushed the test-compute_ctl branch from 5841011 to 99d27fa Compare June 5, 2023 19:41

lubennikovaav added 2 commits June 5, 2023 21:35

fix flaky test_single_branch_get_tenant_size_grows on v15

177e02c

pytest codestyle fix

1282305

lubennikovaav mentioned this pull request Jun 6, 2023

Improve compute.log readability #4430

Open

lubennikovaav approved these changes Jun 6, 2023

View reviewed changes

lubennikovaav merged commit df3bae2 into main Jun 6, 2023

lubennikovaav deleted the test-compute_ctl branch June 6, 2023 13:59

ololobus mentioned this pull request Jun 6, 2023

Add idle compute restart time test #1514

Merged

YanicNeon mentioned this pull request Aug 23, 2023

Duplicate entries in postgresql.conf #5088

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `compute_ctl` to manage Postgres in tests. #3886

Use `compute_ctl` to manage Postgres in tests. #3886

hlinnaka commented Mar 27, 2023 •

edited by lubennikovaav

Loading

ololobus left a comment

hlinnaka commented Jun 6, 2023

Use compute_ctl to manage Postgres in tests. #3886

Use compute_ctl to manage Postgres in tests. #3886

Conversation

hlinnaka commented Mar 27, 2023 • edited by lubennikovaav Loading

ololobus left a comment

Choose a reason for hiding this comment

hlinnaka commented Jun 6, 2023

Use `compute_ctl` to manage Postgres in tests. #3886

Use `compute_ctl` to manage Postgres in tests. #3886

hlinnaka commented Mar 27, 2023 •

edited by lubennikovaav

Loading