Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_scrubber_tenant_snapshot is flaky #10462

Open
erikgrinaker opened this issue Jan 21, 2025 · 0 comments
Open

test_scrubber_tenant_snapshot is flaky #10462

erikgrinaker opened this issue Jan 21, 2025 · 0 comments
Assignees
@arssher
Labels
a/test/flaky Area: related to flaky tests c/storage Component: storage t/bug Issue Type: Bug triaged bugs that were already triaged

Comments

@erikgrinaker
Copy link
Contributor

https://neonprod.grafana.net/d/fddp4rvg7k2dcf/regression-test-failures?orgId=1&var-test_name=test_scrubber_tenant_snapshot%5B4%5D&from=now-7d&to=now&timezone=utc&var-restrict=true&var-max_count=100&var-reference=$__all&var-ignore_reference=refs%2Fpull%2F0000%2Fmerge

/github/home/.cache/pypoetry/virtualenvs/non-package-mode-_pxWMzVK-py3.11/lib/python3.11/site-packages/allure_commons/_allure.py:221: in __call__
    return self._fixture_function(*args, **kwargs)
test_runner/fixtures/neon_fixtures.py:1541: in neon_env_builder
    with NeonEnvBuilder(
test_runner/fixtures/neon_fixtures.py:937: in __exit__
    self.env.storage_controller.assert_no_errors()
test_runner/fixtures/neon_fixtures.py:1711: in assert_no_errors
    assert_no_errors(
test_runner/fixtures/utils.py:527: in assert_no_errors
    assert not errors, f"First log error on {service}: {errors[0]}\nHint: use scripts/check_allowed_errors.sh to test any new allowed_error you add"
E   AssertionError: First log error on storage_controller: (305, '2025-01-21T10:56:05.395898Z  WARN process_result{seq=1 tenant_id=ad2c90375ee6c2cf04de792e9b0898ac shard_id=0004}: Reconcile error: receive body: error sending request for url (http://localhost:31629/v1/tenant/ad2c90375ee6c2cf04de792e9b0898ac-0004/location_config): client error (Connect)\n')
E   Hint: use scripts/check_allowed_errors.sh to test any new allowed_error you add
@erikgrinaker erikgrinaker added a/test/flaky Area: related to flaky tests c/storage Component: storage t/bug Issue Type: Bug labels Jan 21, 2025
@jcsp jcsp added the triaged bugs that were already triaged label Jan 28, 2025
@jcsp jcsp assigned arssher and unassigned problame Jan 29, 2025
@arssher arssher mentioned this issue Jan 30, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 31, 2025
@arssher
Allow pageserver unreachable errors in test_scrubber_tenant_snapshot (#…
765ba43 
…10585)

## Problem

test_scrubber_tenant_snapshot restarts pageservers, but log validation
fails tests on any non white listed storcon warnings, making the test
flaky.

## Summary of changes

Allow warns like
2025-01-29T12:37:42.622179Z WARN reconciler{seq=1
tenant_id=2011077aea9b4e8a60e8e8a19407634c shard_id=0004}: Call to node
2 (localhost:15352) management API failed, will retry (attempt 1):
receive body: error sending request for url
(http://localhost:15352/v1/tenant/2011077aea9b4e8a60e8e8a19407634c-0004/location_config):
client error (Connect)

ref #10462
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arssher arssher

Labels
a/test/flaky Area: related to flaky tests c/storage Component: storage t/bug Issue Type: Bug triaged bugs that were already triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@erikgrinaker @jcsp @problame @arssher