Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sensors/test_status: QA Notice leads to a false fail detection #1

Closed
Totktonada opened this issue Oct 11, 2021 · 0 comments · Fixed by #138
Closed

sensors/test_status: QA Notice leads to a false fail detection #1

Totktonada opened this issue Oct 11, 2021 · 0 comments · Fixed by #138
Assignees
@lastoCHka42
Labels
bug Something isn't working

Comments

@Totktonada
Copy link
Member

Totktonada commented Oct 11, 2021 •
edited
Loading 

[044] replication/box_set_replication_stress.test.lua memtx           [ pass ]
[044] replication/box_set_replication_stress.test.lua vinyl           
[044] * [QA Notice]
[044] *
[044] * The server 'master_quorum1' does not stop during 5 seconds after the 15 (SIGTERM) signal.
[044] * Info: process 36076 [unknown; unknown]
[044] * Sending SIGKILL...
[044] *
[044] [ pass ]

https://github.com/tarantool/tarantool/runs/3814050318

When there is no particular status, we don't search it somewhere below by the log, but assume fail:

multivac/multivac/sensors/test_status.py

Line 15 in 06a1e7e

STATUS_RE = r'((Test timeout of \d+ secs reached\t)?\[ (?P<status>[^ ]+) \]|)'

multivac/multivac/sensors/test_status.py

Line 64 in 06a1e7e

status = m.group('status') or 'fail'

Because usually such line break means that test-run print some information about a fail.

Given QA notice may actually be result of some problem, but we have a notice about low sampling resolution, which definitely should not be detected as a test fail.

In fact, there is no much sense to print a line in two chunks in test-run (see here and here), because now the output is anyway line buffered (due to parallel running of tests). So maybe it worth to resolve it from the test-run side.
@Totktonada Totktonada added the bug Something isn't working label Oct 11, 2021
@Totktonada Totktonada mentioned this issue Oct 20, 2021
Closed
lastoCHka42 added a commit that referenced this issue Aug 29, 2023
@lastoCHka42
sensors/test_status: get the late status
8096b19 
Sometimes we get the test result not in the same line, but few lines
later, i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
@lastoCHka42 lastoCHka42 mentioned this issue Aug 29, 2023
Merged
lastoCHka42 added a commit that referenced this issue Aug 30, 2023
@lastoCHka42
sensors/test_status: get the late status
5c0bad4 
Sometimes we get the test result not in the same line, but few lines
later, i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
lastoCHka42 added a commit that referenced this issue Aug 30, 2023
@lastoCHka42
sensors/test_status: get the late status
b91830a 
Sometimes we get the test result but few lines later after test name,
i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
lastoCHka42 added a commit that referenced this issue Aug 30, 2023
@lastoCHka42
sensors/test_status: get the late status
e0b9dfb 
Sometimes we get the test result few lines later after test name,
i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
lastoCHka42 added a commit that referenced this issue Aug 31, 2023
@lastoCHka42
sensors/test_status: get the late status
9ffc7ef 
Sometimes we get the test result few lines later after test name,
i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
lastoCHka42 added a commit that referenced this issue Aug 31, 2023
@lastoCHka42
sensors/test_status: get the late status
1e16a3b 
Sometimes we get the test result few lines later after test name,
i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
lastoCHka42 added a commit that referenced this issue Sep 1, 2023
@lastoCHka42
sensors/test_status: get the late status
927df01 
Sometimes we get the test result few lines later after test name,
i.e. when there are QA Notice before the test result. This patch
saves all the tests without the result to the dictionary and finds the
result without test with the same worker ID.

Resolves #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lastoCHka42 lastoCHka42

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sensors/test_status: get the late status tarantool/multivac
4 participants
@Totktonada @NickVolynkin @TarantoolBot @lastoCHka42