Skip to content

Commit

Permalink
Handle lack of a last result file at a worker hang
Browse files Browse the repository at this point in the history
When there is no output from workers during a long time (10 seconds by
default or 60 seconds when --long argument is passed), test-run prints a
warning and shows amount of lines in the temporary result file. It is
useful to understand on which statement a test hungs.

I reproduced the problem, when mangled tarantool to ignore SIGTERM and
SIGINT signals and run a simple 'tarantool = core' test. The test
successfully passes, but the worker stucks in waiting for stopping the
tarantool server.

This particular case should be resolved in PR #186, but just because the
timeout for stopping the server is less than the warning delay. This
assumption looks fragile, especially if we'll want to make some of those
timeouts / delays configurable. Let's handle the situation when the file
does not exist.

Found while looking into
https://github.com/tarantool/tarantool/issues/5573

Fixes #245
  • Loading branch information
Totktonada authored and avtikhon committed Dec 1, 2020
1 parent cbdef68 commit 85333e7
Showing 1 changed file with 11 additions and 6 deletions.
17 changes: 11 additions & 6 deletions listeners.py
Original file line number Diff line number Diff line change
Expand Up @@ -282,23 +282,28 @@ def process_timeout(self, delta_seconds):
return

is_warning = self.inactivity < self.kill_timeout
color_schema = 'test_var' if is_warning else 'error'

color_stdout(
"No output during {0.inactivity:.0f} seconds. "
"Will abort after {0.kill_timeout:.0f} seconds without output. "
"List of workers not reporting the status:\n".format(self),
schema=('test_var' if is_warning else 'error'))
schema=color_schema)

hung_tasks = [task for worker_id, task
in self.worker_current_task.iteritems()
if worker_id in worker_ids]
for task in hung_tasks:
with open(task.task_tmp_result, 'r') as f:
lines = sum(1 for _ in f)
color_stdout("- {0} [{1}, {2}] at {3}:{4}\n".format(
result_file = task.task_tmp_result
result_file_summary = '(no result file {})'.format(result_file)
if os.path.exists(result_file):
with open(result_file, 'r') as f:
lines = sum(1 for _ in f)
result_file_summary = 'at {}:{}'.format(result_file,
lines)
color_stdout('- {} [{}, {}] {}\n'.format(
task.worker_name, task.task_name, task.task_param,
task.task_tmp_result, lines),
schema=('test_var' if is_warning else 'error'))
result_file_summary), schema=color_schema)

self.warned_seconds_ago = 0.0

Expand Down

0 comments on commit 85333e7

Please sign in to comment.