List tasks information for hung workers #110
Conversation
|
Checked on some hung tests / result empty. Looks like needs to flush. |
listeners.py
Outdated
| task_name, task_param = self.get_task_by_worker_id(worker_id) | ||
| color_stdout("[{0:03d}] {1} {2}\n".format(worker_id, | ||
| task_name, | ||
| task_param or ''), |
dispatcher.py
Outdated
| @@ -78,10 +78,14 @@ def __init__(self, task_groups, max_workers_cnt, randomize): | |||
| self.worker_next_id = 1 | |||
|
|
|||
| tasks_cnt = 0 | |||
| self.current_task_queue = SimpleQueue() | |||
There was a problem hiding this comment.
Why the new queue is needed?
There was a problem hiding this comment.
Sergei (@sergw) helps me to remember the context: workers are separate processes, so we need a queue to pass information about a test that a worker receive.
I propose the following approach. A worker sends an object of specific type to the results queue (see 'Worker results' at worker.py:113) when receive a task. The HangWatcher listener (listeners.py) checks for this type of an object in process_result and saves necessary info. Then it can be used in process_timeout method.
So, we’ll avoid usage of extra queue and we’ll reuse our flexible approach to handle events depending of its type.
dispatcher.py
Outdated
| @@ -78,10 +78,14 @@ def __init__(self, task_groups, max_workers_cnt, randomize): | |||
| self.worker_next_id = 1 | |||
|
|
|||
| tasks_cnt = 0 | |||
| self.current_task_queue = SimpleQueue() | |||
There was a problem hiding this comment.
Sergei (@sergw) helps me to remember the context: workers are separate processes, so we need a queue to pass information about a test that a worker receive.
I propose the following approach. A worker sends an object of specific type to the results queue (see 'Worker results' at worker.py:113) when receive a task. The HangWatcher listener (listeners.py) checks for this type of an object in process_result and saves necessary info. Then it can be used in process_timeout method.
So, we’ll avoid usage of extra queue and we’ll reuse our flexible approach to handle events depending of its type.
listeners.py
Outdated
| task_name, | ||
| task_param or ''), | ||
| schema='test_var') | ||
| task_path = "{0:03d}_{1}".format(worker_id, |
There was a problem hiding this comment.
I think it is better to get a path to result file from a worker and don’t try to reconstruct it here.
lib/worker.py
Outdated
| @@ -285,6 +286,8 @@ def run_loop(self, task_queue, result_queue): | |||
| schema='test_var') | |||
| self.stop_worker(task_queue, result_queue) | |||
| break | |||
|
|
|||
| current_task_queue.put((self.id, task_id)) | |||
There was a problem hiding this comment.
Please, create a subclass of BaseWorkerMessage to hold the data.
sergw
force-pushed
the
list-tasks-for-hung-workers
branch
from
0483493 to
b559025
Compare
sergw
force-pushed
the
list-tasks-for-hung-workers
branch
from
b559025 to
974dbcc
Compare
If HungListener found some hung tests it will print information about: - worker id - test name - test params - last 15 lines from .result file Closes #107
sergw
force-pushed
the
list-tasks-for-hung-workers
branch
2 times, most recently
from
6c3791a to
54dcd33
Compare
There was a problem hiding this comment.
Thanks, Sergei!
The patch looks good for me. I have one tiny comment, though.
I filed the follow up issue #119 to show output of hang 'core = app' tests. @Korablev77 asks me about that before.
listeners.py
Outdated
| in self.worker_current_task.iteritems() | ||
| if worker_id in worker_ids] | ||
| for current_task in hung_tasks: | ||
| color_stdout("[{0:03d}] {1} {2}\n".format(current_task.worker_id, |
There was a problem hiding this comment.
Maybe it would be look more habitual if we'll use the same encoding (one-line yaml) as in reproduce files, like [app-tap/tap.test.lua, null].
sergw
force-pushed
the
list-tasks-for-hung-workers
branch
from
54dcd33 to
68e41d1
Compare
If HungListener found some hung tests it will print information about:
Closes #107