Send SIGKILL if server does not stop after timeout

If a tarantool instance does not handle SIGTERM correctly, it is the bug
in tarantool. A testing system should reveal problems rather than hide
them. However this patch doing that and I'll explain why.

Many tests we have perform instance starting / stopping: they are
written to spot different problems, but not this one. If everything else
except hanging at stop is okay, there is no reason to fail the test. We
print a notice on the terminal in the case and proceed further.

In future, when all problems of this kind will be resolved, we can
interpret presence of such notices as error. However now it'll not add
anything for quality of our testing.

We had a problem of this kind ([1]) in the past and now have another one
([2]). Until it will be resolved, it worth to handle the situation on
the testing system side.

[1]: tarantool/tarantool#4127
[2]: https://github.com/tarantool/tarantool/issues/5573

Part of #157

Co-authored-by: Alexander Turenko <[email protected]>

lib/tarantool_server.py

@@ -15,6 +15,7 @@
 
 from gevent import socket
 from greenlet import GreenletExit
+from threading import Timer
 
 try:
     from cStringIO import StringIO
@@ -23,7 +24,9 @@
 
 from lib.admin_connection import AdminConnection, AdminAsyncConnection
 from lib.box_connection import BoxConnection
-from lib.colorer import color_stdout, color_log
+from lib.colorer import color_stdout
+from lib.colorer import color_log
+from lib.colorer import qa_notice
 from lib.preprocessor import TestState
 from lib.server import Server
 from lib.server import DEFAULT_SNAPSHOT_NAME
@@ -1018,9 +1021,26 @@ def stop(self, silent=True, signal=signal.SIGTERM):
                 self.process.send_signal(signal)
             except OSError:
                 pass
+
+            # Waiting for stopping the server. If the timeout
+            # reached, send SIGKILL.
+            timeout = 5
+
+            def kill():
+                qa_notice('The server \'{}\' does not stop during {} '
+                          'seconds after the {} ({}) signal.\n'
+                          'Info: {}\n'
+                          'Sending SIGKILL...'.format(
+                              self.name, timeout, signal, signame(signal),
+                              format_process(self.process.pid)))
+                self.process.kill()
+
+            timer = Timer(timeout, kill)
+            timer.start()
             if self.crash_detector is not None:
                 save_join(self.crash_detector)
             self.wait_stop()
+            timer.cancel()
 
         self.status = None
         if re.search(r'^/', str(self._admin.port)):