Refactor of BitcoinTestFramework main function.

doc/test-wrapper.md

@@ -0,0 +1,122 @@
+Test Wrapper for Interactive Environments
+=========================================
+
+This document describes the usage of the `TestWrapper` submodule in the `test_framework` module of the functional test framework.
+
+The TestWrapper submodule extends the `BitcoinTestFramework` functionality to external interactive environments for prototyping and educational purposes. Just like `BitcoinTestFramework`, the TestWrapper allows the user to:
+
+* Manage regtest bitcoind subprocesses.
+* Access RPC interfaces of these bitcoind instances.
+* Log events to functional test logging utility.
+
+The `TestWrapper` can be useful in interactive environments such as the Python3 command-line interpreter or [Jupyter](https://jupyter.org/) notebooks running a Python3 kernel, where is is necessary to extend the object lifetime of the underlying `BitcoinTestFramework` between user inputs.
+
+## 1. Requirements
+
+* Python3
+* `bitcoind` built in the same bitcoin repository as the TestWrapper.
+
+## 2. Importing TestWrapper from the Bitcoin Core repository
+
+We can import the TestWrapper by adding the path of the Bitcoin Core `test_framework` module to the beginning of the PATH variable, and then importing the `TestWrapper` class from the `test_wrapper` sub-package.
+
+```
+>>> import sys
+>>> sys.path.insert(0, "/path/to/bitcoin/test/functional/test_framework")
+>>> from test_framework.test_wrapper import TestWrapper
+```
+
+The following TestWrapper methods manage the lifetime of the underlying bitcoind processes and logging utilities.
+
+* `TestWrapper.setup()`
+* `TestWrapper.shutdown()`
+
+The TestWrapper inherits all BitcoinTestFramework members and methods, such as:
+* `TestWrapper.nodes[index].rpc_method()`
+* `TestWrapper.log.info("Custom log message")`
+
+The following sections demonstrate how to initialize, run and shutdown a TestWrapper object in an interactive Python3 environment.
+
+## 3. Initializing a TestWrapper object
+
+```
+>>> test = TestWrapper()
+>>> test.setup("num_nodes"=2)
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Initializing test directory /path/to/bitcoin_func_test_XXXXXXX
+```
+The TestWrapper supports all functional test parameters of the Bitcoin TestFramework class. The full set of argument keywords which can be used to initialize the TestWrapper can be found [here](../test/functional/test_framework/test_wrapper.py).
+
+**Note: Running multiple instances of TestWrapper is not allowed.**
+This also ensures that logging remains consolidated in the same temporary folder. If you need more bitcoind nodes than set by default (1), simply increase the `num_nodes` parameter during setup.
+
+```
+>>> test2 = TestWrapper()
+>>> test2.setup()
+TestWrapper is already running!
+```
+
+## 4. Interacting with the TestWrapper
+
+Unlike the BitcoinTestFramework class, the TestWrapper keeps the underlying Bitcoind subprocesses (nodes) and logging utilities running, until the user explicitly shuts down the TestWrapper object.
+
+During the time between the `setup` and `shutdown` calls, all `bitcoind` node processes and BitcoinTestFramework convenience methods can be accessed interactively.
+
+**Example: Mining a regtest chain**
+
+By default, the TestWrapper nodes are initialized with a clean chain. This means that each node has at block height 0 after initialization of the TestWrapper.
+
+```
+>>> test.nodes[0].getblockchaininfo()["blocks"]
+0
+```
+
+We now generate 101 regtest blocks, and send these to a wallet address owned by the first node.
+
+```
+>>> address = test.nodes[0].getnewaddress()
+>>> test.nodes[0].generatetoaddress(101, address)
+['2b98dd0044aae6f1cca7f88a0acf366a4bfe053c7f7b00da3c0d115f03d67efb', ...
+```
+Since the two nodes are each initialized to establish a connection to the other during `setup`, the second node will receive the newly mined blocks after they propagate.
+
+```
+>>> test.nodes[1].getblockchaininfo()["blocks"]
+101
+```
+The block rewards of the first block are now spendable by the wallet of the first node.
+
+```
+>>> test.nodes[0].getbalance()
+Decimal('50.00000000')
+```
+
+We can also log custom events to the logger.
+
+```
+>>> TestWrapper.log.info("Successfully mined regtest chain!")
+```
+
+**Note: Please also consider the functional test [readme](../test/functional/README.md), which provides an overview of the test-framework**. Modules such as [key.py](../test/functional/test_framework/key.py), [script.py](../test/functional/test_framework/script.py) and [messages.py](../test/functional/test_framework/messages.py) are especially useful in constructing objects which can be passed to the bitcoind nodes managed by a running TestWrapper object.
+
+## 5. Shutting the TestWrapper down
+
+Shutting down the TestWrapper will safely tear down all running bitcoind instances and remove all temporary data and logging directories.
+
+```
+>>> test.shutdown()
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Cleaning up /path/to/bitcoin_func_test_XXXXXXX on exit
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful
+```
+To prevent the logs from being removed after a shutdown, simply set the `TestWrapper.options.nocleanup` member to `True`.
+```
+>>> test.options.nocleanup = True
+>>> test.shutdown()
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Not cleaning up dir /path/to/bitcoin_func_test_XXXXXXX on exit
+20XX-XX-24TXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful
+```
+
+The following utility consolidates logs from the bitcoind nodes and the underlying BitcoinTestFramework:
+
+* `/path/to/bitcoin/test/functional/combine_logs.py '/path/to/bitcoin_func_test_XXXXXXX'`

test/functional/test_framework/mininode.py

@@ -480,7 +480,8 @@ def close(self, timeout=10):
         wait_until(lambda: not self.network_event_loop.is_running(), timeout=timeout)
         self.network_event_loop.close()
         self.join(timeout)
-
+        # Safe to remove event loop.
+        NetworkThread.network_event_loop = None
 
 class P2PDataStore(P2PInterface):
     """A P2P data store class.

test/functional/test_framework/test_framework.py

@@ -100,11 +100,36 @@ def __init__(self):
         self.bind_to_localhost_only = True
         self.set_test_params()
 
-        assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()"
-
     def main(self):
         """Main function. This should not be overridden by the subclass test scripts."""
 
+        assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()"
+
+        self.parse_args()
+
+        try:
+            self.setup()
+            self.run_test()
+        except BaseException as e:
+            self.success = TestStatus.FAILED
+            if isinstance(e, JSONRPCException):
+                self.log.error("JSONRPC error")
+            elif isinstance(e, SkipTest):
+                self.log.warning("Test Skipped: %s" % e.message)
+                self.success = TestStatus.SKIPPED
+            elif isinstance(e, AssertionError):
+                self.log.error("Assertion failed")
+            elif isinstance(e, KeyError):
+                self.log.error("Key error")
+            elif isinstance(e, Exception):
+                self.log.error("Unexpected exception caught during testing")
+            elif isinstance(e, KeyboardInterrupt):
+                self.log.warning("Exiting after keyboard interrupt")
+        finally:
+            exit_code = self.shutdown()
+            sys.exit(exit_code)
+
+    def parse_args(self):
         parser = argparse.ArgumentParser(usage="%(prog)s [options]")
         parser.add_argument("--nocleanup", dest="nocleanup", default=False, action="store_true",
                             help="Leave bitcoinds and test.* datadir on exit or error")
@@ -135,6 +160,9 @@ def main(self):
         self.add_options(parser)
         self.options = parser.parse_args()
 
+    def setup(self):
+        """Call this method to startup the test-framework object with options set."""
+
         PortSeed.n = self.options.port_seed
 
         check_json_precision()
@@ -181,33 +209,20 @@ def main(self):
         self.network_thread = NetworkThread()
         self.network_thread.start()
 
-        success = TestStatus.FAILED
+        if self.options.usecli:
+            if not self.supports_cli:
+                raise SkipTest("--usecli specified but test does not support using CLI")
+            self.skip_if_no_cli()
+        self.skip_test_if_missing_module()
+        self.setup_chain()
+        self.setup_network()
 
-        try:
-            if self.options.usecli:
-                if not self.supports_cli:
-                    raise SkipTest("--usecli specified but test does not support using CLI")
-                self.skip_if_no_cli()
-            self.skip_test_if_missing_module()
-            self.setup_chain()
-            self.setup_network()
-            self.run_test()
-            success = TestStatus.PASSED
-        except JSONRPCException:
-            self.log.exception("JSONRPC error")
-        except SkipTest as e:
-            self.log.warning("Test Skipped: %s" % e.message)
-            success = TestStatus.SKIPPED
-        except AssertionError:
-            self.log.exception("Assertion failed")
-        except KeyError:
-            self.log.exception("Key error")
-        except Exception:
-            self.log.exception("Unexpected exception caught during testing")
-        except KeyboardInterrupt:
-            self.log.warning("Exiting after keyboard interrupt")
-
-        if success == TestStatus.FAILED and self.options.pdbonfailure:
+        self.success = TestStatus.PASSED
+
+    def shutdown(self):
+        """Call this method to shutdown the test-framework object."""
+
+        if self.success == TestStatus.FAILED and self.options.pdbonfailure:
             print("Testcase failed. Attaching python debugger. Enter ? for help")
             pdb.set_trace()
 
@@ -225,7 +240,7 @@ def main(self):
         should_clean_up = (
             not self.options.nocleanup and
             not self.options.noshutdown and
-            success != TestStatus.FAILED and
+            self.success != TestStatus.FAILED and
             not self.options.perf
         )
         if should_clean_up:
@@ -238,20 +253,26 @@ def main(self):
             self.log.warning("Not cleaning up dir {}".format(self.options.tmpdir))
             cleanup_tree_on_exit = False
 
-        if success == TestStatus.PASSED:
+        if self.success == TestStatus.PASSED:
             self.log.info("Tests successful")
             exit_code = TEST_EXIT_PASSED
-        elif success == TestStatus.SKIPPED:
+        elif self.success == TestStatus.SKIPPED:
             self.log.info("Test skipped")
             exit_code = TEST_EXIT_SKIPPED
         else:
             self.log.error("Test failed. Test logging available at %s/test_framework.log", self.options.tmpdir)
             self.log.error("Hint: Call {} '{}' to consolidate all logs".format(os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + "/../combine_logs.py"), self.options.tmpdir))
             exit_code = TEST_EXIT_FAILED
-        logging.shutdown()
         if cleanup_tree_on_exit:
             shutil.rmtree(self.options.tmpdir)
-        sys.exit(exit_code)
+
+        for h in list(self.log.handlers):
+            h.flush()
+            h.close()
+            self.log.removeHandler(h)
+
+        self.nodes.clear()
+        return exit_code
 
     # Methods to override in subclass test scripts.
     def set_test_params(self):

test/functional/test_framework/test_wrapper.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+# Copyright (c) 2014-2019 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""Wrapper Class for BitcoinTestFramework.
+
+The TestWrapper class extends the BitcoinTestFramework
+rpc & daemon process management functionality to external
+python environments.
+
+It is a singleton class, which ensures that users only
+start a single TestWrapper at a time."""
+
+import argparse
+from os import getpid
+from os.path import abspath, join
+
+from test_framework.test_framework import BitcoinTestFramework
+
+class TestWrapper:
+
+    class __TestWrapper(BitcoinTestFramework):
+
+        def set_test_params(self):
+            pass
+
+        def run_test(self):
+            pass
+
+        def setup(self, **kwargs):
+
+            if self.running:
+                print("TestWrapper is already running!")
+                return
+
+            self.setup_clean_chain = kwargs.get('setup_clean_chain',True)
+            self.num_nodes = kwargs.get('num_nodes', 1)
+            self.network_thread = kwargs.get('network_thread', None)
+            self.rpc_timeout = kwargs.get('rpc_timeout', 60)
+            self.supports_cli = kwargs.get('supports_cli', False)
+            self.bind_to_localhost_only = kwargs.get('bind_to_localhost_only', True)
+
+            self.options = argparse.Namespace
+            self.options.nocleanup = kwargs.get('nocleanup', False)
+            self.options.noshutdown = kwargs.get('noshutdown', False)
+            self.options.cachedir = kwargs.get('cachedir', abspath(join(__file__ ,"../../../..") + "/test/cache"))
+            self.options.tmpdir = kwargs.get('tmpdir', None)
+            self.options.loglevel = kwargs.get('loglevel', 'INFO')
+            self.options.trace_rpc = kwargs.get('trace_rpc', False)
+            self.options.port_seed = kwargs.get('port_seed', getpid())
+            self.options.coveragedir = kwargs.get('coveragedir', None)
+            self.options.configfile = kwargs.get('configfile', abspath(join(__file__ ,"../../../..") + "/test/config.ini"))
+            self.options.pdbonfailure = kwargs.get('pdbonfailure', False)
+            self.options.usecli = kwargs.get('usecli', False)
+            self.options.perf = kwargs.get('perf', False)
+            self.options.randomseed = kwargs.get('randomseed', None)
+
+            self.options.bitcoind = kwargs.get('bitcoind', abspath(join(__file__ ,"../../../..") +  "/src/bitcoind"))
+            self.options.bitcoincli = kwargs.get('bitcoincli', None)
+
+            super().setup()
+            self.running = True
+
+        def shutdown(self):
+            if not self.running:
+                print("TestWrapper is not running!")
+            else:
+                super().shutdown()
+                self.running = False
+
+    instance = None
+
+    def __new__(cls):
+        if not TestWrapper.instance:
+            TestWrapper.instance = TestWrapper.__TestWrapper()
+            TestWrapper.instance.running = False
+        return TestWrapper.instance
+
+    def __getattr__(self, name):
+        return getattr(self.instance, name)
+
+    def __setattr__(self, name, value):
+        return setattr(self.instance, name, value)