Cluster Classes and Testing Implementation

[vschaffn]

Resolves GlacioHack/xdem#681.

Overview

This PR introduces a set of classes designed to handle the execution of tasks in either a basic or a multi-processing cluster. It also includes a series of tests that verify the correct behavior of the cluster functionalities.

Key changes

• ClusterGenerator: A factory class that decides which type of cluster to create based on the given configuration (basic or multiprocessing). It can initialize a BasicCluster or MpCluster with a specified number of worker processes.
• AbstractCluster: The base class for both BasicCluster and MpCluster. It defines methods to launch tasks, retrieve results, and manage resources such as opening and closing the cluster.
• BasicCluster: A subclass of AbstractCluster where tasks are run synchronously (i.e., in a single process). The launch_task method simply executes the provided function immediately.
• MpCluster: A subclass of AbstractCluster that utilizes Python’s multiprocessing module for parallel task execution. The cluster pool is initialized with a defined number of worker processes, and tasks are executed asynchronously using apply_async. The results of tasks can be retrieved via the get_res method.

Test Suite

TestClusterGenerator: A test suite implemented with pytest to validate the functionality of the cluster system.

• test_basic_cluster: Verifies that tasks are executed synchronously in the BasicCluster.
• test_mp_cluster_task: Confirms that tasks are launched asynchronously in the MpCluster.
• test_mp_cluster_parallelism: Tests that multiple tasks can be executed concurrently with multiple workers in MpCluster.
• test_mp_cluster_termination: Ensures that the cluster terminates correctly and no tasks can be launched after termination.

[rhugonnet]

Perfect, all good for me!