diff --git a/doc/api/smartsim_api.rst b/doc/api/smartsim_api.rst index 045c06b1b..745ad3990 100644 --- a/doc/api/smartsim_api.rst +++ b/doc/api/smartsim_api.rst @@ -25,6 +25,7 @@ Experiment Experiment.finished Experiment.get_status Experiment.reconnect_orchestrator + Experiment.preview Experiment.summary .. autoclass:: Experiment diff --git a/doc/changelog.rst b/doc/changelog.rst index b702d1a95..ce485ec39 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -18,6 +18,7 @@ To be released at some future point in time Description +- Preview entities on experiment before start - Change default path for entities - Drop Python 3.8 support - Update watchdog dependency @@ -41,6 +42,8 @@ Description Detailed Notes +- Added preview functionality to Experiment, including preview of all entities, + active infrastructure and client configuration. (SmartSim-PR525_) - The default path for an entity is now the path to the experiment / the entity name. create_database and create_ensemble now have path arguments. All path arguments are compatible with relative paths. Relative paths are @@ -104,6 +107,7 @@ Detailed Notes handler. SmartSim will now attempt to kill any launched jobs before calling the previously registered signal handler. (SmartSim-PR535_) +.. _SmartSim-PR525: https://github.com/CrayLabs/SmartSim/pull/525 .. _SmartSim-PR533: https://github.com/CrayLabs/SmartSim/pull/533 .. _SmartSim-PR544: https://github.com/CrayLabs/SmartSim/pull/544 .. _SmartSim-PR540: https://github.com/CrayLabs/SmartSim/pull/540 diff --git a/doc/experiment.rst b/doc/experiment.rst index 73ba08812..3a26b5b68 100644 --- a/doc/experiment.rst +++ b/doc/experiment.rst @@ -126,6 +126,9 @@ the ``Experiment`` post-creation methods. * - ``get_status`` - ``exp.get_status(*args)`` - Retrieve Entity Status + * - ``preview`` + - ``exp.preview(*args, ...)`` + - Preview an experiment .. _orchestrator_exp_docs: @@ -329,6 +332,9 @@ Example *Generating* - the ``Orchestrator`` output directory - the ``Model`` output directory + *Preview* + - the ``Orchestrator`` contents + - the ``Model`` contents *Starting* - an in-memory database (standalone ``Orchestrator``) - an application (``Model``) @@ -418,6 +424,68 @@ Generating The ``Experiment.generate`` call places the `.err` and `.out` log files in the entity subdirectories within the main ``Experiment`` directory. +Preview +======= +.. compound:: + Optionally, we preview the Experiment. The ``Experiment.preview`` aggregates multiple pieces of information to give users + insight into what and how entities will be launched before the experiment is started. Any instance of ``Model``, ``Ensemble``, + or ``Orchestrator`` created by the Experiment can be passed as an argument to the preview method. + We preview the ``Orchestrator`` and ``Model`` entities by passing the ``Orchestrator`` and ``Model`` instances to ``exp.preview``: + + .. literalinclude:: tutorials/doc_examples/experiment_doc_examples/exp.py + :language: python + :linenos: + :lines: 20-21 + + * `verbosity_level="info"` instructs preview to display user-defined fields and entities. + * `verbosity_level="debug"` instructs preview to display user-defined field and entities and auto-generated fields. + * `verbosity_level="developer"` instructs preview to display user-defined field and entities, auto-generated fields, and run commands. + * `output_format="plain_text"` sets the output format. The only accepted output format is 'plain_text'. + * `output_filename="test_name.txt"` specifies name of file and extension to write preview data to. If no output filename is set, the preview will be output to stdout. + +When executed, the preview shows the following in stdout: + +:: + + === Experiment Overview === + + Experiment Name: example-experiment + Experiment Path: absolute/path/to/SmartSim/example-experiment + Launcher: local + + === Entity Preview === + + == Orchestrators == + + = Database Identifier: orchestrator = + Path: absolute/path/to/SmartSim/example-experiment/orchestrator + Shards: 1 + TCP/IP Port(s): + 6379 + Network Interface: ib0 + Type: redis + Executable: absolute/path/to/SmartSim/smartsim/_core/bin/redis-server + + == Models == + + = Model Name: hello_world = + Path: absolute/path/to/SmartSim/example-experiment/hello_world + Executable: /bin/echo + Executable Arguments: + Hello + World + Client Configuration: + Database Identifier: orchestrator + Database Backend: redis + TCP/IP Port(s): + 6379 + Type: Standalone + Outgoing Key Collision Prevention (Key Prefixing): + Tensors: Off + Datasets: Off + ML Models/Torch Scripts: Off + Aggregation Lists: Off + Starting ======== .. compound:: @@ -428,7 +496,7 @@ Starting .. literalinclude:: tutorials/doc_examples/experiment_doc_examples/exp.py :language: python :linenos: - :lines: 20-21 + :lines: 23-24 Stopping ======== @@ -439,7 +507,7 @@ Stopping .. literalinclude:: tutorials/doc_examples/experiment_doc_examples/exp.py :language: python :linenos: - :lines: 23-26 + :lines: 26-27 Notice that we use the ``Experiment.summary`` function to print the summary of the workflow. @@ -454,4 +522,4 @@ When you run the experiment, the following output will appear:: .. note:: Failure to tear down the ``Orchestrator`` at the end of an ``Experiment`` may lead to ``Orchestrator`` launch failures if another ``Experiment`` is - started on the same node. \ No newline at end of file + started on the same node. diff --git a/doc/tutorials/doc_examples/experiment_doc_examples/exp.py b/doc/tutorials/doc_examples/experiment_doc_examples/exp.py index 7a36262be..611cdabe9 100644 --- a/doc/tutorials/doc_examples/experiment_doc_examples/exp.py +++ b/doc/tutorials/doc_examples/experiment_doc_examples/exp.py @@ -17,10 +17,13 @@ # Generate the output directory exp.generate(standalone_database, model, overwrite=True) +# Preview the experiment +exp.preview(standalone_database, model, verbosity_level="debug") + # Launch the Orchestrator then Model instance exp.start(standalone_database, model) # Clobber the Orchestrator exp.stop(standalone_database) # Log the summary of the Experiment -smartsim_logger.info(exp.summary()) \ No newline at end of file +smartsim_logger.info(exp.summary()) diff --git a/smartsim/_core/control/controller.py b/smartsim/_core/control/controller.py index a1fb044b6..83402ea97 100644 --- a/smartsim/_core/control/controller.py +++ b/smartsim/_core/control/controller.py @@ -145,7 +145,8 @@ def start( self.poll(5, True, kill_on_interrupt=kill_on_interrupt) @property - def active_orch_dict(self) -> t.Dict[str, Job]: + def active_orchestrator_jobs(self) -> t.Dict[str, Job]: + """Return active orchestrator jobs.""" return {**self._jobs.db_jobs} @property diff --git a/smartsim/_core/control/previewrenderer.py b/smartsim/_core/control/previewrenderer.py index c32fafe56..18568b1cd 100644 --- a/smartsim/_core/control/previewrenderer.py +++ b/smartsim/_core/control/previewrenderer.py @@ -57,11 +57,15 @@ class Verbosity(str, Enum): @pass_eval_context def as_toggle(_eval_ctx: u.F, value: bool) -> str: + """Return "On" if value returns True, + and "Off" is value returns False. + """ return "On" if value else "Off" @pass_eval_context def get_ifname(_eval_ctx: u.F, value: t.List[str]) -> str: + """Extract Network Interface from orchestrator run settings.""" if value: for val in value: if "ifname=" in val: @@ -72,6 +76,7 @@ def get_ifname(_eval_ctx: u.F, value: t.List[str]) -> str: @pass_eval_context def get_dbtype(_eval_ctx: u.F, value: str) -> str: + """Extract data base type.""" if value: if "-cli" in value: db_type, _ = value.split("/")[-1].split("-", 1) @@ -81,14 +86,34 @@ def get_dbtype(_eval_ctx: u.F, value: str) -> str: @pass_eval_context def is_list(_eval_ctx: u.F, value: str) -> bool: + """Return True if item is of type list, and False + otherwise, to determine how Jinja template should + render an item. + """ return isinstance(value, list) +def render_to_file(content: str, filename: str) -> None: + """Output preview to a file if an output filename + is specified. + + :param content: The rendered preview. + :type content: str + :param filename: The name of the file to write the preview to. + "type filename: str + """ + filename = find_available_filename(filename) + + with open(filename, "w", encoding="utf-8") as prev_file: + prev_file.write(content) + + def render( exp: "Experiment", manifest: t.Optional[Manifest] = None, verbosity_level: Verbosity = Verbosity.INFO, output_format: Format = Format.PLAINTEXT, + output_filename: t.Optional[str] = None, active_dbjobs: t.Optional[t.Dict[str, Job]] = None, ) -> str: """ @@ -103,9 +128,11 @@ def render( :type output_format: _OutputFormatString """ - verbosity_level = _check_verbosity_level(Verbosity(verbosity_level)) + verbosity_level = Verbosity(verbosity_level) + + _check_output_format(output_format) - loader = jinja2.PackageLoader("templates") + loader = jinja2.PackageLoader("smartsim.templates") env = jinja2.Environment(loader=loader, autoescape=True) env.filters["as_toggle"] = as_toggle @@ -114,17 +141,14 @@ def render( env.filters["is_list"] = is_list env.globals["Verbosity"] = Verbosity - version = f"_{output_format}" - tpl_path = f"preview/base{version}.template" - - _check_output_format(output_format) + tpl_path = f"preview/{output_format.value}/base.template" tpl = env.get_template(tpl_path) if verbosity_level == Verbosity.INFO: logger.warning( - "Only showing user set parameters. Some entity fields are " - "truncated. To view truncated fields: use verbosity_level " + "Only showing user set parameters. Some internal entity " + "fields are truncated. To view truncated fields: use verbosity_level " "'developer' or 'debug.'" ) @@ -135,18 +159,15 @@ def render( config=CONFIG, verbosity_level=verbosity_level, ) - return rendered_preview - -def preview_to_file(content: str, filename: str) -> None: - """ - Output preview to a file if an output filename - is specified. - """ - filename = find_available_filename(filename) - - with open(filename, "w", encoding="utf-8") as prev_file: - prev_file.write(content) + if output_filename: + render_to_file( + rendered_preview, + output_filename, + ) + else: + logger.info(rendered_preview) + return rendered_preview def find_available_filename(filename: str) -> str: @@ -168,20 +189,5 @@ def _check_output_format(output_format: Format) -> None: Check that a valid file output format is given. """ if not output_format == Format.PLAINTEXT: - raise PreviewFormatError( - f"The only valid output format currently available is {Format.PLAINTEXT}" - ) - - -def _check_verbosity_level( - verbosity_level: Verbosity, -) -> Verbosity: - """ - Check that the given verbosity level is valid. - """ - if not isinstance(verbosity_level, Verbosity): - - logger.warning(f"'{verbosity_level}' is an unsupported verbosity level.\ - Setting verbosity to: {Verbosity.INFO}") - return Verbosity.INFO - return verbosity_level + raise PreviewFormatError(f"The only valid output format currently available \ +is {Format.PLAINTEXT.value}") diff --git a/smartsim/entity/dbobject.py b/smartsim/entity/dbobject.py index 63f6486bc..ff18da1cd 100644 --- a/smartsim/entity/dbobject.py +++ b/smartsim/entity/dbobject.py @@ -197,8 +197,6 @@ def __init__( if not script and not script_path: raise ValueError("Either script or script_path must be provided") - self.script_path = script_path - @property def script(self) -> t.Optional[str]: return self.func @@ -278,7 +276,6 @@ def __init__( self.min_batch_timeout = min_batch_timeout self.tag = tag self.inputs, self.outputs = self._check_tensor_args(inputs, outputs) - self.model_file = model_file @property def model(self) -> t.Optional[bytes]: diff --git a/smartsim/entity/ensemble.py b/smartsim/entity/ensemble.py index fa757ae49..df244072e 100644 --- a/smartsim/entity/ensemble.py +++ b/smartsim/entity/ensemble.py @@ -103,6 +103,7 @@ def __init__( self._key_prefixing_enabled = True self.batch_settings = init_default({}, batch_settings, BatchSettings) self.run_settings = init_default({}, run_settings, RunSettings) + self.replicas: str super().__init__(name, str(path), perm_strat=perm_strat, **kwargs) @@ -120,6 +121,7 @@ def _initialize_entities(self, **kwargs: t.Any) -> None: """ strategy = self._set_strategy(kwargs.pop("perm_strat")) replicas = kwargs.pop("replicas", None) + self.replicas = replicas # if a ensemble has parameters and run settings, create # the ensemble and assign run_settings to each member diff --git a/smartsim/error/errors.py b/smartsim/error/errors.py index 1580b35dd..6493d42a8 100644 --- a/smartsim/error/errors.py +++ b/smartsim/error/errors.py @@ -151,8 +151,8 @@ class SmartSimCLIActionCancelled(SmartSimError): """Raised when a `smart` CLI command is terminated""" -class PreviewException(Exception): - """Raised when a part of preview isn't support by SmartSim yet""" +class PreviewException(SSUnsupportedError): + """Raised when a part of preview isn't supported by SmartSim yet""" class PreviewFormatError(PreviewException): diff --git a/smartsim/experiment.py b/smartsim/experiment.py index e810dc3f4..1a952245b 100644 --- a/smartsim/experiment.py +++ b/smartsim/experiment.py @@ -24,6 +24,8 @@ # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# pylint: disable=too-many-lines + import os import os.path as osp import typing as t @@ -868,8 +870,8 @@ def reconnect_orchestrator(self, checkpoint: str) -> Orchestrator: def preview( self, *args: t.Any, - output_format: previewrenderer.Format = previewrenderer.Format.PLAINTEXT, verbosity_level: previewrenderer.Verbosity = previewrenderer.Verbosity.INFO, + output_format: previewrenderer.Format = previewrenderer.Format.PLAINTEXT, output_filename: t.Optional[str] = None, ) -> None: """Preview entity information prior to launch. This method @@ -877,40 +879,39 @@ def preview( into what and how entities will be launched. Any instance of ``Model``, ``Ensemble``, or ``Orchestrator`` created by the Experiment can be passed as an argument to the preview method. - :param output_filename: Specify name of file and extension to write - preview data to. If no output filename is set, the preview will be - output to stdout. Defaults to None. - :type output_filename: str - :param output_format: Set output format. The possible accepted - output formats are `json`, `xml`, `html`, `plain_text`, `color_text`. - Defaults to 'plain_text'. - :type output_type: str - :param verbosity_level: Specify the verbosity level: - info: Display User defined fields and entities - debug: Display user defined field and entities and auto generated + + Verbosity levels: + - info: Display user-defined fields and entities + - debug: Display user-defined field and entities and auto-generated fields. - developer: Display user defined field and entities, auto generated + - developer: Display user-defined field and entities, auto-generated fields, and run commands. - Defaults to info. + + :param verbosity_level: verbosity level specified by user, defaults to info. :type verbosity_level: str + :param output_format: Set output format. The possible accepted + output formats are 'plain_text'. + Defaults to 'plain_text'. + :type output_format: str + :param output_filename: Specify name of file and extension to write + preview data to. If no output filename is set, the preview will be + output to stdout. Defaults to None. + :type output_filename: str """ - # Retrive any active db jobs - active_dbjobs = self._control.active_orch_dict + # Retrieve any active orchestrator jobs + active_dbjobs = self._control.active_orchestrator_jobs preview_manifest = Manifest(*args) - rendered_preview = previewrenderer.render( + previewrenderer.render( self, preview_manifest, verbosity_level, output_format, + output_filename, active_dbjobs, ) - if output_filename: - previewrenderer.preview_to_file(rendered_preview, output_filename) - else: - logger.info(rendered_preview) @property def launcher(self) -> str: diff --git a/templates/templates/preview/activeinfra_plain_text.template b/smartsim/templates/templates/preview/plain_text/activeinfra.template similarity index 92% rename from templates/templates/preview/activeinfra_plain_text.template rename to smartsim/templates/templates/preview/plain_text/activeinfra.template index 48a6ec008..8f403fbc0 100644 --- a/templates/templates/preview/activeinfra_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/activeinfra.template @@ -1,7 +1,7 @@ = Database Identifier: {{ db.entity.db_identifier }} = Shards: {{ db.entity.num_shards }} - TCP/IP Port(s): + TCP/IP Port(s): {%- for port in db.entity.ports %} {{ port }} {%- endfor %} diff --git a/templates/templates/preview/base_plain_text.template b/smartsim/templates/templates/preview/plain_text/base.template similarity index 70% rename from templates/templates/preview/base_plain_text.template rename to smartsim/templates/templates/preview/plain_text/base.template index db5ebb39f..a77ca0b97 100644 --- a/templates/templates/preview/base_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/base.template @@ -1,5 +1,5 @@ -{% include "preview/experiment_plain_text.template" %} +{% include "preview/plain_text/experiment.template" %} {%- if manifest.has_deployable or active_dbjobs %} === Entity Preview === @@ -8,7 +8,7 @@ == Active Infrastructure == {%- for name, db in active_dbjobs.items() %} - {% include "preview/activeinfra_plain_text.template" %} + {% include "preview/plain_text/activeinfra.template" %} {%- endfor %} {%- endif %} {%- if manifest.dbs %} @@ -18,7 +18,7 @@ {%- if db.is_active() %} WARNING: Cannot preview {{ db.name }}, because it is already started. {%- else %} - {% include "preview/orchestrator_plain_text.template" %} + {% include "preview/plain_text/orchestrator.template" %} {%- endif %} {%- endfor %} {%- endif %} @@ -28,15 +28,14 @@ {%- for model in manifest.models %} = Model Name: {{ model.name }} = - {%- include "preview/model_plain_text.template" %} + {%- include "preview/plain_text/model.template" %} {%- if model.run_settings.colocated_db_settings or manifest.dbs %} - Client Configuration: {%- if model.run_settings.colocated_db_settings %} - {%- include "preview/clientconfigcolo_plain_text.template" %} + {%- include "preview/plain_text/clientconfigcolo.template" %} {%- endif %} {%- if manifest.dbs %} - {%- include "preview/clientconfig_plain_text.template" %} + {%- include "preview/plain_text/clientconfig.template" %} {%- endif %} {%- endif %} {%- endfor %} @@ -45,9 +44,8 @@ {%- if manifest.ensembles %} == Ensembles == - {%- for ensemble in manifest.ensembles %} - {% include "preview/ensemble_plain_text.template" %} + {%- include "preview/plain_text/ensemble.template" %} {%- endfor %} {%- endif %} diff --git a/templates/templates/preview/orchestrator_plain_text.template b/smartsim/templates/templates/preview/plain_text/clientconfig.template similarity index 55% rename from templates/templates/preview/orchestrator_plain_text.template rename to smartsim/templates/templates/preview/plain_text/clientconfig.template index 41c4056b7..a157a9e92 100644 --- a/templates/templates/preview/orchestrator_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/clientconfig.template @@ -1,7 +1,7 @@ {%- if verbosity_level == Verbosity.INFO %} -{%- include "preview/orchestrator_plain_text_info.template" -%} +{%- include "preview/plain_text/clientconfig_info.template" -%} {%- endif %} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} -{%- include "preview/orchestrator_plain_text_debug.template" -%} +{%- include "preview/plain_text/clientconfig_debug.template" -%} {%- endif %} diff --git a/templates/templates/preview/clientconfig_plain_text_debug.template b/smartsim/templates/templates/preview/plain_text/clientconfig_debug.template similarity index 91% rename from templates/templates/preview/clientconfig_plain_text_debug.template rename to smartsim/templates/templates/preview/plain_text/clientconfig_debug.template index 191ed9c08..51dafd0d1 100644 --- a/templates/templates/preview/clientconfig_plain_text_debug.template +++ b/smartsim/templates/templates/preview/plain_text/clientconfig_debug.template @@ -9,12 +9,8 @@ {%- for port in db.ports %} {{ port }} {%- endfor %} - {%- if db.num_shards > 2 %} - Type: Clustered - {%- else %} Type: Standalone {%- endif %} - {%- endif %} {%- endfor %} {%- if model.incoming_entities %} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} diff --git a/templates/templates/preview/clientconfig_plain_text_info.template b/smartsim/templates/templates/preview/plain_text/clientconfig_info.template similarity index 87% rename from templates/templates/preview/clientconfig_plain_text_info.template rename to smartsim/templates/templates/preview/plain_text/clientconfig_info.template index 5f71a9958..164f4bd4a 100644 --- a/templates/templates/preview/clientconfig_plain_text_info.template +++ b/smartsim/templates/templates/preview/plain_text/clientconfig_info.template @@ -8,11 +8,7 @@ {%- for port in db.ports %} {{ port }} {%- endfor %} - {%- if db.num_shards > 2 %} - Type: Clustered - {%- else %} Type: Standalone - {%- endif %} {%- endfor %} {%- if model.query_key_prefixing() %} Outgoing Key Collision Prevention (Key Prefixing): diff --git a/templates/templates/preview/clientconfigcolo_plain_text.template b/smartsim/templates/templates/preview/plain_text/clientconfigcolo.template similarity index 56% rename from templates/templates/preview/clientconfigcolo_plain_text.template rename to smartsim/templates/templates/preview/plain_text/clientconfigcolo.template index f203965d8..2c2630d4b 100644 --- a/templates/templates/preview/clientconfigcolo_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/clientconfigcolo.template @@ -1,7 +1,7 @@ {%- if verbosity_level == Verbosity.INFO %} -{%- include "preview/clientconfigcolo_plain_text_info.template" %} +{%- include "preview/plain_text/clientconfigcolo_info.template" %} {% endif %} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} -{%- include "preview/clientconfigcolo_plain_text_debug.template" %} +{%- include "preview/plain_text/clientconfigcolo_debug.template" %} {%- endif %} diff --git a/templates/templates/preview/clientconfigcolo_plain_text_debug.template b/smartsim/templates/templates/preview/plain_text/clientconfigcolo_debug.template similarity index 100% rename from templates/templates/preview/clientconfigcolo_plain_text_debug.template rename to smartsim/templates/templates/preview/plain_text/clientconfigcolo_debug.template diff --git a/templates/templates/preview/clientconfigcolo_plain_text_info.template b/smartsim/templates/templates/preview/plain_text/clientconfigcolo_info.template similarity index 100% rename from templates/templates/preview/clientconfigcolo_plain_text_info.template rename to smartsim/templates/templates/preview/plain_text/clientconfigcolo_info.template diff --git a/templates/templates/preview/ensemble_plain_text.template b/smartsim/templates/templates/preview/plain_text/ensemble.template similarity index 56% rename from templates/templates/preview/ensemble_plain_text.template rename to smartsim/templates/templates/preview/plain_text/ensemble.template index bc4545cb3..10840a8c2 100644 --- a/templates/templates/preview/ensemble_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/ensemble.template @@ -1,7 +1,7 @@ {%- if verbosity_level == Verbosity.INFO %} -{%- include "preview/ensemble_plain_text_info.template" -%} +{%- include "preview/plain_text/ensemble_info.template" -%} {%- endif %} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} -{%- include "preview/ensemble_plain_text_debug.template" -%} +{%- include "preview/plain_text/ensemble_debug.template" -%} {%- endif %} diff --git a/templates/templates/preview/ensemble_plain_text_debug.template b/smartsim/templates/templates/preview/plain_text/ensemble_debug.template similarity index 67% rename from templates/templates/preview/ensemble_plain_text_debug.template rename to smartsim/templates/templates/preview/plain_text/ensemble_debug.template index 6b417fe56..3c87a20b2 100644 --- a/templates/templates/preview/ensemble_plain_text_debug.template +++ b/smartsim/templates/templates/preview/plain_text/ensemble_debug.template @@ -1,12 +1,20 @@ - {%- for ensemble in manifest.ensembles %} + {% for ensemble in manifest.ensembles %} = Ensemble Name: {{ ensemble.name }} = + {%- if ensemble.path %} + Path: {{ ensemble.path }} + {%- endif %} Members: {{ ensemble|length }} {%- if ensemble.params %} Ensemble Parameters: {%- for key, value in ensemble.params.items() %} {{ key }}: {{ value | join(", ") | wordwrap(150) | safe | replace('\n', '\n ') }} - {%- endfor %} + {%- endfor %} + {%- endif %} + {%- if ensemble.replicas %} + Replicas: {{ ensemble.replicas }} + {%- elif ensemble.perm_strat %} + Permutation Strategy: {{ ensemble.perm_strat }} {%- endif %} {%- if ensemble.batch_settings %} Batch Launch: True @@ -23,34 +31,32 @@ {%- for model in ensemble.entities %} - Model Name: {{ model.name }} - - {%- include 'preview/model_plain_text.template' %} + {%- include 'preview/plain_text/model.template' %} {%- if model.run_settings.colocated_db_settings or manifest.dbs %} - Client Configuration: {%- if model.run_settings.colocated_db_settings %} - {%- include "preview/clientconfigcolo_plain_text.template" %} + {%- include "preview/plain_text/clientconfigcolo.template" %} {%- endif %} {%- if manifest.dbs %} - {%- include "preview/clientconfig_plain_text.template" %} + {%- include "preview/plain_text/clientconfig.template" %} {%- endif %} {%- endif %} - {%- endfor %} + {%- endfor %} {%- endif %} {%- if verbosity_level == Verbosity.DEVELOPER %} {%- for model in ensemble.entities %} - Model Name: {{ model.name }} - - {%- include 'preview/model_plain_text_debug.template' %} + {%- include 'preview/plain_text/model_debug.template' %} {%- if model.run_settings.colocated_db_settings or manifest.dbs %} - Client Configuration: {%- if model.run_settings.colocated_db_settings %} - {%- include "preview/clientconfigcolo_plain_text.template" %} + {%- include "preview/plain_text/clientconfigcolo.template" %} {%- endif %} {%- if manifest.dbs %} - {%- include "preview/clientconfig_plain_text.template" %} + {%- include "preview/plain_text/clientconfig.template" %} {%- endif %} {%- endif %} {%- endfor %} {%- endif %} -{%- endfor %} +{% endfor %} diff --git a/templates/templates/preview/ensemble_plain_text_info.template b/smartsim/templates/templates/preview/plain_text/ensemble_info.template similarity index 75% rename from templates/templates/preview/ensemble_plain_text_info.template rename to smartsim/templates/templates/preview/plain_text/ensemble_info.template index 1113bfcf1..2b501d704 100644 --- a/templates/templates/preview/ensemble_plain_text_info.template +++ b/smartsim/templates/templates/preview/plain_text/ensemble_info.template @@ -1,4 +1,3 @@ - = Ensemble Name: {{ ensemble.name }} = Members: {{ ensemble|length }} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} @@ -12,41 +11,40 @@ {%- if ensemble.models | length > 2 %} {% set model = ensemble.models[0] %} - Model Name: {{ model.name }} - - {%- include 'preview/model_plain_text.template' %} + {%- include 'preview/plain_text/model.template' %} {%- if model.run_settings.colocated_db_settings or manifest.dbs %} - Client Configuration: {%- if model.run_settings.colocated_db_settings %} - {%- include "preview/clientconfigcolo_plain_text.template" %} + {%- include "preview/plain_text/clientconfigcolo.template" %} {%- endif %} {%- if manifest.dbs %} - {%- include "preview/clientconfig_plain_text.template" %} + {%- include "preview/plain_text/clientconfig.template" %} {%- endif %} {%- endif %} ... {% set model = ensemble.models[(ensemble.models | length)-1] %} - Model Name: {{ model.name }} - - {%- include 'preview/model_plain_text.template' %} + {%- include 'preview/plain_text/model.template' %} {% if model.run_settings.colocated_db_settings or manifest.dbs %} Client Configuration: {%- if model.run_settings.colocated_db_settings %} - {%- include "preview/clientconfigcolo_plain_text.template" %} + {%- include "preview/plain_text/clientconfigcolo.template" %} {%- endif %} {%- if manifest.dbs %} - {%- include "preview/clientconfig_plain_text.template" %} + {%- include "preview/plain_text/clientconfig.template" %} {%- endif %} {%- endif %} {%- else %} {% for model in ensemble %} - Model Name: {{ model.name }} - - {%- include 'preview/model_plain_text.template' %} + {%- include 'preview/plain_text/model.template' %} {% if model.run_settings.colocated_db_settings or manifest.dbs %} Client Configuration: {%- if model.run_settings.colocated_db_settings %} - {%- include "preview/clientconfigcolo_plain_text.template" %} + {%- include "preview/plain_text/clientconfigcolo.template" %} {%- endif %} {%- if manifest.dbs %} - {%- include "preview/clientconfig_plain_text.template" %} + {%- include "preview/plain_text/clientconfig.template" %} {%- endif %} {%- endif %} {% endfor %} diff --git a/templates/templates/preview/experiment_plain_text.template b/smartsim/templates/templates/preview/plain_text/experiment.template similarity index 100% rename from templates/templates/preview/experiment_plain_text.template rename to smartsim/templates/templates/preview/plain_text/experiment.template diff --git a/templates/templates/preview/model_plain_text.template b/smartsim/templates/templates/preview/plain_text/model.template similarity index 58% rename from templates/templates/preview/model_plain_text.template rename to smartsim/templates/templates/preview/plain_text/model.template index 32d9c2e0b..77b595486 100644 --- a/templates/templates/preview/model_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/model.template @@ -1,7 +1,7 @@ {%- if verbosity_level == Verbosity.INFO %} -{%- include "preview/model_plain_text_info.template" -%} +{%- include "preview/plain_text/model_info.template" -%} {%- endif %} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} -{%- include "preview/model_plain_text_debug.template" -%} +{%- include "preview/plain_text/model_debug.template" -%} {%- endif -%} diff --git a/templates/templates/preview/model_plain_text_debug.template b/smartsim/templates/templates/preview/plain_text/model_debug.template similarity index 84% rename from templates/templates/preview/model_plain_text_debug.template rename to smartsim/templates/templates/preview/plain_text/model_debug.template index 3d50431d2..186746186 100644 --- a/templates/templates/preview/model_plain_text_debug.template +++ b/smartsim/templates/templates/preview/plain_text/model_debug.template @@ -1,5 +1,8 @@ {%- if model is defined %} + {%- if model.path %} + Path: {{ model.path }} + {%- endif %} Executable: {{ model.run_settings.exe[0] }} Executable Arguments: {%- for param in model.run_settings.exe_args %} @@ -66,25 +69,29 @@ Unix Socket: {{ model.run_settings.colocated_db_settings.unix_socket }} {%- endif %} {%- if model.run_settings.colocated_db_settings.ifname %} + {%- if model.run_settings.colocated_db_settings.ifname | is_list %} Network Interface Name: {{ model.run_settings.colocated_db_settings.ifname[0] }} + {%- else %} + Network Interface Name: {{ model.run_settings.colocated_db_settings.ifname }} + {%- endif %} {%- endif %} CPUs: {{ model.run_settings.colocated_db_settings.cpus }} Custom Pinning: {{ model.run_settings.colocated_db_settings.custom_pinning }} {%- endif %} - {%- if model.run_settings.colocated_db_settings['db_scripts'] %} + {%- if model._db_scripts %} Torch Scripts: - {%- for script in model.run_settings.colocated_db_settings['db_scripts'] %} + {%- for script in model._db_scripts%} Name: {{ script.name }} - Path: {{ script.script_path }} + Path: {{ script.file }} Backend: {{ script.device }} Devices Per Node: {{ script.devices_per_node }} {%- endfor %} {%- endif %} - {%- if model.run_settings.colocated_db_settings['db_models'] %} + {%- if model._db_models %} ML Models: - {%- for mlmodel in model.run_settings.colocated_db_settings['db_models'] %} + {%- for mlmodel in model._db_models %} Name: {{ mlmodel.name }} - Path: {{ mlmodel.model_file }} + Path: {{ mlmodel.file }} Backend: {{ mlmodel.backend }} Device: {{ mlmodel.device }} Devices Per Node: {{ mlmodel.devices_per_node }} @@ -99,9 +106,9 @@ Outputs: {{ output }} {%- endfor %} - {%- endfor %} - {%- endif %} - {%- if model.query_key_prefixing()%} + {%- endfor %} + {%- endif %} + {%- if model.query_key_prefixing()%} Key Prefix: {{ model.name }} - {%- endif %} + {%- endif %} {%- endif %} diff --git a/templates/templates/preview/model_plain_text_info.template b/smartsim/templates/templates/preview/plain_text/model_info.template similarity index 100% rename from templates/templates/preview/model_plain_text_info.template rename to smartsim/templates/templates/preview/plain_text/model_info.template diff --git a/templates/templates/preview/clientconfig_plain_text.template b/smartsim/templates/templates/preview/plain_text/orchestrator.template similarity index 55% rename from templates/templates/preview/clientconfig_plain_text.template rename to smartsim/templates/templates/preview/plain_text/orchestrator.template index b454d2cc8..7965790e6 100644 --- a/templates/templates/preview/clientconfig_plain_text.template +++ b/smartsim/templates/templates/preview/plain_text/orchestrator.template @@ -1,7 +1,7 @@ {%- if verbosity_level == Verbosity.INFO %} -{%- include "preview/clientconfig_plain_text_info.template" -%} +{%- include "preview/plain_text/orchestrator_info.template" -%} {%- endif %} {%- if verbosity_level == Verbosity.DEBUG or verbosity_level == Verbosity.DEVELOPER %} -{%- include "preview/clientconfig_plain_text_debug.template" -%} +{%- include "preview/plain_text/orchestrator_debug.template" -%} {%- endif %} diff --git a/templates/templates/preview/orchestrator_plain_text_debug.template b/smartsim/templates/templates/preview/plain_text/orchestrator_debug.template similarity index 93% rename from templates/templates/preview/orchestrator_plain_text_debug.template rename to smartsim/templates/templates/preview/plain_text/orchestrator_debug.template index 94943af16..127a4949e 100644 --- a/templates/templates/preview/orchestrator_plain_text_debug.template +++ b/smartsim/templates/templates/preview/plain_text/orchestrator_debug.template @@ -1,5 +1,8 @@ = Database Identifier: {{ db.name }} = + {%- if db.path %} + Path: {{ db.path }} + {%- endif %} Shards: {{ db.num_shards }} TCP/IP Port(s): {%- for port in db.ports %} diff --git a/templates/templates/preview/orchestrator_plain_text_info.template b/smartsim/templates/templates/preview/plain_text/orchestrator_info.template similarity index 100% rename from templates/templates/preview/orchestrator_plain_text_info.template rename to smartsim/templates/templates/preview/plain_text/orchestrator_info.template diff --git a/smartsim_db.dat b/smartsim_db.dat deleted file mode 100644 index 1bc387357..000000000 Binary files a/smartsim_db.dat and /dev/null differ diff --git a/tests/on_wlm/test_preview_wlm.py b/tests/on_wlm/test_preview_wlm.py new file mode 100644 index 000000000..72fc7564c --- /dev/null +++ b/tests/on_wlm/test_preview_wlm.py @@ -0,0 +1,407 @@ +# BSD 2-Clause License +# +# Copyright (c) 2021-2023, Hewlett Packard Enterprise +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# 1. Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from os import path as osp + +import numpy as np +import pytest +from jinja2.filters import FILTERS + +from smartsim import Experiment +from smartsim._core import Manifest, previewrenderer +from smartsim._core.config import CONFIG +from smartsim.database import Orchestrator +from smartsim.settings import QsubBatchSettings, RunSettings + +pytestmark = pytest.mark.slow_tests + +on_wlm = (pytest.test_launcher in pytest.wlm_options,) + + +@pytest.fixture +def choose_host(): + def _choose_host(wlmutils, index: int = 0): + hosts = wlmutils.get_test_hostlist() + if hosts: + return hosts[index] + return None + + return _choose_host + + +def add_batch_resources(wlmutils, batch_settings): + if isinstance(batch_settings, QsubBatchSettings): + for key, value in wlmutils.get_batch_resources().items(): + batch_settings.set_resource(key, value) + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_wlm_run_commands_cluster_orc_model( + test_dir, coloutils, fileutils, wlmutils +): + """ + Test preview of wlm run command and run aruguments on a + orchestrator and model + """ + + exp_name = "test-preview-orc-model" + launcher = wlmutils.get_test_launcher() + test_port = wlmutils.get_test_port() + test_script = fileutils.get_test_conf_path("smartredis/multidbid.py") + exp = Experiment(exp_name, launcher=launcher, exp_path=test_dir) + + network_interface = wlmutils.get_test_interface() + orc = exp.create_database( + wlmutils.get_test_port(), + db_nodes=3, + batch=False, + interface=network_interface, + single_cmd=True, + hosts=wlmutils.get_test_hostlist(), + db_identifier="testdb_reg", + ) + + db_args = { + "port": test_port, + "db_cpus": 1, + "debug": True, + "db_identifier": "testdb_colo", + } + + # Create model with colocated database + smartsim_model = coloutils.setup_test_colo( + fileutils, "uds", exp, test_script, db_args, on_wlm=on_wlm + ) + + preview_manifest = Manifest(orc, smartsim_model) + + # Execute preview method + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + # Evaluate output + assert "Run Command" in output + assert "Run Arguments" in output + assert "ntasks" in output + assert "nodes" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_model_on_wlm(fileutils, test_dir, wlmutils): + """ + Test preview of wlm run command and run aruguments for a model + """ + exp_name = "test-preview-model-wlm" + exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) + + script = fileutils.get_test_conf_path("sleep.py") + settings1 = wlmutils.get_base_run_settings("python", f"{script} --time=5") + settings2 = wlmutils.get_base_run_settings("python", f"{script} --time=5") + M1 = exp.create_model("m1", path=test_dir, run_settings=settings1) + M2 = exp.create_model("m2", path=test_dir, run_settings=settings2) + + preview_manifest = Manifest(M1, M2) + + # Execute preview method + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + assert "Run Command" in output + assert "Run Arguments" in output + assert "nodes" in output + assert "ntasks" in output + assert "time" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_batch_model(fileutils, test_dir, wlmutils): + """Test the preview of a model with batch settings""" + + exp_name = "test-batch-model" + exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) + + script = fileutils.get_test_conf_path("sleep.py") + batch_settings = exp.create_batch_settings(nodes=1, time="00:01:00") + + batch_settings.set_account(wlmutils.get_test_account()) + add_batch_resources(wlmutils, batch_settings) + run_settings = wlmutils.get_run_settings("python", f"{script} --time=5") + model = exp.create_model( + "model", path=test_dir, run_settings=run_settings, batch_settings=batch_settings + ) + model.set_path(test_dir) + + preview_manifest = Manifest(model) + + # Execute preview method + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + assert "Batch Launch: True" in output + assert "Batch Command" in output + assert "Batch Arguments" in output + assert "nodes" in output + assert "time" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_batch_ensemble(fileutils, test_dir, wlmutils): + """Test preview of a batch ensemble""" + + exp_name = "test-preview-batch-ensemble" + exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) + + script = fileutils.get_test_conf_path("sleep.py") + settings = wlmutils.get_run_settings("python", f"{script} --time=5") + M1 = exp.create_model("m1", path=test_dir, run_settings=settings) + M2 = exp.create_model("m2", path=test_dir, run_settings=settings) + + batch = exp.create_batch_settings(nodes=1, time="00:01:00") + add_batch_resources(wlmutils, batch) + + batch.set_account(wlmutils.get_test_account()) + ensemble = exp.create_ensemble("batch-ens", batch_settings=batch) + ensemble.add_model(M1) + ensemble.add_model(M2) + ensemble.set_path(test_dir) + + preview_manifest = Manifest(ensemble) + + # Execute preview method + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + assert "Batch Launch: True" in output + assert "Batch Command" in output + assert "Batch Arguments" in output + assert "nodes" in output + assert "time" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_launch_command(test_dir, wlmutils, choose_host): + """Test preview launch command for orchestrator, models, and + ensembles""" + # Prepare entities + test_launcher = wlmutils.get_test_launcher() + test_interface = wlmutils.get_test_interface() + test_port = wlmutils.get_test_port() + exp_name = "test_preview_launch_command" + exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) + # create regular database + orc = exp.create_database( + port=test_port, + interface=test_interface, + hosts=choose_host(wlmutils), + ) + + model_params = {"port": 6379, "password": "unbreakable_password"} + rs1 = RunSettings("bash", "multi_tags_template.sh") + rs2 = exp.create_run_settings("echo", ["spam", "eggs"]) + + hello_world_model = exp.create_model( + "echo-hello", run_settings=rs1, params=model_params + ) + + spam_eggs_model = exp.create_model("echo-spam", run_settings=rs2) + + # setup ensemble parameter space + learning_rate = list(np.linspace(0.01, 0.5)) + train_params = {"LR": learning_rate} + + run = exp.create_run_settings(exe="python", exe_args="./train-model.py") + + ensemble = exp.create_ensemble( + "Training-Ensemble", + params=train_params, + params_as_args=["LR"], + run_settings=run, + perm_strategy="random", + n_models=4, + ) + + preview_manifest = Manifest(orc, spam_eggs_model, hello_world_model, ensemble) + + # Execute preview method + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + assert "orchestrator" in output + assert "echo-spam" in output + assert "echo-hello" in output + + assert "Training-Ensemble" in output + assert "me: Training-Ensemble_0" in output + assert "Training-Ensemble_1" in output + assert "Training-Ensemble_2" in output + assert "Training-Ensemble_3" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_batch_launch_command(fileutils, test_dir, wlmutils): + """Test the preview of a model with batch settings""" + + exp_name = "test-batch-entities" + exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) + + script = fileutils.get_test_conf_path("sleep.py") + batch_settings = exp.create_batch_settings(nodes=1, time="00:01:00") + + batch_settings.set_account(wlmutils.get_test_account()) + add_batch_resources(wlmutils, batch_settings) + run_settings = wlmutils.get_run_settings("python", f"{script} --time=5") + model = exp.create_model( + "model", path=test_dir, run_settings=run_settings, batch_settings=batch_settings + ) + model.set_path(test_dir) + + orc = Orchestrator( + wlmutils.get_test_port(), + db_nodes=3, + batch=True, + interface="lo", + launcher="slurm", + run_command="srun", + ) + orc.set_batch_arg("account", "ACCOUNT") + + preview_manifest = Manifest(orc, model) + # Execute preview method + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + # Evaluate output + assert "Batch Launch: True" in output + assert "Batch Command" in output + assert "Batch Arguments" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_ensemble_batch(test_dir, wlmutils): + """ + Test preview of client configuration and key prefixing in Ensemble preview + """ + # Prepare entities + test_launcher = wlmutils.get_test_launcher() + exp = Experiment( + "test-preview-ensemble-clientconfig", exp_path=test_dir, launcher=test_launcher + ) + # Create Orchestrator + db = exp.create_database(port=6780, interface="lo") + exp.generate(db, overwrite=True) + rs1 = exp.create_run_settings("echo", ["hello", "world"]) + # Create ensemble + batch_settings = exp.create_batch_settings(nodes=1, time="00:01:00") + batch_settings.set_account(wlmutils.get_test_account()) + add_batch_resources(wlmutils, batch_settings) + ensemble = exp.create_ensemble( + "fd_simulation", run_settings=rs1, batch_settings=batch_settings, replicas=2 + ) + # enable key prefixing on ensemble + ensemble.enable_key_prefixing() + exp.generate(ensemble, overwrite=True) + rs2 = exp.create_run_settings("echo", ["spam", "eggs"]) + # Create model + ml_model = exp.create_model("tf_training", rs2) + + for sim in ensemble.entities: + ml_model.register_incoming_entity(sim) + + exp.generate(ml_model, overwrite=True) + + preview_manifest = Manifest(db, ml_model, ensemble) + + # Call preview renderer for testing output + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + # Evaluate output + assert "Client Configuration" in output + assert "Database Identifier" in output + assert "Database Backend" in output + assert "Type" in output + + +@pytest.mark.skipif( + pytest.test_launcher not in pytest.wlm_options, + reason="Not testing WLM integrations", +) +def test_preview_ensemble_db_script(wlmutils, test_dir): + """ + Test preview of a torch script on a model in an ensemble. + """ + # Initialize the Experiment and set the launcher to auto + test_launcher = wlmutils.get_test_launcher() + exp = Experiment("getting-started", launcher=test_launcher) + + orch = exp.create_database(db_identifier="test_db1") + orch_2 = exp.create_database(db_identifier="test_db2", db_nodes=3) + # Initialize a RunSettings object + model_settings = exp.create_run_settings(exe="python", exe_args="params.py") + model_settings_2 = exp.create_run_settings(exe="python", exe_args="params.py") + model_settings_3 = exp.create_run_settings(exe="python", exe_args="params.py") + # Initialize a Model object + model_instance = exp.create_model("model_name", model_settings) + model_instance_2 = exp.create_model("model_name_2", model_settings_2) + batch = exp.create_batch_settings(time="24:00:00", account="test") + ensemble = exp.create_ensemble( + "ensemble", batch_settings=batch, run_settings=model_settings_3, replicas=2 + ) + ensemble.add_model(model_instance) + ensemble.add_model(model_instance_2) + + # TorchScript string + torch_script_str = "def negate(x):\n\treturn torch.neg(x)\n" + + # Attach TorchScript to Model + model_instance.add_script( + name="example_script", + script=torch_script_str, + device="GPU", + devices_per_node=2, + first_device=0, + ) + preview_manifest = Manifest(ensemble, orch, orch_2) + + # Call preview renderer for testing output + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + + # Evaluate output + assert "Torch Script" in output diff --git a/tests/test_preview.py b/tests/test_preview.py index b95c3ec75..3c7bed6fe 100644 --- a/tests/test_preview.py +++ b/tests/test_preview.py @@ -26,22 +26,26 @@ import pathlib import sys +import typing as t from os import path as osp +import jinja2 import numpy as np import pytest -from jinja2 import Template -from jinja2.filters import FILTERS +import smartsim import smartsim._core._cli.utils as _utils from smartsim import Experiment from smartsim._core import Manifest, previewrenderer from smartsim._core.config import CONFIG +from smartsim._core.control.controller import Controller +from smartsim._core.control.job import Job from smartsim.database import Orchestrator +from smartsim.entity.entity import SmartSimEntity from smartsim.error.errors import PreviewFormatError from smartsim.settings import QsubBatchSettings, RunSettings -on_wlm = (pytest.test_launcher in pytest.wlm_options,) +pytestmark = pytest.mark.group_b @pytest.fixture @@ -55,12 +59,122 @@ def _choose_host(wlmutils, index: int = 0): return _choose_host +@pytest.fixture +def preview_object(test_dir) -> t.Dict[str, Job]: + """ + Bare bones orch + """ + rs = RunSettings(exe="echo", exe_args="ifname=lo") + s = SmartSimEntity(name="faux-name", path=test_dir, run_settings=rs) + o = Orchestrator() + o.entity = s + s.db_identifier = "test_db_id" + s.ports = [1235] + s.num_shards = 1 + job = Job("faux-name", "faux-step-id", s, "slurm", True) + active_dbjobs: t.Dict[str, Job] = {"mock_job": job} + return active_dbjobs + + +@pytest.fixture +def preview_object_multidb(test_dir) -> t.Dict[str, Job]: + """ + Bare bones orch + """ + rs = RunSettings(exe="echo", exe_args="ifname=lo") + s = SmartSimEntity(name="faux-name", path=test_dir, run_settings=rs) + o = Orchestrator() + o.entity = s + s.db_identifier = "testdb_reg" + s.ports = [8750] + s.num_shards = 1 + job = Job("faux-name", "faux-step-id", s, "slurm", True) + + rs2 = RunSettings(exe="echo", exe_args="ifname=lo") + s2 = SmartSimEntity(name="faux-name_2", path=test_dir, run_settings=rs) + o2 = Orchestrator() + o2.entity = s2 + s2.db_identifier = "testdb_reg2" + s2.ports = [8752] + s2.num_shards = 1 + job2 = Job("faux-name_2", "faux-step-id_2", s2, "slurm", True) + + active_dbjobs: t.Dict[str, Job] = {"mock_job": job, "mock_job2": job2} + return active_dbjobs + + def add_batch_resources(wlmutils, batch_settings): if isinstance(batch_settings, QsubBatchSettings): for key, value in wlmutils.get_batch_resources().items(): batch_settings.set_resource(key, value) +def test_get_ifname_filter(): + """Test get_ifname filter""" + + # Test input and expected output + value_dict = ( + (["+ifname=ib0"], "ib0"), + ("", ""), + ("+ifnameib0", ""), + ("=ib0", ""), + (["_ifname=bad_if_key"], "bad_if_key"), + (["ifname=mock_if_name"], "mock_if_name"), + ("IFname=case_sensitive_key", ""), + ("xfname=not_splittable", ""), + (None, ""), + ) + + template_str = "{{ value | get_ifname }}" + template_dict = {"ts": template_str} + + loader = jinja2.DictLoader(template_dict) + env = jinja2.Environment(loader=loader, autoescape=True) + env.filters["get_ifname"] = previewrenderer.get_ifname + + t = env.get_template("ts") + + for input, expected_output in value_dict: + output = t.render(value=input) + # assert that that filter output matches expected output + assert output == expected_output + + +def test_get_dbtype_filter(): + """Test get_dbtype filter to extract database backend from config""" + + template_str = "{{ config | get_dbtype }}" + template_dict = {"ts": template_str} + loader = jinja2.DictLoader(template_dict) + env = jinja2.Environment(loader=loader, autoescape=True) + env.filters["get_dbtype"] = previewrenderer.get_dbtype + + t = env.get_template("ts") + output = t.render(config=CONFIG.database_cli) + + assert output in CONFIG.database_cli + # Test empty input + test_string = "" + output = t.render(config=test_string) + assert output == "" + # Test empty path + test_string = "SmartSim/smartsim/_core/bin/" + output = t.render(config=test_string) + assert output == "" + # Test no hyphen + test_string = "SmartSim/smartsim/_core/bin/rediscli" + output = t.render(config=test_string) + assert output == "" + # Test no LHS + test_string = "SmartSim/smartsim/_core/bin/redis-" + output = t.render(config=test_string) + assert output == "" + # Test no RHS + test_string = "SmartSim/smartsim/_core/bin/-cli" + output = t.render(config=test_string) + assert output == "" + + def test_experiment_preview(test_dir, wlmutils): """Test correct preview output fields for Experiment preview""" # Prepare entities @@ -154,7 +268,9 @@ def test_preview_to_file(test_dir, wlmutils): path = pathlib.Path(test_dir) / filename # Execute preview method exp.preview( - output_format="plain_text", output_filename=str(path), verbosity_level="debug" + output_format=previewrenderer.Format.PLAINTEXT, + output_filename=str(path), + verbosity_level="debug", ) # Evaluate output @@ -162,236 +278,6 @@ def test_preview_to_file(test_dir, wlmutils): assert path.is_file() -def test_active_orch_dict_property(wlmutils, test_dir, choose_host): - """Ensure db_jobs remaines unchanched after deletion - of active_orch_dict property stays intace when retrieving db_jobs""" - - # Retrieve parameters from testing environment - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - - # start a new Experiment for this section - exp = Experiment( - "test-active-orch-dict-property", exp_path=test_dir, launcher=test_launcher - ) - - # create and start an instance of the Orchestrator database - db = exp.create_database( - port=test_port + 3, - interface=test_interface, - db_identifier="testdb_reg", - hosts=choose_host(wlmutils, 1), - ) - - # create database with different db_id - db2 = exp.create_database( - port=test_port + 5, - interface=test_interface, - db_identifier="testdb_reg2", - hosts=choose_host(wlmutils, 2), - ) - exp.start(db, db2) - - # Remove a job from active_orch_dict - active_orch_dict = exp._control.active_orch_dict - del active_orch_dict["testdb_reg2_0"] - - # assert that db_jobs is not affected by deletion - assert len(active_orch_dict) == 1 - assert len(exp._control._jobs.db_jobs) == 2 - - exp.stop(db, db2) - - -def test_preview_active_infrastructure(wlmutils, test_dir, choose_host): - """Test active infrastructure without other orchestrators""" - - # Prepare entities - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - exp_name = "test_active_infrastructure_preview" - exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) - - orc = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - db_identifier="orc_1", - ) - - # Start the orchestrator - exp.start(orc) - - assert orc.is_active() == True - - # Retrieve started manifest from experiment - active_dbjobs = exp._control.active_orch_dict - - # Execute method for template rendering - output = previewrenderer.render( - exp, active_dbjobs=active_dbjobs, verbosity_level="debug" - ) - - assert "Active Infrastructure" in output - assert "Database Identifier" in output - assert "Shards" in output - assert "Network Interface" in output - assert "Type" in output - assert "TCP/IP" in output - - exp.stop(orc) - - -def test_preview_orch_active_infrastructure(wlmutils, test_dir, choose_host): - """ - Test correct preview output properties for active infrastructure preview - with other orchestrators - """ - # Prepare entities - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - exp_name = "test_orchestrator_active_infrastructure_preview" - exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) - - orc = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - db_identifier="orc_1", - ) - - exp.start(orc) - - assert orc.is_active() == True - - orc2 = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - db_identifier="orc_2", - ) - - orc3 = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - db_identifier="orc_3", - ) - - # Retreive any active jobs - active_dbjobs = exp._control.active_orch_dict - - preview_manifest = Manifest(orc2, orc3) - - # Execute method for template rendering - output = previewrenderer.render( - exp, preview_manifest, active_dbjobs=active_dbjobs, verbosity_level="debug" - ) - - assert "Active Infrastructure" in output - assert "Database Identifier" in output - assert "Shards" in output - assert "Network Interface" in output - assert "Type" in output - assert "TCP/IP" in output - - exp.stop(orc) - - -def test_preview_multidb_active_infrastructure(wlmutils, test_dir, choose_host): - """multiple started databases active infrastructure""" - - # Retrieve parameters from testing environment - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - - # start a new Experiment for this section - exp = Experiment( - "test_preview_multidb_active_infrastructure", - exp_path=test_dir, - launcher=test_launcher, - ) - - # create and start an instance of the Orchestrator database - db = exp.create_database( - port=test_port, - interface=test_interface, - db_identifier="testdb_reg", - hosts=choose_host(wlmutils, 1), - ) - - # create database with different db_id - db2 = exp.create_database( - port=test_port + 1, - interface=test_interface, - db_identifier="testdb_reg2", - hosts=choose_host(wlmutils, 2), - ) - exp.start(db, db2) - - # Retreive any active jobs - active_dbjobs = exp._control.active_orch_dict - - # Execute method for template rendering - output = previewrenderer.render( - exp, active_dbjobs=active_dbjobs, verbosity_level="debug" - ) - - assert "Active Infrastructure" in output - assert "Database Identifier" in output - assert "Shards" in output - assert "Network Interface" in output - assert "Type" in output - assert "TCP/IP" in output - - assert "testdb_reg" in output - assert "testdb_reg2" in output - assert "Ochestrators" not in output - - exp.stop(db, db2) - - -def test_preview_active_infrastructure_orchestrator_error( - wlmutils, test_dir, choose_host -): - """Demo error when trying to preview a started orchestrator""" - # Prepare entities - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - exp_name = "test_active_infrastructure_preview_orch_error" - exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) - - orc = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - db_identifier="orc_1", - ) - # Start the orchestrator - exp.start(orc) - - assert orc.is_active() == True - - # Retrieve any active jobs - active_dbjobs = exp._control.active_orch_dict - - preview_manifest = Manifest(orc) - - # Execute method for template rendering - output = previewrenderer.render( - exp, preview_manifest, active_dbjobs=active_dbjobs, verbosity_level="debug" - ) - - assert "WARNING: Cannot preview orc_1, because it is already started" in output - - exp.stop(orc) - - def test_model_preview(test_dir, wlmutils): """ Test correct preview output fields for Model preview @@ -891,8 +777,9 @@ def test_preview_colocated_db_script_ensemble(fileutils, test_dir, wlmutils, mlu test_device = mlutils.get_test_device() test_num_gpus = mlutils.get_test_num_gpus() if pytest.test_device == "GPU" else 1 + expected_torch_script = "torchscript.py" test_script = fileutils.get_test_conf_path("run_dbscript_smartredis.py") - torch_script = fileutils.get_test_conf_path("torchscript.py") + torch_script = fileutils.get_test_conf_path(expected_torch_script) # Create SmartSim Experiment exp = Experiment(exp_name, launcher=test_launcher, exp_path=test_dir) @@ -979,41 +866,203 @@ def test_preview_colocated_db_script_ensemble(fileutils, test_dir, wlmutils, mlu assert "Devices Per Node" in output assert cm_name2 in output - assert torch_script in output + assert expected_torch_script in output assert test_device in output assert cm_name1 in output -def test_verbosity_info_ensemble(test_dir, wlmutils): - """ - Test preview of separate model entity and ensemble entity - with verbosity level set to info - """ - exp_name = "test-model-and-ensemble" - test_dir = pathlib.Path(test_dir) / exp_name - test_dir.mkdir(parents=True) - test_launcher = wlmutils.get_test_launcher() - exp = Experiment(exp_name, exp_path=str(test_dir), launcher=test_launcher) - - rs1 = exp.create_run_settings("echo", ["hello", "world"]) - rs2 = exp.create_run_settings("echo", ["spam", "eggs"]) - - hw_name = "echo-hello" - se_name = "echo-spam" - ens_name = "echo-ensemble" - hello_world_model = exp.create_model(hw_name, run_settings=rs1) - spam_eggs_model = exp.create_model(se_name, run_settings=rs2) - hello_ensemble = exp.create_ensemble(ens_name, run_settings=rs1, replicas=3) +def test_preview_active_infrastructure(wlmutils, test_dir, preview_object): + """Test active infrastructure without other orchestrators""" - exp.generate(hello_world_model, spam_eggs_model, hello_ensemble) + # Prepare entities + test_launcher = wlmutils.get_test_launcher() + exp_name = "test_active_infrastructure_preview" + exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) - preview_manifest = Manifest(hello_world_model, spam_eggs_model, hello_ensemble) - output = previewrenderer.render(exp, preview_manifest, verbosity_level="info") + # Execute method for template rendering + output = previewrenderer.render( + exp, active_dbjobs=preview_object, verbosity_level="debug" + ) - assert "Executable" not in output - assert "Executable Arguments" not in output + assert "Active Infrastructure" in output + assert "Database Identifier" in output + assert "Shards" in output + assert "Network Interface" in output + assert "Type" in output + assert "TCP/IP" in output - assert "echo_ensemble_1" not in output + +def test_preview_orch_active_infrastructure( + wlmutils, test_dir, choose_host, preview_object +): + """ + Test correct preview output properties for active infrastructure preview + with other orchestrators + """ + # Prepare entities + test_launcher = wlmutils.get_test_launcher() + test_interface = wlmutils.get_test_interface() + test_port = wlmutils.get_test_port() + exp_name = "test_orchestrator_active_infrastructure_preview" + exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) + + orc2 = exp.create_database( + port=test_port, + interface=test_interface, + hosts=choose_host(wlmutils), + db_identifier="orc_2", + ) + + orc3 = exp.create_database( + port=test_port, + interface=test_interface, + hosts=choose_host(wlmutils), + db_identifier="orc_3", + ) + + preview_manifest = Manifest(orc2, orc3) + + # Execute method for template rendering + output = previewrenderer.render( + exp, preview_manifest, active_dbjobs=preview_object, verbosity_level="debug" + ) + + assert "Active Infrastructure" in output + assert "Database Identifier" in output + assert "Shards" in output + assert "Network Interface" in output + assert "Type" in output + assert "TCP/IP" in output + + +def test_preview_multidb_active_infrastructure( + wlmutils, test_dir, choose_host, preview_object_multidb +): + """multiple started databases active infrastructure""" + + # Retrieve parameters from testing environment + test_launcher = wlmutils.get_test_launcher() + test_interface = wlmutils.get_test_interface() + test_port = wlmutils.get_test_port() + + # start a new Experiment for this section + exp = Experiment( + "test_preview_multidb_active_infrastructure", + exp_path=test_dir, + launcher=test_launcher, + ) + + # Execute method for template rendering + output = previewrenderer.render( + exp, active_dbjobs=preview_object_multidb, verbosity_level="debug" + ) + + assert "Active Infrastructure" in output + assert "Database Identifier" in output + assert "Shards" in output + assert "Network Interface" in output + assert "Type" in output + assert "TCP/IP" in output + + assert "testdb_reg" in output + assert "testdb_reg2" in output + assert "Ochestrators" not in output + + +def test_preview_active_infrastructure_orchestrator_error( + wlmutils, test_dir, choose_host, monkeypatch: pytest.MonkeyPatch +): + """Demo error when trying to preview a started orchestrator""" + # Prepare entities + test_launcher = wlmutils.get_test_launcher() + test_interface = wlmutils.get_test_interface() + test_port = wlmutils.get_test_port() + exp_name = "test_active_infrastructure_preview_orch_error" + exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) + + monkeypatch.setattr( + smartsim.database.orchestrator.Orchestrator, "is_active", lambda x: True + ) + + orc = exp.create_database( + port=test_port, + interface=test_interface, + hosts=choose_host(wlmutils), + db_identifier="orc_1", + ) + + # Retrieve any active jobs + active_dbjobs = exp._control.active_orchestrator_jobs + + preview_manifest = Manifest(orc) + + # Execute method for template rendering + output = previewrenderer.render( + exp, preview_manifest, active_dbjobs=active_dbjobs, verbosity_level="debug" + ) + + assert "WARNING: Cannot preview orc_1, because it is already started" in output + + +def test_active_orchestrator_jobs_property( + wlmutils, + test_dir, + preview_object, +): + """Ensure db_jobs remaines unchanged after deletion + of active_orchestrator_jobs property stays intact when retrieving db_jobs""" + + # Retrieve parameters from testing environment + test_launcher = wlmutils.get_test_launcher() + + # start a new Experiment for this section + exp = Experiment( + "test-active_orchestrator_jobs-property", + exp_path=test_dir, + launcher=test_launcher, + ) + + controller = Controller() + controller._jobs.db_jobs = preview_object + + # Modify the returned job collection + active_orchestrator_jobs = exp._control.active_orchestrator_jobs + active_orchestrator_jobs["test"] = "test_value" + + # Verify original collection is not also modified + assert not exp._control.active_orchestrator_jobs.get("test", None) + + +def test_verbosity_info_ensemble(test_dir, wlmutils): + """ + Test preview of separate model entity and ensemble entity + with verbosity level set to info + """ + exp_name = "test-model-and-ensemble" + test_dir = pathlib.Path(test_dir) / exp_name + test_dir.mkdir(parents=True) + test_launcher = wlmutils.get_test_launcher() + exp = Experiment(exp_name, exp_path=str(test_dir), launcher=test_launcher) + + rs1 = exp.create_run_settings("echo", ["hello", "world"]) + rs2 = exp.create_run_settings("echo", ["spam", "eggs"]) + + hw_name = "echo-hello" + se_name = "echo-spam" + ens_name = "echo-ensemble" + hello_world_model = exp.create_model(hw_name, run_settings=rs1) + spam_eggs_model = exp.create_model(se_name, run_settings=rs2) + hello_ensemble = exp.create_ensemble(ens_name, run_settings=rs1, replicas=3) + + exp.generate(hello_world_model, spam_eggs_model, hello_ensemble) + + preview_manifest = Manifest(hello_world_model, spam_eggs_model, hello_ensemble) + output = previewrenderer.render(exp, preview_manifest, verbosity_level="info") + + assert "Executable" not in output + assert "Executable Arguments" not in output + + assert "echo_ensemble_1" not in output def test_verbosity_info_colocated_db_model_ensemble( @@ -1175,158 +1224,7 @@ def test_verbosity_info_ensemble(test_dir, wlmutils): assert "Outgoing Key Collision Prevention (Key Prefixing)" in output -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_preview_wlm_run_commands_cluster_orc_model( - test_dir, coloutils, fileutils, wlmutils -): - """ - Test preview of wlm run command and run aruguments on a - orchestrator and model - """ - - exp_name = "test-preview-orc-model" - launcher = wlmutils.get_test_launcher() - test_port = wlmutils.get_test_port() - test_script = fileutils.get_test_conf_path("smartredis/multidbid.py") - exp = Experiment(exp_name, launcher=launcher, exp_path=test_dir) - - network_interface = wlmutils.get_test_interface() - orc = exp.create_database( - wlmutils.get_test_port(), - db_nodes=3, - batch=False, - interface=network_interface, - single_cmd=True, - hosts=wlmutils.get_test_hostlist(), - db_identifier="testdb_reg", - ) - - db_args = { - "port": test_port, - "db_cpus": 1, - "debug": True, - "db_identifier": "testdb_colo", - } - - # Create model with colocated database - smartsim_model = coloutils.setup_test_colo( - fileutils, "uds", exp, test_script, db_args, on_wlm=on_wlm - ) - - preview_manifest = Manifest(orc, smartsim_model) - - # Execute preview method - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - - # Evaluate output - assert "Run Command" in output - assert "Run Arguments" in output - assert "ntasks" in output - assert "nodes" in output - - -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_preview_model_on_wlm(fileutils, test_dir, wlmutils): - """ - Test preview of wlm run command and run aruguments for a model - """ - exp_name = "test-preview-model-wlm" - exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) - - script = fileutils.get_test_conf_path("sleep.py") - settings1 = wlmutils.get_base_run_settings("python", f"{script} --time=5") - settings2 = wlmutils.get_base_run_settings("python", f"{script} --time=5") - M1 = exp.create_model("m1", path=test_dir, run_settings=settings1) - M2 = exp.create_model("m2", path=test_dir, run_settings=settings2) - - preview_manifest = Manifest(M1, M2) - - # Execute preview method - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - - assert "Run Command" in output - assert "Run Arguments" in output - assert "nodes" in output - assert "ntasks" in output - assert "time" in output - - -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_preview_batch_model(fileutils, test_dir, wlmutils): - """Test the preview of a model with batch settings""" - - exp_name = "test-batch-model" - exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) - - script = fileutils.get_test_conf_path("sleep.py") - batch_settings = exp.create_batch_settings(nodes=1, time="00:01:00") - - batch_settings.set_account(wlmutils.get_test_account()) - add_batch_resources(wlmutils, batch_settings) - run_settings = wlmutils.get_run_settings("python", f"{script} --time=5") - model = exp.create_model( - "model", path=test_dir, run_settings=run_settings, batch_settings=batch_settings - ) - model.set_path(test_dir) - - preview_manifest = Manifest(model) - - # Execute preview method - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - - assert "Batch Launch: True" in output - assert "Batch Command" in output - assert "Batch Arguments" in output - assert "nodes" in output - assert "time" in output - - -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_preview_batch_ensemble(fileutils, test_dir, wlmutils): - """Test preview of a batch ensemble""" - - exp_name = "test-preview-batch-ensemble" - exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) - - script = fileutils.get_test_conf_path("sleep.py") - settings = wlmutils.get_run_settings("python", f"{script} --time=5") - M1 = exp.create_model("m1", path=test_dir, run_settings=settings) - M2 = exp.create_model("m2", path=test_dir, run_settings=settings) - - batch = exp.create_batch_settings(nodes=1, time="00:01:00") - add_batch_resources(wlmutils, batch) - - batch.set_account(wlmutils.get_test_account()) - ensemble = exp.create_ensemble("batch-ens", batch_settings=batch) - ensemble.add_model(M1) - ensemble.add_model(M2) - ensemble.set_path(test_dir) - - preview_manifest = Manifest(ensemble) - - # Execute preview method - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - - assert "Batch Launch: True" in output - assert "Batch Command" in output - assert "Batch Arguments" in output - assert "nodes" in output - assert "time" in output - - -def test_output_format_error(): +def test_check_output_format_error(): """ Test error when invalid ouput format is given. """ @@ -1343,165 +1241,34 @@ def test_output_format_error(): ) -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_preview_launch_command(test_dir, wlmutils, choose_host): - """Test preview launch command for orchestrator, models, and - ensembles""" +def test_check_verbosity_level_error(): + """ + Testing that an error does occur when a string verbosity is passed + """ # Prepare entities - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - exp_name = "test_preview_launch_command" - exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) - # create regular database - orc = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - ) - - model_params = {"port": 6379, "password": "unbreakable_password"} - rs1 = RunSettings("bash", "multi_tags_template.sh") - rs2 = exp.create_run_settings("echo", ["spam", "eggs"]) - - hello_world_model = exp.create_model( - "echo-hello", run_settings=rs1, params=model_params - ) - - spam_eggs_model = exp.create_model("echo-spam", run_settings=rs2) - - # setup ensemble parameter space - learning_rate = list(np.linspace(0.01, 0.5)) - train_params = {"LR": learning_rate} - - run = exp.create_run_settings(exe="python", exe_args="./train-model.py") - - ensemble = exp.create_ensemble( - "Training-Ensemble", - params=train_params, - params_as_args=["LR"], - run_settings=run, - perm_strategy="random", - n_models=4, - ) - - preview_manifest = Manifest(orc, spam_eggs_model, hello_world_model, ensemble) - - # Execute preview method - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - - assert "orchestrator" in output - assert "echo-spam" in output - assert "echo-hello" in output - - assert "Training-Ensemble" in output - assert "me: Training-Ensemble_0" in output - assert "Training-Ensemble_1" in output - assert "Training-Ensemble_2" in output - assert "Training-Ensemble_3" in output - - -def add_batch_resources(wlmutils, batch_settings): - if isinstance(batch_settings, QsubBatchSettings): - for key, value in wlmutils.get_batch_resources().items(): - batch_settings.set_resource(key, value) - - -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_preview_batch_launch_command(fileutils, test_dir, wlmutils): - """Test the preview of a model with batch settings""" - - exp_name = "test-batch-entities" - exp = Experiment(exp_name, launcher=wlmutils.get_test_launcher(), exp_path=test_dir) - - script = fileutils.get_test_conf_path("sleep.py") - batch_settings = exp.create_batch_settings(nodes=1, time="00:01:00") - - batch_settings.set_account(wlmutils.get_test_account()) - add_batch_resources(wlmutils, batch_settings) - run_settings = wlmutils.get_run_settings("python", f"{script} --time=5") - model = exp.create_model( - "model", path=test_dir, run_settings=run_settings, batch_settings=batch_settings - ) - model.set_path(test_dir) - - orc = Orchestrator( - wlmutils.get_test_port(), - db_nodes=3, - batch=True, - interface="lo", - launcher="slurm", - run_command="srun", - ) - orc.set_batch_arg("account", "ACCOUNT") + exp_name = "test_verbosity_level_error" + exp = Experiment(exp_name) - preview_manifest = Manifest(orc, model) # Execute preview method - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - - # Evaluate output - assert "Batch Launch: True" in output - assert "Batch Command" in output - assert "Batch Arguments" in output + with pytest.raises(ValueError) as ex: + exp.preview(verbosity_level="hello") -@pytest.mark.skipif( - pytest.test_launcher not in pytest.wlm_options, - reason="Not testing WLM integrations", -) -def test_ensemble_batch(test_dir, wlmutils): +def test_check_verbosity_level(): """ - Test preview of client configuration and key prefixing in Ensemble preview + Testing that an error doesnt occur when a string verbosity is passed """ # Prepare entities - test_launcher = wlmutils.get_test_launcher() - exp = Experiment( - "test-preview-ensemble-clientconfig", exp_path=test_dir, launcher=test_launcher - ) - # Create Orchestrator - db = exp.create_database(port=6780, interface="lo") - exp.generate(db, overwrite=True) - rs1 = exp.create_run_settings("echo", ["hello", "world"]) - # Create ensemble - batch_settings = exp.create_batch_settings(nodes=1, time="00:01:00") - batch_settings.set_account(wlmutils.get_test_account()) - add_batch_resources(wlmutils, batch_settings) - ensemble = exp.create_ensemble( - "fd_simulation", run_settings=rs1, batch_settings=batch_settings, replicas=2 - ) - # enable key prefixing on ensemble - ensemble.enable_key_prefixing() - exp.generate(ensemble, overwrite=True) - rs2 = exp.create_run_settings("echo", ["spam", "eggs"]) - # Create model - ml_model = exp.create_model("tf_training", rs2) - - for sim in ensemble.entities: - ml_model.register_incoming_entity(sim) - - exp.generate(ml_model, overwrite=True) - - preview_manifest = Manifest(db, ml_model, ensemble) - - # Call preview renderer for testing output - output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") + exp_name = "test_verbosity_level" + exp = Experiment(exp_name) - # Evaluate output - assert "Client Configuration" in output - assert "Database Identifier" in output - assert "Database Backend" in output - assert "Type" in output + # Execute preview method + exp.preview(verbosity_level="info") def test_preview_colocated_db_singular_model(wlmutils, test_dir): """Test preview behavior when a colocated db is only added to - one model. The expected behviour is that + one model. The expected behviour is that both models are colocated """ test_launcher = wlmutils.get_test_launcher() @@ -1527,87 +1294,37 @@ def test_preview_colocated_db_singular_model(wlmutils, test_dir): assert "Client Configuration" in output -def test_get_dbtype_filter(): - """Test get_dbtype filter to extract database backend from config""" - - template_str = "{{ config | get_dbtype }}" - FILTERS["get_dbtype"] = previewrenderer.get_dbtype - output = Template(template_str).render(config=CONFIG.database_cli) - assert output in CONFIG.database_cli - # Test empty input - test_string = "" - output = Template(template_str).render(config=test_string) - assert output == "" - # Test empty path - test_string = "SmartSim/smartsim/_core/bin/" - output = Template(template_str).render(config=test_string) - assert output == "" - # Test no hyphen - test_string = "SmartSim/smartsim/_core/bin/rediscli" - output = Template(template_str).render(config=test_string) - assert output == "" - # Test no LHS - test_string = "SmartSim/smartsim/_core/bin/redis-" - output = Template(template_str).render(config=test_string) - assert output == "" - # Test no RHS - test_string = "SmartSim/smartsim/_core/bin/-cli" - output = Template(template_str).render(config=test_string) - assert output == "" +def test_preview_db_script(wlmutils, test_dir): + """ + Test preview of model instance with a torch script. + """ + test_launcher = wlmutils.get_test_launcher() + # Initialize the Experiment and set the launcher to auto + exp = Experiment("getting-started", launcher=test_launcher) -def test_get_ifname_filter(wlmutils, test_dir, choose_host): - """Test get_ifname filter""" - # Prepare entities - test_launcher = wlmutils.get_test_launcher() - test_interface = wlmutils.get_test_interface() - test_port = wlmutils.get_test_port() - exp_name = "test-get-ifname-filter" - exp = Experiment(exp_name, exp_path=test_dir, launcher=test_launcher) + # Initialize a RunSettings object + model_settings = exp.create_run_settings(exe="python", exe_args="params.py") - orc = exp.create_database( - port=test_port, - interface=test_interface, - hosts=choose_host(wlmutils), - db_identifier="orc_1", - ) - # Start the orchestrator - exp.start(orc) + # Initialize a Model object + model_instance = exp.create_model("model_name", model_settings) + model_instance.colocate_db_tcp() - assert orc.is_active() == True + # TorchScript string + torch_script_str = "def negate(x):\n\treturn torch.neg(x)\n" - active_dbjobs = exp._control.active_orch_dict + # Attach TorchScript to Model + model_instance.add_script( + name="example_script", + script=torch_script_str, + device="GPU", + devices_per_node=2, + first_device=0, + ) + preview_manifest = Manifest(model_instance) - template_str = "{{db_exe_args | get_ifname}}" + # Call preview renderer for testing output + output = previewrenderer.render(exp, preview_manifest, verbosity_level="debug") - for db in active_dbjobs.values(): - FILTERS["get_ifname"] = previewrenderer.get_ifname - output = Template(template_str).render( - db_exe_args=db.entity.run_settings.exe_args - ) - assert output == test_interface[0] - # Test empty input string - test_string = "" - output = Template(template_str).render(db_exe_args=test_string) - assert output == "" - # Test input with no '=' delimiter - test_string = ["+ifnameib0"] - output = Template(template_str).render(db_exe_args=test_string) - assert output == "" - # Test input with empty RHS - test_string = ["=ib0"] - output = Template(template_str).render(db_exe_args=test_string) - assert output == "" - # Test input with empty LHS - test_string = ["+ifname="] - output = Template(template_str).render(db_exe_args=test_string) - assert output == "" - # Test input with no matching item - test_string = [ - "+name=orc_1_0", - "+port=6780", - ] - output = Template(template_str).render(db_exe_args=test_string) - assert output == "" - - exp.stop(orc) + # Evaluate output + assert "Torch Script" in output