Add predefined log modes (#3221)

- add predefined log modes ("default", "concise", and "verbose") to
simulator and dynamic log config admin commands config argument
- move simulator log_config.json to fuel/utils
- update documentation
- also applied to 2.5 branch
(#3237)


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Quick tests passed locally by running `./runtest.sh`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated.

docs/user_guide/configurations/logging_configuration.rst

@@ -20,7 +20,8 @@ Logging Configuration and Features
 Default Logging Configuration
 =============================
 
-The default logging configuration json file **log_config.json.default** is divided into 3 main sections: formatters, handlers, and loggers.
+The default logging configuration json file (**log_config.json.default**, ``default``) is divided into 3 main sections: formatters, handlers, and loggers.
+This file can be found at :github_nvflare_link:`log_config.json <nvflare/fuel/utils/log_config.json>`.
 See the `configuration dictionary schema <(https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema)>`_ for more details.
 
 .. code-block:: json
@@ -255,7 +256,7 @@ The following log file handlers are pre-configured:
 - jsonFileHandler with jsonFormatter to write json formatted logs to ``log.json``
 - FLFileHandler with baseFormatter and FLFilter to write fl training and custom logs to ``log_fl.txt``
 
-
+.. _loggers:
 Loggers
 =======
 
@@ -288,7 +289,7 @@ Modifying Logging Configurations
 Simulator log configuration
 ===========================
 
-Users can specify a log configuration file in the simulator command with the ``-l`` simulator argument:
+Users can specify a log configuration in the simulator command with the ``-l`` simulator argument:
 
 .. code-block:: shell
 
@@ -301,6 +302,13 @@ Or using the ``log_config`` argument of the Job API simulator run:
     job.simulator_run("/tmp/nvflare/hello-numpy-sag", log_config="log_config.json")
 
 
+The log config argument be one of the following:
+
+- path to a log configuration json file (``/path/to/my_log_config.json``)
+- preconfigured log mode (``default``, ``concise``, ``verbose``)
+- log level name or number (``debug``, ``info``, ``warning``, ``error``, ``critical``, ``30``)
+
+
 POC log configurations
 ======================
 If you search the POC workspace, you will find the following:
@@ -342,16 +350,23 @@ We also recommend using the :ref:`Dynamic Logging Configuration Commands <dynami
 Dynamic Logging Configuration Commands
 **************************************
 
-We provide two admin commands to enable users to dynamically configure the site or job level logging.
+When running the FLARE system (POC mode or production mode), there are two sets of logs: the site logs and job logs.
+The current site log configuration will be used for the site logs as well as the log config of any new job started on that site.
+In order to access the generated logs in the workspaces refer to :ref:`access_server_workspace` and :ref:`client_workspace`.
+
+We provide two admin commands to enable users to dynamically configure the site or job level logging when running the FLARE system.
+Note these command effects will last until reconfiguration or as long as the corresponding site or job is running.
+However these commands do not overwrite the log configuration file in the workspace- the log configuration file can be reloaded using "reload".
 
 - **target**: ``server``, ``client <clients>...``, or ``all``
 - **config**: log configuration
 
     - path to a json log configuration file (``/path/to/my_log_config.json``)
+    - predefined log mode (``default``, ``concise``, ``verbose``)
     - log level name/number (``debug``, ``INFO``, ``30``)
-    - read the current log configuration file (``reload``)
+    - read the current log configuration file from the workspace (``reload``)
 
-To configure the target site logging (does not affect jobs):
+To configure the target site logging (does not affect currently running jobs):
 
 .. code-block:: shell
 

examples/tutorials/logging.ipynb

@@ -23,7 +23,12 @@
    "source": [
     "## Simulator Logging\n",
     "\n",
-    "To get started, let's run the **hello-numpy-sag** job in the simulator and take a look at the default logging output:"
+    "To get started, let's run the **hello-numpy-sag** job in the simulator and take a look at the default logging output.\n",
+    "\n",
+    "The simulator `--log_config` (`-l`) argument can be used to set the log config mode ('concise', 'full', 'verbose'), filepath to a log config json file, or level (info, debug, error, etc.).\n",
+    "For this example we first show the `\"full\"` mode to compare it with our custom configuration.\n",
+    "\n",
+    "Note: this argument is defaulted to `\"concise\"` in the simulator mode, however feel free to experiment with the other modes or levels."
    ]
   },
   {
@@ -34,18 +39,18 @@
    "outputs": [],
    "source": [
     "!mkdir -p hello-numpy-sag-workspace\n",
-    "!nvflare simulator -w hello-numpy-sag-workspace -n 2 -t 2 ../hello-world/hello-numpy-sag/jobs/hello-numpy-sag"
+    "!nvflare simulator -w hello-numpy-sag-workspace -n 2 -t 2 -l full ../hello-world/hello-numpy-sag/jobs/hello-numpy-sag "
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "80ee3335",
+   "id": "3bb2c22c",
    "metadata": {},
    "source": [
     "Notice how the output contains lots of logs from both the FLARE system, as well as the training workflow.\n",
     "Additionally, the different level of logs (eg. INFO, WARNING, ERROR) have different console colors.\n",
     "\n",
-    "We can view the default configuration used in this run and the generated log files in the workspace:"
+    "We now will cover custom log configurations using the log config json file option. First lets look at the default configuration used in this run and the generated log files in the workspace:"
    ]
   },
   {
@@ -269,7 +274,7 @@
    "id": "d3b43521",
    "metadata": {},
    "source": [
-    "Compare this to the original output from the first command, and note the differences in the log output.\n",
+    "Compare this to the original output from the first command, and note the differences in the log output. Note that this concise format can also be achieved using the simulator \"concise\" mode, however this helps cover how to customize the logs using the file.\n",
     "\n",
     "In addition to the consoleHandler, all the other formatters, filters, handlers, and loggers can all also be customized just as easily.\n",
     "\n",
@@ -303,6 +308,7 @@
     "\n",
     "- **target**: server, client <clients>..., or all\n",
     "- **config**: log configuration\n",
+    "    - log mode (concise, full, verbose)\n",
     "    - path to a json log configuration file (/path/to/my_log_config.json)\n",
     "    - log level name/number (debug, INFO, 30)\n",
     "    - read the current log configuration file (reload)\n",
@@ -368,6 +374,44 @@
    "source": [
     "!tree /tmp/nvflare/poc"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b50c4865",
+   "metadata": {},
+   "source": [
+    "## Defining Loggers in the Hierarchy\n",
+    "\n",
+    "When defining new loggers, we provide several functions to help adhere to the FLARE package logger hierarchy. For example say we have the following module at `my_package.my_module`:\n",
+    "\n",
+    "- [get_obj_logger](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.fuel.utils.log_utils.html#nvflare.fuel.utils.log_utils.get_obj_logger) for classes. Ex: \n",
+    "```\n",
+    "        class MyClass:\n",
+    "            def __init__(self):\n",
+    "            self.logger = get_obj_logger(self) # my_package.my_module.MyClass\n",
+    "```\n",
+    "\n",
+    "- [get_script_logger](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.fuel.utils.log_utils.html#nvflare.fuel.utils.log_utils.get_script_logger) for scripts (if not in a package, default to custom.<script_file_name>). Ex:\n",
+    "```\n",
+    "        if __name__ == \"__main__\":\n",
+    "            logger = get_script_logger() # my_package.my_module\n",
+    "```\n",
+    "\n",
+    "- [get_module_logger](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.fuel.utils.log_utils.html#nvflare.fuel.utils.log_utils.get_module_logger) for modules. Ex:\n",
+    "```\n",
+    "        def my_function():\n",
+    "            logger = get_module_logger(name=\"my_function\") # my_package.my_module.my_function\n",
+    "```\n",
+    "\n",
+    "\n",
+    "For more information, refer to the [Logging Configuration Documentation](https://nvflare.readthedocs.io/en/main/user_guide/configurations/logging_configuration.html#loggers) for definining loggers. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ac669133",
+   "metadata": {},
+   "source": []
   }
  ],
  "metadata": {

nvflare/fuel/utils/log_utils.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import copy
 import inspect
 import json
 import logging
@@ -19,9 +20,40 @@
 import re
 from logging import Logger
 from logging.handlers import RotatingFileHandler
+from typing import Union
 
 from nvflare.apis.workspace import Workspace
 
+DEFAULT_LOG_JSON = "log_config.json"
+
+
+class LogMode:
+    RELOAD = "reload"
+    FULL = "full"
+    CONCISE = "concise"
+    VERBOSE = "verbose"
+
+
+# Predefined log dicts based from DEFAULT_LOG_JSON
+with open(os.path.join(os.path.dirname(__file__), DEFAULT_LOG_JSON), "r") as f:
+    default_log_dict = json.load(f)
+
+concise_log_dict = copy.deepcopy(default_log_dict)
+concise_log_dict["formatters"]["consoleFormatter"]["fmt"] = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+concise_log_dict["handlers"]["consoleHandler"]["filters"] = ["FLFilter"]
+
+verbose_log_dict = copy.deepcopy(default_log_dict)
+verbose_log_dict["formatters"]["consoleFormatter"][
+    "fmt"
+] = "%(asctime)s - %(identity)s - %(name)s - %(levelname)s - %(message)s"
+verbose_log_dict["loggers"]["root"]["level"] = "DEBUG"
+
+logmode_config_dict = {
+    LogMode.FULL: default_log_dict,
+    LogMode.CONCISE: concise_log_dict,
+    LogMode.VERBOSE: verbose_log_dict,
+}
+
 
 class ANSIColor:
     # Basic ANSI color codes
@@ -229,6 +261,7 @@ def matches_name(self, name, logger_names) -> bool:
 
 
 def get_module_logger(module=None, name=None):
+    # Get module logger name adhering to logger hierarchy. Optionally add name as a suffix.
     if module is None:
         caller_globals = inspect.stack()[1].frame.f_globals
         module = caller_globals.get("__name__", "")
@@ -237,11 +270,12 @@ def get_module_logger(module=None, name=None):
 
 
 def get_obj_logger(obj):
-    return logging.getLogger(f"{obj.__module__}.{obj.__class__.__qualname__}")
+    # Get object logger name adhering to logger hierarchy.
+    return logging.getLogger(f"{obj.__module__}.{obj.__class__.__qualname__}") if obj else None
 
 
 def get_script_logger():
-    # Get script logger name based on filename and package. If not in a package, default to custom.
+    # Get script logger name adhering to logger hierarchy. Based on package and filename. If not in a package, default to custom.
     caller_frame = inspect.stack()[1]
     package = caller_frame.frame.f_globals.get("__package__", "")
     file = caller_frame.frame.f_globals.get("__file__", "")
@@ -278,40 +312,40 @@ def apply_log_config(dict_config, dir_path: str = "", file_prefix: str = ""):
     logging.config.dictConfig(dict_config)
 
 
-def dynamic_log_config(config: str, workspace: Workspace, job_id: str = None):
-    # Dynamically configure log given a config (filepath, levelname, levelnumber, 'reload'), apply the config to the proper locations.
-    if not isinstance(config, str):
-        raise ValueError(
-            f"Unsupported config type. Expect config to be string filepath, levelname, levelnumber, or 'reload' but got {type(config)}"
-        )
+def dynamic_log_config(config: Union[dict, str], dir_path: str, reload_path: str):
+    # Dynamically configure log given a config (dict, filepath, LogMode, or level), apply the config to the proper locations.
 
-    if config == "reload":
-        config = workspace.get_log_config_file_path()
+    if isinstance(config, dict):
+        apply_log_config(config, dir_path)
+    elif isinstance(config, str):
+        # Handle pre-defined LogModes
+        if config == LogMode.RELOAD:
+            config = reload_path
+        elif log_config := logmode_config_dict.get(config):
+            apply_log_config(copy.deepcopy(log_config), dir_path)
+            return
 
-    if os.path.isfile(config):
-        # Read confg file
-        with open(config, "r") as f:
-            dict_config = json.load(f)
+        # Read config file
+        if os.path.isfile(config):
+            with open(config, "r") as f:
+                dict_config = json.load(f)
 
-        if job_id:
-            dir_path = workspace.get_run_dir(job_id)
+            apply_log_config(dict_config, dir_path)
         else:
-            dir_path = workspace.get_root_dir()
+            # If logging is not yet configured, use default config
+            if not logging.getLogger().hasHandlers():
+                apply_log_config(default_log_dict, dir_path)
 
-        apply_log_config(dict_config, dir_path)
-
-    else:
-        # Set level of root logger based on levelname or levelnumber
-        if config.isdigit():
-            level = int(config)
-            if not (0 <= level <= 50):
-                raise ValueError(f"Invalid logging level: {level}")
-        else:
-            level = getattr(logging, config.upper(), None)
-            if level is None:
+            # Set level of root logger based on levelname or levelnumber
+            level = int(config) if config.isdigit() else getattr(logging, config.upper(), None)
+            if level is None or not (0 <= level <= 50):
                 raise ValueError(f"Invalid logging level: {config}")
 
-        logging.getLogger().setLevel(level)
+            logging.getLogger().setLevel(level)
+    else:
+        raise ValueError(
+            f"Unsupported config type. Expect config to be a dict, filepath, level, or LogMode but got {type(config)}"
+        )
 
 
 def add_log_file_handler(log_file_name):

nvflare/job_config/api.py

@@ -533,7 +533,7 @@ def simulator_run(
             n_clients: number of clients.
             threads: number of threads.
             gpu: gpu assignments for simulating clients, comma separated
-            log_config: log config json file path
+            log_config: log config mode ('concise', 'default', 'verbose'), filepath, or level
 
         Returns:
 

nvflare/private/fed/app/simulator/simulator.py

@@ -18,6 +18,7 @@
 import sys
 from sys import platform
 
+from nvflare.fuel.utils.log_utils import LogMode
 from nvflare.private.fed.app.simulator.simulator_runner import SimulatorRunner
 from nvflare.private.fed.app.utils import version_check
 
@@ -29,7 +30,13 @@ def define_simulator_parser(simulator_parser):
     simulator_parser.add_argument("-c", "--clients", type=str, help="client names list")
     simulator_parser.add_argument("-t", "--threads", type=int, help="number of parallel running clients")
     simulator_parser.add_argument("-gpu", "--gpu", type=str, help="list of GPU Device Ids, comma separated")
-    simulator_parser.add_argument("-l", "--log_config", type=str, help="log config file path")
+    simulator_parser.add_argument(
+        "-l",
+        "--log_config",
+        type=str,
+        default=LogMode.CONCISE,
+        help="log config mode ('concise', 'full', 'verbose'), filepath, or level",
+    )
     simulator_parser.add_argument("-m", "--max_clients", type=int, default=100, help="max number of clients")
     simulator_parser.add_argument(
         "--end_run_for_all",