Skip to content

Commit

Permalink
vdk-structlog: put vdk init logs config behind flag (#3107)
Browse files Browse the repository at this point in the history 
## Why?

Configuring the logs in the vdk_initialize hook causes commands like vdk
server and vdk info to have the same verbose logging format as data job
logs.

## What?

- Put the log configuration for vdk_initialize behind a config option
and disable it by default for local runs and enable it by default for
cloud runs. Configuring the logging format in vdk_initialize might still
be useful for cloud runs and we don't care about other vdk commands
output in cloud environments

- Move config classes and functions into separate file

## How was this tested

```sh
(venv) 11:01:14 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=True 
(venv) 11:01:19 ~ $ vdk create
2024-02-15 11:01:21,922 [VDK] [INFO ] vdk.internal.builtin_plugins.b builtin_hook_impl.py :98   vdk_initialize   - Setting: OP_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681, ATTEMPT_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681-aa061, EXECUTION_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681
2024-02-15 11:01:21,923 [VDK] [INFO ] vdk.internal.cli_entry                 cli_entry.py :135  vdk_main         - Start CLI vdk with args ['create']
Job Name: 

(venv) 11:01:52 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=False
(venv) 11:01:58 ~ $ vdk create                                 
Job Name: 
```

```sh
(venv) 11:03:25 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=True
(venv) 11:03:27 ~ $ vdk version
2024-02-15 11:03:30,267 [VDK] [INFO ] vdk.internal.builtin_plugins.b builtin_hook_impl.py :98   vdk_initialize   - Setting: OP_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810, ATTEMPT_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810-3568d, EXECUTION_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810
2024-02-15 11:03:30,267 [VDK] [INFO ] vdk.internal.cli_entry                 cli_entry.py :135  vdk_main         - Start CLI vdk with args ['version']
2024-02-15 11:03:30,292 [VDK] [INFO ] vdk.internal.builtin_plugins.v           version.py :85   version          - Versatile Data Kit (VDK)
Version: 0.3.1177416838
Build details: RELEASE_VERSION=0.3.1177416838, BUILD_DATE=Thu Feb 15 07:57:26 UTC 2024, BUILD_MACHINE_INFO=Linux runner-trzkizxz-project-28359933-concurrent-0rvcnn 5.4.235-144.344.amzn2.x86_64 #1 SMP Sun Mar 12 12:50:22 UTC 2023 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=6177494231, GIT_COMMIT_SHA=a4c5a23b69ed2bd135b7f14658abd18c143e8759, GIT_BRANCH=main
Python version: 3.11.7 64bit (/Users/mdilyan/Projects/versatile-data-kit/projects/vdk-plugins/vdk-structlog/venv/bin/python)

Installed plugins:
vdk-ingest-http (from package vdk-ingest-http, version 0.2.1156222304)
vdk-sqlite (from package vdk-sqlite, version 0.1.1156222304)
vdk-server (from package vdk-server, version 0.1.1174992336)
vdk-structlog (from package vdk-structlog, version 0.1.0)
vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-ingest-file (from package vdk-ingest-file, version 0.1.948436673)
--------------------------------------------------------------------------------

(venv) 11:04:04 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=False
(venv) 11:04:09 ~ $ vdk version                                
Versatile Data Kit (VDK)
Version: 0.3.1177416838
Build details: RELEASE_VERSION=0.3.1177416838, BUILD_DATE=Thu Feb 15 07:57:26 UTC 2024, BUILD_MACHINE_INFO=Linux runner-trzkizxz-project-28359933-concurrent-0rvcnn 5.4.235-144.344.amzn2.x86_64 #1 SMP Sun Mar 12 12:50:22 UTC 2023 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=6177494231, GIT_COMMIT_SHA=a4c5a23b69ed2bd135b7f14658abd18c143e8759, GIT_BRANCH=main
Python version: 3.11.7 64bit (/Users/mdilyan/Projects/versatile-data-kit/projects/vdk-plugins/vdk-structlog/venv/bin/python)

Installed plugins:
vdk-ingest-http (from package vdk-ingest-http, version 0.2.1156222304)
vdk-sqlite (from package vdk-sqlite, version 0.1.1156222304)
vdk-server (from package vdk-server, version 0.1.1174992336)
vdk-structlog (from package vdk-structlog, version 0.1.0)
vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.948436673)
vdk-ingest-file (from package vdk-ingest-file, version 0.1.948436673)
--------------------------------------------------------------------------------
```
CI/CD

## What kind of change is this?

Bugfix

Signed-off-by: Dilyan Marinov <[email protected]>
Co-authored-by: Dilyan Marinov <[email protected]>
  • Loading branch information
@DeltaMichael
DeltaMichael and Dilyan Marinov authored Feb 16, 2024
1 parent 395beed commit 5feed5b
Show file tree
Hide file tree
Showing 5 changed files with 274 additions and 224 deletions.
3 changes: 2 additions & 1 deletion projects/vdk-plugins/vdk-structlog/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,12 @@ pip install vdk-structlog

| Name | Description | Example Value | Possible Values |
|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| use_structlog | Use the structlog logging config instead of using the one in vdk-core | "True" | "True", "False" |
| use_structlog | Use the structlog logging config instead of using the one in vdk-core (True by default). | "True" | "True", "False" |
| structlog_metadata | Configure the metadata that will be output along with the log message | "timestamp, level, logger_name, file_name, vdk_job_name | Any combination of the following: "timestamp, level, logger_name, file_name, line_number, function_name, vdk_job_name, vdk_step_name, vdk_step_type". Can be expanded by extra params and bound key-value pairs. See the bound logger examples for more information |
| structlog_format | Configure the logging output format. Available formats: json, console, ltsv | "console" | "console", "json", "ltsv" |
| structlog_console_log_pattern | Custom format string for console logging, applied only when`logging_format` is 'console'. Overrides `logging_metadata`. Note: For config.ini, %-signs should be escaped by doubling, e.g. %(asctime)s should become %%(asctime)s | "%(asctime)s %(name)-12s %(levelname)-8s %(message)s" | Any valid Python logging format string |
| structlog_config_preset | Choose a configuration preset. Any config options set together with the preset will override the preset options. Available presets: LOCAL, CLOUD. | "CLOUD" | "console", "json", "ltsv" |
| structlog_format_init_logs | Set to True to apply structlog formatting options to vdk initialization logs | "True" | "True", "False" |
| log_level_module | Configure the log level of different Python modules separately | "a.b.c=INFO;foo.bar=ERROR" | Semicolon-separated list of pairs of Python module paths and log level labels |
| syslog_host | Syslog host to which logs are emitted | "syslog.vmware.com" | Any valid host name |
| syslog_port | Syslog port used to emit logs | 514 | Any valid port number |
Expand Down
1 change: 1 addition & 0 deletions projects/vdk-plugins/vdk-structlog/src/vdk/plugin/structlog/constants.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
STRUCTLOG_LOGGING_FORMAT_KEY = "structlog_format"
STRUCTLOG_CONSOLE_LOG_PATTERN = "structlog_console_custom_format"
STRUCTLOG_CONFIG_PRESET = "structlog_config_preset"
STRUCTLOG_FORMAT_INIT_LOGS = "structlog_format_init_logs"

STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES = ["console", "json", "ltsv"]
STRUCTLOG_LOGGING_FORMAT_DEFAULT = "console"
Expand Down
189 changes: 189 additions & 0 deletions projects/vdk-plugins/vdk-structlog/src/vdk/plugin/structlog/structlog_config.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
# Copyright 2021-2024 VMware, Inc.
# SPDX-License-Identifier: Apache-2.0
from vdk.internal.builtin_plugins.config import vdk_config
from vdk.internal.core.config import Configuration
from vdk.internal.core.config import ConfigurationBuilder
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_ENABLED
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_HOST
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_PORT
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_PROTOCOL
from vdk.plugin.structlog.constants import DETAILED_LOGGING_FORMAT
from vdk.plugin.structlog.constants import JSON_STRUCTLOG_LOGGING_METADATA_DEFAULT
from vdk.plugin.structlog.constants import STRUCTLOG_CONFIG_PRESET
from vdk.plugin.structlog.constants import STRUCTLOG_CONSOLE_LOG_PATTERN
from vdk.plugin.structlog.constants import STRUCTLOG_FORMAT_INIT_LOGS
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_FORMAT_DEFAULT
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_FORMAT_KEY
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_METADATA_ALL_KEYS
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_METADATA_KEY
from vdk.plugin.structlog.constants import STRUCTLOG_USE_STRUCTLOG
from vdk.plugin.structlog.constants import SYSLOG_ENABLED_KEY
from vdk.plugin.structlog.constants import SYSLOG_HOST_KEY
from vdk.plugin.structlog.constants import SYSLOG_PORT_KEY
from vdk.plugin.structlog.constants import SYSLOG_PROTOCOL_KEY


class StructlogConfig:
def __init__(self, configuration: Configuration):
presets = {
"LOCAL": {
STRUCTLOG_USE_STRUCTLOG: configuration.get_value(
STRUCTLOG_USE_STRUCTLOG
),
STRUCTLOG_LOGGING_METADATA_KEY: ",".join(
list(JSON_STRUCTLOG_LOGGING_METADATA_DEFAULT.keys())
),
STRUCTLOG_LOGGING_FORMAT_KEY: STRUCTLOG_LOGGING_FORMAT_DEFAULT,
STRUCTLOG_CONSOLE_LOG_PATTERN: "",
STRUCTLOG_CONFIG_PRESET: configuration.get_value(
STRUCTLOG_CONFIG_PRESET
),
SYSLOG_HOST_KEY: DEFAULT_SYSLOG_HOST,
SYSLOG_PORT_KEY: DEFAULT_SYSLOG_PORT,
SYSLOG_PROTOCOL_KEY: DEFAULT_SYSLOG_PROTOCOL,
SYSLOG_ENABLED_KEY: DEFAULT_SYSLOG_ENABLED,
vdk_config.LOG_LEVEL_VDK.lower(): configuration.get_value(
vdk_config.LOG_LEVEL_VDK.lower()
),
vdk_config.LOG_LEVEL_MODULE.lower(): configuration.get_value(
vdk_config.LOG_LEVEL_MODULE.lower()
),
STRUCTLOG_FORMAT_INIT_LOGS: False,
},
"CLOUD": {
STRUCTLOG_USE_STRUCTLOG: configuration.get_value(
STRUCTLOG_USE_STRUCTLOG
),
STRUCTLOG_LOGGING_METADATA_KEY: "",
STRUCTLOG_LOGGING_FORMAT_KEY: STRUCTLOG_LOGGING_FORMAT_DEFAULT,
STRUCTLOG_CONSOLE_LOG_PATTERN: DETAILED_LOGGING_FORMAT,
STRUCTLOG_CONFIG_PRESET: configuration.get_value(
STRUCTLOG_CONFIG_PRESET
),
SYSLOG_HOST_KEY: DEFAULT_SYSLOG_HOST,
SYSLOG_PORT_KEY: DEFAULT_SYSLOG_PORT,
SYSLOG_PROTOCOL_KEY: DEFAULT_SYSLOG_PROTOCOL,
SYSLOG_ENABLED_KEY: DEFAULT_SYSLOG_ENABLED,
vdk_config.LOG_LEVEL_VDK.lower(): configuration.get_value(
vdk_config.LOG_LEVEL_VDK.lower()
),
vdk_config.LOG_LEVEL_MODULE.lower(): configuration.get_value(
vdk_config.LOG_LEVEL_MODULE.lower()
),
STRUCTLOG_FORMAT_INIT_LOGS: True,
},
}

self._config = presets[configuration.get_value(STRUCTLOG_CONFIG_PRESET)]

for key in configuration.list_config_keys():
if not configuration.is_default(key):
self._config[key] = configuration.get_value(key)

def get_use_structlog(self) -> bool:
return self._config[STRUCTLOG_USE_STRUCTLOG]

def get_structlog_logging_metadata(self) -> str:
return self._config[STRUCTLOG_LOGGING_METADATA_KEY]

def get_structlog_logging_format(self) -> str:
return self._config[STRUCTLOG_LOGGING_FORMAT_KEY]

def get_structlog_console_log_pattern(self) -> str:
return self._config[STRUCTLOG_CONSOLE_LOG_PATTERN]

def get_structlog_config_preset(self) -> str:
return self._config[STRUCTLOG_CONFIG_PRESET]

def get_syslog_host(self) -> str:
return self._config[SYSLOG_HOST_KEY]

def get_syslog_port(self) -> int:
return self._config[SYSLOG_PORT_KEY]

def get_syslog_protocol(self) -> str:
return self._config[SYSLOG_PROTOCOL_KEY]

def get_syslog_enabled(self) -> bool:
return self._config[SYSLOG_ENABLED_KEY]

def get_log_level_vdk(self) -> str:
return self._config[vdk_config.LOG_LEVEL_VDK.lower()]

def get_log_level_module(self) -> str:
return self._config[vdk_config.LOG_LEVEL_MODULE.lower()]

def get_format_init_logs(self) -> str:
return self._config[STRUCTLOG_FORMAT_INIT_LOGS]


def add_definitions(config_builder: ConfigurationBuilder):
config_builder.add(
key=STRUCTLOG_LOGGING_METADATA_KEY,
default_value=",".join(list(JSON_STRUCTLOG_LOGGING_METADATA_DEFAULT.keys())),
description=(
f"Possible values: {STRUCTLOG_LOGGING_METADATA_ALL_KEYS}"
"User-defined key-value pairs added to the logger's context will be displayed after the metadata, "
"but before the message"
"Keys for user-defined key-value pairs have to be added in this config option for the values to be "
"displayed in the metadata"
),
)

config_builder.add(
key=STRUCTLOG_CONSOLE_LOG_PATTERN,
default_value="",
description="Custom format string for console logging. Leave empty for default format.",
)

config_builder.add(
key=STRUCTLOG_LOGGING_FORMAT_KEY,
default_value=STRUCTLOG_LOGGING_FORMAT_DEFAULT,
description=(
f"Controls the logging output format. Possible values: {STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES}"
),
)

config_builder.add(
key=SYSLOG_HOST_KEY,
default_value=DEFAULT_SYSLOG_HOST,
description="Hostname of the Syslog server.",
)

config_builder.add(
key=SYSLOG_PORT_KEY,
default_value=DEFAULT_SYSLOG_PORT,
description="Port of the Syslog server.",
)

config_builder.add(
key=SYSLOG_PROTOCOL_KEY,
default_value=DEFAULT_SYSLOG_PROTOCOL,
description="Syslog protocol (UDP or TCP).",
)

config_builder.add(
key=SYSLOG_ENABLED_KEY,
default_value=DEFAULT_SYSLOG_ENABLED,
description="Enable Syslog logging (True or False).",
)

config_builder.add(
key=STRUCTLOG_CONFIG_PRESET,
default_value="LOCAL",
description="Choose configuration preset. Any config options set together with the preset will override "
"the preset options. Available presets: LOCAL, CLOUD",
)

config_builder.add(
key=STRUCTLOG_USE_STRUCTLOG,
default_value=True,
description="Use the structlog logging config instead of using the one in vdk-core",
)

config_builder.add(
key=STRUCTLOG_FORMAT_INIT_LOGS,
default_value=False,
description="Set to True to apply structlog formatting options to the vdk initialization logs",
)
Loading

0 comments on commit 5feed5b

Please sign in to comment.