Skip to content

Commit

Permalink
vdk-structlog: add default logging format values (#3055)
Browse files Browse the repository at this point in the history 
## Why?

For users who do not wish to configure each structlog option separately,
introduce presets for different cases.

## What?

Introduce CLOUD and LOCAL presets.

The CLOUD preset hardcodes the following options

```sh
export VDK_STRUCTLOG_FORMAT=console
export VDK_STRUCTLOG_CONSOLE_LOG_PATTERN="%(asctime)s [VDK] %(vdk_job_name)s [%(levelname)-5.5s] %(name)-30.30s %(filename)20.20s:%(lineno)-4.4s %(funcName)-16.16s[id:%(attempt_id)s]- %(message)s"
```

LOCAL just uses the defaults set in the configuration code

Refactor structlog plugin to eliminate repeating code
Remove syslog config tests
Fix bug with vdk fields crashing logging for custom format
Add tests for vdk fields with custom format

## How was this tested?

Functional tests
CI

## What kind of change is this?

Feature/non-breaking

Signed-off-by: Dilyan Marinov <[email protected]>
  • Loading branch information
@DeltaMichael
DeltaMichael authored Feb 12, 2024
1 parent 7c993c9 commit d0ef338
Show file tree
Hide file tree
Showing 6 changed files with 406 additions and 293 deletions.
22 changes: 12 additions & 10 deletions projects/vdk-plugins/vdk-structlog/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,18 @@ pip install vdk-structlog

(`vdk config-help` is a useful command to browse all config options of your installation of vdk)

| Name | Description | Example Value | Possible Values |
|----------------------------|-------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| logging_metadata | Configure the metadata that will be output along with the log message | "timestamp, level, logger_name, file_name, vdk_job_name | Any combination of the following: "timestamp, level, logger_name, file_name, line_number, function_name, vdk_job_name, vdk_step_name, vdk_step_type". Can be expanded by extra params and bound key-value pairs. See the bound logger examples for more information |
| logging_format | Configure the logging output format. Available formats: json, console | "console" | "console", "json", "ltsv" |
| custom_console_log_pattern | Custom format string for console logging, applied only when`logging_format` is 'console'. Overrides `logging_metadata`. | "%(asctime)s %(name)-12s %(levelname)-8s %(message)s" | Any valid Python logging format string |
| log_level_module | Configure the log level of different Python modules separately | "a.b.c=INFO;foo.bar=ERROR" | Semicolon-separated list of pairs of Python module paths and log level labels |
| syslog_host | Syslog host to which logs are emitted | "syslog.vmware.com" | Any valid host name |
| syslog_port | Syslog port used to emit logs | 514 | Any valid port number |
| syslog_protocol | Protocol used to emit logs | "UDP" | "TCP", "UDP" |
| syslog_enabled | Enable/disable syslog | "True" | "True", "False" |
| Name | Description | Example Value | Possible Values |
|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| use_structlog | Use the structlog logging config instead of using the one in vdk-core | "True" | "True", "False" |
| structlog_metadata | Configure the metadata that will be output along with the log message | "timestamp, level, logger_name, file_name, vdk_job_name | Any combination of the following: "timestamp, level, logger_name, file_name, line_number, function_name, vdk_job_name, vdk_step_name, vdk_step_type". Can be expanded by extra params and bound key-value pairs. See the bound logger examples for more information |
| structlog_format | Configure the logging output format. Available formats: json, console, ltsv | "console" | "console", "json", "ltsv" |
| structlog_console_log_pattern | Custom format string for console logging, applied only when`logging_format` is 'console'. Overrides `logging_metadata`. Note: For config.ini, %-signs should be escaped by doubling, e.g. %(asctime)s should become %%(asctime)s | "%(asctime)s %(name)-12s %(levelname)-8s %(message)s" | Any valid Python logging format string |
| structlog_config_preset | Choose a configuration preset. Any config options set together with the preset will override the preset options. Available presets: LOCAL, CLOUD. | "CLOUD" | "console", "json", "ltsv" |
| log_level_module | Configure the log level of different Python modules separately | "a.b.c=INFO;foo.bar=ERROR" | Semicolon-separated list of pairs of Python module paths and log level labels |
| syslog_host | Syslog host to which logs are emitted | "syslog.vmware.com" | Any valid host name |
| syslog_port | Syslog port used to emit logs | 514 | Any valid port number |
| syslog_protocol | Protocol used to emit logs | "UDP" | "TCP", "UDP" |
| syslog_enabled | Enable/disable syslog | "True" | "True", "False" |

### Example: Configure Custom Console Format

Expand Down
17 changes: 13 additions & 4 deletions projects/vdk-plugins/vdk-structlog/src/vdk/plugin/structlog/constants.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,17 @@
import logging
import socket

STRUCTLOG_USE_STRUCTLOG = "use_structlog"
STRUCTLOG_LOGGING_METADATA_KEY = "structlog_metadata"
STRUCTLOG_LOGGING_FORMAT_KEY = "structlog_format"
STRUCTLOG_CONSOLE_LOG_PATTERN = "structlog_console_custom_format"
STRUCTLOG_CONFIG_PRESET = "structlog_config_preset"

STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES = ["console", "json", "ltsv"]
STRUCTLOG_LOGGING_FORMAT_DEFAULT = "console"

STRUCTLOG_LOGGING_METADATA_JOB = {
"attempt_id": "%(attempt_id)s",
"vdk_job_name": "%(vdk_job_name)s",
"vdk_step_name": "%(vdk_step_name)s",
"vdk_step_type": "%(vdk_step_type)s",
Expand Down Expand Up @@ -44,6 +47,7 @@
}

CONSOLE_STRUCTLOG_LOGGING_METADATA_JOB = {
"attempt_id": "[id:%(attempt_id)s]",
"vdk_job_name": "%(vdk_job_name)s",
"vdk_step_name": "%(vdk_step_name)s",
"vdk_step_type": "%(vdk_step_type)s",
Expand All @@ -66,10 +70,10 @@
RECORD_DEFAULT_FIELDS = set(vars(logging.LogRecord("", "", "", "", "", "", "")))

# Syslog constants
SYSLOG_HOST_KEY = "SYSLOG_HOST"
SYSLOG_PORT_KEY = "SYSLOG_PORT"
SYSLOG_PROTOCOL_KEY = "SYSLOG_PROTOCOL"
SYSLOG_ENABLED_KEY = "SYSLOG_ENABLED"
SYSLOG_HOST_KEY = "syslog_host"
SYSLOG_PORT_KEY = "syslog_port"
SYSLOG_PROTOCOL_KEY = "syslog_protocol"
SYSLOG_ENABLED_KEY = "syslog_enabled"

# Default values for Syslog
DEFAULT_SYSLOG_HOST = "localhost"
Expand All @@ -78,3 +82,8 @@
DEFAULT_SYSLOG_ENABLED = False

SYSLOG_PROTOCOLS = {"UDP": socket.SOCK_DGRAM, "TCP": socket.SOCK_STREAM}

DETAILED_LOGGING_FORMAT = (
"%(asctime)s [VDK] %(vdk_job_name)s [%(levelname)-5.5s] %(name)-30.30s %(filename)20.20s:%("
"lineno)-4.4s %(funcName)-16.16s[id:%(attempt_id)s]- %(message)s"
)
Loading

0 comments on commit d0ef338

Please sign in to comment.