Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels #7024

Merged
merged 5 commits into from
Oct 22, 2021
Merged

🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels #7024

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
82272f3
Airbyte CDK native logger #1279 - transforming Python log levels to A…
vitaliizazmic Oct 13, 2021
fe762cb
Merge branch 'master' into vitalii/1279_cdk_standard_logger_levels
vitaliizazmic Oct 18, 2021
549f05b
Airbyte CDK native logger #1279 - add test for level types transforming
vitaliizazmic Oct 18, 2021
619c932
CDK standard logger #1279 - test for transform critical level to fatal
vitaliizazmic Oct 20, 2021
1e9250e
Merge branch 'master' into vitalii/1279_cdk_standard_logger_levels
vitaliizazmic Oct 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions airbyte-cdk/python/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Changelog

## 0.1.29
Transforming Python log levels to Airbyte protocol log levels

## 0.1.28
Added `check_config_against_spec` parameter to `Connector` abstract class
to allow skipping validating the input config against the spec for non-`check` calls
Expand Down
13 changes: 12 additions & 1 deletion airbyte-cdk/python/airbyte_cdk/logger.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,21 @@ def init_logger(name: str):
class AirbyteLogFormatter(logging.Formatter):
"""Output log records using AirbyteMessage"""

# Transforming Python log levels to Airbyte protocol log levels
level_mapping = {
logging.FATAL: "FATAL",
logging.ERROR: "ERROR",
logging.WARNING: "WARN",
logging.INFO: "INFO",
logging.DEBUG: "DEBUG",
TRACE_LEVEL_NUM: "TRACE",
}

def format(self, record: logging.LogRecord) -> str:
"""Return a JSON representation of the log message"""
message = super().format(record)
log_message = AirbyteMessage(type="LOG", log=AirbyteLogMessage(level=record.levelname, message=message))
airbyte_level = self.level_mapping.get(record.levelno, "INFO")
log_message = AirbyteMessage(type="LOG", log=AirbyteLogMessage(level=airbyte_level, message=message))
return log_message.json(exclude_unset=True)


Expand Down
2 changes: 0 additions & 2 deletions airbyte-cdk/python/airbyte_cdk/models/airbyte_protocol.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,8 @@ class Config:

class Level(Enum):
FATAL = "FATAL"
CRITICAL = "CRITICAL"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the critical log level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davinchia I transform levels by them numeric value. CRITICAL numeric value is equal to FATAL numeric value and will be transform to FATAL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a test for that I am missing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davinchia I've added test for transforming CRITICAL level

ERROR = "ERROR"
WARN = "WARN"
WARNING = "WARNING"
INFO = "INFO"
DEBUG = "DEBUG"
TRACE = "TRACE"
Expand Down
2 changes: 1 addition & 1 deletion airbyte-cdk/python/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

setup(
name="airbyte-cdk",
version="0.1.28",
version="0.1.29",
description="A framework for writing Airbyte Connectors.",
long_description=README,
long_description_content_type="text/markdown",
Expand Down
11 changes: 11 additions & 0 deletions airbyte-cdk/python/unit_tests/test_logger.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,17 @@ def test_formatter(logger, caplog):
assert message == "Test formatter"


def test_level_transform(logger, caplog):
formatter = AirbyteLogFormatter()
logger.warn("Test level transform")
record = caplog.records[0]
formatted_record = formatter.format(record)
formatted_record_data = json.loads(formatted_record)
log = formatted_record_data.get("log")
level = log.get("level")
assert level == "WARN"


def test_trace(logger, caplog):
logger.trace("Test trace 1")
record = caplog.records[0]
Expand Down
2 changes: 0 additions & 2 deletions airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,8 @@ class Config:

class Level(Enum):
FATAL = "FATAL"
CRITICAL = "CRITICAL"
ERROR = "ERROR"
WARN = "WARN"
WARNING = "WARNING"
INFO = "INFO"
DEBUG = "DEBUG"
TRACE = "TRACE"
Expand Down
2 changes: 0 additions & 2 deletions airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,8 @@ definitions:
type: string
enum:
- FATAL
- CRITICAL
- ERROR
- WARN
- WARNING
- INFO
- DEBUG
- TRACE
Expand Down