🎉 CDK: use standard logger with custom handler #6085

vitaliizazmic · 2021-09-15T14:09:02Z

What

Use standard logger with custom handler

Closes #1279

How

Create according to issue AirbyteLogFormatter and AirbyteNativeLogger, which inherits logging.Logger.

airbyte-cdk/python/airbyte_cdk/logger.py

keu · 2021-09-15T14:44:24Z

airbyte-cdk/python/airbyte_cdk/logger.py

+        logging.Logger.__init__(self, name)
+        self.valid_log_types = {"FATAL": 50, "ERROR": 40, "WARN": 30, "INFO": 20, "DEBUG": 10, "TRACE": 5}
+
+    def log_by_prefix(self, msg, default_level):


add docstrings please

keu · 2021-09-15T14:45:27Z

airbyte-cdk/python/CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog

+## 0.1.19
+Use standard logger with custom handler


Suggested change

Use standard logger with custom handler

Use python standard logging instead of custom class

keu

can we have simple test for this?

gaart

Agree with @keu comments

vitaliizazmic · 2021-09-17T13:10:12Z

can we have simple test for this?

I need advice how to better test logging.

keu · 2021-09-23T13:19:03Z

airbyte-cdk/python/airbyte_cdk/logger.py

+def get_logger():
+    logging.setLoggerClass(AirbyteNativeLogger)
+    logging.addLevelName(TRACE_LEVEL_NUM, "TRACE")
+    logger = logging.getLogger("airbyte.source.zabbix")


again, why did you hardcode this name?

also, you need to set up this for root-level logger only. We are not particularly interested in return result, this will configure global object, so all loggers derived from root will follow this configuration, i.e.

logging.info(...)

will work as intended

keu · 2021-09-23T14:03:14Z

airbyte-cdk/python/airbyte_cdk/sources/streams/core.py

@@ -45,7 +45,7 @@ class Stream(ABC):
    """

    # Use self.logger in subclasses to log any messages
-    logger = AirbyteLogger()  # TODO use native "logging" loggers with custom handlers
+    logger = get_logger()


Suggested change

logger = get_logger()

@property

def logger(self):

return logging.getLogger(f'streams.{self.name}')

keu · 2021-09-23T14:10:24Z

airbyte-cdk/python/airbyte_cdk/logger.py

+    logging.setLoggerClass(AirbyteNativeLogger)
+    logging.addLevelName(TRACE_LEVEL_NUM, "TRACE")
+    logger = logging.getLogger("airbyte.source.zabbix")
+    logger.setLevel(TRACE_LEVEL_NUM)


also please use dict like configuration

LOGGING_CONFIG = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'airbyte': { 'class': 'AirbyteLogFormatter, 'format': '%(message)s' }, }, 'handlers': { 'console': { 'class': 'logging.StreamHandler', 'formatter': 'simple', 'level': 'TRACE' }, }, 'root': { 'handlers': ['console'], 'level': 'TRACE' } } logging.config.dictConfig(config)

keu · 2021-09-23T14:13:03Z

airbyte-cdk/python/airbyte_cdk/logger.py

+    return logger
+
+
+class AirbyteLogFormatter(logging.Formatter):


keu · 2021-09-23T14:13:11Z

airbyte-cdk/python/airbyte_cdk/logger.py

+
+
+class AirbyteLogFormatter(logging.Formatter):
+    def format(self, record):


annotations

keu · 2021-09-23T14:13:43Z

airbyte-cdk/python/airbyte_cdk/logger.py

+
+
+class AirbyteNativeLogger(logging.Logger):
+    def __init__(self, name):


docstrings explaining why do we this class would be nice

keu · 2021-09-23T14:34:51Z

airbyte-cdk/python/airbyte_cdk/logger.py

+
+class AirbyteNativeLogger(logging.Logger):
+    def __init__(self, name):
+        logging.Logger.__init__(self, name)


Suggested change

logging.Logger.__init__(self, name)

super().__init__(name)

keu

see comments

CLAassistant · 2021-09-27T17:25:32Z

All committers have signed the CLA.

keu · 2021-10-04T08:59:58Z

@vitaliizazmic to test logging try to mock stdout, init logging and log different messages with different levels

davinchia · 2021-10-12T06:46:43Z

airbyte-cdk/python/airbyte_cdk/entrypoint.py


 class AirbyteEntrypoint(object):
    def __init__(self, source: Source):
        self.source = source
+        self.logger = init_logger(source.name)


# Conflicts: # airbyte-cdk/python/CHANGELOG.md # airbyte-cdk/python/setup.py

vitaliizazmic · 2021-10-12T14:38:22Z

/publish-cdk dry-run=false

❌ https://github.com/airbytehq/airbyte/actions/runs/1333576525
❌ https://github.com/airbytehq/airbyte/actions/runs/1333576525
❌ https://github.com/airbytehq/airbyte/actions/runs/1333576525
❌ https://github.com/airbytehq/airbyte/actions/runs/1333576525

vitaliizazmic · 2021-10-12T16:32:15Z

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/1334000460
✅ https://github.com/airbytehq/airbyte/actions/runs/1334000460

vitaliizazmic · 2021-10-12T17:21:13Z

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/1334153204
✅ https://github.com/airbytehq/airbyte/actions/runs/1334153204

vitaliizazmic added 5 commits September 15, 2021 00:07

CDK - using native logger with custom formatter

bb3bb9d

CDK - using native logger(fix exception and add trace)

c51d7b2

CDK - using native logger in AirbyteEntrypoint

db606c5

CDK - CHANGELOG.md

3ba9c94

CDK - remove unnecessary imports

06873c0

github-actions bot added the CDK Connector Development Kit label Sep 15, 2021

vitaliizazmic requested review from eliziario, gaart and keu September 15, 2021 14:09

keu reviewed Sep 15, 2021

View reviewed changes

airbyte-cdk/python/airbyte_cdk/logger.py Outdated Show resolved Hide resolved

keu reviewed Sep 15, 2021

View reviewed changes

airbyte-cdk/python/airbyte_cdk/logger.py Outdated Show resolved Hide resolved

keu reviewed Sep 15, 2021

View reviewed changes

airbyte-cdk/python/airbyte_cdk/logger.py Outdated Show resolved Hide resolved

keu reviewed Sep 15, 2021

View reviewed changes

airbyte-cdk/python/airbyte_cdk/logger.py Outdated Show resolved Hide resolved

keu reviewed Sep 15, 2021

View reviewed changes

keu suggested changes Sep 15, 2021

View reviewed changes

gaart reviewed Sep 16, 2021

View reviewed changes

CDK - fixing according to PR review

1a76a88

vitaliizazmic requested a review from keu September 17, 2021 13:10

vitaliizazmic self-assigned this Sep 17, 2021

keu reviewed Sep 23, 2021

View reviewed changes

airbyte-cdk/python/airbyte_cdk/logger.py

return logger

class AirbyteLogFormatter(logging.Formatter):

Copy link

Contributor

keu Sep 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docstrings

keu reviewed Sep 23, 2021

View reviewed changes

keu suggested changes Sep 23, 2021

View reviewed changes

vitaliizazmic requested a review from gaart September 30, 2021 16:17

vitaliizazmic added 2 commits October 7, 2021 21:21

CDK standard logger #1279 - tests

b9c6e61

CDK standard logger #1279 - reformat

dde75aa

keu approved these changes Oct 8, 2021

View reviewed changes

vitaliizazmic requested review from sherifnada, davinchia and Phlair October 8, 2021 14:40

davinchia reviewed Oct 12, 2021

View reviewed changes

davinchia approved these changes Oct 12, 2021

View reviewed changes

Airbyte CDK #1279 - improve docstrings

f9354f4

github-actions bot added the area/protocol label Oct 12, 2021

vitaliizazmic added 2 commits October 12, 2021 17:28

Airbyte CDK #1279 - improve log levels

c5ce983

Merge branch 'master' into vitalii/1279_cdk_standard_logger

7103780

# Conflicts: # airbyte-cdk/python/CHANGELOG.md # airbyte-cdk/python/setup.py

vitaliizazmic temporarily deployed to more-secrets October 12, 2021 14:34 Inactive

vitaliizazmic added 2 commits October 12, 2021 19:30

Airbyte CDK #1279 - fix init get source name

00e1d26

Merge branch 'master' into vitalii/1279_cdk_standard_logger

e3803f2

vitaliizazmic temporarily deployed to more-secrets October 12, 2021 16:33 Inactive

Airbyte CDK #1279 - update test licence

249dd02

vitaliizazmic temporarily deployed to more-secrets October 12, 2021 16:52 Inactive

Airbyte CDK #1279 - bump version

8f1f4b8

vitaliizazmic temporarily deployed to more-secrets October 12, 2021 17:22 Inactive

vitaliizazmic temporarily deployed to more-secrets October 12, 2021 17:32 Inactive

vitaliizazmic merged commit d5c0499 into master Oct 12, 2021

vitaliizazmic deleted the vitalii/1279_cdk_standard_logger branch October 12, 2021 17:56

This was referenced Oct 13, 2021

🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels #7024

Merged

🐛 Airbyte CDK: fixing integration test failing #7078

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 CDK: use standard logger with custom handler #6085

🎉 CDK: use standard logger with custom handler #6085

vitaliizazmic commented Sep 15, 2021

keu Sep 15, 2021

keu Sep 15, 2021 •

edited

Loading

keu left a comment •

edited

Loading

gaart left a comment

vitaliizazmic commented Sep 17, 2021

keu Sep 23, 2021

keu Sep 23, 2021 •

edited

Loading

keu Sep 23, 2021

keu Sep 23, 2021

keu Sep 23, 2021

keu Sep 23, 2021

keu Sep 23, 2021

keu Sep 23, 2021

keu left a comment

CLAassistant commented Sep 27, 2021 •

edited

Loading

keu commented Oct 4, 2021

davinchia Oct 12, 2021

vitaliizazmic commented Oct 12, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 12, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 12, 2021 •

edited by github-actions bot

Loading

	Use standard logger with custom handler
	Use python standard logging instead of custom class

-    logger = get_logger()
+    @property
+    def logger(self):
+        return logging.getLogger(f'streams.{self.name}')



		class AirbyteLogFormatter(logging.Formatter):
		def format(self, record):



		class AirbyteNativeLogger(logging.Logger):
		def __init__(self, name):

🎉 CDK: use standard logger with custom handler #6085

🎉 CDK: use standard logger with custom handler #6085

Conversation

vitaliizazmic commented Sep 15, 2021

What

How

Choose a reason for hiding this comment

keu Sep 15, 2021 • edited Loading

Choose a reason for hiding this comment

keu left a comment • edited Loading

Choose a reason for hiding this comment

gaart left a comment

Choose a reason for hiding this comment

vitaliizazmic commented Sep 17, 2021

Choose a reason for hiding this comment

keu Sep 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keu left a comment

Choose a reason for hiding this comment

CLAassistant commented Sep 27, 2021 • edited Loading

keu commented Oct 4, 2021

Choose a reason for hiding this comment

vitaliizazmic commented Oct 12, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Oct 12, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Oct 12, 2021 • edited by github-actions bot Loading

keu Sep 15, 2021 •

edited

Loading

keu left a comment •

edited

Loading

keu Sep 23, 2021 •

edited

Loading

CLAassistant commented Sep 27, 2021 •

edited

Loading

vitaliizazmic commented Oct 12, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 12, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 12, 2021 •

edited by github-actions bot

Loading