-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add event_logger to test_connection and create_database commands #13468
Conversation
…into hugh/event-logger-refactor
Co-authored-by: Beto Dealmeida <[email protected]>
…e/incubator-superset into hugh/event-logger-refactor
Co-authored-by: Beto Dealmeida <[email protected]>
…into hugh/event-logger-refactor
…into hugh/event-logger-refactor
…into hugh/event-logger-refactor
…into hugh/use-event-db
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left a few comments and mentioned a couple nits, but I have one concern with the rollback()
that's being called inside the event_logger
context manager. We should move it out and call it before logging, to ensure that it goes through even if logging fails.
with event_logger( | ||
action="db_connection_failed", | ||
engine=database.db_engine_spec.__name__, | ||
): | ||
db.session.rollback() | ||
raise DatabaseConnectionFailedError() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For places where we do a rollback we should it outside the context manager, otherwise if the logging fails the transaction won't be rolled back:
with event_logger( | |
action="db_connection_failed", | |
engine=database.db_engine_spec.__name__, | |
): | |
db.session.rollback() | |
raise DatabaseConnectionFailedError() | |
db.session.rollback() | |
event_logger.log_with_context( | |
action="db_connection_failed", | |
engine=database.db_engine_spec.__name__, | |
) | |
raise DatabaseConnectionFailedError() |
Reversing the order ensures that even if the logging fails the rollback will still go through.
with event_logger( | ||
action=f"db_creation_failed.{ex.__class__.__name__}", | ||
engine=database.db_engine_spec.__name__, | ||
): | ||
raise DatabaseCreateFailedError() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Total nit, but in places where we just raise an exception inside the context manager like here and in lines 94-97 I think it would make more sense to use the log_with_context
function instead:
with event_logger( | |
action=f"db_creation_failed.{ex.__class__.__name__}", | |
engine=database.db_engine_spec.__name__, | |
): | |
raise DatabaseCreateFailedError() | |
event_logger.log_with_context( | |
action=f"db_creation_failed.{ex.__class__.__name__}", | |
engine=database.db_engine_spec.__name__, | |
) | |
raise DatabaseCreateFailedError() |
if database is None: | ||
raise DBAPIError("Database is not found", None, None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can actually remove this and fix the type annotation in DatabaseDAO.build_db_for_connection_test
, the class method never returns None
.
tests/databases/commands_tests.py
Outdated
def test_connection_db_exception( | ||
self, mock_event_logger, mock_build_db_for_connection_test | ||
): | ||
"""Test that users can't export databases they don't have access to""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update docstring here and in the other 2 tests 😄
tests/databases/commands_tests.py
Outdated
with self.assertRaises(DatabaseTestConnectionUnexpectedError): | ||
command_without_db_name.run() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that we're using pytest
, it's better to run tests using plain assert
s instead, since it gives more context when the test fails. Also, we can check that the exception raised has the right error message:
with self.assertRaises(DatabaseTestConnectionUnexpectedError): | |
command_without_db_name.run() | |
with pytest.raises(DatabaseTestConnectionUnexpectedError) as excinfo: | |
command_without_db_name.run() | |
assert str(excinfo.value) == ( | |
"Unexpected error occurred, please check your logs for details" | |
) |
Same in the other 2 tests below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
logger.exception(ex.exception) | ||
event_logger.log_with_context( | ||
action=f"db_creation_failed.{ex.__class__.__name__}", | ||
engine=database.db_engine_spec.__name__, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hughhhh what did you want to do about making the engine string consistent around each of these logs so that we can group by the same engine later when observing the logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I ended up changing all the engine references in the event_logger.log_with_context
to pull data from the database.db_engine_spec.__name__
I say we go with this for now and see the data then see if we need to create some mapping either in the service or downstream
apache#13468) Co-authored-by: Beto Dealmeida <[email protected]>
SUMMARY
Leverage refactored
event_logger
with context managementDependent on #13441
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TEST PLAN
ADDITIONAL INFORMATION