Database: minimize PostgreSQL logs for Django backend #5056
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Would be good to try and merge this soon so I can also include this in the v1.6.5 backport release that is planned for tomorrow. |
sphuber
force-pushed
the
fix/5055/postgresql-schema-check-logs
branch
from
f1d8b77 to
0413555
Compare
For the Django backend, we are recording the current version of the database schema manually in our own `DbSetting` table. We use this to determine whether the database schema version is compatible with that of the code that is installed. Since the way the version and generation of the schema are stored change over the years, we had to rely on a try-except method to determine which way was the case. The original try was for the original way of v0.x of `aiida-core`, and if that failed we would fall back to the modern method of storing the info introduced in `aiida-core==1.0`. However, each time the except was hit, meaning a programming error was encountered in the PostgreSQL statement, a number of warning are logged to PostgreSQL. Not only can they appear in stdout if no log file has been configed for the PostgreSQL server (can be the case for conda installations) the log file will also accumulate these necessary logs, which are produced everytime the database version is checked, i.e., when a profile is loaded, which is often. It might not be trivial to get completely rid of these errors being logged but we can invert the try-except logic. Assuming that the majority of users are now on v1.0 or above, they would always be hitting the except. By first checking for the schema-style of v1.0 instead, only users of v0.x will be getting the logs, which should be negligible.
Codecov Report
@@ Coverage Diff @@
## develop #5056 +/- ##
===========================================
- Coverage 80.23% 80.22% -0.01%
===========================================
Files 515 515
Lines 36757 36757
===========================================
- Hits 29490 29485 -5
- Misses 7267 7272 +5
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
giovannipizzi
approved these changes
Aug 9, 2021
sphuber
added a commit
that referenced
this pull request
Aug 9, 2021
For the Django backend, we are recording the current version of the database schema manually in our own `DbSetting` table. We use this to determine whether the database schema version is compatible with that of the code that is installed. Since the way the version and generation of the schema are stored change over the years, we had to rely on a try-except method to determine which way was the case. The original try was for the original way of v0.x of `aiida-core`, and if that failed we would fall back to the modern method of storing the info introduced in `aiida-core==1.0`. However, each time the except was hit, meaning a programming error was encountered in the PostgreSQL statement, a number of warning are logged to PostgreSQL. Not only can they appear in stdout if no log file has been configured for the PostgreSQL server (can be the case for conda installations) the log file will also accumulate these necessary logs, which are produced every time the database version is checked, i.e., when a profile is loaded, which is often. It might not be trivial to get completely rid of these errors being logged but we can invert the try-except logic. Assuming that the majority of users are now on v1.0 or above, they would always be hitting the except. By first checking for the schema-style of v1.0 instead, only users of v0.x will be getting the logs, which should be negligible. Cherry-pick: 9285245
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #5055
For the Django backend, we are recording the current version of the
database schema manually in our own
DbSettingtable. We use this todetermine whether the database schema version is compatible with that of
the code that is installed.
Since the way the version and generation of the schema are stored change
over the years, we had to rely on a try-except method to determine which
way was the case. The original try was for the original way of v0.x of
aiida-core, and if that failed we would fall back to the modern methodof storing the info introduced in
aiida-core==1.0. However, each timethe except was hit, meaning a programming error was encountered in the
PostgreSQL statement, a number of warning are logged to PostgreSQL. Not
only can they appear in stdout if no log file has been configed for the
PostgreSQL server (can be the case for conda installations) the log file
will also accumulate these necessary logs, which are produced everytime
the database version is checked, i.e., when a profile is loaded, which
is often.
It might not be trivial to get completely rid of these errors being
logged but we can invert the try-except logic. Assuming that the
majority of users are now on v1.0 or above, they would always be hitting
the except. By first checking for the schema-style of v1.0 instead, only
users of v0.x will be getting the logs, which should be negligible.