Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crawl log #231

Merged
merged 5 commits into from
Feb 25, 2023
Merged

Crawl log #231

merged 5 commits into from
Feb 25, 2023

Conversation

ikreymer
Copy link
Member

Write crawl log to {coll}/logs/crawl-{iso-timestamp}.log from when crawl is started to just before WACZ creation starts (to allow storing log in WACZ).

Still requires py-wacz support to add logs.

@ikreymer ikreymer requested a review from tw4l February 23, 2023 17:27
@ikreymer ikreymer linked an issue Feb 23, 2023 that may be closed by this pull request
Copy link
Member

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Code looks good and worked well when I tested 🎉

Copy link
Member

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thought: the timestamp included in the filename has some colons, which are reserved characters in a number of file systems. Might be better to remove the colons or replace them with dashes?

- write most of the crawl log to '{coll}/logs/crawl-{iso-timestamp}.log', part of #230
- close log before wacz file is generated to allow storing log in wacz
wacz: add logs/ directory to wacz with new py-wacz
@ikreymer ikreymer marked this pull request as ready for review February 24, 2023 23:42
@ikreymer ikreymer merged commit 63717c4 into main Feb 25, 2023
@ikreymer ikreymer deleted the crawl-log branch February 25, 2023 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Logging: Log to file and store logs to WACZ
2 participants