Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stream file output (instead of RAM) #28

Closed
stooit opened this issue Jul 18, 2019 · 2 comments
Closed

Stream file output (instead of RAM) #28

stooit opened this issue Jul 18, 2019 · 2 comments
Labels
enhancement New feature or request
Milestone
0.4.0

Comments

@stooit
Copy link
Contributor

stooit commented Jul 18, 2019

Currently data rows and errors are stored in memory during a run.

To avoid excess resource usage this should stream output to files instead. This impacts for both the crawler and migrate commands
@stooit stooit added the enhancement New feature or request label Jul 18, 2019
@stooit stooit added this to the 0.3.0 milestone Jul 18, 2019
@stooit stooit modified the milestones: 0.3.0, 0.4.0 Aug 31, 2019
@stooit
Copy link
Contributor Author

stooit commented Aug 31, 2019

Less of an issue with performance enhancements, change of crawler in use, and cache

@stooit stooit mentioned this issue Nov 6, 2019
Closed
@stooit
Copy link
Contributor Author

stooit commented Apr 27, 2020

Largely resolved by #115, closing for now

@stooit stooit closed this as completed Apr 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
0.4.0
Development

No branches or pull requests
1 participant
@stooit