Skip to content
This repository has been archived by the owner on Oct 26, 2023. It is now read-only.

Handle large history export #20

Open
hisabimbola opened this issue Jun 11, 2016 · 0 comments
Open

Handle large history export #20

hisabimbola opened this issue Jun 11, 2016 · 0 comments

Comments

@hisabimbola
Copy link
Owner

If the history being exported is very large, memory consumption might be very large because we storing all history in memory and then writing it to a file after all histories have been fetched. This can be very costly and if an error occurs when fetching messages, all messages will be lost.

Also, another risk for this is when #5 is implemented, memory usage can be very high.

A simple solution is to write the history to file in batch, once gotten from slack API and append subsequent messages to end of the file.

Another elegant solution is to open a NodeJS writable stream and close when done. Writing to file will be faster.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant