Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch loading #5

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Batch loading #5

wants to merge 12 commits into from

Conversation

Iaxama
Copy link
Collaborator

@Iaxama Iaxama commented Aug 19, 2020

This PR adds the possibility to load large ROS bags datasets without filling your RAM completely.
To achieve that the bag is read in its entirety but only the info about a certain chunk of data is retrieved. These infos contain the start and end timestamp of the data contained in that chunk and the position in the file where the chunk can be read. The data can therefore be read on request providing the desired time range. All the data outside that time range can be offloaded from RAM for retrieving it later. This can be done through the DataProvider class which needs to be initialized with the proper function to load the data. In fact that class only provides an interface for any module that needs to retrieve data on request, in order to allow extension of current implementation to formats other than ROS bags.

Being a major change further testing is recommended before merging.

@Iaxama Iaxama added the enhancement New feature or request label Aug 19, 2020
@Iaxama Iaxama requested a review from simbamford August 19, 2020 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants