Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove arrow file generation when vaex 4.0.0 is available #405

Open
ajstewart opened this issue Nov 2, 2020 · 1 comment
Open

Remove arrow file generation when vaex 4.0.0 is available #405

ajstewart opened this issue Nov 2, 2020 · 1 comment
Assignees
Labels
enhancement New feature or request on hold Issue will be addressed but is currently on hold.

Comments

@ajstewart
Copy link
Contributor

ajstewart commented Nov 2, 2020

vaex 4.0.0 includes the ability to open parquet files in an out of core context. This means that the arrow file will no longer be required when this is released.

However from testing it is still beneficial to create a measurements.arrow file that contains all the association information already merged - as this can still take a few minutes to generate with vaex, but is negligible in the context of a pipeline run.

vaex 4.0.0 is currently at the alpha stage, so will wait for a more stable version.

@ajstewart ajstewart added the enhancement New feature or request label Nov 2, 2020
@ajstewart ajstewart self-assigned this Nov 2, 2020
@ajstewart ajstewart added the on hold Issue will be addressed but is currently on hold. label Nov 5, 2020
@ajstewart
Copy link
Contributor Author

This is closely related to issue askap-vast/vast-tools#225 where I have been testing the performance of vaex 4.0.0 on using the parquets.

From recent testing, the performance of the arrow file is still much better compared to the parquets and hence the arrow file is still beneficial.

So for now this remains on hold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request on hold Issue will be addressed but is currently on hold.
Projects
None yet
Development

No branches or pull requests

1 participant