Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for Float 32-bit. #6109

Open
radeusgd opened this issue Mar 28, 2023 · 1 comment
Open

In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for Float 32-bit. #6109

radeusgd opened this issue Mar 28, 2023 · 1 comment
Labels
-libs Libraries: New libraries to be implemented l-table-column-datatypes p-lowest Should be completed at some point x-on-hold

Comments

@radeusgd
Copy link
Member

radeusgd commented Mar 28, 2023 •
edited
Loading

With #5159 we can set our storage to accept integers of limited size.

For now however, we've been still storing them as 64-bit ints, because having alternative storages requires lots of boilerplate and there was not enough time to implement it.

This task is to actually attempt it and alter the storage to store 32-bit integers as Java int arrays, and 16-bit as short arrays.

Similarly, it should add support for storing Float 32-bit as float arrays (currently only double are supported).

To do so and keep it maintainable, we may need to create a code generator, to avoid duplicating tons of code.

Value

  • We can store more versatile datatypes (support for 32-bit floats apart from only 64-bit ones).
  • Our storage will be more compact, saving memory used by Tables.
@radeusgd radeusgd mentioned this issue Mar 28, 2023
Closed
8 tasks
@radeusgd radeusgd added p-lowest Should be completed at some point -libs Libraries: New libraries to be implemented l-table-column-datatypes labels Mar 28, 2023
@radeusgd radeusgd changed the title In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for 32-bit floats In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for 32-bit floats. Mar 28, 2023
@radeusgd radeusgd mentioned this issue Mar 28, 2023
Closed
@radeusgd radeusgd changed the title In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for 32-bit floats. In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for Float 32-bit. Mar 28, 2023
@radeusgd radeusgd mentioned this issue Mar 28, 2023
Closed
2 tasks
@jdunkerley jdunkerley moved this from ❓New to 📤 Backlog in Issues Board Mar 28, 2023
@radeusgd radeusgd mentioned this issue Apr 13, 2023
Closed
@radeusgd radeusgd mentioned this issue Aug 10, 2023
Merged
5 tasks
mergify bot pushed a commit that referenced this issue Aug 22, 2023
@radeusgd
Add size-limited strings and varying bit-width integer Value_Types to…
2385f5b 
… in-memory backend and check for ArithmeticOverflow in LongStorage (#7557)

- Closes #5159
- Now data downloaded from the database can keep the type much closer to the original type (like string length limits or smaller integer types).
- Cast also exposes these types.
- The integers are still all stored as 64-bit Java `long`s, we just check their bounds. Changing underlying storage for memory efficiency may come in the future: #6109
- Fixes #7565
- Fixes #7529 by checking for arithmetic overflow in in-memory integer arithmetic operations that could overflow. Adds a documentation note saying that the behaviour for Database backends is unspecified and depends on particular database.
@jdunkerley
Copy link
Member

Subject to review when we look at Apache Arrow

@jdunkerley jdunkerley moved this from 📤 Backlog to ❓New in Issues Board Jan 9, 2024
@radeusgd radeusgd mentioned this issue Jul 29, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
-libs Libraries: New libraries to be implemented l-table-column-datatypes p-lowest Should be completed at some point x-on-hold
Projects
Issues Board
Status: ❓New
Milestone
No milestone
Development

No branches or pull requests
2 participants
@radeusgd @jdunkerley