Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to read a single value or a specific range? #258

Open
Simone98 opened this issue Dec 14, 2022 · 6 comments
Open

How to read a single value or a specific range? #258

Simone98 opened this issue Dec 14, 2022 · 6 comments

Comments

@Simone98
Copy link

Simone98 commented Dec 14, 2022

Hi, I'm opening a hdf5 of 40gb, and I'm trying to get some data without opening all the dataset.
I have this:
image
And with them i can read data from a 3 dimensional array.
image

Storing all the dataset in an array and then cicling it is impossible, because it's way too much for the array maximum limit.
Is there a way to get only the wanted values using three indexes?

Sorry if I used the wrong place to ask.

@LiorBanai
Copy link
Owner

Hi. This is good place to ask.

I can add partial read but I don't understand the current dataset structure. Could you add example file (small one) and give example of slice data you would like to read?

I can work on this with good datase for testing.

@Simone98
Copy link
Author

HRS_ITA_1KM_LEAFWET_HOURLY_CREATED.zip

This is an example file.
It has a small dataset of float[,,].

The real one has these dimensions:
float[timespans, lat, long] => float[16775,1281,1729]

It would be super useful in terms of memory to read only 1 timespan for each cycle, in that case i would get only 1281*1729 floats everytime, without being worry about the size of the array.

@LiorBanai
Copy link
Owner

Thanks. I'll work on the next week

@LiorBanai
Copy link
Owner

@Simone98 btw if you only need read of hdf5 I high recommend https://github.com/Apollo3zehn/HDF5.NET library as it has pure .net implementation and it seems to support partial read per the readmd

@Simone98
Copy link
Author

Thank you for the suggestion @LiorBanai. After hours spent on the various docs I successfully read the files without loading all of it at once.

Thank you for the help, I solved my problem with the Apollo3zehn solution.

@LiorBanai
Copy link
Owner

Glad to know it helped you. I do planned to add that ability also to my library later on but it is good you are not blocked on this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants