Re: Interests to create chunks for a compression data
Den 6 sep. 2016 6:33 em skrev "MEYNARD Rolih" <[hidden email]>:
> I would like to know the reasons why it is required to create chunks for compression on HDF5 ?
> For example Why is not possible to compress dataset without creating of chunks ?
I would assume that without chunks, the entire dataset would need to be decompressed just to access a single value. It's therefore better to create chunks, such that when accessing a value, only the chunk in which it lies must be decompressed.
you can make a chunk as large as 4GB, which may cover the entire
dataset at once, if you prefer so. For datasets larger than 4GB, then
chunking it into sections that can be accessed with 32-bit indices seems
If you always read an entire dataset at once, then using a single chunk
may be just fine. However, if you ever use hyperslabs and you want to
read only a part of a bigger datasets, then the ability to read only
those parts, and even more, to decompress only those parts that are of
interested instead of needing to decompress the entire dataset will be
beneficial. Also, if you have multiple chunks, then each chunk can be
decompressed independently, thus in parallel (though I don't know if
there are filters implemented that way for serial HDF5).
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362