Understanding about space allocation issue on compression in parallel

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Understanding about space allocation issue on compression in parallel

MEYNARD Rolih
Hi,

Before compression, the size of chunks is the same. But, I would like  
to know if after compression chunks are the  same size ?

Secondly, I need to understand the issue about space allocation on  
compression in parallel:
If I have well understood: data are firstly compressed then can be  
write in parallel on file. But, process don't know where to write  
chunks in file when they are compressed. The reason is they don't know  
the new size of chunks compressed.
Is it right what I mean ?

Thank you in advance,

Rolih


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Understanding about space allocation issue on compression in parallel

Gerd Heber
Hi Rolih,
 
> Before compression, the size of chunks is the same. But, I would like to know if after
> compression chunks are the  same size ?

No, the size of the compressed chunks will generally be different for different
chunks (unless they contain the exact same data). In the worst case, the size
of a compressed chunk could be larger than the original chunk size.

> Secondly, I need to understand the issue about space allocation on compression in parallel:

Compression is currently (!) not supported for parallel writes.
However, you can read compressed datasets in parallel.

Best, G.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...