Quantcast

Performance in the disk storage

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Performance in the disk storage

MEYNARD Rolih

Hi,

I have realised compression tests from only HDF5 library.
In these tests the size of chunks are varied in the following way:

1) Firstly, I have changed the size of chunks with even values: 10*10,  
12*12, 14*14, 16*16, etc;
2)Secondly, I have modified the size of chunks with odd values: 11*11,  
13*13, 15*15, 17*17, etc.

I have got 2 graphic lines. Please find enclosed the file called  
"dataset_square_sdisk_pair_impair_1000.png"

The x numbers are the size of chunks in octets.
The y numbers are the size of file in the disk in bytes


-The red line represents chunks with an even size (10*10,12*12,...);
-The green line represents chunks with an odd size (11*11, 13*13, etc).

My question is: Do we have a reason that would explain why we observe  
a better performance in the disk storage with chunks of an even size  
that an odd size ?

Thanks you,

Rolih


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

dataset_square_sdisk_pair_impair_1000.png (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance in the disk storage

Werner Benger

Hi,

 the performance is probably best if the chunk size is an integer multiple of the disk's block size, which is usually a power of two, unless you're limited by caching and memory buffer / copying.

If you used compression during your disk tests, which compression filters did you use? The read / write performance of all those various filters and their settings would be quite interesting to see. I've got best performance with the LZ4 filter (which was actually recommended here in the mailing list some time ago), it achieves write speed nearly as good as uncompressed data. Whereas the intrinsic zip filters achieve 2x as good compression rate, but at the cost of being significantly slower, in some cases even 200x slower.

      Werner


On 06.09.2016 15:20, MEYNARD Rolih wrote:

Hi,

I have realised compression tests from only HDF5 library.
In these tests the size of chunks are varied in the following way:

1) Firstly, I have changed the size of chunks with even values: 10*10, 12*12, 14*14, 16*16, etc;
2)Secondly, I have modified the size of chunks with odd values: 11*11, 13*13, 15*15, 17*17, etc.

I have got 2 graphic lines. Please find enclosed the file called "dataset_square_sdisk_pair_impair_1000.png"

The x numbers are the size of chunks in octets.
The y numbers are the size of file in the disk in bytes


-The red line represents chunks with an even size (10*10,12*12,...);
-The green line represents chunks with an odd size (11*11, 13*13, etc).

My question is: Do we have a reason that would explain why we observe a better performance in the disk storage with chunks of an even size that an odd size ?

Thanks you,

Rolih



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance in the disk storage

MEYNARD Rolih
Thank you Werner for your answer,

Actually, I have only used ZLIB filter with as level of compression 6  
(setdeflate(6))...

Quoting Werner Benger <[hidden email]>:

> Hi,
>
>  the performance is probably best if the chunk size is an integer  
> multiple of the disk's block size, which is usually a power of two,  
> unless you're limited by caching and memory buffer / copying.
>
> If you used compression during your disk tests, which compression  
> filters did you use? The read / write performance of all those  
> various filters and their settings would be quite interesting to  
> see. I've got best performance with the LZ4 filter (which was  
> actually recommended here in the mailing list some time ago), it  
> achieves write speed nearly as good as uncompressed data. Whereas  
> the intrinsic zip filters achieve 2x as good compression rate, but  
> at the cost of being significantly slower, in some cases even 200x  
> slower.
>
>       Werner
>
>
> On 06.09.2016 15:20, MEYNARD Rolih wrote:
>>
>> Hi,
>>
>> I have realised compression tests from only HDF5 library.
>> In these tests the size of chunks are varied in the following way:
>>
>> 1) Firstly, I have changed the size of chunks with even values:  
>> 10*10, 12*12, 14*14, 16*16, etc;
>> 2)Secondly, I have modified the size of chunks with odd values:  
>> 11*11, 13*13, 15*15, 17*17, etc.
>>
>> I have got 2 graphic lines. Please find enclosed the file called  
>> "dataset_square_sdisk_pair_impair_1000.png"
>>
>> The x numbers are the size of chunks in octets.
>> The y numbers are the size of file in the disk in bytes
>>
>>
>> -The red line represents chunks with an even size (10*10,12*12,...);
>> -The green line represents chunks with an odd size (11*11, 13*13, etc).
>>
>> My question is: Do we have a reason that would explain why we  
>> observe a better performance in the disk storage with chunks of an  
>> even size that an odd size ?
>>
>> Thanks you,
>>
>> Rolih
>>
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [hidden email]
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>
> --
> ___________________________________________________________________________
> Dr. Werner Benger                Visualization Research
> Center for Computation & Technology at Louisiana State University (CCT/LSU)
> 2019  Digital Media Center, Baton Rouge, Louisiana 70803
> Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...