[hdf-forum] HDF5 performance issue

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] HDF5 performance issue

Nikhil Laghave
Hello All,

I have been trying to improve the IO performance in our nuclear physics code but
HDF5 is consistently performing much slower than binary in terms of IO even when
I use multiple processors in HDF5 as against single processor in HDF5.

To get some expert opinion, I have written 2 small programs that does only the
IO. Can some one please have a look at the code and try running it ?

When I run these 2 codes, binary program writes 2GB file(root only) is about 10
seconds. On the contrary, HDF5 program takes about 40 seconds to write this 2 GB
with 23 processors.

I can put the code as an attachment, but the files required to be read(64 MB)
cannot be attached. Could anyone suggest how can I send these files ?

--
Regards,
Nikhil

Regards,
Nikhil




----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] HDF5 performance issue

Ger van Diepen
Hi Nikhil,

No programs were attached.

Have you done an fsync in your binary program to be sure your data are really written to disk? Otherwise you only measure the time it takes to copy your data to the system's file buffers.

Do you write from your 23 processors into a single HDF5 file or multiple files?

Cheers,
Ger


 
>>> "Nikhil Laghave" <nikhill at iastate.edu> 09/12/08 1:48 AM >>>
Hello All,

I have been trying to improve the IO performance in our nuclear physics code but
HDF5 is consistently performing much slower than binary in terms of IO even when
I use multiple processors in HDF5 as against single processor in HDF5.

To get some expert opinion, I have written 2 small programs that does only the
IO. Can some one please have a look at the code and try running it ?

When I run these 2 codes, binary program writes 2GB file(root only) is about 10
seconds. On the contrary, HDF5 program takes about 40 seconds to write this 2 GB
with 23 processors.

I can put the code as an attachment, but the files required to be read(64 MB)
cannot be attached. Could anyone suggest how can I send these files ?

--
Regards,
Nikhil

Regards,
Nikhil




----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.