[hdf-forum] Performance anomalies

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] Performance anomalies

Nikhil Laghave
Hello,

I wanted some expert opinion for some HDF5 work that I am doing. To simulate
the IO of our nuclear physics code, I had made a test program. Now I have a
HDF5 file(quite large). So the program reads this HDF5 file(Here I measure the
read performance) and the I write out this data into another HDF5 file(Here I
measure the write performance).

Now on measuring the timings, I am getting some weird timings. For ~2GB file,
the read time is usually quite fast(with some outliers) but the write times are
quite large. For ex. Read time = 4 secs and Write time = 40 secs(approx). The
wrapper that I have written for the reads and writes are very similar and I
don't see much implementation difference in the reads and writes. Could someone
give me some suggestion as to what I may be doing wrong ?

One more thing is that although the file size is around 2GB, one read/write
operation only involves around 9MB of data....For example, 15 processors
writing 9 MB and this runs in a loop for 15 times(15*15*9 = ~2GB). Could this
small size be causing the slow speed ?


I am attaching the code for this program and a small HDF5 file to illustrate
how it works. I would appreciate if someone can point what I am doing wrong.

--
Regards,
Nikhil
-------------- next part --------------
A non-text attachment was scrubbed...
Name: HDFIO.zip
Type: application/x-zip-compressed
Size: 999454 bytes
Desc: not available
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20081103/5f32fc55/attachment.bin>
-------------- next part --------------
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.

Reply | Threaded
Open this post in threaded view
|

[hdf-forum] Performance anomalies

Rob Latham
On Mon, Nov 03, 2008 at 02:05:50PM -0600, Nikhil Laghave wrote:

> Now on measuring the timings, I am getting some weird timings. For
> ~2GB file, the read time is usually quite fast(with some outliers)
> but the write times are quite large. For ex. Read time = 4 secs and
> Write time = 40 secs(approx). The wrapper that I have written for
> the reads and writes are very similar and I don't see much
> implementation difference in the reads and writes. Could someone
> give me some suggestion as to what I may be doing wrong ?
>
> One more thing is that although the file size is around 2GB, one
> read/write operation only involves around 9MB of data....For
> example, 15 processors writing 9 MB and this runs in a loop for 15
> times(15*15*9 = ~2GB). Could this small size be causing the slow
> speed ?

9 MB of data per process should be just fine (well, not knowing
anything about your parallel file system or network, but generally
anything over 1MB is good -- more is always better, though :> )

One thing that immediately jumps out at me is that you have explicitly
enabled independent i/o in both the read and write cases
(H5FD_MPIO_INDEPENDENT_F).  Wouldn't it be better if you used
collective I/O ?   I only looked at your code quickly... am I missing
some some aspect of your hyperslab layout that would make indep better
than collective?

Is that enough to explain a 10x performance difference between reads
and writes?  Maybe... depends on your file system.   What FS are you
using?  Also, what MPI implementation is this?

==rob

--
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.