[hdf-forum] HDF5 Performance

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] HDF5 Performance

Nikhil Laghave
Hello,

I am sorry I did not attach any files since it wasn't possible to attach a 64
MB file. I am attaching the program for a smaller run.

The folder ptestrun contains the program to read a binary file "mfdn.smwf001"
and write the same binary file into another file "binopfile". It also uses a
HDF5 file hdfref, to read the lengths being read and written by the processors.
The program does IO in exactly the same way our actual code does. IO happens as
follows.
1. Proc 0 reads its part of the vector. Then it reads the vector for the
remaining processors and sends(tag=1) it to the respective processors.
2. Proc 0 writes its part of the vector. Then it writes the vector for the
remaining processors that it receives(tag=2) from the respective processors.

The folder testrun contains the program to write the output file in a HDF5
file. It reads the contents from a HDF5 file "mfdn.smwf001" and writes the
dataset named "EigenVectors__" to a new HDF5 file "hdfopfile". The lengths
written by various procs can stored in the dataset "LENGTHS". The IO happens as
follows.

1. Get the dataset dims from the datasets named "EigenVectors__"
2. Create the HDF5 o/p file accordingly.
3. Read the datasets from "mfdn.smwf001" and write it to "hdfopfile"

Although this is a small run and may not fully utilize PHDF5 but may be someone
can tell if I am making any mistakes that is leading to the slow IO. I can't
send the larger runs as attachments since they are around 64 MB and 2 GB.

This program should run exactly on 5 processors.

Thanks a lot. Hope to get some pointers for improving the code.

Regards,
Nikhil








----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] HDF5 Performance

Rob Latham
On Fri, Sep 12, 2008 at 12:55:04PM -0500, Nikhil Laghave wrote:
> Hello,
>
> I am sorry I did not attach any files since it wasn't possible to attach a 64
> MB file. I am attaching the program for a smaller run.

Hi.  I do this all the time too -- but there was no code in your
message.

> The folder ptestrun contains the program to read a binary file "mfdn.smwf001"
> and write the same binary file into another file "binopfile". It also uses a
> HDF5 file hdfref, to read the lengths being read and written by the
> processors.  The program does IO in exactly the same way our actual
> code does. IO happens as
> follows.
> 1. Proc 0 reads its part of the vector. Then it reads the vector for the
> remaining processors and sends(tag=1) it to the respective processors.
> 2. Proc 0 writes its part of the vector. Then it writes the vector for the
> remaining processors that it receives(tag=2) from the respective processors.

This approach of having rank 0 drive the entire I/O process is going
to give you pretty poor performance, right?  

You're on the right track, using MPI to carry out some coordination
before performing I/O.   Just take that a little further.

parallel HDF5 really needs two things to perform well:
- simultaneous accesses from all processes (via hyperslabs)
- collective I/O

> The folder testrun contains the program to write the output file in a HDF5
> file. It reads the contents from a HDF5 file "mfdn.smwf001" and writes the
> dataset named "EigenVectors__" to a new HDF5 file "hdfopfile". The lengths
> written by various procs can stored in the dataset "LENGTHS". The IO happens as
> follows.
>
> 1. Get the dataset dims from the datasets named "EigenVectors__"

OK, you can do this with rank 0.  I'd suggest rank 0 broadcast the
result to everyone.  You could either use MPI_Bcast or MPI_Scatter.
Use MPI_Bcast if each process has to compute their 'offset' into the
dataset as well.  Use MPI_Scatter if all they need is a length.

> 2. Create the HDF5 o/p file accordingly.
> 3. Read the datasets from "mfdn.smwf001" and write it to "hdfopfile"

Once all processes have the lenghts and have computed their offsets,
then you can create hyperslabs to set up a collective read from
mfdn.smwf001 and a collective write to hdfopfile.

> can tell if I am making any mistakes that is leading to the slow IO. I can't
> send the larger runs as attachments since they are around 64 MB and 2 GB.

Maybe your slow performance is because you are you are using the
parallel HDF5 interface to carry out essentially serial I/O?   Do
follow up with your code.

==rob

--
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.