[hdf-forum] performance on lustre filesystems?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] performance on lustre filesystems?

Steve Heistand
Hi folks,
has anyone gotten good performance with parallel HDF operations
on a lustre filesystem? When our client count grows large we are seeing
really really bad IO rates and sometimes even failures when doing
parallel IO operations.
Thought I would ask the group to see if anyone is seeing good
lustre performance.

thanks.

s

--
************************************************************************
 Steve Heistand                           NASA Ames Research Center
 SciCon Group                             Mail Stop 258-6
 steve.heistand at nasa.gov  (650) 604-4369  Moffett Field, CA 94035-1000
************************************************************************
 "Any opinions expressed are those of our alien overloads, not my own."

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: OpenPGP digital signature
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20081125/41b8b594/attachment.bin>

Reply | Threaded
Open this post in threaded view
|

[hdf-forum] performance on lustre filesystems?

Brock Palen
Eh not so much here ether.

Looking at some data from sun a while back, it really depends on your  
MPI-IO layer. For example sun was working on a ROMIO adio driver for  
lustre that is only in the absolute newest versions.  Previous that  
ROMIO would use the UFS adio driver and performance was poor.

This data was for flash IO in parallel to hdf5.
Talk to your MPI vendor about what integration their MPI-IO system  
has with lustre.

I personally have been disappointed for how slow it has been with how  
quickly lustre has be installed on so many systems. I am glad to see  
the work finally done,

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985



On Nov 25, 2008, at 10:22 AM, Steve Heistand wrote:

> Hi folks,
> has anyone gotten good performance with parallel HDF operations
> on a lustre filesystem? When our client count grows large we are  
> seeing
> really really bad IO rates and sometimes even failures when doing
> parallel IO operations.
> Thought I would ask the group to see if anyone is seeing good
> lustre performance.
>
> thanks.
>
> s
>
> --
> **********************************************************************
> **
>  Steve Heistand                           NASA Ames Research Center
>  SciCon Group                             Mail Stop 258-6
>  steve.heistand at nasa.gov  (650) 604-4369  Moffett Field, CA 94035-1000
> **********************************************************************
> **
>  "Any opinions expressed are those of our alien overloads, not my  
> own."
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] performance on lustre filesystems?

Quincey Koziol
Hi all,
        Hopefully, we (that is, The HDF Group) will be getting some funding  
to address this issue, probably in the first half of '09.

                Quincey Koziol

On Nov 25, 2008, at 10:34 AM, Brock Palen wrote:

> Eh not so much here ether.
>
> Looking at some data from sun a while back, it really depends on  
> your MPI-IO layer. For example sun was working on a ROMIO adio  
> driver for lustre that is only in the absolute newest versions.  
> Previous that ROMIO would use the UFS adio driver and performance  
> was poor.
>
> This data was for flash IO in parallel to hdf5.
> Talk to your MPI vendor about what integration their MPI-IO system  
> has with lustre.
>
> I personally have been disappointed for how slow it has been with  
> how quickly lustre has be installed on so many systems. I am glad to  
> see the work finally done,
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
> On Nov 25, 2008, at 10:22 AM, Steve Heistand wrote:
>
>> Hi folks,
>> has anyone gotten good performance with parallel HDF operations
>> on a lustre filesystem? When our client count grows large we are  
>> seeing
>> really really bad IO rates and sometimes even failures when doing
>> parallel IO operations.
>> Thought I would ask the group to see if anyone is seeing good
>> lustre performance.
>>
>> thanks.
>>
>> s
>>
>> --
>> ************************************************************************
>> Steve Heistand                           NASA Ames Research Center
>> SciCon Group                             Mail Stop 258-6
>> steve.heistand at nasa.gov  (650) 604-4369  Moffett Field, CA 94035-1000
>> ************************************************************************
>> "Any opinions expressed are those of our alien overloads, not my  
>> own."
>>
>
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] performance on lustre filesystems?

Elena Pourmal
All,

It will be very helpful  if  this community can share  with us (The  
HDF Group) "simple" HDF5 C and/or Fortran examples that represent I/O  
patterns of applications with poor performance.

Thanks in advance,

Elena


On Nov 25, 2008, at 12:42 PM, Quincey Koziol wrote:

> Hi all,
> Hopefully, we (that is, The HDF Group) will be getting some funding  
> to address this issue, probably in the first half of '09.
>
> Quincey Koziol
>
> On Nov 25, 2008, at 10:34 AM, Brock Palen wrote:
>
>> Eh not so much here ether.
>>
>> Looking at some data from sun a while back, it really depends on  
>> your MPI-IO layer. For example sun was working on a ROMIO adio  
>> driver for lustre that is only in the absolute newest versions.  
>> Previous that ROMIO would use the UFS adio driver and performance  
>> was poor.
>>
>> This data was for flash IO in parallel to hdf5.
>> Talk to your MPI vendor about what integration their MPI-IO system  
>> has with lustre.
>>
>> I personally have been disappointed for how slow it has been with  
>> how quickly lustre has be installed on so many systems. I am glad  
>> to see the work finally done,
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>> On Nov 25, 2008, at 10:22 AM, Steve Heistand wrote:
>>
>>> Hi folks,
>>> has anyone gotten good performance with parallel HDF operations
>>> on a lustre filesystem? When our client count grows large we are  
>>> seeing
>>> really really bad IO rates and sometimes even failures when doing
>>> parallel IO operations.
>>> Thought I would ask the group to see if anyone is seeing good
>>> lustre performance.
>>>
>>> thanks.
>>>
>>> s
>>>
>>> --
>>> ************************************************************************
>>> Steve Heistand                           NASA Ames Research Center
>>> SciCon Group                             Mail Stop 258-6
>>> steve.heistand at nasa.gov  (650) 604-4369  Moffett Field, CA  
>>> 94035-1000
>>> ************************************************************************
>>> "Any opinions expressed are those of our alien overloads, not my  
>>> own."
>>>
>>
>>
>> ----------------------------------------------------------------------
>> This mailing list is for HDF software users discussion.
>> To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org
>> .
>> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>>
>>
>
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] performance on lustre filesystems?

Ruth Aydt
Administrator
In reply to this post by Steve Heistand
Steve - I forwarded your query on to one of our lustre-based users who  
has seen performance issues but who is not on the forum list.  He  
replied:

 > We haven't seen the kinds of failure that Steve is seeing, but you  
might recommend that he try the MPI-POSIX VFD and make sure the stripe  
count is an even multiple of the number of processors.  As for the  
reduced performance at high concurrency, that looks like something bad  
is happening with the lock manager.

As Quincey noted, we are hoping to get some funding next year to work  
on improving HDF5 performance on lustre.  We believe it will be  
possible to customize/tune HDF5's interaction with the MPI and lustre  
layers to "serve" data in the manner needed for lustre to perform  
better.  Further changes, such as two-phase or server-directed  I/O  
have also been discussed as ways to improve performance.

The actual work undertaken will depend on the priorities of the client  
and the funding they have available.

Because the anticipated funding is limited, we are also trying to  
identify other users whose work is hampered by HDF5 performance on  
lustre.   If there is sufficient need, we may be able to make a strong  
case to other sources for additional support of this effort.

I encourage those of you who are using HDF5 on lustre to email me  
directly. I'll send you a questionnaire that will help us prioritize  
efforts and possibly secure additional funding.    << I haven't yet  
written the questions so it may be a couple of days before you hear  
back!  >>

  -Ruth

------------------------------------------------------------
Ruth Aydt
Director of Sponsored Projects and Business Development
The HDF Group
aydt at hdfgroup.org      (217)265-7837
------------------------------------------------------------


On Nov 25, 2008, at 9:22 AM, Steve Heistand wrote:

> Hi folks,
> has anyone gotten good performance with parallel HDF operations
> on a lustre filesystem? When our client count grows large we are  
> seeing
> really really bad IO rates and sometimes even failures when doing
> parallel IO operations.
> Thought I would ask the group to see if anyone is seeing good
> lustre performance.
>
> thanks.
>
> s
>
> --
> ************************************************************************
> Steve Heistand                           NASA Ames Research Center
> SciCon Group                             Mail Stop 258-6
> steve.heistand at nasa.gov  (650) 604-4369  Moffett Field, CA 94035-1000
> ************************************************************************
> "Any opinions expressed are those of our alien overloads, not my own."
>






----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.