Quantcast

Turn off file locking

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Turn off file locking

Krueger, Brendan K
Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to write and read data files.  My current computing resources aren't sufficient for much longer, so I got an account on a new computing resource.  Unfortunately, the administrators on this new system have disabled file locking to improve performance of their Lustre file system.  HDF5 demands file locks, which I don't need, and I can't find a way to turn that off.  Is there a way to tell HDF5 to not request a file lock, so that I can run on a system that doesn't allow file locking?

Thank you,
Brendan

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Turn off file locking

Roger Leigh
On 28/06/2016 17:09, Krueger, Brendan K wrote:

> Hello,
>
> I've written a code that uses HDF5 (serial mode, C++ if that matters) to
> write and read data files.  My current computing resources aren't
> sufficient for much longer, so I got an account on a new computing
> resource.  Unfortunately, the administrators on this new system have
> disabled file locking to improve performance of their Lustre file
> system.  HDF5 demands file locks, which I don't need, and I can't find a
> way to turn that off.  Is there a way to tell HDF5 to not request a file
> lock, so that I can run on a system that doesn't allow file locking?

If you're on Linux, you could look at the "eatmydata" command.  This
LD_PRELOADs a small library which overrides the glibc locking calls and
simply stubs them out.  Depending upon how the locking is implemented by
HDF5, this may or may not be a possible solution, but thought it might
be something you might find worth investigating.


Regards,
Roger

--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: (01382) 386364

The University of Dundee is a registered Scottish Charity, No: SC015096

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Turn off file locking

Dana Robinson
In reply to this post by Krueger, Brendan K
Hi Brendan,

There is a source patch for HDF5 1.10 that disables file locking located here:

https://www.hdfgroup.org/HDF5/release/obtainsrc5110.html#patch

Does that solve your problem?

Dana Robinson
Software Engineer
The HDF Group

From: Hdf-forum <[hidden email]> on behalf of "Krueger, Brendan K" <[hidden email]>
Reply-To: HDF List <[hidden email]>
Date: Tuesday, June 28, 2016 at 12:09
To: HDF List <[hidden email]>
Subject: [Hdf-forum] Turn off file locking

Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to write and read data files.  My current computing resources aren't sufficient for much longer, so I got an account on a new computing resource.  Unfortunately, the administrators on this new system have disabled file locking to improve performance of their Lustre file system.  HDF5 demands file locks, which I don't need, and I can't find a way to turn that off.  Is there a way to tell HDF5 to not request a file lock, so that I can run on a system that doesn't allow file locking?

Thank you,
Brendan

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Turn off file locking

Krueger, Brendan K
Hi Dana,

I installed HDF5 using the patch that you pointed me to and ran a test case.  It looks like everything works.  Thank you very much!

Brendan

---------------------------------
Brendan K. Krueger
Scientist
XCP-2: Eulerian Codes
Los Alamos National Laboratory
www.linkedin.com/in/bkkrueger
(505) 667-0243

From: Hdf-forum <[hidden email]> on behalf of Dana Robinson <[hidden email]>
Reply-To: HDF Users Discussion List <[hidden email]>
Date: Tuesday, June 28, 2016 at 11:45 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] Turn off file locking

Hi Brendan,

There is a source patch for HDF5 1.10 that disables file locking located here:


Does that solve your problem?

Dana Robinson
Software Engineer
The HDF Group

From: Hdf-forum <[hidden email]> on behalf of "Krueger, Brendan K" <[hidden email]>
Reply-To: HDF List <[hidden email]>
Date: Tuesday, June 28, 2016 at 12:09
To: HDF List <[hidden email]>
Subject: [Hdf-forum] Turn off file locking

Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to write and read data files.  My current computing resources aren't sufficient for much longer, so I got an account on a new computing resource.  Unfortunately, the administrators on this new system have disabled file locking to improve performance of their Lustre file system.  HDF5 demands file locks, which I don't need, and I can't find a way to turn that off.  Is there a way to tell HDF5 to not request a file lock, so that I can run on a system that doesn't allow file locking?

Thank you,
Brendan

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...