HDF5 files on NFS?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

HDF5 files on NFS?

Rafal Lichwala
Hi All,

In case my previous question was lost between other issues I sent
before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is
not possible?
How to achieve this? Is there just some "switch" before compiling the
library source?

Regards,
Rafal


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Nelson, Jarom
I haven't had trouble using the serial HDF5 library on NFS. For parallel HDF5, I believe there are certain versions of NFS and specific configurations required, and from what I understand it will certainly not be fast. I think you really need a file system that is built for parallel I/O to use parallel HDF5 effectively.

I don't know the answer regarding the requirement of flock().

Jarom

-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of Rafal Lichwala
Sent: Friday, March 31, 2017 2:07 AM
To: [hidden email]
Subject: [Hdf-forum] HDF5 files on NFS?

Hi All,

In case my previous question was lost between other issues I sent before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is not possible?
How to achieve this? Is there just some "switch" before compiling the library source?

Regards,
Rafal


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Dave Allured - NOAA Affiliate
In reply to this post by Rafal Lichwala
Rafal,

There is a patch to disable flock for the previous release, hdf5 1.10.0-patch1.  This might be an interim solution for you.  Please see this recent message from HDF5 support:


I am not sure what kind of flock treatment is in the recently announced 1.10.1-pre1 pre-release.

--Dave


On Fri, Mar 31, 2017 at 3:06 AM, Rafal Lichwala <[hidden email]> wrote:
Hi All,

In case my previous question was lost between other issues I sent before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is not possible?
How to achieve this? Is there just some "switch" before compiling the library source?

Regards,
Rafal


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Sean Byland
In reply to this post by Rafal Lichwala

I'm not familiar with the ramifications, and doing so is kind of a hack, but it appears you can disable flock() by passing "ac_cv_func_flock=no" to "configure"  before compiling.


Sean


From: Hdf-forum <[hidden email]> on behalf of Rafal Lichwala <[hidden email]>
Sent: Friday, March 31, 2017 4:06:57 AM
To: [hidden email]
Subject: [Hdf-forum] HDF5 files on NFS?
 
Hi All,

In case my previous question was lost between other issues I sent
before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is
not possible?
How to achieve this? Is there just some "switch" before compiling the
library source?

Regards,
Rafal


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Rafal Lichwala
Thank you All for the hints!
I will try both: the patch and configure switch to disable flock().
Thanks!

Regards,
Rafal


W dniu 2017-03-31 o 22:48, Sean Byland pisze:

> I'm not familiar with the ramifications, and doing so is kind of a hack,
> but it appears you can disable flock() by passing "ac_cv_func_flock=no"
> to "configure"  before compiling.
>
>
> Sean
>
> ------------------------------------------------------------------------
> *From:* Hdf-forum <[hidden email]> on behalf of
> Rafal Lichwala <[hidden email]>
> *Sent:* Friday, March 31, 2017 4:06:57 AM
> *To:* [hidden email]
> *Subject:* [Hdf-forum] HDF5 files on NFS?
>
> Hi All,
>
> In case my previous question was lost between other issues I sent
> before, I'd like to ask again with hope someone knows the answer here:
>
> Is is possible to use HDF5 library on NFS filesystems where flock() is
> not possible?
> How to achieve this? Is there just some "switch" before compiling the
> library source?
>
> Regards,
> Rafal
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>

--

***
Rafał Lichwała
Poznańskie Centrum Superkomputerowo-Sieciowe
ul. Jana Pawła II nr 10
61-139 Poznań
e-mail: [hidden email]
***

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Dana Robinson
Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an environment variable that you can use to disable file locking (HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

On 4/2/17, 11:38 PM, "Hdf-forum on behalf of Rafal Lichwala" <[hidden email] on behalf of [hidden email]> wrote:

    Thank you All for the hints!
    I will try both: the patch and configure switch to disable flock().
    Thanks!
   
    Regards,
    Rafal
   
   
    W dniu 2017-03-31 o 22:48, Sean Byland pisze:
    > I'm not familiar with the ramifications, and doing so is kind of a hack,
    > but it appears you can disable flock() by passing "ac_cv_func_flock=no"
    > to "configure"  before compiling.
    >
    >
    > Sean
    >
    > ------------------------------------------------------------------------
    > *From:* Hdf-forum <[hidden email]> on behalf of
    > Rafal Lichwala <[hidden email]>
    > *Sent:* Friday, March 31, 2017 4:06:57 AM
    > *To:* [hidden email]
    > *Subject:* [Hdf-forum] HDF5 files on NFS?
    >
    > Hi All,
    >
    > In case my previous question was lost between other issues I sent
    > before, I'd like to ask again with hope someone knows the answer here:
    >
    > Is is possible to use HDF5 library on NFS filesystems where flock() is
    > not possible?
    > How to achieve this? Is there just some "switch" before compiling the
    > library source?
    >
    > Regards,
    > Rafal
    >
    >
    > _______________________________________________
    > Hdf-forum is for HDF software users discussion.
    > [hidden email]
    > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    > Twitter: https://twitter.com/hdf5
    >
    >
    > _______________________________________________
    > Hdf-forum is for HDF software users discussion.
    > [hidden email]
    > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    > Twitter: https://twitter.com/hdf5
    >
   
    --
   
    ***
    Rafał Lichwała
    Poznańskie Centrum Superkomputerowo-Sieciowe
    ul. Jana Pawła II nr 10
    61-139 Poznań
    e-mail: [hidden email]
    ***
   
    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    [hidden email]
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Dave Allured - NOAA Affiliate
In reply to this post by Rafal Lichwala
Dana,

Is HDF5_USE_FILE_LOCKING applied at configure time, build time, or run time?

--Dave


On Mon, Apr 3, 2017 at 4:27 PM, Dana Robinson <[hidden email]> wrote:
Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an environment variable that you can use to disable file locking (HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

<snip>

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Dave Allured - NOAA Affiliate
Sorry, I am confused about the meaning of "FALSE" in this context.  Is that actual setting the five-character string "FALSE", or the digit "0"?  Thanks for your help.

--Dave


On Mon, Apr 3, 2017 at 5:14 PM, Dave Allured - NOAA Affiliate <[hidden email]> wrote:
Dana,

Is HDF5_USE_FILE_LOCKING applied at configure time, build time, or run time?

--Dave


On Mon, Apr 3, 2017 at 4:27 PM, Dana Robinson <[hidden email]> wrote:
Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an environment variable that you can use to disable file locking (HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

<snip>


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 files on NFS?

Dana Robinson

Hi Dave,

 

The environment variable is checked at runtime when the file is opened.

 

We check for the five-character string "FALSE" (case-sensitive). No other false-y value will work.

 

Dana

 

 

 

From: Hdf-forum <[hidden email]> on behalf of Dave Allured - NOAA Affiliate <[hidden email]>
Reply-To: HDF List <[hidden email]>
Date: Monday, April 3, 2017 at 4:23 PM
To: HDF List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 files on NFS?

 

Sorry, I am confused about the meaning of "FALSE" in this context.  Is that actual setting the five-character string "FALSE", or the digit "0"?  Thanks for your help.

 

--Dave

 

 

On Mon, Apr 3, 2017 at 5:14 PM, Dave Allured - NOAA Affiliate <[hidden email]> wrote:

Dana,

 

Is HDF5_USE_FILE_LOCKING applied at configure time, build time, or run time?

 

--Dave

 

 

On Mon, Apr 3, 2017 at 4:27 PM, Dana Robinson <[hidden email]> wrote:

Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an environment variable that you can use to disable file locking (HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

 

<snip>

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...