Quantcast

HDF5 -- multithreading support or not?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

HDF5 -- multithreading support or not?

Isaac Gerg
Hi Folks,

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Gerd Heber

Did you build the HDF5 library with thread-safety enabled?

 

https://www.hdfgroup.org/hdf5-quest.html#tsafe

 

Yes, the documentation is inconsistent. L

 

G.

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 5:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Isaac Gerg
Did you build the HDF5 library with thread-safety enabled?

Yes I did.

On Mon, Aug 29, 2016 at 9:59 AM, Gerd Heber <[hidden email]> wrote:

Did you build the HDF5 library with thread-safety enabled?

 

https://www.hdfgroup.org/hdf5-quest.html#tsafe

 

Yes, the documentation is inconsistent. L

 

G.

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 5:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Gerd Heber

Can you send us an example that fails (occasionally)? G.

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 9:05 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

 

Did you build the HDF5 library with thread-safety enabled?

 

Yes I did.

 

On Mon, Aug 29, 2016 at 9:59 AM, Gerd Heber <[hidden email]> wrote:

Did you build the HDF5 library with thread-safety enabled?

 

https://www.hdfgroup.org/hdf5-quest.html#tsafe

 

Yes, the documentation is inconsistent. L

 

G.

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 5:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Dana Robinson
In reply to this post by Isaac Gerg

Hi Isaac,

 

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :)

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

 

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

 

Dana Robinson

Software Engineer

The HDF Group

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Isaac Gerg
Hi Dana,

I am not sure I understand your question.  In each thread, I open an hdf file, read from it, and then call H5close().  

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

I did not know about this.

In reading the documentation...

> "This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

Thank you,
Isaac

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :)

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

 

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

 

Dana Robinson

Software Engineer

The HDF Group

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Dana Robinson

Hi Isaac,

 

It sounds like you want to call H5Fclose, not H5close. Try just closing the file and not calling H5close at all. Does that work?

 

Dana

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:25 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Dana,

 

I am not sure I understand your question.  In each thread, I open an hdf file, read from it, and then call H5close().  

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

I did not know about this.

 

In reading the documentation...

 

> "This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

 

Thank you,

Isaac

 

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :)

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

 

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

 

Dana Robinson

Software Engineer

The HDF Group

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Isaac Gerg
Thanks Dana, I see what you mean and where I misread.

On Mon, Aug 29, 2016 at 11:29 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

It sounds like you want to call H5Fclose, not H5close. Try just closing the file and not calling H5close at all. Does that work?

 

Dana

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:25 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Dana,

 

I am not sure I understand your question.  In each thread, I open an hdf file, read from it, and then call H5close().  

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

I did not know about this.

 

In reading the documentation...

 

> "This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

 

Thank you,

Isaac

 

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :)

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

 

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

 

Dana Robinson

Software Engineer

The HDF Group

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Isaac Gerg
In reply to this post by Dana Robinson
Interesting... I am using H5Fclose().  

If 2 different threads call this function for the same file (but different file handles), does HDF handle this?

On Mon, Aug 29, 2016 at 11:29 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

It sounds like you want to call H5Fclose, not H5close. Try just closing the file and not calling H5close at all. Does that work?

 

Dana

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:25 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Dana,

 

I am not sure I understand your question.  In each thread, I open an hdf file, read from it, and then call H5close().  

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

I did not know about this.

 

In reading the documentation...

 

> "This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

 

Thank you,

Isaac

 

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :)

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

 

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

 

Dana Robinson

Software Engineer

The HDF Group

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HDF5 -- multithreading support or not?

Dana Robinson

Hi Isaac,

 

Yes. There can be multiple file IDs for the same physical file. Closing one of them will not affect the other.

 

Dana

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:57 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

 

Interesting... I am using H5Fclose().  

 

If 2 different threads call this function for the same file (but different file handles), does HDF handle this?

 

On Mon, Aug 29, 2016 at 11:29 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

It sounds like you want to call H5Fclose, not H5close. Try just closing the file and not calling H5close at all. Does that work?

 

Dana

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:25 AM
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Dana,

 

I am not sure I understand your question.  In each thread, I open an hdf file, read from it, and then call H5close().  

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

I did not know about this.

 

In reading the documentation...

 

> "This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

 

Thank you,

Isaac

 

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <[hidden email]> wrote:

Hi Isaac,

 

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :)

 

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

 

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

 

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

 

Dana Robinson

Software Engineer

The HDF Group

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: [hidden email]
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

 

Hi Folks,

 

I am a need for a windows application to have two threads read the same hdf5 file as read-only.  I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

 

I have built a test in C++, using the hdf5 C API, to determine if the read is possible.  The test occasional segfaults on H5Close().  I am trying to determine if the segfault is due to HDF5 or my own error.

 

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

 

Thanks in advance,

Isaac


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...