H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

Deepak 8 Kumar
Hello!

I have a HDF5 based application to read the hdf5 file which has almost 3200 groups and 56000 attributes. The application is using the standard hdf5 api and it took almost 11 minutes to read only the groups and attributes. I used StopWatch and observed that H5Aopen_name() is taking almost 97 percent of the total time. I am using HDF 1.10.1 Windows10 x64.

My question is that is this the expected behavior here with H5Aopen_name()  or I am not  reading it properly?
What approach we should take in this kind of file with large number of attributes?
Any insight is greatly appreciated.

Thanks,
Deepak Kumar
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

Quincey Koziol-3
Hi Deepak,
Can you try using the “latest” version of the format (https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetLibverBounds ) and see if that helps?

Quincey

On Jul 13, 2017, at 11:14 AM, Deepak 8 Kumar <[hidden email]> wrote:

Hello!

I have a HDF5 based application to read the hdf5 file which has almost 3200 groups and 56000 attributes. The application is using the standard hdf5 api and it took almost 11 minutes to read only the groups and attributes. I used StopWatch and observed that H5Aopen_name() is taking almost 97 percent of the total time. I am using HDF 1.10.1 Windows10 x64.

My question is that is this the expected behavior here with H5Aopen_name()  or I am not  reading it properly?
What approach we should take in this kind of file with large number of attributes?
Any insight is greatly appreciated.

Thanks,
Deepak Kumar_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

Miller, Mark C.

Hi Quincey,

 

Quick question...I understand how libver bounds can improve a "data producer" in that it effects both memory and file data structures used.

 

But, how does it effect a "data consumer", especially reading an "old" file using "old structures"? Is there a conversion step during the read to new memory data structures?

 

Mark

 

 

"Hdf-forum on behalf of Quincey Koziol" wrote:

 

Hi Deepak,

Can you try using the “latest” version of the format (https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetLibverBounds ) and see if that helps?

 

Quincey

 

On Jul 13, 2017, at 11:14 AM, Deepak 8 Kumar <[hidden email]> wrote:

 

Hello!

I have a HDF5 based application to read the hdf5 file which has almost 3200 groups and 56000 attributes. The application is using the standard hdf5 api and it took almost 11 minutes to read only the groups and attributes. I used StopWatch and observed that H5Aopen_name() is taking almost 97 percent of the total time. I am using HDF 1.10.1 Windows10 x64.

My question is that is this the expected behavior here with H5Aopen_name()  or I am not  reading it properly?
What approach we should take in this kind of file with large number of attributes?
Any insight is greatly appreciated.

Thanks,
Deepak Kumar_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

Quincey Koziol-3
Hi Mark,

On Jul 13, 2017, at 1:33 PM, Miller, Mark C. <[hidden email]> wrote:

Hi Quincey,
 
Quick question...I understand how libver bounds can improve a "data producer" in that it effects both memory and file data structures used.
 
But, how does it effect a "data consumer", especially reading an "old" file using "old structures"? Is there a conversion step during the read to new memory data structures?

It’ll help data consumers that are reading from the new data structures.  As you imply, it won’t help readers of older files.  (Although running h5repack on those files with the “latest format” command line option would create a file with the new data structures)

Quincey

 
Mark
 
 
"Hdf-forum on behalf of Quincey Koziol" wrote:
 
Hi Deepak, 
Can you try using the “latest” version of the format (https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetLibverBounds ) and see if that helps?
 
Quincey
 
On Jul 13, 2017, at 11:14 AM, Deepak 8 Kumar <[hidden email]> wrote:
 
Hello! 

I have a HDF5 based application to read the hdf5 file which has almost 3200 groups and 56000 attributes. The application is using the standard hdf5 api and it took almost 11 minutes to read only the groups and attributes. I used StopWatch and observed that H5Aopen_name() is taking almost 97 percent of the total time. I am using HDF 1.10.1 Windows10 x64. 

My question is that is this the expected behavior here with H5Aopen_name()  or I am not  reading it properly? 
What approach we should take in this kind of file with large number of attributes? 
Any insight is greatly appreciated. 

Thanks, 
Deepak Kumar_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
 
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

Deepak 8 Kumar
Hello Everyone,

Thank you for all the help and suggestion. I actually figure out that I am not calling the H5Aclose(attribute_id) and H5Tclose(attribute_type_id) and is the primary reason for H5Aopen_name() taking that much time. Now the time taken to load that many groups and attributes is 1 seconds or sometimes even less than that. Also, I did checked with H5Aopen_idx() and I found that the time taken is almost same.

Thank you




From:        Quincey Koziol <[hidden email]>
To:        HDF Users Discussion List <[hidden email]>
Date:        07/13/2017 01:50 PM
Subject:        Re: [Hdf-forum] H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.
Sent by:        "Hdf-forum" <[hidden email]>




Hi Mark,

On Jul 13, 2017, at 1:33 PM, Miller, Mark C. <miller86@...> wrote:

Hi Quincey,
 
Quick question...I understand how libver bounds can improve a "data producer" in that it effects both memory and file data structures used.
 
But, how does it effect a "data consumer", especially reading an "old" file using "old structures"? Is there a conversion step during the read to new memory data structures?

It’ll help data consumers that are reading from the new data structures.  As you imply, it won’t help readers of older files.  (Although running h5repack on those files with the “latest format” command line option would create a file with the new data structures)

Quincey

 
Mark
 
 
"Hdf-forum on behalf of Quincey Koziol" wrote:
 
Hi Deepak,
Can you try using the “latest” version of the format (https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetLibverBounds ) and see if that helps?
 
Quincey
 
On Jul 13, 2017, at 11:14 AM, Deepak 8 Kumar <Kumar_Deepak_8@...> wrote:
 
Hello!

I have a HDF5 based application to read the hdf5 file which has almost 3200 groups and 56000 attributes. The application is using the standard hdf5 api and it took almost 11 minutes to read only the groups and attributes. I used StopWatch and observed that H5Aopen_name() is taking almost 97 percent of the total time. I am using HDF 1.10.1 Windows10 x64.


My question is that is this the expected behavior here with H5Aopen_name()  or I am not  reading it properly?
What approach we should take in this kind of file with large number of attributes?

Any insight is greatly appreciated.


Thanks,

Deepak Kumar
_______________________________________________
Hdf-forum is for HDF software users discussion.

[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:
https://twitter.com/hdf5
 
_______________________________________________
Hdf-forum is for HDF software users discussion.

[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:
https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.hdfgroup.org_mailman_listinfo_hdf-2Dforum-5Flists.hdfgroup.org&d=DQICAg&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=aVpsmDSm2bBgznM4DES61bNo7E_uMhhiIutsE14aYRg&m=Qm4TbicN-tFw0xk0PiiN4rg0OUZJpGERAVjI8eOqqRw&s=GazDncbchTfD5DrkEeZh7ZvgIK_XFWcv78tzTgsBemM&e=
Twitter:
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_hdf5&d=DQICAg&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=aVpsmDSm2bBgznM4DES61bNo7E_uMhhiIutsE14aYRg&m=Qm4TbicN-tFw0xk0PiiN4rg0OUZJpGERAVjI8eOqqRw&s=AoQIiXWq9uQ5Xi3CAkIEy6eSGL0S4AkQ39RrTbW3nLg&e=

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.

Deepak 8 Kumar
In reply to this post by Quincey Koziol-3
Hello Everyone,

Thank you for all the help and suggestion. I actually figure out that I am not calling the H5Aclose(attribute_id) and H5Tclose(attribute_type_id) and is the primary reason for H5Aopen_name() taking that much time. Now the time taken to load that many groups and attributes is 1 seconds or sometimes even less than that. Also, I did checked with H5Aopen_idx() and I found that the time taken is almost same.

Thank you



From:        Quincey Koziol <[hidden email]>
To:        HDF Users Discussion List <[hidden email]>
Date:        07/13/2017 01:50 PM
Subject:        Re: [Hdf-forum] H5Aopen_name() is taking almost 11 minutes to read 5600 attributes.
Sent by:        "Hdf-forum" <[hidden email]>




Hi Mark,

On Jul 13, 2017, at 1:33 PM, Miller, Mark C. <miller86@...> wrote:

Hi Quincey,
 
Quick question...I understand how libver bounds can improve a "data producer" in that it effects both memory and file data structures used.
 
But, how does it effect a "data consumer", especially reading an "old" file using "old structures"? Is there a conversion step during the read to new memory data structures?

It’ll help data consumers that are reading from the new data structures.  As you imply, it won’t help readers of older files.  (Although running h5repack on those files with the “latest format” command line option would create a file with the new data structures)

Quincey

 
Mark
 
 
"Hdf-forum on behalf of Quincey Koziol" wrote:
 
Hi Deepak,
Can you try using the “latest” version of the format (https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetLibverBounds ) and see if that helps?
 
Quincey
 
On Jul 13, 2017, at 11:14 AM, Deepak 8 Kumar <Kumar_Deepak_8@...> wrote:
 
Hello!

I have a HDF5 based application to read the hdf5 file which has almost 3200 groups and 56000 attributes. The application is using the standard hdf5 api and it took almost 11 minutes to read only the groups and attributes. I used StopWatch and observed that H5Aopen_name() is taking almost 97 percent of the total time. I am using HDF 1.10.1 Windows10 x64.


My question is that is this the expected behavior here with H5Aopen_name()  or I am not  reading it properly?
What approach we should take in this kind of file with large number of attributes?

Any insight is greatly appreciated.


Thanks,

Deepak Kumar
_______________________________________________
Hdf-forum is for HDF software users discussion.

[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:
https://twitter.com/hdf5
 
_______________________________________________
Hdf-forum is for HDF software users discussion.

[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:
https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.hdfgroup.org_mailman_listinfo_hdf-2Dforum-5Flists.hdfgroup.org&d=DQICAg&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=aVpsmDSm2bBgznM4DES61bNo7E_uMhhiIutsE14aYRg&m=Qm4TbicN-tFw0xk0PiiN4rg0OUZJpGERAVjI8eOqqRw&s=GazDncbchTfD5DrkEeZh7ZvgIK_XFWcv78tzTgsBemM&e=
Twitter:
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_hdf5&d=DQICAg&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=aVpsmDSm2bBgznM4DES61bNo7E_uMhhiIutsE14aYRg&m=Qm4TbicN-tFw0xk0PiiN4rg0OUZJpGERAVjI8eOqqRw&s=AoQIiXWq9uQ5Xi3CAkIEy6eSGL0S4AkQ39RrTbW3nLg&e=

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5