How much time should it take to iterate over groups and get data

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How much time should it take to iterate over groups and get data

asdaqsamir
We have measured the time it takes to collect all the groups in hdf5 file hierarchy. We did this loading time measurement on files located in the computer and same files located on the remote server. The time it takes to load all the group hierarchy is about 10 times more in the case of server than loading straight from the computer. This time measurement was only for reading group names from the file and not attributes or datasets. We also observed that it took almost 5 seconds to load a local file with 20,000 groups with about 0.25(ms)/group. Are there any ideas to improve these times for loading a group hierarchy?

We are considering all the group hierarchy and their attributes into a single large byte array stored in the dataset of a special group. This should speed up loading up all the hierarchy. Any opinions on this approach?
 
Thank you
Asdaq Samir
Electrical Engineer | Caterpillar Inc.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5