How much time should it take to iterate over groups and get data
We have measured the time it takes to collect
all the groups in hdf5 file hierarchy. We did this loading time measurement
on files located in the computer and same files located on the remote server.
The time it takes to load all the group hierarchy is about 10 times more
in the case of server than loading straight from the computer. This time
measurement was only for reading group names from the file and not attributes
or datasets. We also observed that it took almost 5 seconds to load a local
file with 20,000 groups with about 0.25(ms)/group. Are there any ideas
to improve these times for loading a group hierarchy?
We are considering all the group hierarchy
and their attributes into a single large byte array stored in the dataset
of a special group. This should speed up loading up all the hierarchy.
Any opinions on this approach?