meaning of H5S_ALL_F for mem_space_id in H5Dwrite

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

meaning of H5S_ALL_F for mem_space_id in H5Dwrite

Frank Loeffler
Hi,

I want to select a hyperslab in a file, but use all of the memory space,
so my call to H5Dwrite looks like this (fortran):

call h5dwrite_f(dset_id, dtreal_id, real(mygrid(grid)%u(:,1,lm)), &
                array_dims, error, H5S_ALL_F,dspace_id )

Things to note here:
- I use H5S_ALL_F (otherwise the default) for the memspace selection
- I use dspace_id for the file space selection, which has been populated
  using an earlier call to h5sselect_hyperslab_f.

What I see is that, while the file space selection works fine (the
correct hyperslab is selected and written to, the offset of the
hyperslab is also applied to the memory space. I didn't expect this,
because I specifically pass H5S_ALL_F, which according to the
documentation:

  H5S_ALL in C (H5S_ALL_F in FORTRAN) is the default value and indicates
that the whole dataspace in memory is selected for the I/O operation.

After not finding an error in my code I dove into the hdf5 source and
found this in H5Dio.c:685

    /* Initialize dataspace information */
    if(!file_space)
        file_space = dataset->shared->space;
    if(!mem_space)
        mem_space = file_space;

Note the last line. While if file_space is 0 (H5S_ALL), the entire space
of the dataset is used, the same is not true for the memory space: if
you happen to set file_space (as I do), but not mem_space (as I do),
then mem_space is *not* set to the size of the memory dataspace, but to
the (passed) size of the file space. Which means in that case any
selection applied to the file dataspace is also applied to the memory
data space.

Am I correct that this is a bug in the HDF5 implementation, or is this
intended behavior? If it is intended, isn't this then a bug in the
documentation?

thanks, Frank


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: meaning of H5S_ALL_F for mem_space_id in H5Dwrite

Frank Loeffler
On Fri, Jun 09, 2017 at 03:36:29PM -0500, Frank Loeffler wrote:
...
>Which means in that case any selection applied to the file dataspace is
>also applied to the memory data space.
>
>Am I correct that this is a bug in the HDF5 implementation, or is this
>intended behavior? If it is intended, isn't this then a bug in the
>documentation?

Does anyone know or just want to comment?

Frank


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: meaning of H5S_ALL_F for mem_space_id in H5Dwrite

Elena Pourmal
This behavior is documented in H5Dwrite https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Write

(yep...confusing)

Elena
On Jun 14, 2017, at 10:33 AM, Frank Loeffler <[hidden email]> wrote:

On Fri, Jun 09, 2017 at 03:36:29PM -0500, Frank Loeffler wrote:
...
Which means in that case any selection applied to the file dataspace is also applied to the memory data space.

Am I correct that this is a bug in the HDF5 implementation, or is this intended behavior? If it is intended, isn't this then a bug in the documentation?

Does anyone know or just want to comment?

Frank

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: meaning of H5S_ALL_F for mem_space_id in H5Dwrite

Frank Loeffler
On Thu, Jun 15, 2017 at 09:21:40PM +0000, Elena Pourmal wrote:
>This behavior is documented in H5Dwrite https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Write
>
>(yep...confusing)

Hi Elena,

Thanks. I must have missed that. This is indeed confusing. I am
currently also looking at the tutorial at

  https://support.hdfgroup.org/HDF5/Tutor/rdwt.html

which states: "Note that H5S_ALL is passed in for the memory and file
dataspace parameters in the read and write calls. This indicates that
the entire dataspace of the dataset will be read or written to."

While this is technically correct (since in the examples both the memory
and the file dataspace are specified as H5S_ALL), this is indeed a
confusing statement. Maybe a note could be added that 'H5S_ALL' by
itself does not necessarily have that meaning, and the meaning of
specifying H5S_ALL for one dataspace indeed depends on the other
dataspace.

In general, this is quite a confusing choice of definition for H5S_ALL.
Is there another define that does what I actually thought H5S_ALL does:
simply selecting the entire available space, independent of whatever was
selected for the other data space? My best guess is: no, since I don't
see anything mentioned in H5Spublic.h. It would be a useful addition.

There is of course the simple workaround of specifying a space for both,
but requiring a separate data space for a simple array just because the
other space has an offset seems a bit awkward.

Frank


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: meaning of H5S_ALL_F for mem_space_id in H5Dwrite

Elena Pourmal
Hi Frank,

> On Jun 15, 2017, at 4:39 PM, Frank Loeffler <[hidden email]> wrote:
>
> On Thu, Jun 15, 2017 at 09:21:40PM +0000, Elena Pourmal wrote:
>> This behavior is documented in H5Dwrite https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Write
>>
>> (yep...confusing)
>
> Hi Elena,
>
> Thanks. I must have missed that.
It is hard to find. I had to think little-bit before I recalled where to find this info ;-)

> This is indeed confusing. I am currently also looking at the tutorial at
>
> https://support.hdfgroup.org/HDF5/Tutor/rdwt.html
>
> which states: "Note that H5S_ALL is passed in for the memory and file dataspace parameters in the read and write calls. This indicates that the entire dataspace of the dataset will be read or written to."
>
> While this is technically correct (since in the examples both the memory and the file dataspace are specified as H5S_ALL), this is indeed a confusing statement. Maybe a note could be added that 'H5S_ALL' by itself does not necessarily have that meaning, and the meaning of specifying H5S_ALL for one dataspace indeed depends on the other data space.
>
Thank you for suggestion! We definitely need to have a better explanation how selections and data spaces work. I entered JIRA report HDFFV-10229.

> In general, this is quite a confusing choice of definition for H5S_ALL. Is there another define that does what I actually thought H5S_ALL does:
> simply selecting the entire available space, independent of whatever was selected for the other data space? My best guess is: no, since I don't see anything mentioned in H5Spublic.h. It would be a useful addition.
Noted.
>
> There is of course the simple workaround of specifying a space for both, but requiring a separate data space for a simple array just because the other space has an offset seems a bit awkward.
Agree.

Thank you for reporting!

Elena
>
> Frank
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5