Quantcast

Infer Datatype

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Infer Datatype

Matt Wood
Hi,

I am storing 1000's of datasets of different types. In particular two of my types are LONG(any long) and DATETIME(long value representing #nano seconds since UNIX EPOCH). While I experiment LONG uses the H5T_STD_I64BE datatype and currently DATETIME uses H5T_STD_I64LE so I can use the endian flag to distinguish between the two.

Now when this data is read by various systems (in particular Java) and I am looking for an intuitive and efficient way to distinguish between these two datatypes. My current options are:

  1. Make DATETIME signed and change EPOCH to 1900 or something.
  2. Maked DATATIME little endian, as above.
  3. Use a committed datatype.

Have I missed something and is there an alternative way to achieve this?

Thanks,

Matt






_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Reading VLEN string attribute values.

Matt Wood
Hi,

I am using HDF.PInvoke 1.10.0.4 but I am struggling to read the string value of an attribute when it is variable length.

Ultimately I believe I need to call:

GCHandle hnd = GCHandle.Alloc(data, GCHandleType.Pinned);
int err = H5A.read(aid, typeId, hnd.AddrOfPinnedObject());
hnd.Free();


Where data is a suitable object.

I am confident that the attribute id (aid) is correct because it works for all the other attributes types.

I am not so confident on the typeId, I have tried:

   H5T.get_type(aid);
* H5T.get_native_type(tid);
   H5T.create(H5T.class_t.VLEN, H5T.VARIABLE);
   H5T.create(H5T.class_t.STRING, H5T.VARIABLE);


Likewise for the data and I have tried:

*  data = new byte[256];
    data = new IntPtr[1]; => data[0] = GCHandle.Alloc(byte[256], GCHandleType.Pinned).AddrOfPinnedObject()


The only time some non-zero values were returned was case (*) but then I couldn't relate the values to the expected text.

Thanks,

Matt

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reading VLEN string attribute values.

Gerd Heber

What’s the storage layout of the attribute?

(i.e., is it H5S_SIMPLE or H5S_SCALAR?)

 

G.

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Matt Wood
Sent: Thursday, March 30, 2017 9:42 AM
To: [hidden email]
Subject: [Hdf-forum] Reading VLEN string attribute values.

 

Hi,

I am using HDF.PInvoke 1.10.0.4 but I am struggling to read the string value of an attribute when it is variable length.

Ultimately I believe I need to call:

GCHandle hnd = GCHandle.Alloc(data, GCHandleType.Pinned);
int err = H5A.read(aid, typeId, hnd.AddrOfPinnedObject());
hnd.Free();


Where data is a suitable object.

I am confident that the attribute id (aid) is correct because it works for all the other attributes types.

I am not so confident on the typeId, I have tried:

   H5T.get_type(aid);
* H5T.get_native_type(tid);
   H5T.create(H5T.class_t.VLEN, H5T.VARIABLE);
   H5T.create(H5T.class_t.STRING, H5T.VARIABLE);


Likewise for the data and I have tried:

*  data = new byte[256];
    data = new IntPtr[1]; => data[0] = GCHandle.Alloc(byte[256], GCHandleType.Pinned).AddrOfPinnedObject()


The only time some non-zero values were returned was case (*) but then I couldn't relate the values to the expected text.

Thanks,

Matt


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reading VLEN string attribute values.

Vincent

Try something like this, if you want a single string (not tested):

 

int byteLength = Marshal.SizeOf(Of IntPtr);

IntPtr intPtr = Marshal.AllocHGlobal(byteLength);

 

H5A.read(attributeId, attributeTypeId, intPtr);

 

string result = Marshal.PtrToStringAnsi(intPtr);

 

If you want a more general way to read multiple strings at once, the code must be modified slightly.

 

Best regards,

Vincent

 

Von: Hdf-forum [mailto:[hidden email]] Im Auftrag von Gerd Heber
Gesendet: Donnerstag, 30. März 2017 16:58
An: HDF Users Discussion List <[hidden email]>
Betreff: Re: [Hdf-forum] Reading VLEN string attribute values.

 

What’s the storage layout of the attribute?

(i.e., is it H5S_SIMPLE or H5S_SCALAR?)

 

G.

 

From: Hdf-forum [[hidden email]] On Behalf Of Matt Wood
Sent: Thursday, March 30, 2017 9:42 AM
To: [hidden email]
Subject: [Hdf-forum] Reading VLEN string attribute values.

 

Hi,

I am using HDF.PInvoke 1.10.0.4 but I am struggling to read the string value of an attribute when it is variable length.

Ultimately I believe I need to call:

GCHandle hnd = GCHandle.Alloc(data, GCHandleType.Pinned);
int err = H5A.read(aid, typeId, hnd.AddrOfPinnedObject());
hnd.Free();


Where data is a suitable object.

I am confident that the attribute id (aid) is correct because it works for all the other attributes types.

I am not so confident on the typeId, I have tried:

   H5T.get_type(aid);
* H5T.get_native_type(tid);
   H5T.create(H5T.class_t.VLEN, H5T.VARIABLE);
   H5T.create(H5T.class_t.STRING, H5T.VARIABLE);


Likewise for the data and I have tried:

*  data = new byte[256];
    data = new IntPtr[1]; => data[0] = GCHandle.Alloc(byte[256], GCHandleType.Pinned).AddrOfPinnedObject()


The only time some non-zero values were returned was case (*) but then I couldn't relate the values to the expected text.

Thanks,

Matt


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reading VLEN string attribute values.

Matt Wood
In reply to this post by Matt Wood
The output from h5dump is:

ATTRIBUTE "ORIGIN" {
    DATATYPE  H5T_STRING {
       STRSIZE H5T_VARIABLE;
       STRPAD H5T_STR_NULLTERM;
       CSET H5T_CSET_ASCII;
       CTYPE H5T_C_S1;
    }
    DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
    DATA {
    (0): "BASIC"
    }
}

And I tried:

int byteLength = Marshal.SizeOf(typeof(IntPtr));
IntPtr intPtr = Marshal.AllocHGlobal(byteLength);
H5A.read(aid, typeId, intPtr);
string result = Marshal.PtrToStringAnsi(intPtr);

And result contains "\u0001\u0002\b\n\n".

Matt


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reading VLEN string attribute values.

Gerd Heber
The dataspace is simple, i.e., it's an array which in this case
happens to have only one element. (This is NOT the same as a scalar attribute!)
Have a look at H5DreadTest2 in

https://github.com/HDFGroup/HDF.PInvoke/blob/master/UnitTests/H5DTest/H5Dread.cs

        [TestMethod]
        public void H5DreadTest2()
        {
            hid_t mem_type = H5T.create(H5T.class_t.STRING, H5T.VARIABLE);
            Assert.IsTrue(H5T.set_cset(mem_type, H5T.cset_t.UTF8) >= 0);
            Assert.IsTrue(H5T.set_strpad(mem_type, H5T.str_t.NULLTERM) >= 0);

            hid_t fspace = H5D.get_space(m_v0_utf8_dset);
            Assert.IsTrue(fspace >= 0);

            hssize_t count = H5S.get_simple_extent_npoints(fspace);
            Assert.IsTrue(count > 0);
            Assert.IsTrue(H5S.close(fspace) >= 0);

            IntPtr[] rdata = new IntPtr[count];
            GCHandle hnd = GCHandle.Alloc(rdata, GCHandleType.Pinned);
            Assert.IsTrue(H5D.read(m_v0_utf8_dset, mem_type, H5S.ALL, H5S.ALL,
                H5P.DEFAULT, hnd.AddrOfPinnedObject()) >= 0);
           
            for (int i = 0; i < rdata.Length; ++i)
            {
                int len = 0;
                while (Marshal.ReadByte(rdata[i], len) != 0) { ++len; }
                byte[] buffer = new byte[len];
                Marshal.Copy(rdata[i], buffer, 0, buffer.Length);
                string s = Encoding.UTF8.GetString(buffer);

                Assert.IsTrue(s == ((string)m_utf8strings[i]));

                Assert.IsTrue(H5.free_memory(rdata[i]) >= 0);
            }

            ...

            hnd.Free();
        }
    }


(The example is for datasets, but applies the same way for attributes.)

Your snippet is almost correct (i.e., allocate an IntPtr array of size 1).
The part that's wrong is that data needs no initialization or byte array allocation.
Reading a simple dataset of variable-length strings returns a pointer array.
(You must free the strings later or leak memory!)

G.



-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of Matt Wood
Sent: Thursday, March 30, 2017 10:24 AM
To: [hidden email]
Subject: Re: [Hdf-forum] Reading VLEN string attribute values.

The output from h5dump is:

ATTRIBUTE "ORIGIN" {
    DATATYPE  H5T_STRING {
       STRSIZE H5T_VARIABLE;
       STRPAD H5T_STR_NULLTERM;
       CSET H5T_CSET_ASCII;
       CTYPE H5T_C_S1;
    }
    DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
    DATA {
    (0): "BASIC"
    }
}

And I tried:

int byteLength = Marshal.SizeOf(typeof(IntPtr)); IntPtr intPtr = Marshal.AllocHGlobal(byteLength); H5A.read(aid, typeId, intPtr); string result = Marshal.PtrToStringAnsi(intPtr);

And result contains "\u0001\u0002\b\n\n".

Matt


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reading VLEN string attribute values.

Matt Wood
In reply to this post by Matt Wood

Many thanks, that worked a treat. I have attached my working method below.

Matt

/// <summary>
/// Helper method that extracts a string array value from a variable length attribute.
/// See https://github.com/HDFGroup/HDF.PInvoke/blob/master/UnitTests/H5DTest/H5Dread.cs
/// </summary>
/// <param name="aid">The id of the attribute.</param>
/// <param name="tid">The type id of the attribute.</param>
/// <returns>An array of strings.</returns>
public string[] GetStrings(long aid, long tid)
{
     long space = H5A.get_space(aid);
     // TODO close space
     long count = H5S.get_simple_extent_npoints(space);
     // Create pointer [] to receive the addresses.
     IntPtr[] rdata = new IntPtr[count];
     // Create array for result.
     string[] result = new string[count];
     GCHandle hnd = GCHandle.Alloc(rdata, GCHandleType.Pinned);
     // Call the function with address of pointer [].
     H5A.read(aid, tid, hnd.AddrOfPinnedObject());
     hnd.Free();
     // For each pointer extract string.
     for (int ii = 0; ii < rdata.Length; ++ii)
     {
    
     int len = 0;
    
     // Find the end of the string (\0)
     
    while (Marshal.ReadByte(rdata[ii], len) != 0) { ++len; }
          byte[] buffer = new byte[len];
          // Copy to buffer
          Marshal.Copy(rdata[ii], buffer, 0, buffer.Length);
          // Encode the string.
          result[ii] = Encoding.UTF8.GetString(buffer);
     }
     return result;
}


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reading VLEN string attribute values.

Gerd Heber

Glad to hear that it works.

 

Don’t forget to free the buffer(s) that the HDF5 library

allocated for those strings or you’ll have created a memory leak:

 

Assert.IsTrue(H5.free_memory(rdata[i]) >= 0);

 

G.

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of Matt Wood
Sent: Thursday, March 30, 2017 11:23 AM
To: [hidden email]
Subject: Re: [Hdf-forum] Reading VLEN string attribute values.

 

Many thanks, that worked a treat. I have attached my working method below.

Matt

/// <summary>
/// Helper method that extracts a string array value from a variable length attribute.
/// See https://github.com/HDFGroup/HDF.PInvoke/blob/master/UnitTests/H5DTest/H5Dread.cs
/// </summary>
/// <param name="aid">The id of the attribute.</param>
/// <param name="tid">The type id of the attribute.</param>
/// <returns>An array of strings.</returns>
public string[] GetStrings(long aid, long tid)
{
     long space = H5A.get_space(aid);
     // TODO close space
     long count = H5S.get_simple_extent_npoints(space);
     // Create pointer [] to receive the addresses.
     IntPtr[] rdata = new IntPtr[count];
     // Create array for result.
     string[] result = new string[count];
     GCHandle hnd = GCHandle.Alloc(rdata, GCHandleType.Pinned);
     // Call the function with address of pointer [].
     H5A.read(aid, tid, hnd.AddrOfPinnedObject());
     hnd.Free();
     // For each pointer extract string.
     for (int ii = 0; ii < rdata.Length; ++ii)
     {
          int len = 0;
          // Find the end of the string (\0)
          while (Marshal.ReadByte(rdata[ii], len) != 0) { ++len; }
          byte[] buffer = new byte[len];
          // Copy to buffer
          Marshal.Copy(rdata[ii], buffer, 0, buffer.Length);
          // Encode the string.
          result[ii] = Encoding.UTF8.GetString(buffer);
     }
     return result;
}


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...