User-defined filter parameter parsing broken in h5repack?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

User-defined filter parameter parsing broken in h5repack?

Elvis Stansvik
Consider the attached noop_plugin.c. It contains a dynamically loaded filter plugin, with filter number 42, which does nothing except print out the received `cd_values` array in the filter function.

Build the test plugin (on *NIX) with `make`.

Then test the following:

estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1 input.hdf5 output.hdf5
1
estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2 input.hdf5 output.hdf5
2 2
estan@newton:~/noop_plugin$

Notice how in the second invokation (with UD=42,2,1,2), the cd_values contains { 2, 2 }, despite { 1, 2 } being passed on the command line.

Has anyone seen this strangeness before, or can reproduce?

Thanks in advance,
Elvis Stansvik

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

noop_plugin.tar.gz (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Elvis Stansvik
2016-08-22 10:16 GMT+02:00 Elvis Stansvik <[hidden email]>:
Consider the attached noop_plugin.c. It contains a dynamically loaded filter plugin, with filter number 42, which does nothing except print out the received `cd_values` array in the filter function.

Build the test plugin (on *NIX) with `make`.

Then test the following:

estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1 input.hdf5 output.hdf5
1
estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2 input.hdf5 output.hdf5
2 2
estan@newton:~/noop_plugin$

Notice how in the second invokation (with UD=42,2,1,2), the cd_values contains { 2, 2 }, despite { 1, 2 } being passed on the command line.

I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.

Elvis
 

Has anyone seen this strangeness before, or can reproduce?

Thanks in advance,
Elvis Stansvik


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Andrea Bedini
I can reproduce on OS X, same version of HDF5. Compiling the library with --enable-debug and getting the trace (HDF5_DEBUG=“trace”) shows the following call:

H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0, cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;

So I bet it's a bug in h5repack.

Andrea


> On 22 Aug 2016, at 4:18 PM, Elvis Stansvik <[hidden email]> wrote:
>
> 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <[hidden email]>:
> Consider the attached noop_plugin.c. It contains a dynamically loaded filter plugin, with filter number 42, which does nothing except print out the received `cd_values` array in the filter function.
>
> Build the test plugin (on *NIX) with `make`.
>
> Then test the following:
>
> estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1 input.hdf5 output.hdf5
> 1
> estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2 input.hdf5 output.hdf5
> 2 2
> estan@newton:~/noop_plugin$
>
> Notice how in the second invokation (with UD=42,2,1,2), the cd_values contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
>
> I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
>
> Elvis
>  
>
> Has anyone seen this strangeness before, or can reproduce?
>
> Thanks in advance,
> Elvis Stansvik
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

--
Andrea Bedini
@andreabedini, http://www.andreabedini.com

See the impact of my research at https://impactstory.org/AndreaBedini
use https://keybase.io/andreabedini to send me encrypted messages
Key fingerprint = 17D5 FB49 FA18 A068 CF53  C5C2 9503 64C1 B2D5 9591




_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Elvis Stansvik
2016-08-22 14:21 GMT+02:00 Andrea Bedini <[hidden email]>:
I can reproduce on OS X, same version of HDF5. Compiling the library with --enable-debug and getting the trace (HDF5_DEBUG=“trace”) shows the following call:

H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0, cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;

So I bet it's a bug in h5repack.

Thanks for testing Andrea. Yes I'm pretty sure it's a parsing bug. The parsing code in tools/h5repack/h5repack_parse.c looks pretty gnarly and I haven't fully understood it yet. But I suspect it might be ignoring anything past the first user parameter, so anything I'm seeing past that is just uninialized memory.. not sure though.

Elvis


Andrea


> On 22 Aug 2016, at 4:18 PM, Elvis Stansvik <[hidden email]> wrote:
>
> 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <[hidden email]>:
> Consider the attached noop_plugin.c. It contains a dynamically loaded filter plugin, with filter number 42, which does nothing except print out the received `cd_values` array in the filter function.
>
> Build the test plugin (on *NIX) with `make`.
>
> Then test the following:
>
> estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1 input.hdf5 output.hdf5
> 1
> estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2 input.hdf5 output.hdf5
> 2 2
> estan@newton:~/noop_plugin$
>
> Notice how in the second invokation (with UD=42,2,1,2), the cd_values contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
>
> I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
>
> Elvis
>
>
> Has anyone seen this strangeness before, or can reproduce?
>
> Thanks in advance,
> Elvis Stansvik
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

--
Andrea Bedini
@andreabedini, http://www.andreabedini.com

See the impact of my research at https://impactstory.org/AndreaBedini
use https://keybase.io/andreabedini to send me encrypted messages
Key fingerprint = 17D5 FB49 FA18 A068 CF53  C5C2 9503 64C1 B2D5 9591




_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Bálint Balázs
In reply to this post by Elvis Stansvik
Hi Elvis,

I also had this problem a few weeks ago on Windows 7, both version 1.8.15 and 1.8.17. I managed to fix it in version 1.8.17, it was indeed in h5repack_parse.c, as you suspected. Please find attached the corrected file, hopefully if will work with 1.8.16 as well.

Balint



2016-08-22 14:43 GMT+02:00 <[hidden email]>:
Send Hdf-forum mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: HDF5 running on ARM V 7 (Gerd Heber)
   2. Re: User-defined filter parameter parsing broken in h5repack?
      (Elvis Stansvik)


----------------------------------------------------------------------




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20160822/d3e1f46f/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 22 Aug 2016 14:42:21 +0200
From: Elvis Stansvik <[hidden email]>
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] User-defined filter parameter parsing broken
        in h5repack?
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

2016-08-22 14:21 GMT+02:00 Andrea Bedini <[hidden email]>:

> I can reproduce on OS X, same version of HDF5. Compiling the library with
> --enable-debug and getting the trace (HDF5_DEBUG=?trace?) shows the
> following call:
>
> H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0,
> cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;
>
> So I bet it's a bug in h5repack.
>

Thanks for testing Andrea. Yes I'm pretty sure it's a parsing bug. The
parsing code in tools/h5repack/h5repack_parse.c looks pretty gnarly and I
haven't fully understood it yet. But I suspect it might be ignoring
anything past the first user parameter, so anything I'm seeing past that is
just uninialized memory.. not sure though.

Elvis


> Andrea
>
>
> > On 22 Aug 2016, at 4:18 PM, Elvis Stansvik <[hidden email]>
> wrote:
> >
> > 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <[hidden email]>
> :
> > Consider the attached noop_plugin.c. It contains a dynamically loaded
> filter plugin, with filter number 42, which does nothing except print out
> the received `cd_values` array in the filter function.
> >
> > Build the test plugin (on *NIX) with `make`.
> >
> > Then test the following:
> >
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
> input.hdf5 output.hdf5
> > 1
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2
> input.hdf5 output.hdf5
> > 2 2
> > estan@newton:~/noop_plugin$
> >
> > Notice how in the second invokation (with UD=42,2,1,2), the cd_values
> contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
> >
> > I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
> >
> > Elvis
> >
> >
> > Has anyone seen this strangeness before, or can reproduce?
> >
> > Thanks in advance,
> > Elvis Stansvik
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> > Twitter: https://twitter.com/hdf5
>
> --
> Andrea Bedini
> @andreabedini, http://www.andreabedini.com
>
> See the impact of my research at https://impactstory.org/AndreaBedini
> use https://keybase.io/andreabedini to send me encrypted messages
> Key fingerprint = 17D5 FB49 FA18 A068 CF53  C5C2 9503 64C1 B2D5 9591
>
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20160822/812a0c74/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 86, Issue 20
*****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

h5repack_parse.zip (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Elvis Stansvik
2016-08-22 15:09 GMT+02:00 Bálint Balázs <[hidden email]>:
Hi Elvis,

I also had this problem a few weeks ago on Windows 7, both version 1.8.15 and 1.8.17. I managed to fix it in version 1.8.17, it was indeed in h5repack_parse.c, as you suspected. Please find attached the corrected file, hopefully if will work with 1.8.16 as well.

Thanks a lot for sharing Balint.

I haven't built HDF5 myself, but using the Ubuntu package, and it seems it's not so easy to just recompile the h5repack tool. But I might try a full build of HDF5 at some point. Your corrected h5repack_parse.c will most probably work fine.

Did you report the issue to the HDF5 maintainers, and do you know if it will be fixed in a future version?

Elvis
 

Balint



2016-08-22 14:43 GMT+02:00 <[hidden email]>:
Send Hdf-forum mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: HDF5 running on ARM V 7 (Gerd Heber)
   2. Re: User-defined filter parameter parsing broken in h5repack?
      (Elvis Stansvik)


----------------------------------------------------------------------




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20160822/d3e1f46f/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 22 Aug 2016 14:42:21 +0200
From: Elvis Stansvik <[hidden email]>
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] User-defined filter parameter parsing broken
        in h5repack?
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

2016-08-22 14:21 GMT+02:00 Andrea Bedini <[hidden email]>:

> I can reproduce on OS X, same version of HDF5. Compiling the library with
> --enable-debug and getting the trace (HDF5_DEBUG=?trace?) shows the

> following call:
>
> H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0,
> cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;
>
> So I bet it's a bug in h5repack.
>

Thanks for testing Andrea. Yes I'm pretty sure it's a parsing bug. The
parsing code in tools/h5repack/h5repack_parse.c looks pretty gnarly and I
haven't fully understood it yet. But I suspect it might be ignoring
anything past the first user parameter, so anything I'm seeing past that is
just uninialized memory.. not sure though.

Elvis


> Andrea
>
>
> > On 22 Aug 2016, at 4:18 PM, Elvis Stansvik <[hidden email]>
> wrote:
> >
> > 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <[hidden email]>
> :
> > Consider the attached noop_plugin.c. It contains a dynamically loaded
> filter plugin, with filter number 42, which does nothing except print out
> the received `cd_values` array in the filter function.
> >
> > Build the test plugin (on *NIX) with `make`.
> >
> > Then test the following:
> >
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
> input.hdf5 output.hdf5
> > 1
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2
> input.hdf5 output.hdf5
> > 2 2
> > estan@newton:~/noop_plugin$
> >
> > Notice how in the second invokation (with UD=42,2,1,2), the cd_values
> contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
> >
> > I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
> >
> > Elvis
> >
> >
> > Has anyone seen this strangeness before, or can reproduce?
> >
> > Thanks in advance,
> > Elvis Stansvik
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> > Twitter: https://twitter.com/hdf5
>
> --
> Andrea Bedini
> @andreabedini, http://www.andreabedini.com
>
> See the impact of my research at https://impactstory.org/AndreaBedini
> use https://keybase.io/andreabedini to send me encrypted messages
> Key fingerprint = 17D5 FB49 FA18 A068 CF53  C5C2 9503 64C1 B2D5 9591
>
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20160822/812a0c74/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 86, Issue 20
*****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Bálint Balázs
In reply to this post by Elvis Stansvik
Hi Elvis,

I hope your rebuild will work.

I actually haven't reported this yet (will do now), but I think regarding the nature of this change, it's very likely they will include it in a future version.

Balint


2016-08-22 15:48 GMT+02:00 <[hidden email]>:



Thanks a lot for sharing Balint.

I haven't built HDF5 myself, but using the Ubuntu package, and it seems
it's not so easy to just recompile the h5repack tool. But I might try a
full build of HDF5 at some point. Your corrected h5repack_parse.c will most
probably work fine.

Did you report the issue to the HDF5 maintainers, and do you know if it
will be fixed in a future version?

Elvis


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: User-defined filter parameter parsing broken in h5repack?

Elvis Stansvik
2016-08-23 13:16 GMT+02:00 Bálint Balázs <[hidden email]>:
Hi Elvis,

I hope your rebuild will work.

I actually haven't reported this yet (will do now), but I think regarding the nature of this change, it's very likely they will include it in a future version.

Alright, thanks a lot.

Elvis
 

Balint


2016-08-22 15:48 GMT+02:00 <[hidden email]>:



Thanks a lot for sharing Balint.

I haven't built HDF5 myself, but using the Ubuntu package, and it seems
it's not so easy to just recompile the h5repack tool. But I might try a
full build of HDF5 at some point. Your corrected h5repack_parse.c will most
probably work fine.

Did you report the issue to the HDF5 maintainers, and do you know if it
will be fixed in a future version?

Elvis


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...