Collective IO and filters

classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
I observe this comment in the H5Z-blosc code:

    /* Allocate an output buffer exactly as long as the input data; if
       the result is larger, we simply return 0.  The filter is flagged
       as optional, so HDF5 marks the chunk as uncompressed and
       proceeds.
    */

In my current setup, I have not marked the filter with
H5Z_FLAG_MANDATORY, for this reason.  Is this comment accurate for the
collective filtered path, or is it possible that the zero return code
is being treated as "compressed data is zero bytes long"?



On Thu, Nov 9, 2017 at 1:37 PM, Michael K. Edwards
<[hidden email]> wrote:

> Thank you for the explanation.  That's consistent with what I see when
> I add a debug printf into H5D__construct_filtered_io_info_list().  So
> I'm now looking into the filter situation.  It's possible that the
> H5Z-blosc glue is mishandling the case where the compressed data is
> larger than the uncompressed data.
>
> About to write 12 of 20
> About to write 0 of 20
> About to write 0 of 20
> About to write 8 of 20
> Rank 0 selected 12 of 20
> Rank 1 selected 8 of 20
> HDF5-DIAG: Error detected in HDF5 (1.11.0) MPI-process 0:
>   #000: H5Dio.c line 319 in H5Dwrite(): can't prepare for writing data
>     major: Dataset
>     minor: Write failed
>   #001: H5Dio.c line 395 in H5D__pre_write(): can't write data
>     major: Dataset
>     minor: Write failed
>   #002: H5Dio.c line 836 in H5D__write(): can't write data
>     major: Dataset
>     minor: Write failed
>   #003: H5Dmpio.c line 1019 in H5D__chunk_collective_write(): write error
>     major: Dataspace
>     minor: Write failed
>   #004: H5Dmpio.c line 934 in H5D__chunk_collective_io(): couldn't
> finish filtered linked chunk MPI-IO
>     major: Low-level I/O
>     minor: Can't get value
>   #005: H5Dmpio.c line 1474 in
> H5D__link_chunk_filtered_collective_io(): couldn't process chunk entry
>     major: Dataset
>     minor: Write failed
>   #006: H5Dmpio.c line 3278 in
> H5D__filtered_collective_chunk_entry_io(): couldn't unfilter chunk for
> modifying
>     major: Data filters
>     minor: Filter operation failed
>   #007: H5Z.c line 1256 in H5Z_pipeline(): filter returned failure during read
>     major: Data filters
>     minor: Read failed
>
>
>
> On Thu, Nov 9, 2017 at 1:02 PM, Jordan Henderson
> <[hidden email]> wrote:
>> For the purpose of collective I/O it is true that all ranks must call
>> H5Dwrite() so that they can participate in those collective operations that
>> are necessary (the file space re-allocation and so on). However, even though
>> they called H5Dwrite() with a valid memspace, the fact that they have a NONE
>> selection in the given file space should cause their chunk-file mapping
>> struct (see lines 357-385 of H5Dpkg.h for the struct's definition and the
>> code for H5D__link_chunk_filtered_collective_io() to see how it uses this
>> built up list of chunks selected in the file) to contain no entries in the
>> "fm->sel_chunks" field. That alone should mean that during the chunk
>> redistribution, they will not actually send anything at all to any of the
>> ranks. They only participate there for the sake that, were the method of
>> redistribution modified, ranks which previously had no chunks selected could
>> potentially be given some chunks to work on.
>>
>>
>> For all practical purposes, every single chunk_entry seen in the list from
>> rank 0's perspective should be a valid I/O caused by some rank writing some
>> positive amount of bytes to the chunk. On rank 0's side, you should be able
>> to check the io_size field of each of the chunk_entry entries and see how
>> big the I/O is from the "original_owner" to that chunk. If any of these are
>> 0, something is likely very wrong. If that is indeed the case, you could
>> likely pull a hacky workaround by manually removing them from the list, but
>> I'd be more concerned about the root of the problem if there are zero-size
>> I/O chunk_entry entries being added to the list.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
That does appear to have been the problem.  I modified H5Z-blosc to
allocate enough room for the BLOSC header, and to fall back to memcpy
mode (clevel=0) if the data expands during "compressed" encoding.
This unblocks me, though I think it might be a good idea for the
collective filtered I/O path to handle H5Z_FLAG_OPTIONAL properly.

Would it be helpful for me to send a patch once I've cleaned up my
debugging goop?  What's a good way to do that -- github pull request?
Do you need a contributor agreement / copyright assignment / some such
thing?


On Thu, Nov 9, 2017 at 1:44 PM, Michael K. Edwards
<[hidden email]> wrote:

> I observe this comment in the H5Z-blosc code:
>
>     /* Allocate an output buffer exactly as long as the input data; if
>        the result is larger, we simply return 0.  The filter is flagged
>        as optional, so HDF5 marks the chunk as uncompressed and
>        proceeds.
>     */
>
> In my current setup, I have not marked the filter with
> H5Z_FLAG_MANDATORY, for this reason.  Is this comment accurate for the
> collective filtered path, or is it possible that the zero return code
> is being treated as "compressed data is zero bytes long"?
>
>
>
> On Thu, Nov 9, 2017 at 1:37 PM, Michael K. Edwards
> <[hidden email]> wrote:
>> Thank you for the explanation.  That's consistent with what I see when
>> I add a debug printf into H5D__construct_filtered_io_info_list().  So
>> I'm now looking into the filter situation.  It's possible that the
>> H5Z-blosc glue is mishandling the case where the compressed data is
>> larger than the uncompressed data.
>>
>> About to write 12 of 20
>> About to write 0 of 20
>> About to write 0 of 20
>> About to write 8 of 20
>> Rank 0 selected 12 of 20
>> Rank 1 selected 8 of 20
>> HDF5-DIAG: Error detected in HDF5 (1.11.0) MPI-process 0:
>>   #000: H5Dio.c line 319 in H5Dwrite(): can't prepare for writing data
>>     major: Dataset
>>     minor: Write failed
>>   #001: H5Dio.c line 395 in H5D__pre_write(): can't write data
>>     major: Dataset
>>     minor: Write failed
>>   #002: H5Dio.c line 836 in H5D__write(): can't write data
>>     major: Dataset
>>     minor: Write failed
>>   #003: H5Dmpio.c line 1019 in H5D__chunk_collective_write(): write error
>>     major: Dataspace
>>     minor: Write failed
>>   #004: H5Dmpio.c line 934 in H5D__chunk_collective_io(): couldn't
>> finish filtered linked chunk MPI-IO
>>     major: Low-level I/O
>>     minor: Can't get value
>>   #005: H5Dmpio.c line 1474 in
>> H5D__link_chunk_filtered_collective_io(): couldn't process chunk entry
>>     major: Dataset
>>     minor: Write failed
>>   #006: H5Dmpio.c line 3278 in
>> H5D__filtered_collective_chunk_entry_io(): couldn't unfilter chunk for
>> modifying
>>     major: Data filters
>>     minor: Filter operation failed
>>   #007: H5Z.c line 1256 in H5Z_pipeline(): filter returned failure during read
>>     major: Data filters
>>     minor: Read failed
>>
>>
>>
>> On Thu, Nov 9, 2017 at 1:02 PM, Jordan Henderson
>> <[hidden email]> wrote:
>>> For the purpose of collective I/O it is true that all ranks must call
>>> H5Dwrite() so that they can participate in those collective operations that
>>> are necessary (the file space re-allocation and so on). However, even though
>>> they called H5Dwrite() with a valid memspace, the fact that they have a NONE
>>> selection in the given file space should cause their chunk-file mapping
>>> struct (see lines 357-385 of H5Dpkg.h for the struct's definition and the
>>> code for H5D__link_chunk_filtered_collective_io() to see how it uses this
>>> built up list of chunks selected in the file) to contain no entries in the
>>> "fm->sel_chunks" field. That alone should mean that during the chunk
>>> redistribution, they will not actually send anything at all to any of the
>>> ranks. They only participate there for the sake that, were the method of
>>> redistribution modified, ranks which previously had no chunks selected could
>>> potentially be given some chunks to work on.
>>>
>>>
>>> For all practical purposes, every single chunk_entry seen in the list from
>>> rank 0's perspective should be a valid I/O caused by some rank writing some
>>> positive amount of bytes to the chunk. On rank 0's side, you should be able
>>> to check the io_size field of each of the chunk_entry entries and see how
>>> big the I/O is from the "original_owner" to that chunk. If any of these are
>>> 0, something is likely very wrong. If that is indeed the case, you could
>>> likely pull a hacky workaround by manually removing them from the list, but
>>> I'd be more concerned about the root of the problem if there are zero-size
>>> I/O chunk_entry entries being added to the list.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Jordan Henderson

As the filtered collective path simply calls through the filter pipeline by way of the H5Z_pipeline() function, it would seem that either the filter pipeline itself is not handling this case correctly, or this is somewhat unexpected behavior for the pipeline to deal with.


Either way, I think a pull request/diff file would be very useful for going over this. If you're able to generate a diff between what you have now and the current develop branch/H5Z-blosc code and put it here that would be useful. I don't think that there should be too much in the way of logistics for getting this code in, we just want to make sure that we approach the solution in the right way without breaking something else.


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
In reply to this post by Michael K. Edwards
Just so it's clear:  the fixes are mostly in the plugins and in how
PETSc calls into the HDF5 code.  (It should probably never have mixed
simple and null dataspaces in one collective write.)  The fixes to
HDF5 itself are:

* Dana's observations with regard to the H5MM APIs:
  * the inappropriate assert(size > 0) in H5MM_[mc]alloc in the
develop branch; and
  * the recommendation to use H5allocate/resize/free_memory() rather
than the private APIs.
* The recommendation to sort by chunk address within each owner's
range of chunk entries, to avoid risk of deadlock in the
H5D__chunk_redistribute_shared_chunks() code.

I haven't switched to H5allocate/resize/free_memory() yet, but here's
(minimally tested) code to handle the other two issues.

Cheers,
- Michael

diff --git a/src/H5Dmpio.c b/src/H5Dmpio.c
index 79572c0..60e9f03 100644
--- a/src/H5Dmpio.c
+++ b/src/H5Dmpio.c
@@ -2328,14 +2328,22 @@
H5D__cmp_filtered_collective_io_info_entry(const void
*filtered_collective_io_in
 static int
 H5D__cmp_filtered_collective_io_info_entry_owner(const void
*filtered_collective_io_info_entry1, const void
*filtered_collective_io_info_entry2)
 {
-    int owner1 = -1, owner2 = -1;
+    int owner1 = -1, owner2 = -1, delta = 0;
+    haddr_t addr1 = HADDR_UNDEF, addr2 = HADDR_UNDEF;

     FUNC_ENTER_STATIC_NOERR

     owner1 = ((const H5D_filtered_collective_io_info_t *)
filtered_collective_io_info_entry1)->owners.original_owner;
     owner2 = ((const H5D_filtered_collective_io_info_t *)
filtered_collective_io_info_entry2)->owners.original_owner;
-
-    FUNC_LEAVE_NOAPI(owner1 - owner2)
+    if (owner1 != owner2) {
+        delta = owner1 - owner2;
+    } else {
+        addr1 = ((const H5D_filtered_collective_io_info_t *)
filtered_collective_io_info_entry1)->chunk_states.new_chunk.offset;
+        addr2 = ((const H5D_filtered_collective_io_info_t *)
filtered_collective_io_info_entry2)->chunk_states.new_chunk.offset;
+        delta = H5F_addr_cmp(addr1, addr2);
+    }
+
+    FUNC_LEAVE_NOAPI(delta)
 } /* end H5D__cmp_filtered_collective_io_info_entry_owner() */

 ^L
diff --git a/src/H5MM.c b/src/H5MM.c
index ee3b28f..3f06850 100644
--- a/src/H5MM.c
+++ b/src/H5MM.c
@@ -268,8 +268,6 @@ H5MM_malloc(size_t size)
 {
     void *ret_value = NULL;

-    HDassert(size);
-
     /* Use FUNC_ENTER_NOAPI_NOINIT_NOERR here to avoid performance issues */
     FUNC_ENTER_NOAPI_NOINIT_NOERR

@@ -357,8 +355,6 @@ H5MM_calloc(size_t size)
 {
     void *ret_value = NULL;

-    HDassert(size);
-
     /* Use FUNC_ENTER_NOAPI_NOINIT_NOERR here to avoid performance issues */
     FUNC_ENTER_NOAPI_NOINIT_NOERR


On Thu, Nov 9, 2017 at 2:27 PM, Michael K. Edwards
<[hidden email]> wrote:

> That does appear to have been the problem.  I modified H5Z-blosc to
> allocate enough room for the BLOSC header, and to fall back to memcpy
> mode (clevel=0) if the data expands during "compressed" encoding.
> This unblocks me, though I think it might be a good idea for the
> collective filtered I/O path to handle H5Z_FLAG_OPTIONAL properly.
>
> Would it be helpful for me to send a patch once I've cleaned up my
> debugging goop?  What's a good way to do that -- github pull request?
> Do you need a contributor agreement / copyright assignment / some such
> thing?
>
>
> On Thu, Nov 9, 2017 at 1:44 PM, Michael K. Edwards
> <[hidden email]> wrote:
>> I observe this comment in the H5Z-blosc code:
>>
>>     /* Allocate an output buffer exactly as long as the input data; if
>>        the result is larger, we simply return 0.  The filter is flagged
>>        as optional, so HDF5 marks the chunk as uncompressed and
>>        proceeds.
>>     */
>>
>> In my current setup, I have not marked the filter with
>> H5Z_FLAG_MANDATORY, for this reason.  Is this comment accurate for the
>> collective filtered path, or is it possible that the zero return code
>> is being treated as "compressed data is zero bytes long"?
>>
>>
>>
>> On Thu, Nov 9, 2017 at 1:37 PM, Michael K. Edwards
>> <[hidden email]> wrote:
>>> Thank you for the explanation.  That's consistent with what I see when
>>> I add a debug printf into H5D__construct_filtered_io_info_list().  So
>>> I'm now looking into the filter situation.  It's possible that the
>>> H5Z-blosc glue is mishandling the case where the compressed data is
>>> larger than the uncompressed data.
>>>
>>> About to write 12 of 20
>>> About to write 0 of 20
>>> About to write 0 of 20
>>> About to write 8 of 20
>>> Rank 0 selected 12 of 20
>>> Rank 1 selected 8 of 20
>>> HDF5-DIAG: Error detected in HDF5 (1.11.0) MPI-process 0:
>>>   #000: H5Dio.c line 319 in H5Dwrite(): can't prepare for writing data
>>>     major: Dataset
>>>     minor: Write failed
>>>   #001: H5Dio.c line 395 in H5D__pre_write(): can't write data
>>>     major: Dataset
>>>     minor: Write failed
>>>   #002: H5Dio.c line 836 in H5D__write(): can't write data
>>>     major: Dataset
>>>     minor: Write failed
>>>   #003: H5Dmpio.c line 1019 in H5D__chunk_collective_write(): write error
>>>     major: Dataspace
>>>     minor: Write failed
>>>   #004: H5Dmpio.c line 934 in H5D__chunk_collective_io(): couldn't
>>> finish filtered linked chunk MPI-IO
>>>     major: Low-level I/O
>>>     minor: Can't get value
>>>   #005: H5Dmpio.c line 1474 in
>>> H5D__link_chunk_filtered_collective_io(): couldn't process chunk entry
>>>     major: Dataset
>>>     minor: Write failed
>>>   #006: H5Dmpio.c line 3278 in
>>> H5D__filtered_collective_chunk_entry_io(): couldn't unfilter chunk for
>>> modifying
>>>     major: Data filters
>>>     minor: Filter operation failed
>>>   #007: H5Z.c line 1256 in H5Z_pipeline(): filter returned failure during read
>>>     major: Data filters
>>>     minor: Read failed
>>>
>>>
>>>
>>> On Thu, Nov 9, 2017 at 1:02 PM, Jordan Henderson
>>> <[hidden email]> wrote:
>>>> For the purpose of collective I/O it is true that all ranks must call
>>>> H5Dwrite() so that they can participate in those collective operations that
>>>> are necessary (the file space re-allocation and so on). However, even though
>>>> they called H5Dwrite() with a valid memspace, the fact that they have a NONE
>>>> selection in the given file space should cause their chunk-file mapping
>>>> struct (see lines 357-385 of H5Dpkg.h for the struct's definition and the
>>>> code for H5D__link_chunk_filtered_collective_io() to see how it uses this
>>>> built up list of chunks selected in the file) to contain no entries in the
>>>> "fm->sel_chunks" field. That alone should mean that during the chunk
>>>> redistribution, they will not actually send anything at all to any of the
>>>> ranks. They only participate there for the sake that, were the method of
>>>> redistribution modified, ranks which previously had no chunks selected could
>>>> potentially be given some chunks to work on.
>>>>
>>>>
>>>> For all practical purposes, every single chunk_entry seen in the list from
>>>> rank 0's perspective should be a valid I/O caused by some rank writing some
>>>> positive amount of bytes to the chunk. On rank 0's side, you should be able
>>>> to check the io_size field of each of the chunk_entry entries and see how
>>>> big the I/O is from the "original_owner" to that chunk. If any of these are
>>>> 0, something is likely very wrong. If that is indeed the case, you could
>>>> likely pull a hacky workaround by manually removing them from the list, but
>>>> I'd be more concerned about the root of the problem if there are zero-size
>>>> I/O chunk_entry entries being added to the list.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
In reply to this post by Jordan Henderson
And here's the change to H5Z-blosc (still using the private H5MM APIs):

diff --git a/src/blosc_filter.c b/src/blosc_filter.c
index bfd8c3e..9bc1a42 100644
--- a/src/blosc_filter.c
+++ b/src/blosc_filter.c
@@ -16,6 +16,7 @@
 #include <string.h>
 #include <errno.h>
 #include "hdf5.h"
+#include "H5MMprivate.h"
 #include "blosc_filter.h"

 #if defined(__GNUC__)
@@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
   /* We're compressing */
   if (!(flags & H5Z_FLAG_REVERSE)) {

-    /* Allocate an output buffer exactly as long as the input data; if
-       the result is larger, we simply return 0.  The filter is flagged
-       as optional, so HDF5 marks the chunk as uncompressed and
-       proceeds.
+    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes
+       larger than the input data, to accommodate the BLOSC header.
+       If compression with the requested parameters causes the data itself
+       to grow (thereby causing the compressed data, with header, to exceed
+       the output buffer size), fall back to memcpy mode (clevel=0).
     */

-    outbuf_size = (*buf_size);
+    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;

 #ifdef BLOSC_DEBUG
     fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",
     nbytes, outbuf_size);
 #endif

-    outbuf = malloc(outbuf_size);
+    outbuf = H5MM_malloc(outbuf_size);

     if (outbuf == NULL) {
       PUSH_ERR("blosc_filter", H5E_CALLBACK,
@@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

     blosc_set_compressor(compname);
     status = blosc_compress(clevel, doshuffle, typesize, nbytes,
-                            *buf, outbuf, nbytes);
+                            *buf, outbuf, outbuf_size);
+    if (status < 0) {
+      status = blosc_compress(0, doshuffle, typesize, nbytes,
+                              *buf, outbuf, outbuf_size);
+    }
     if (status < 0) {
       PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");
       goto failed;
@@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
     /* declare dummy variables */
     size_t cbytes, blocksize;

-    free(outbuf);
+    H5MM_xfree(outbuf);

     /* Extract the exact outbuf_size from the buffer header.
      *
@@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
     fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",
nbytes, outbuf_size);
 #endif

-    outbuf = malloc(outbuf_size);
+    if (outbuf_size == 0) {
+      H5MM_xfree(*buf);
+      *buf = NULL;
+      *buf_size = outbuf_size;
+      return 0;  /* Size of compressed/decompressed data */
+    }
+
+    outbuf = H5MM_malloc(outbuf_size);

     if (outbuf == NULL) {
       PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate
decompression buffer");
@@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
   } /* compressing vs decompressing */

   if (status != 0) {
-    free(*buf);
+    H5MM_xfree(*buf);
     *buf = outbuf;
     *buf_size = outbuf_size;
     return status;  /* Size of compressed/decompressed data */
   }

   failed:
-  free(outbuf);
+  H5MM_xfree(outbuf);
   return 0;

 } /* End filter function */

On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson
<[hidden email]> wrote:

> As the filtered collective path simply calls through the filter pipeline by
> way of the H5Z_pipeline() function, it would seem that either the filter
> pipeline itself is not handling this case correctly, or this is somewhat
> unexpected behavior for the pipeline to deal with.
>
>
> Either way, I think a pull request/diff file would be very useful for going
> over this. If you're able to generate a diff between what you have now and
> the current develop branch/H5Z-blosc code and put it here that would be
> useful. I don't think that there should be too much in the way of logistics
> for getting this code in, we just want to make sure that we approach the
> solution in the right way without breaking something else.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Jordan Henderson
Thanks! I'll discuss this with others and see what the best way to proceed forward from this is. I think this has been a very productive discussion and very useful feedback.

From: Michael K. Edwards <[hidden email]>
Sent: Thursday, November 9, 2017 5:01:33 PM
To: Jordan Henderson
Cc: HDF Users Discussion List
Subject: Re: [Hdf-forum] Collective IO and filters
 
And here's the change to H5Z-blosc (still using the private H5MM APIs):

diff --git a/src/blosc_filter.c b/src/blosc_filter.c
index bfd8c3e..9bc1a42 100644
--- a/src/blosc_filter.c
+++ b/src/blosc_filter.c
@@ -16,6 +16,7 @@
 #include <string.h>
 #include <errno.h>
 #include "hdf5.h"
+#include "H5MMprivate.h"
 #include "blosc_filter.h"

 #if defined(__GNUC__)
@@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
   /* We're compressing */
   if (!(flags & H5Z_FLAG_REVERSE)) {

-    /* Allocate an output buffer exactly as long as the input data; if
-       the result is larger, we simply return 0.  The filter is flagged
-       as optional, so HDF5 marks the chunk as uncompressed and
-       proceeds.
+    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes
+       larger than the input data, to accommodate the BLOSC header.
+       If compression with the requested parameters causes the data itself
+       to grow (thereby causing the compressed data, with header, to exceed
+       the output buffer size), fall back to memcpy mode (clevel=0).
     */

-    outbuf_size = (*buf_size);
+    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;

 #ifdef BLOSC_DEBUG
     fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",
     nbytes, outbuf_size);
 #endif

-    outbuf = malloc(outbuf_size);
+    outbuf = H5MM_malloc(outbuf_size);

     if (outbuf == NULL) {
       PUSH_ERR("blosc_filter", H5E_CALLBACK,
@@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

     blosc_set_compressor(compname);
     status = blosc_compress(clevel, doshuffle, typesize, nbytes,
-                            *buf, outbuf, nbytes);
+                            *buf, outbuf, outbuf_size);
+    if (status < 0) {
+      status = blosc_compress(0, doshuffle, typesize, nbytes,
+                              *buf, outbuf, outbuf_size);
+    }
     if (status < 0) {
       PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");
       goto failed;
@@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
     /* declare dummy variables */
     size_t cbytes, blocksize;

-    free(outbuf);
+    H5MM_xfree(outbuf);

     /* Extract the exact outbuf_size from the buffer header.
      *
@@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
     fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",
nbytes, outbuf_size);
 #endif

-    outbuf = malloc(outbuf_size);
+    if (outbuf_size == 0) {
+      H5MM_xfree(*buf);
+      *buf = NULL;
+      *buf_size = outbuf_size;
+      return 0;  /* Size of compressed/decompressed data */
+    }
+
+    outbuf = H5MM_malloc(outbuf_size);

     if (outbuf == NULL) {
       PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate
decompression buffer");
@@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
   } /* compressing vs decompressing */

   if (status != 0) {
-    free(*buf);
+    H5MM_xfree(*buf);
     *buf = outbuf;
     *buf_size = outbuf_size;
     return status;  /* Size of compressed/decompressed data */
   }

   failed:
-  free(outbuf);
+  H5MM_xfree(outbuf);
   return 0;

 } /* End filter function */

On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson
<[hidden email]> wrote:
> As the filtered collective path simply calls through the filter pipeline by
> way of the H5Z_pipeline() function, it would seem that either the filter
> pipeline itself is not handling this case correctly, or this is somewhat
> unexpected behavior for the pipeline to deal with.
>
>
> Either way, I think a pull request/diff file would be very useful for going
> over this. If you're able to generate a diff between what you have now and
> the current develop branch/H5Z-blosc code and put it here that would be
> useful. I don't think that there should be too much in the way of logistics
> for getting this code in, we just want to make sure that we approach the
> solution in the right way without breaking something else.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
It's exciting to be able to show the collective filtered IO feature as
part of a full software stack.  Thank you for your hard work on this,
and please let me know what more I can do to help keep it on glide
path for release.


On Thu, Nov 9, 2017 at 3:22 PM, Jordan Henderson
<[hidden email]> wrote:

> Thanks! I'll discuss this with others and see what the best way to proceed
> forward from this is. I think this has been a very productive discussion and
> very useful feedback.
> ________________________________
> From: Michael K. Edwards <[hidden email]>
> Sent: Thursday, November 9, 2017 5:01:33 PM
> To: Jordan Henderson
> Cc: HDF Users Discussion List
> Subject: Re: [Hdf-forum] Collective IO and filters
>
> And here's the change to H5Z-blosc (still using the private H5MM APIs):
>
> diff --git a/src/blosc_filter.c b/src/blosc_filter.c
> index bfd8c3e..9bc1a42 100644
> --- a/src/blosc_filter.c
> +++ b/src/blosc_filter.c
> @@ -16,6 +16,7 @@
>  #include <string.h>
>  #include <errno.h>
>  #include "hdf5.h"
> +#include "H5MMprivate.h"
>  #include "blosc_filter.h"
>
>  #if defined(__GNUC__)
> @@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>    /* We're compressing */
>    if (!(flags & H5Z_FLAG_REVERSE)) {
>
> -    /* Allocate an output buffer exactly as long as the input data; if
> -       the result is larger, we simply return 0.  The filter is flagged
> -       as optional, so HDF5 marks the chunk as uncompressed and
> -       proceeds.
> +    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes
> +       larger than the input data, to accommodate the BLOSC header.
> +       If compression with the requested parameters causes the data itself
> +       to grow (thereby causing the compressed data, with header, to exceed
> +       the output buffer size), fall back to memcpy mode (clevel=0).
>      */
>
> -    outbuf_size = (*buf_size);
> +    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;
>
>  #ifdef BLOSC_DEBUG
>      fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",
>      nbytes, outbuf_size);
>  #endif
>
> -    outbuf = malloc(outbuf_size);
> +    outbuf = H5MM_malloc(outbuf_size);
>
>      if (outbuf == NULL) {
>        PUSH_ERR("blosc_filter", H5E_CALLBACK,
> @@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>
>      blosc_set_compressor(compname);
>      status = blosc_compress(clevel, doshuffle, typesize, nbytes,
> -                            *buf, outbuf, nbytes);
> +                            *buf, outbuf, outbuf_size);
> +    if (status < 0) {
> +      status = blosc_compress(0, doshuffle, typesize, nbytes,
> +                              *buf, outbuf, outbuf_size);
> +    }
>      if (status < 0) {
>        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");
>        goto failed;
> @@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>      /* declare dummy variables */
>      size_t cbytes, blocksize;
>
> -    free(outbuf);
> +    H5MM_xfree(outbuf);
>
>      /* Extract the exact outbuf_size from the buffer header.
>       *
> @@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>      fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",
> nbytes, outbuf_size);
>  #endif
>
> -    outbuf = malloc(outbuf_size);
> +    if (outbuf_size == 0) {
> +      H5MM_xfree(*buf);
> +      *buf = NULL;
> +      *buf_size = outbuf_size;
> +      return 0;  /* Size of compressed/decompressed data */
> +    }
> +
> +    outbuf = H5MM_malloc(outbuf_size);
>
>      if (outbuf == NULL) {
>        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate
> decompression buffer");
> @@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>    } /* compressing vs decompressing */
>
>    if (status != 0) {
> -    free(*buf);
> +    H5MM_xfree(*buf);
>      *buf = outbuf;
>      *buf_size = outbuf_size;
>      return status;  /* Size of compressed/decompressed data */
>    }
>
>    failed:
> -  free(outbuf);
> +  H5MM_xfree(outbuf);
>    return 0;
>
>  } /* End filter function */
>
> On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson
> <[hidden email]> wrote:
>> As the filtered collective path simply calls through the filter pipeline
>> by
>> way of the H5Z_pipeline() function, it would seem that either the filter
>> pipeline itself is not handling this case correctly, or this is somewhat
>> unexpected behavior for the pipeline to deal with.
>>
>>
>> Either way, I think a pull request/diff file would be very useful for
>> going
>> over this. If you're able to generate a diff between what you have now and
>> the current develop branch/H5Z-blosc code and put it here that would be
>> useful. I don't think that there should be too much in the way of
>> logistics
>> for getting this code in, we just want to make sure that we approach the
>> solution in the right way without breaking something else.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
It's good to have a reference example when testing an integration like
this.  I've attached the patch I've been using against the "maint"
(3.8.x) branch of PETSc.  It's obviously not suitable for integration
(it blindly applies ZFP to floating point Vecs and BloscLZ to integer
Vecs), but it does exercise the code paths in interesting ways.

Here's how I configure the HDF5 develop branch (for debug purposes):

./configure --prefix=/usr/local 'MAKE=/usr/bin/gmake' 'CC=mpicc'
'CFLAGS=-fPIC -fstack-protector -g3 -fopenmp' 'AR=/usr/bin/ar'
'ARFLAGS=cr' 'CXX=mpicxx' 'CXXFLAGS=-fstack-protector -g -fopenmp
-fPIC' 'F90=mpif90' 'F90FLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'
'F77=mpif90' 'FFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'
'FC=mpif90' 'FCFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'
'--enable-shared' '--with-default-api-version=v18' '--enable-parallel'
'--enable-fortran' 'F9X=mpif90' '--with-zlib=yes' '--with-szlib=yes'

And here's how I configure and run PETSc (again, for debug purposes):

./configure --without-x --with-openmp
--with-blaslapack-dir=/opt/intel/mkl --with-hdf5 --download-p4est=yes
--download-triangle=yes --download-pragmatic=yes --download-metis=yes
--download-eigen=yes
make PETSC_DIR=/home/centos/p4est/petsc/petsc PETSC_ARCH=arch-linux2-c-debug all
cd src/snes/examples/tutorials
make PETSC_DIR=/home/centos/p4est/petsc/petsc
PETSC_ARCH=arch-linux2-c-debug ex12
/usr/local/bin/mpiexec -n 4 ./ex12 -run_type full
-variable_coefficient nonlinear -nonzero_initial_guess 1 -interpolate
1 -petscspace_order 2 -snes_max_it 10 -snes_type fas
-snes_linesearch_type bt -snes_fas_levels 3 -fas_coarse_snes_type
newtonls -fas_coarse_snes_linesearch_type basic -fas_coarse_ksp_type
cg -fas_coarse_pc_type jacobi -fas_coarse_snes_monitor_short
-fas_levels_snes_max_it 4 -fas_levels_snes_type newtonls
-fas_levels_snes_linesearch_type bt -fas_levels_ksp_type cg
-fas_levels_pc_type jacobi -fas_levels_snes_monitor_short
-fas_levels_cycle_snes_linesearch_type bt -snes_monitor_short
-snes_converged_reason -snes_view -simplex 0 -petscspace_poly_tensor
-dm_plex_convert_type p4est -dm_forest_minimum_refinement 0
-dm_forest_initial_refinement 2 -dm_forest_maximum_refinement 4
-dm_p4est_refine_pattern hash -dm_view_hierarchy

Basically this is a smoke test for various shapes and sizes of object
that occur in an adaptive mesh refinement use case.  The
"-dm_view_hierarchy" flag is what triggers the write of three HDF5
files.  The typical structure looks like this:

[centos@centos74 tutorials]$ h5dump -pH ex12-2.h5
HDF5 "ex12-2.h5" {
GROUP "/" {
   GROUP "fields" {
      DATASET "solution error" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 636 ) / ( 636 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 636 )
            SIZE 1566 (3.249:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32013
               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP
               PARAMS { 5374064 91252346 10163 -924844032 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
      }
   }
   GROUP "geometry" {
      DATASET "vertices" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 170, 2 ) / ( 170, 2 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 170, 2 )
            SIZE 3318 (0.820:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32013
               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP
               PARAMS { 5374064 91252346 -1879048169 -924844022 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
      }
   }
   GROUP "labels" {
      GROUP "Face Sets" {
         GROUP "1" {
            DATASET "indices" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }
               STORAGE_LAYOUT {
                  CHUNKED ( 20, 1 )
                  SIZE 96 (0.833:1 COMPRESSION)
               }
               FILTERS {
                  USER_DEFINED_FILTER {
                     FILTER_ID 32001
                     COMMENT blosc
                     PARAMS { 2 2 4 80 5 1 0 }
                  }
               }
               FILLVALUE {
                  FILL_TIME H5D_FILL_TIME_IFSET
                  VALUE  H5D_FILL_VALUE_DEFAULT
               }
               ALLOCATION_TIME {
                  H5D_ALLOC_TIME_EARLY
               }
            }
         }
         GROUP "2" {
            DATASET "indices" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }
               STORAGE_LAYOUT {
                  CHUNKED ( 20, 1 )
                  SIZE 96 (0.833:1 COMPRESSION)
               }
               FILTERS {
                  USER_DEFINED_FILTER {
                     FILTER_ID 32001
                     COMMENT blosc
                     PARAMS { 2 2 4 80 5 1 0 }
                  }
               }
               FILLVALUE {
                  FILL_TIME H5D_FILL_TIME_IFSET
                  VALUE  H5D_FILL_VALUE_DEFAULT
               }
               ALLOCATION_TIME {
                  H5D_ALLOC_TIME_EARLY
               }
            }
         }
         GROUP "3" {
            DATASET "indices" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 14, 1 ) / ( 14, 1 ) }
               STORAGE_LAYOUT {
                  CHUNKED ( 14, 1 )
                  SIZE 72 (0.778:1 COMPRESSION)
               }
               FILTERS {
                  USER_DEFINED_FILTER {
                     FILTER_ID 32001
                     COMMENT blosc
                     PARAMS { 2 2 4 56 5 1 0 }
                  }
               }
               FILLVALUE {
                  FILL_TIME H5D_FILL_TIME_IFSET
                  VALUE  H5D_FILL_VALUE_DEFAULT
               }
               ALLOCATION_TIME {
                  H5D_ALLOC_TIME_EARLY
               }
            }
         }
         GROUP "4" {
            DATASET "indices" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 22, 1 ) / ( 22, 1 ) }
               STORAGE_LAYOUT {
                  CHUNKED ( 22, 1 )
                  SIZE 104 (0.846:1 COMPRESSION)
               }
               FILTERS {
                  USER_DEFINED_FILTER {
                     FILTER_ID 32001
                     COMMENT blosc
                     PARAMS { 2 2 4 88 5 1 0 }
                  }
               }
               FILLVALUE {
                  FILL_TIME H5D_FILL_TIME_IFSET
                  VALUE  H5D_FILL_VALUE_DEFAULT
               }
               ALLOCATION_TIME {
                  H5D_ALLOC_TIME_EARLY
               }
            }
         }
      }
      GROUP "marker" {
         GROUP "1" {
            DATASET "indices" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 80, 1 ) / ( 80, 1 ) }
               STORAGE_LAYOUT {
                  CHUNKED ( 80, 1 )
                  SIZE 336 (0.952:1 COMPRESSION)
               }
               FILTERS {
                  USER_DEFINED_FILTER {
                     FILTER_ID 32001
                     COMMENT blosc
                     PARAMS { 2 2 4 320 5 1 0 }
                  }
               }
               FILLVALUE {
                  FILL_TIME H5D_FILL_TIME_IFSET
                  VALUE  H5D_FILL_VALUE_DEFAULT
               }
               ALLOCATION_TIME {
                  H5D_ALLOC_TIME_EARLY
               }
            }
         }
      }
   }
   GROUP "topology" {
      DATASET "cells" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 1136, 1 )
            SIZE 1476 (3.079:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 4 4544 5 1 0 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
         ATTRIBUTE "cell_dim" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SCALAR
         }
      }
      DATASET "cones" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 636, 1 )
            SIZE 155 (16.413:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 4 2544 5 1 0 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
      }
      DATASET "order" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 636, 1 )
            SIZE 755 (3.370:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 4 2544 5 1 0 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
      }
      DATASET "orientation" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 1136, 1 )
            SIZE 216 (21.037:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 4 4544 5 1 0 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
      }
   }
   GROUP "vertex_fields" {
      DATASET "solution error_potential" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 170 ) / ( 170 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 170 )
            SIZE 617 (2.204:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32013
               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP
               PARAMS { 5374064 91252346 2707 -924844032 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_EARLY
         }
         ATTRIBUTE "vector_field_type" {
            DATATYPE  H5T_STRING {
               STRSIZE 7;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
            DATASPACE  SCALAR
         }
      }
   }
   GROUP "viz" {
      GROUP "topology" {
         DATASET "cells" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 130, 4 ) / ( 130, 4 ) }
            STORAGE_LAYOUT {
               CHUNKED ( 130, 4 )
               SIZE 592 (3.514:1 COMPRESSION)
            }
            FILTERS {
               USER_DEFINED_FILTER {
                  FILTER_ID 32001
                  COMMENT blosc
                  PARAMS { 2 2 4 2080 5 1 0 }
               }
            }
            FILLVALUE {
               FILL_TIME H5D_FILL_TIME_IFSET
               VALUE  H5D_FILL_VALUE_DEFAULT
            }
            ALLOCATION_TIME {
               H5D_ALLOC_TIME_EARLY
            }
            ATTRIBUTE "cell_corners" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SCALAR
            }
            ATTRIBUTE "cell_dim" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SCALAR
            }
         }
      }
   }
}
}




On Thu, Nov 9, 2017 at 3:27 PM, Michael K. Edwards
<[hidden email]> wrote:

> It's exciting to be able to show the collective filtered IO feature as
> part of a full software stack.  Thank you for your hard work on this,
> and please let me know what more I can do to help keep it on glide
> path for release.
>
>
> On Thu, Nov 9, 2017 at 3:22 PM, Jordan Henderson
> <[hidden email]> wrote:
>> Thanks! I'll discuss this with others and see what the best way to proceed
>> forward from this is. I think this has been a very productive discussion and
>> very useful feedback.
>> ________________________________
>> From: Michael K. Edwards <[hidden email]>
>> Sent: Thursday, November 9, 2017 5:01:33 PM
>> To: Jordan Henderson
>> Cc: HDF Users Discussion List
>> Subject: Re: [Hdf-forum] Collective IO and filters
>>
>> And here's the change to H5Z-blosc (still using the private H5MM APIs):
>>
>> diff --git a/src/blosc_filter.c b/src/blosc_filter.c
>> index bfd8c3e..9bc1a42 100644
>> --- a/src/blosc_filter.c
>> +++ b/src/blosc_filter.c
>> @@ -16,6 +16,7 @@
>>  #include <string.h>
>>  #include <errno.h>
>>  #include "hdf5.h"
>> +#include "H5MMprivate.h"
>>  #include "blosc_filter.h"
>>
>>  #if defined(__GNUC__)
>> @@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>>    /* We're compressing */
>>    if (!(flags & H5Z_FLAG_REVERSE)) {
>>
>> -    /* Allocate an output buffer exactly as long as the input data; if
>> -       the result is larger, we simply return 0.  The filter is flagged
>> -       as optional, so HDF5 marks the chunk as uncompressed and
>> -       proceeds.
>> +    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes
>> +       larger than the input data, to accommodate the BLOSC header.
>> +       If compression with the requested parameters causes the data itself
>> +       to grow (thereby causing the compressed data, with header, to exceed
>> +       the output buffer size), fall back to memcpy mode (clevel=0).
>>      */
>>
>> -    outbuf_size = (*buf_size);
>> +    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;
>>
>>  #ifdef BLOSC_DEBUG
>>      fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",
>>      nbytes, outbuf_size);
>>  #endif
>>
>> -    outbuf = malloc(outbuf_size);
>> +    outbuf = H5MM_malloc(outbuf_size);
>>
>>      if (outbuf == NULL) {
>>        PUSH_ERR("blosc_filter", H5E_CALLBACK,
>> @@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>>
>>      blosc_set_compressor(compname);
>>      status = blosc_compress(clevel, doshuffle, typesize, nbytes,
>> -                            *buf, outbuf, nbytes);
>> +                            *buf, outbuf, outbuf_size);
>> +    if (status < 0) {
>> +      status = blosc_compress(0, doshuffle, typesize, nbytes,
>> +                              *buf, outbuf, outbuf_size);
>> +    }
>>      if (status < 0) {
>>        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");
>>        goto failed;
>> @@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>>      /* declare dummy variables */
>>      size_t cbytes, blocksize;
>>
>> -    free(outbuf);
>> +    H5MM_xfree(outbuf);
>>
>>      /* Extract the exact outbuf_size from the buffer header.
>>       *
>> @@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>>      fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",
>> nbytes, outbuf_size);
>>  #endif
>>
>> -    outbuf = malloc(outbuf_size);
>> +    if (outbuf_size == 0) {
>> +      H5MM_xfree(*buf);
>> +      *buf = NULL;
>> +      *buf_size = outbuf_size;
>> +      return 0;  /* Size of compressed/decompressed data */
>> +    }
>> +
>> +    outbuf = H5MM_malloc(outbuf_size);
>>
>>      if (outbuf == NULL) {
>>        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate
>> decompression buffer");
>> @@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,
>>    } /* compressing vs decompressing */
>>
>>    if (status != 0) {
>> -    free(*buf);
>> +    H5MM_xfree(*buf);
>>      *buf = outbuf;
>>      *buf_size = outbuf_size;
>>      return status;  /* Size of compressed/decompressed data */
>>    }
>>
>>    failed:
>> -  free(outbuf);
>> +  H5MM_xfree(outbuf);
>>    return 0;
>>
>>  } /* End filter function */
>>
>> On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson
>> <[hidden email]> wrote:
>>> As the filtered collective path simply calls through the filter pipeline
>>> by
>>> way of the H5Z_pipeline() function, it would seem that either the filter
>>> pipeline itself is not handling this case correctly, or this is somewhat
>>> unexpected behavior for the pipeline to deal with.
>>>
>>>
>>> Either way, I think a pull request/diff file would be very useful for
>>> going
>>> over this. If you're able to generate a diff between what you have now and
>>> the current develop branch/H5Z-blosc code and put it here that would be
>>> useful. I don't think that there should be too much in the way of
>>> logistics
>>> for getting this code in, we just want to make sure that we approach the
>>> solution in the right way without breaking something else.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

0001-Proof-of-concept-no-control-path-demonstration-of-co.patch (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Miller, Mark C.

Hey Michael,

 

This is all in parallel? Thats awesome to get working!

 

Just a very brief look see on the h5dump output suggests the chunking is all Nx1.

 

Filters are applied on a per-chunk basis with a parameter set nonetheless that is fixed for all chunks (unless the filter is encoding params *into* the chunks themselves as opposed to dataset header messages). So, in the case of ZFP, with Nx1 chunking, its not really giving ZFP the greatest opportunity to compress because its essentially limiting its work to one dimension.

 

Next, the chunk size does seem to yield pretty small chunks.

 

Is this a constraint in the way parallel is being handled? Can you easily switch to larger chunks and chunks with non-unity in more dimensions?

 

Mark

 

 

"Hdf-forum on behalf of Michael K. Edwards" wrote:

 

It's good to have a reference example when testing an integration like

this.  I've attached the patch I've been using against the "maint"

(3.8.x) branch of PETSc.  It's obviously not suitable for integration

(it blindly applies ZFP to floating point Vecs and BloscLZ to integer

Vecs), but it does exercise the code paths in interesting ways.

 

Here's how I configure the HDF5 develop branch (for debug purposes):

 

./configure --prefix=/usr/local 'MAKE=/usr/bin/gmake' 'CC=mpicc'

'CFLAGS=-fPIC -fstack-protector -g3 -fopenmp' 'AR=/usr/bin/ar'

'ARFLAGS=cr' 'CXX=mpicxx' 'CXXFLAGS=-fstack-protector -g -fopenmp

-fPIC' 'F90=mpif90' 'F90FLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'F77=mpif90' 'FFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'FC=mpif90' 'FCFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'--enable-shared' '--with-default-api-version=v18' '--enable-parallel'

'--enable-fortran' 'F9X=mpif90' '--with-zlib=yes' '--with-szlib=yes'

 

And here's how I configure and run PETSc (again, for debug purposes):

 

./configure --without-x --with-openmp

--with-blaslapack-dir=/opt/intel/mkl --with-hdf5 --download-p4est=yes

--download-triangle=yes --download-pragmatic=yes --download-metis=yes

--download-eigen=yes

make PETSC_DIR=/home/centos/p4est/petsc/petsc PETSC_ARCH=arch-linux2-c-debug all

cd src/snes/examples/tutorials

make PETSC_DIR=/home/centos/p4est/petsc/petsc

PETSC_ARCH=arch-linux2-c-debug ex12

/usr/local/bin/mpiexec -n 4 ./ex12 -run_type full

-variable_coefficient nonlinear -nonzero_initial_guess 1 -interpolate

1 -petscspace_order 2 -snes_max_it 10 -snes_type fas

-snes_linesearch_type bt -snes_fas_levels 3 -fas_coarse_snes_type

newtonls -fas_coarse_snes_linesearch_type basic -fas_coarse_ksp_type

cg -fas_coarse_pc_type jacobi -fas_coarse_snes_monitor_short

-fas_levels_snes_max_it 4 -fas_levels_snes_type newtonls

-fas_levels_snes_linesearch_type bt -fas_levels_ksp_type cg

-fas_levels_pc_type jacobi -fas_levels_snes_monitor_short

-fas_levels_cycle_snes_linesearch_type bt -snes_monitor_short

-snes_converged_reason -snes_view -simplex 0 -petscspace_poly_tensor

-dm_plex_convert_type p4est -dm_forest_minimum_refinement 0

-dm_forest_initial_refinement 2 -dm_forest_maximum_refinement 4

-dm_p4est_refine_pattern hash -dm_view_hierarchy

 

Basically this is a smoke test for various shapes and sizes of object

that occur in an adaptive mesh refinement use case.  The

"-dm_view_hierarchy" flag is what triggers the write of three HDF5

files.  The typical structure looks like this:

 

[centos@centos74 tutorials]$ h5dump -pH ex12-2.h5

HDF5 "ex12-2.h5" {

GROUP "/" {

   GROUP "fields" {

      DATASET "solution error" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 636 ) / ( 636 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636 )

            SIZE 1566 (3.249:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 10163 -924844032 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "geometry" {

      DATASET "vertices" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 170, 2 ) / ( 170, 2 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 170, 2 )

            SIZE 3318 (0.820:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 -1879048169 -924844022 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "labels" {

      GROUP "Face Sets" {

         GROUP "1" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 20, 1 )

                  SIZE 96 (0.833:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 80 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "2" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 20, 1 )

                  SIZE 96 (0.833:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 80 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "3" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 14, 1 ) / ( 14, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 14, 1 )

                  SIZE 72 (0.778:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 56 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "4" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 22, 1 ) / ( 22, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 22, 1 )

                  SIZE 104 (0.846:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 88 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

      }

      GROUP "marker" {

         GROUP "1" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 80, 1 ) / ( 80, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 80, 1 )

                  SIZE 336 (0.952:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 320 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

      }

   }

   GROUP "topology" {

      DATASET "cells" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 1136, 1 )

            SIZE 1476 (3.079:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 4544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

         ATTRIBUTE "cell_dim" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SCALAR

         }

      }

      DATASET "cones" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636, 1 )

            SIZE 155 (16.413:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 2544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

      DATASET "order" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636, 1 )

            SIZE 755 (3.370:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 2544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

      DATASET "orientation" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 1136, 1 )

            SIZE 216 (21.037:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 4544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "vertex_fields" {

      DATASET "solution error_potential" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 170 ) / ( 170 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 170 )

            SIZE 617 (2.204:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 2707 -924844032 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

         ATTRIBUTE "vector_field_type" {

            DATATYPE  H5T_STRING {

               STRSIZE 7;

               STRPAD H5T_STR_NULLTERM;

               CSET H5T_CSET_ASCII;

               CTYPE H5T_C_S1;

            }

            DATASPACE  SCALAR

         }

      }

   }

   GROUP "viz" {

      GROUP "topology" {

         DATASET "cells" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 130, 4 ) / ( 130, 4 ) }

            STORAGE_LAYOUT {

               CHUNKED ( 130, 4 )

               SIZE 592 (3.514:1 COMPRESSION)

            }

            FILTERS {

               USER_DEFINED_FILTER {

                  FILTER_ID 32001

                  COMMENT blosc

                  PARAMS { 2 2 4 2080 5 1 0 }

               }

            }

            FILLVALUE {

               FILL_TIME H5D_FILL_TIME_IFSET

               VALUE  H5D_FILL_VALUE_DEFAULT

            }

            ALLOCATION_TIME {

               H5D_ALLOC_TIME_EARLY

            }

            ATTRIBUTE "cell_corners" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

            ATTRIBUTE "cell_dim" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

         }

      }

   }

}

}

 

 

 

 

On Thu, Nov 9, 2017 at 3:27 PM, Michael K. Edwards

<[hidden email]> wrote:

It's exciting to be able to show the collective filtered IO feature as

part of a full software stack.  Thank you for your hard work on this,

and please let me know what more I can do to help keep it on glide

path for release.

 

 

On Thu, Nov 9, 2017 at 3:22 PM, Jordan Henderson

<[hidden email]> wrote:

Thanks! I'll discuss this with others and see what the best way to proceed

forward from this is. I think this has been a very productive discussion and

very useful feedback.

________________________________

From: Michael K. Edwards <[hidden email]>

Sent: Thursday, November 9, 2017 5:01:33 PM

To: Jordan Henderson

Cc: HDF Users Discussion List

Subject: Re: [Hdf-forum] Collective IO and filters

 

And here's the change to H5Z-blosc (still using the private H5MM APIs):

 

diff --git a/src/blosc_filter.c b/src/blosc_filter.c

index bfd8c3e..9bc1a42 100644

--- a/src/blosc_filter.c

+++ b/src/blosc_filter.c

@@ -16,6 +16,7 @@

  #include <string.h>

  #include <errno.h>

  #include "hdf5.h"

+#include "H5MMprivate.h"

  #include "blosc_filter.h"

 

  #if defined(__GNUC__)

@@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

    /* We're compressing */

    if (!(flags & H5Z_FLAG_REVERSE)) {

 

-    /* Allocate an output buffer exactly as long as the input data; if

-       the result is larger, we simply return 0.  The filter is flagged

-       as optional, so HDF5 marks the chunk as uncompressed and

-       proceeds.

+    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes

+       larger than the input data, to accommodate the BLOSC header.

+       If compression with the requested parameters causes the data itself

+       to grow (thereby causing the compressed data, with header, to exceed

+       the output buffer size), fall back to memcpy mode (clevel=0).

      */

 

-    outbuf_size = (*buf_size);

+    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;

 

  #ifdef BLOSC_DEBUG

      fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",

      nbytes, outbuf_size);

  #endif

 

-    outbuf = malloc(outbuf_size);

+    outbuf = H5MM_malloc(outbuf_size);

 

      if (outbuf == NULL) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK,

@@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

 

      blosc_set_compressor(compname);

      status = blosc_compress(clevel, doshuffle, typesize, nbytes,

-                            *buf, outbuf, nbytes);

+                            *buf, outbuf, outbuf_size);

+    if (status < 0) {

+      status = blosc_compress(0, doshuffle, typesize, nbytes,

+                              *buf, outbuf, outbuf_size);

+    }

      if (status < 0) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");

        goto failed;

@@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

      /* declare dummy variables */

      size_t cbytes, blocksize;

 

-    free(outbuf);

+    H5MM_xfree(outbuf);

 

      /* Extract the exact outbuf_size from the buffer header.

       *

@@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

      fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",

nbytes, outbuf_size);

  #endif

 

-    outbuf = malloc(outbuf_size);

+    if (outbuf_size == 0) {

+      H5MM_xfree(*buf);

+      *buf = NULL;

+      *buf_size = outbuf_size;

+      return 0;  /* Size of compressed/decompressed data */

+    }

+

+    outbuf = H5MM_malloc(outbuf_size);

 

      if (outbuf == NULL) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate

decompression buffer");

@@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

    } /* compressing vs decompressing */

 

    if (status != 0) {

-    free(*buf);

+    H5MM_xfree(*buf);

      *buf = outbuf;

      *buf_size = outbuf_size;

      return status;  /* Size of compressed/decompressed data */

    }

 

    failed:

-  free(outbuf);

+  H5MM_xfree(outbuf);

    return 0;

 

  } /* End filter function */

 

On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson

<[hidden email]> wrote:

As the filtered collective path simply calls through the filter pipeline

by

way of the H5Z_pipeline() function, it would seem that either the filter

pipeline itself is not handling this case correctly, or this is somewhat

unexpected behavior for the pipeline to deal with.

 

 

Either way, I think a pull request/diff file would be very useful for

going

over this. If you're able to generate a diff between what you have now and

the current develop branch/H5Z-blosc code and put it here that would be

useful. I don't think that there should be too much in the way of

logistics

for getting this code in, we just want to make sure that we approach the

solution in the right way without breaking something else.

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Michael K. Edwards
Yes, it's all 4-way parallel (at test scale), and should be exercisable at much larger scale with changes to command-line parameters.  The p4est/p8est code routinely generates Nx1 Vecs because it stores data in "space-filling curve" traversal order.  I should have some examples to post soon that uses DMDA instead, which should afford some scope for exercising 2-D/3-D modes of ZFP/fpzip.  I wanted to make sure we could handle the little corner-case-y structures before stressing it in the direction of scale.

On Nov 14, 2017 3:27 PM, "Miller, Mark C." <[hidden email]> wrote:

Hey Michael,

 

This is all in parallel? Thats awesome to get working!

 

Just a very brief look see on the h5dump output suggests the chunking is all Nx1.

 

Filters are applied on a per-chunk basis with a parameter set nonetheless that is fixed for all chunks (unless the filter is encoding params *into* the chunks themselves as opposed to dataset header messages). So, in the case of ZFP, with Nx1 chunking, its not really giving ZFP the greatest opportunity to compress because its essentially limiting its work to one dimension.

 

Next, the chunk size does seem to yield pretty small chunks.

 

Is this a constraint in the way parallel is being handled? Can you easily switch to larger chunks and chunks with non-unity in more dimensions?

 

Mark

 

 

"Hdf-forum on behalf of Michael K. Edwards" wrote:

 

It's good to have a reference example when testing an integration like

this.  I've attached the patch I've been using against the "maint"

(3.8.x) branch of PETSc.  It's obviously not suitable for integration

(it blindly applies ZFP to floating point Vecs and BloscLZ to integer

Vecs), but it does exercise the code paths in interesting ways.

 

Here's how I configure the HDF5 develop branch (for debug purposes):

 

./configure --prefix=/usr/local 'MAKE=/usr/bin/gmake' 'CC=mpicc'

'CFLAGS=-fPIC -fstack-protector -g3 -fopenmp' 'AR=/usr/bin/ar'

'ARFLAGS=cr' 'CXX=mpicxx' 'CXXFLAGS=-fstack-protector -g -fopenmp

-fPIC' 'F90=mpif90' 'F90FLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'F77=mpif90' 'FFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'FC=mpif90' 'FCFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'--enable-shared' '--with-default-api-version=v18' '--enable-parallel'

'--enable-fortran' 'F9X=mpif90' '--with-zlib=yes' '--with-szlib=yes'

 

And here's how I configure and run PETSc (again, for debug purposes):

 

./configure --without-x --with-openmp

--with-blaslapack-dir=/opt/intel/mkl --with-hdf5 --download-p4est=yes

--download-triangle=yes --download-pragmatic=yes --download-metis=yes

--download-eigen=yes

make PETSC_DIR=/home/centos/p4est/petsc/petsc PETSC_ARCH=arch-linux2-c-debug all

cd src/snes/examples/tutorials

make PETSC_DIR=/home/centos/p4est/petsc/petsc

PETSC_ARCH=arch-linux2-c-debug ex12

/usr/local/bin/mpiexec -n 4 ./ex12 -run_type full

-variable_coefficient nonlinear -nonzero_initial_guess 1 -interpolate

1 -petscspace_order 2 -snes_max_it 10 -snes_type fas

-snes_linesearch_type bt -snes_fas_levels 3 -fas_coarse_snes_type

newtonls -fas_coarse_snes_linesearch_type basic -fas_coarse_ksp_type

cg -fas_coarse_pc_type jacobi -fas_coarse_snes_monitor_short

-fas_levels_snes_max_it 4 -fas_levels_snes_type newtonls

-fas_levels_snes_linesearch_type bt -fas_levels_ksp_type cg

-fas_levels_pc_type jacobi -fas_levels_snes_monitor_short

-fas_levels_cycle_snes_linesearch_type bt -snes_monitor_short

-snes_converged_reason -snes_view -simplex 0 -petscspace_poly_tensor

-dm_plex_convert_type p4est -dm_forest_minimum_refinement 0

-dm_forest_initial_refinement 2 -dm_forest_maximum_refinement 4

-dm_p4est_refine_pattern hash -dm_view_hierarchy

 

Basically this is a smoke test for various shapes and sizes of object

that occur in an adaptive mesh refinement use case.  The

"-dm_view_hierarchy" flag is what triggers the write of three HDF5

files.  The typical structure looks like this:

 

[centos@centos74 tutorials]$ h5dump -pH ex12-2.h5

HDF5 "ex12-2.h5" {

GROUP "/" {

   GROUP "fields" {

      DATASET "solution error" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 636 ) / ( 636 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636 )

            SIZE 1566 (3.249:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 10163 -924844032 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "geometry" {

      DATASET "vertices" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 170, 2 ) / ( 170, 2 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 170, 2 )

            SIZE 3318 (0.820:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 -1879048169 -924844022 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "labels" {

      GROUP "Face Sets" {

         GROUP "1" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 20, 1 )

                  SIZE 96 (0.833:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 80 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "2" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 20, 1 )

                  SIZE 96 (0.833:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 80 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "3" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 14, 1 ) / ( 14, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 14, 1 )

                  SIZE 72 (0.778:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 56 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "4" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 22, 1 ) / ( 22, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 22, 1 )

                  SIZE 104 (0.846:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 88 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

      }

      GROUP "marker" {

         GROUP "1" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 80, 1 ) / ( 80, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 80, 1 )

                  SIZE 336 (0.952:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 320 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

      }

   }

   GROUP "topology" {

      DATASET "cells" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 1136, 1 )

            SIZE 1476 (3.079:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 4544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

         ATTRIBUTE "cell_dim" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SCALAR

         }

      }

      DATASET "cones" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636, 1 )

            SIZE 155 (16.413:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 2544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

      DATASET "order" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636, 1 )

            SIZE 755 (3.370:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 2544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

      DATASET "orientation" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 1136, 1 )

            SIZE 216 (21.037:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 4544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "vertex_fields" {

      DATASET "solution error_potential" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 170 ) / ( 170 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 170 )

            SIZE 617 (2.204:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 2707 -924844032 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

         ATTRIBUTE "vector_field_type" {

            DATATYPE  H5T_STRING {

               STRSIZE 7;

               STRPAD H5T_STR_NULLTERM;

               CSET H5T_CSET_ASCII;

               CTYPE H5T_C_S1;

            }

            DATASPACE  SCALAR

         }

      }

   }

   GROUP "viz" {

      GROUP "topology" {

         DATASET "cells" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 130, 4 ) / ( 130, 4 ) }

            STORAGE_LAYOUT {

               CHUNKED ( 130, 4 )

               SIZE 592 (3.514:1 COMPRESSION)

            }

            FILTERS {

               USER_DEFINED_FILTER {

                  FILTER_ID 32001

                  COMMENT blosc

                  PARAMS { 2 2 4 2080 5 1 0 }

               }

            }

            FILLVALUE {

               FILL_TIME H5D_FILL_TIME_IFSET

               VALUE  H5D_FILL_VALUE_DEFAULT

            }

            ALLOCATION_TIME {

               H5D_ALLOC_TIME_EARLY

            }

            ATTRIBUTE "cell_corners" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

            ATTRIBUTE "cell_dim" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

         }

      }

   }

}

}

 

 

 

 

On Thu, Nov 9, 2017 at 3:27 PM, Michael K. Edwards

<[hidden email]> wrote:

It's exciting to be able to show the collective filtered IO feature as

part of a full software stack.  Thank you for your hard work on this,

and please let me know what more I can do to help keep it on glide

path for release.

 

 

On Thu, Nov 9, 2017 at 3:22 PM, Jordan Henderson

<[hidden email]> wrote:

Thanks! I'll discuss this with others and see what the best way to proceed

forward from this is. I think this has been a very productive discussion and

very useful feedback.

________________________________

From: Michael K. Edwards <[hidden email]>

Sent: Thursday, November 9, 2017 5:01:33 PM

To: Jordan Henderson

Cc: HDF Users Discussion List

Subject: Re: [Hdf-forum] Collective IO and filters

 

And here's the change to H5Z-blosc (still using the private H5MM APIs):

 

diff --git a/src/blosc_filter.c b/src/blosc_filter.c

index bfd8c3e..9bc1a42 100644

--- a/src/blosc_filter.c

+++ b/src/blosc_filter.c

@@ -16,6 +16,7 @@

  #include <string.h>

  #include <errno.h>

  #include "hdf5.h"

+#include "H5MMprivate.h"

  #include "blosc_filter.h"

 

  #if defined(__GNUC__)

@@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

    /* We're compressing */

    if (!(flags & H5Z_FLAG_REVERSE)) {

 

-    /* Allocate an output buffer exactly as long as the input data; if

-       the result is larger, we simply return 0.  The filter is flagged

-       as optional, so HDF5 marks the chunk as uncompressed and

-       proceeds.

+    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes

+       larger than the input data, to accommodate the BLOSC header.

+       If compression with the requested parameters causes the data itself

+       to grow (thereby causing the compressed data, with header, to exceed

+       the output buffer size), fall back to memcpy mode (clevel=0).

      */

 

-    outbuf_size = (*buf_size);

+    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;

 

  #ifdef BLOSC_DEBUG

      fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",

      nbytes, outbuf_size);

  #endif

 

-    outbuf = malloc(outbuf_size);

+    outbuf = H5MM_malloc(outbuf_size);

 

      if (outbuf == NULL) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK,

@@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

 

      blosc_set_compressor(compname);

      status = blosc_compress(clevel, doshuffle, typesize, nbytes,

-                            *buf, outbuf, nbytes);

+                            *buf, outbuf, outbuf_size);

+    if (status < 0) {

+      status = blosc_compress(0, doshuffle, typesize, nbytes,

+                              *buf, outbuf, outbuf_size);

+    }

      if (status < 0) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");

        goto failed;

@@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

      /* declare dummy variables */

      size_t cbytes, blocksize;

 

-    free(outbuf);

+    H5MM_xfree(outbuf);

 

      /* Extract the exact outbuf_size from the buffer header.

       *

@@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

      fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",

nbytes, outbuf_size);

  #endif

 

-    outbuf = malloc(outbuf_size);

+    if (outbuf_size == 0) {

+      H5MM_xfree(*buf);

+      *buf = NULL;

+      *buf_size = outbuf_size;

+      return 0;  /* Size of compressed/decompressed data */

+    }

+

+    outbuf = H5MM_malloc(outbuf_size);

 

      if (outbuf == NULL) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate

decompression buffer");

@@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

    } /* compressing vs decompressing */

 

    if (status != 0) {

-    free(*buf);

+    H5MM_xfree(*buf);

      *buf = outbuf;

      *buf_size = outbuf_size;

      return status;  /* Size of compressed/decompressed data */

    }

 

    failed:

-  free(outbuf);

+  H5MM_xfree(outbuf);

    return 0;

 

  } /* End filter function */

 

On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson

<[hidden email]> wrote:

As the filtered collective path simply calls through the filter pipeline

by

way of the H5Z_pipeline() function, it would seem that either the filter

pipeline itself is not handling this case correctly, or this is somewhat

unexpected behavior for the pipeline to deal with.

 

 

Either way, I think a pull request/diff file would be very useful for

going

over this. If you're able to generate a diff between what you have now and

the current develop branch/H5Z-blosc code and put it here that would be

useful. I don't think that there should be too much in the way of

logistics

for getting this code in, we just want to make sure that we approach the

solution in the right way without breaking something else.

 



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Collective IO and filters

Dana Robinson

FWIW, the H5MM asserts you were tripping over have already been removed from the develop branch.

 

Dana

 

From: Hdf-forum <[hidden email]> on behalf of "Michael K. Edwards" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>, HDF List <[hidden email]>
Date: Tuesday, November 14, 2017 at 15:40
To: "Miller, Mark C." <[hidden email]>
Cc: HDF List <[hidden email]>
Subject: Re: [Hdf-forum] Collective IO and filters

 

Yes, it's all 4-way parallel (at test scale), and should be exercisable at much larger scale with changes to command-line parameters.  The p4est/p8est code routinely generates Nx1 Vecs because it stores data in "space-filling curve" traversal order.  I should have some examples to post soon that uses DMDA instead, which should afford some scope for exercising 2-D/3-D modes of ZFP/fpzip.  I wanted to make sure we could handle the little corner-case-y structures before stressing it in the direction of scale.

 

On Nov 14, 2017 3:27 PM, "Miller, Mark C." <[hidden email]> wrote:

Hey Michael,

 

This is all in parallel? Thats awesome to get working!

 

Just a very brief look see on the h5dump output suggests the chunking is all Nx1.

 

Filters are applied on a per-chunk basis with a parameter set nonetheless that is fixed for all chunks (unless the filter is encoding params *into* the chunks themselves as opposed to dataset header messages). So, in the case of ZFP, with Nx1 chunking, its not really giving ZFP the greatest opportunity to compress because its essentially limiting its work to one dimension.

 

Next, the chunk size does seem to yield pretty small chunks.

 

Is this a constraint in the way parallel is being handled? Can you easily switch to larger chunks and chunks with non-unity in more dimensions?

 

Mark

 

 

"Hdf-forum on behalf of Michael K. Edwards" wrote:

 

It's good to have a reference example when testing an integration like

this.  I've attached the patch I've been using against the "maint"

(3.8.x) branch of PETSc.  It's obviously not suitable for integration

(it blindly applies ZFP to floating point Vecs and BloscLZ to integer

Vecs), but it does exercise the code paths in interesting ways.

 

Here's how I configure the HDF5 develop branch (for debug purposes):

 

./configure --prefix=/usr/local 'MAKE=/usr/bin/gmake' 'CC=mpicc'

'CFLAGS=-fPIC -fstack-protector -g3 -fopenmp' 'AR=/usr/bin/ar'

'ARFLAGS=cr' 'CXX=mpicxx' 'CXXFLAGS=-fstack-protector -g -fopenmp

-fPIC' 'F90=mpif90' 'F90FLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'F77=mpif90' 'FFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'FC=mpif90' 'FCFLAGS=-fPIC -ffree-line-length-0 -g -fopenmp'

'--enable-shared' '--with-default-api-version=v18' '--enable-parallel'

'--enable-fortran' 'F9X=mpif90' '--with-zlib=yes' '--with-szlib=yes'

 

And here's how I configure and run PETSc (again, for debug purposes):

 

./configure --without-x --with-openmp

--with-blaslapack-dir=/opt/intel/mkl --with-hdf5 --download-p4est=yes

--download-triangle=yes --download-pragmatic=yes --download-metis=yes

--download-eigen=yes

make PETSC_DIR=/home/centos/p4est/petsc/petsc PETSC_ARCH=arch-linux2-c-debug all

cd src/snes/examples/tutorials

make PETSC_DIR=/home/centos/p4est/petsc/petsc

PETSC_ARCH=arch-linux2-c-debug ex12

/usr/local/bin/mpiexec -n 4 ./ex12 -run_type full

-variable_coefficient nonlinear -nonzero_initial_guess 1 -interpolate

1 -petscspace_order 2 -snes_max_it 10 -snes_type fas

-snes_linesearch_type bt -snes_fas_levels 3 -fas_coarse_snes_type

newtonls -fas_coarse_snes_linesearch_type basic -fas_coarse_ksp_type

cg -fas_coarse_pc_type jacobi -fas_coarse_snes_monitor_short

-fas_levels_snes_max_it 4 -fas_levels_snes_type newtonls

-fas_levels_snes_linesearch_type bt -fas_levels_ksp_type cg

-fas_levels_pc_type jacobi -fas_levels_snes_monitor_short

-fas_levels_cycle_snes_linesearch_type bt -snes_monitor_short

-snes_converged_reason -snes_view -simplex 0 -petscspace_poly_tensor

-dm_plex_convert_type p4est -dm_forest_minimum_refinement 0

-dm_forest_initial_refinement 2 -dm_forest_maximum_refinement 4

-dm_p4est_refine_pattern hash -dm_view_hierarchy

 

Basically this is a smoke test for various shapes and sizes of object

that occur in an adaptive mesh refinement use case.  The

"-dm_view_hierarchy" flag is what triggers the write of three HDF5

files.  The typical structure looks like this:

 

[centos@centos74 tutorials]$ h5dump -pH ex12-2.h5

HDF5 "ex12-2.h5" {

GROUP "/" {

   GROUP "fields" {

      DATASET "solution error" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 636 ) / ( 636 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636 )

            SIZE 1566 (3.249:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 10163 -924844032 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "geometry" {

      DATASET "vertices" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 170, 2 ) / ( 170, 2 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 170, 2 )

            SIZE 3318 (0.820:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 -1879048169 -924844022 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "labels" {

      GROUP "Face Sets" {

         GROUP "1" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 20, 1 )

                  SIZE 96 (0.833:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 80 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "2" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 20, 1 ) / ( 20, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 20, 1 )

                  SIZE 96 (0.833:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 80 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "3" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 14, 1 ) / ( 14, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 14, 1 )

                  SIZE 72 (0.778:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 56 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

         GROUP "4" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 22, 1 ) / ( 22, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 22, 1 )

                  SIZE 104 (0.846:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 88 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

      }

      GROUP "marker" {

         GROUP "1" {

            DATASET "indices" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 80, 1 ) / ( 80, 1 ) }

               STORAGE_LAYOUT {

                  CHUNKED ( 80, 1 )

                  SIZE 336 (0.952:1 COMPRESSION)

               }

               FILTERS {

                  USER_DEFINED_FILTER {

                     FILTER_ID 32001

                     COMMENT blosc

                     PARAMS { 2 2 4 320 5 1 0 }

                  }

               }

               FILLVALUE {

                  FILL_TIME H5D_FILL_TIME_IFSET

                  VALUE  H5D_FILL_VALUE_DEFAULT

               }

               ALLOCATION_TIME {

                  H5D_ALLOC_TIME_EARLY

               }

            }

         }

      }

   }

   GROUP "topology" {

      DATASET "cells" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 1136, 1 )

            SIZE 1476 (3.079:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 4544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

         ATTRIBUTE "cell_dim" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SCALAR

         }

      }

      DATASET "cones" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636, 1 )

            SIZE 155 (16.413:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 2544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

      DATASET "order" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 636, 1 ) / ( 636, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 636, 1 )

            SIZE 755 (3.370:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 2544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

      DATASET "orientation" {

         DATATYPE  H5T_STD_I32LE

         DATASPACE  SIMPLE { ( 1136, 1 ) / ( 1136, 1 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 1136, 1 )

            SIZE 216 (21.037:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32001

               COMMENT blosc

               PARAMS { 2 2 4 4544 5 1 0 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

      }

   }

   GROUP "vertex_fields" {

      DATASET "solution error_potential" {

         DATATYPE  H5T_IEEE_F64LE

         DATASPACE  SIMPLE { ( 170 ) / ( 170 ) }

         STORAGE_LAYOUT {

            CHUNKED ( 170 )

            SIZE 617 (2.204:1 COMPRESSION)

         }

         FILTERS {

            USER_DEFINED_FILTER {

               FILTER_ID 32013

               COMMENT H5Z-ZFP-0.7.0 (ZFP-0.5.2) github.com/LLNL/H5Z-ZFP

               PARAMS { 5374064 91252346 2707 -924844032 }

            }

         }

         FILLVALUE {

            FILL_TIME H5D_FILL_TIME_IFSET

            VALUE  H5D_FILL_VALUE_DEFAULT

         }

         ALLOCATION_TIME {

            H5D_ALLOC_TIME_EARLY

         }

         ATTRIBUTE "vector_field_type" {

            DATATYPE  H5T_STRING {

               STRSIZE 7;

               STRPAD H5T_STR_NULLTERM;

               CSET H5T_CSET_ASCII;

               CTYPE H5T_C_S1;

            }

            DATASPACE  SCALAR

         }

      }

   }

   GROUP "viz" {

      GROUP "topology" {

         DATASET "cells" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 130, 4 ) / ( 130, 4 ) }

            STORAGE_LAYOUT {

               CHUNKED ( 130, 4 )

               SIZE 592 (3.514:1 COMPRESSION)

            }

            FILTERS {

               USER_DEFINED_FILTER {

                  FILTER_ID 32001

                  COMMENT blosc

                  PARAMS { 2 2 4 2080 5 1 0 }

               }

            }

            FILLVALUE {

               FILL_TIME H5D_FILL_TIME_IFSET

               VALUE  H5D_FILL_VALUE_DEFAULT

            }

            ALLOCATION_TIME {

               H5D_ALLOC_TIME_EARLY

            }

            ATTRIBUTE "cell_corners" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

            ATTRIBUTE "cell_dim" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SCALAR

            }

         }

      }

   }

}

}

 

 

 

 

On Thu, Nov 9, 2017 at 3:27 PM, Michael K. Edwards

<[hidden email]> wrote:

It's exciting to be able to show the collective filtered IO feature as

part of a full software stack.  Thank you for your hard work on this,

and please let me know what more I can do to help keep it on glide

path for release.

 

 

On Thu, Nov 9, 2017 at 3:22 PM, Jordan Henderson

<[hidden email]> wrote:

Thanks! I'll discuss this with others and see what the best way to proceed

forward from this is. I think this has been a very productive discussion and

very useful feedback.

________________________________

From: Michael K. Edwards <[hidden email]>

Sent: Thursday, November 9, 2017 5:01:33 PM

To: Jordan Henderson

Cc: HDF Users Discussion List

Subject: Re: [Hdf-forum] Collective IO and filters

 

And here's the change to H5Z-blosc (still using the private H5MM APIs):

 

diff --git a/src/blosc_filter.c b/src/blosc_filter.c

index bfd8c3e..9bc1a42 100644

--- a/src/blosc_filter.c

+++ b/src/blosc_filter.c

@@ -16,6 +16,7 @@

  #include <string.h>

  #include <errno.h>

  #include "hdf5.h"

+#include "H5MMprivate.h"

  #include "blosc_filter.h"

 

  #if defined(__GNUC__)

@@ -194,20 +195,21 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

    /* We're compressing */

    if (!(flags & H5Z_FLAG_REVERSE)) {

 

-    /* Allocate an output buffer exactly as long as the input data; if

-       the result is larger, we simply return 0.  The filter is flagged

-       as optional, so HDF5 marks the chunk as uncompressed and

-       proceeds.

+    /* Allocate an output buffer BLOSC_MAX_OVERHEAD (currently 16) bytes

+       larger than the input data, to accommodate the BLOSC header.

+       If compression with the requested parameters causes the data itself

+       to grow (thereby causing the compressed data, with header, to exceed

+       the output buffer size), fall back to memcpy mode (clevel=0).

      */

 

-    outbuf_size = (*buf_size);

+    outbuf_size = nbytes + BLOSC_MAX_OVERHEAD;

 

  #ifdef BLOSC_DEBUG

      fprintf(stderr, "Blosc: Compress %zd chunk w/buffer %zd\n",

      nbytes, outbuf_size);

  #endif

 

-    outbuf = malloc(outbuf_size);

+    outbuf = H5MM_malloc(outbuf_size);

 

      if (outbuf == NULL) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK,

@@ -217,7 +219,11 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

 

      blosc_set_compressor(compname);

      status = blosc_compress(clevel, doshuffle, typesize, nbytes,

-                            *buf, outbuf, nbytes);

+                            *buf, outbuf, outbuf_size);

+    if (status < 0) {

+      status = blosc_compress(0, doshuffle, typesize, nbytes,

+                              *buf, outbuf, outbuf_size);

+    }

      if (status < 0) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Blosc compression error");

        goto failed;

@@ -228,7 +234,7 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

      /* declare dummy variables */

      size_t cbytes, blocksize;

 

-    free(outbuf);

+    H5MM_xfree(outbuf);

 

      /* Extract the exact outbuf_size from the buffer header.

       *

@@ -243,7 +249,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

      fprintf(stderr, "Blosc: Decompress %zd chunk w/buffer %zd\n",

nbytes, outbuf_size);

  #endif

 

-    outbuf = malloc(outbuf_size);

+    if (outbuf_size == 0) {

+      H5MM_xfree(*buf);

+      *buf = NULL;

+      *buf_size = outbuf_size;

+      return 0;  /* Size of compressed/decompressed data */

+    }

+

+    outbuf = H5MM_malloc(outbuf_size);

 

      if (outbuf == NULL) {

        PUSH_ERR("blosc_filter", H5E_CALLBACK, "Can't allocate

decompression buffer");

@@ -259,14 +272,14 @@ size_t blosc_filter(unsigned flags, size_t cd_nelmts,

    } /* compressing vs decompressing */

 

    if (status != 0) {

-    free(*buf);

+    H5MM_xfree(*buf);

      *buf = outbuf;

      *buf_size = outbuf_size;

      return status;  /* Size of compressed/decompressed data */

    }

 

    failed:

-  free(outbuf);

+  H5MM_xfree(outbuf);

    return 0;

 

  } /* End filter function */

 

On Thu, Nov 9, 2017 at 2:45 PM, Jordan Henderson

<[hidden email]> wrote:

As the filtered collective path simply calls through the filter pipeline

by

way of the H5Z_pipeline() function, it would seem that either the filter

pipeline itself is not handling this case correctly, or this is somewhat

unexpected behavior for the pipeline to deal with.

 

 

Either way, I think a pull request/diff file would be very useful for

going

over this. If you're able to generate a diff between what you have now and

the current develop branch/H5Z-blosc code and put it here that would be

useful. I don't think that there should be too much in the way of

logistics

for getting this code in, we just want to make sure that we approach the

solution in the right way without breaking something else.

 

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
123