Quantcast

Transposing a 2d matrix via hyperslab

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Transposing a 2d matrix via hyperslab

Isaac Gerg
A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

Thanks,
Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Quincey Koziol-3

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Isaac Gerg
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Werner Benger

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner


On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Isaac Gerg
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:(225)%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:(225)%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Werner Benger

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner


On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Isaac Gerg
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:(225)%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:(225)%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Werner Benger

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.


On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Isaac Gerg
I think of hyperslab as a sort of fancy click+drag selection tool of a tensor (fancy in the sense that you can do strided blocks of selection).   The original problem stemmed from a desire to do matlab shiftdim-like or numpy rollaxis-like operation.  i.e. I have axes in memory of Z,X,Y because of processing efficiency, but wish to store the data X,Y,Z for visualization efficiency.  Its really not a big deal to simply do a memcopy but why waste the memory if I don't need to; this gives us a chance to really do a deep dive on hyperslab to see if it can do the job.

In trying to get this to work with hyperslabbing, this led me to the source for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis under the good).  The solution is rather elegant for them.  Every array in np stores a stride tuple which are the strides over dimension.  So to do a transpose they simply permute this tuple.  Perhaps having a similar interface for HDF is something to be considered?  On the other hand, I could see providing another interface to mangle data as "giving yourself more rope to trip over" so to speak.

In any case, I figured I'd share where the problem came from and in digging around how others (i.e numpy) handle it given their use cases.

Isaac

On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <[hidden email]> wrote:

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.


On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:(225)%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:(225)%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Werner Benger

Hi Isaac,

 just out of curiosity, which operation do you consider where something like Z,X,Y order is so much more efficient to justify such a data transformation at I/O?

        Werner


On 26.04.2017 13:58, Isaac Gerg wrote:
I think of hyperslab as a sort of fancy click+drag selection tool of a tensor (fancy in the sense that you can do strided blocks of selection).   The original problem stemmed from a desire to do matlab shiftdim-like or numpy rollaxis-like operation.  i.e. I have axes in memory of Z,X,Y because of processing efficiency, but wish to store the data X,Y,Z for visualization efficiency.  Its really not a big deal to simply do a memcopy but why waste the memory if I don't need to; this gives us a chance to really do a deep dive on hyperslab to see if it can do the job.

In trying to get this to work with hyperslabbing, this led me to the source for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis under the good).  The solution is rather elegant for them.  Every array in np stores a stride tuple which are the strides over dimension.  So to do a transpose they simply permute this tuple.  Perhaps having a similar interface for HDF is something to be considered?  On the other hand, I could see providing another interface to mangle data as "giving yourself more rope to trip over" so to speak.

In any case, I figured I'd share where the problem came from and in digging around how others (i.e numpy) handle it given their use cases.

Isaac

On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <[hidden email]> wrote:

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.


On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Isaac Gerg
Hi Werner,

I am doing remote sensing into the seafloor with refraction being modeled.  Our beamformer creates imagery via layers and its a useful way to deal with the refraction.  But when you go to display the data, its useful to look at the side projection (where the vertical axis is depth and the horizontal is along track or range).

Isaac

On Wed, Apr 26, 2017 at 3:14 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 just out of curiosity, which operation do you consider where something like Z,X,Y order is so much more efficient to justify such a data transformation at I/O?

        Werner


On 26.04.2017 13:58, Isaac Gerg wrote:
I think of hyperslab as a sort of fancy click+drag selection tool of a tensor (fancy in the sense that you can do strided blocks of selection).   The original problem stemmed from a desire to do matlab shiftdim-like or numpy rollaxis-like operation.  i.e. I have axes in memory of Z,X,Y because of processing efficiency, but wish to store the data X,Y,Z for visualization efficiency.  Its really not a big deal to simply do a memcopy but why waste the memory if I don't need to; this gives us a chance to really do a deep dive on hyperslab to see if it can do the job.

In trying to get this to work with hyperslabbing, this led me to the source for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis under the good).  The solution is rather elegant for them.  Every array in np stores a stride tuple which are the strides over dimension.  So to do a transpose they simply permute this tuple.  Perhaps having a similar interface for HDF is something to be considered?  On the other hand, I could see providing another interface to mangle data as "giving yourself more rope to trip over" so to speak.

In any case, I figured I'd share where the problem came from and in digging around how others (i.e numpy) handle it given their use cases.

Isaac

On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <[hidden email]> wrote:

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.


On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:(225)%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:(225)%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Quincey Koziol-3
In reply to this post by Werner Benger
Hi Werner,

On Apr 26, 2017, at 12:34 AM, Werner Benger <[hidden email]> wrote:

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.


I like the idea quite a lot. :-)  However, it might be quite difficult to implement. :-(  The hyperslabs are “compiled” into [complex] internal data structures as they are built up, in order to make the I/O operations as fast as possible.  It would be possible to defer that compilation until when the actual I/O operation occurs, but that would be another layer of data structures, etc.  I believe it would be worthwhile, particularly so that transposes are possible, but it would definitely be a fair bit of work…

Quincey


On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank" class="">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank" class="">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a moz-do-not-send="true" href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank" class="">+1 225 578 4809                        Fax.: <a moz-do-not-send="true" href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank" class="">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362 
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transposing a 2d matrix via hyperslab

Isaac Gerg
*I* have to chime in and say that I'm happy with the way HDF works now.  Please don't take our single usecase as a need to spend a lot of work on this.  I expected a simple yes or no to my question!  Im totally fine with HDF not being the end all of every usecase.

On Thu, Apr 27, 2017 at 7:13 PM, Quincey Koziol <[hidden email]> wrote:
Hi Werner,

On Apr 26, 2017, at 12:34 AM, Werner Benger <[hidden email]> wrote:

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.


I like the idea quite a lot. :-)  However, it might be quite difficult to implement. :-(  The hyperslabs are “compiled” into [complex] internal data structures as they are built up, in order to make the I/O operations as fast as possible.  It would be possible to defer that compilation until when the actual I/O operation occurs, but that would be another layer of data structures, etc.  I believe it would be worthwhile, particularly so that transposes are possible, but it would definitely be a fair bit of work…

Quincey


On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation:  start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner



On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner,  thanks for the reply.  Let's wrestle with this a bit.  So numpy does their indexing by strides, the stride for each dimension.  hdf5 does their "striding" across the data in the sense of more of a selection than a stride.  I would be inclined to say that even a negative stride wouldnt allow this.  The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[hidden email]> wrote:

Hi Isaac,

 without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner



On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go.  Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[hidden email]> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[hidden email]> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab.   It seems that the grammer provided by start, stride, blocks, and count won't allow for this.  We hope we are wrong!  It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%28225%29%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%28225%29%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:(225)%20578-4809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:(225)%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362 
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...