[Pytables-users] Writing to a dataset with 'wrong' chunksize

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Pytables-users] Writing to a dataset with 'wrong' chunksize

Quincey Koziol
Hi Francesc,

On Dec 3, 2007, at 11:21 AM, Francesc Altet wrote:

> A Monday 03 December 2007, Francesc Altet escrigu?:
>> Ups, I've ended with a similar program and send it to the
>> hdf-forum at hdfgroup.org list past Saturday.  I'm attaching my own
>> version (which is pretty similar to yours).  Sorry for not sending
>> you a copy of my previous message, because it could saved you some
>> work :-/
>
> Well, as Ivan pointed out, a couple of glitches slipped in my program.
> I'm attaching the correct version, but the result is the same, i.e.
> when N=600. I'm getting a segfault both under HDF5 1.6.5 and 1.8.0
> beta5.

        I was able to duplicate the segfault with your program, but it was a  
stack overflow and if you move the "data" array out of main() and  
make it a global variable, things run to completion without error.  
It's _really_ slow and chews _lots_ of memory still (because you are  
slicing the dataset the "wrong" way), but everything seems to be  
working correctly.

        It's somewhat hard to fix the "slicing the wrong way" problem,  
because the library is building a list of all the chunks that will be  
affected by each I/O operation (so that we can do all the I/O on each  
chunk at once) and that has some memory issues when dealing with I/O  
operations that affect so many chunks at once right now.  Building a  
list of all the affected chunks is good for the parallel I/O case,  
but could be avoided in the serial I/O case, I think.  However, that  
would probably make the code difficult to maintain...  :-/

        You could try adjusting the chunk cache size larger, which would  
probably help, if you make it large enough to hold all the chunks for  
the dataset.

        Quincey


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Pytables-users] Writing to a dataset with 'wrong' chunksize

Mike Folk
Fransesc et al:
Just to elaborate  a little bit on Quincey's
"slicing the wrong way" explanation.  (I hope I'm not just confusing matters.)

If possible you want to design the shape of the
chunk so that you get the most useful data with
the fewest number of accesses.  If accesses are
mostly contiguous elements along a certain
dimension, you shape the chunk to contain the
most elements along that dimension.  If accesses
are random shapes and sizes, then it gets a
little tricky -- we just generally recommend a
square (cube, etc.), but that may not be as good
as, say, a shape that has the same proportions as your dataset.

So, for instance if your dataset is 3,000x6,000
(3,000 rows, 6,000 columns) and you always access
a single column, then each chunk should contain
as much of a column as possible, given your best
chunk size.  If we assume a good chunk size is
600 elements, then your chunks would all be
600x1, and accessing any column in its entirety
would take 10 accesses.  Having each chunk be a
part of a row (1x600) would give you the worst
performance in this case, since you'd need to
access 6,000 chunks to access a column.

If accesses are unpredictable, perhaps a chunk
size of 30x60 would be best, as your worst case
performance (for reading a single column or row)
would take 100 accesses.  (By worst case, I'm
thinking of the case where you have to do the
most accesses per useful data element.)

In other cases, such as when you slice it one way
you don't care about performance, but when you
slice it another way you really do, would call
for a chunk to be shaped accordingly.

Mike


At 11:01 AM 12/4/2007, Quincey Koziol wrote:

>Hi Francesc,
>
>On Dec 3, 2007, at 11:21 AM, Francesc Altet wrote:
>
>>A Monday 03 December 2007, Francesc Altet escrigu?:
>>>Ups, I've ended with a similar program and send it to the
>>>hdf-forum at hdfgroup.org list past Saturday.  I'm attaching my own
>>>version (which is pretty similar to yours).  Sorry for not sending
>>>you a copy of my previous message, because it could saved you some
>>>work :-/
>>
>>Well, as Ivan pointed out, a couple of glitches slipped in my program.
>>I'm attaching the correct version, but the result is the same, i.e.
>>when N=600. I'm getting a segfault both under HDF5 1.6.5 and 1.8.0
>>beta5.
>
>         I was able to duplicate the segfault
> with your program, but it was a
>stack overflow and if you move the "data" array out of main() and
>make it a global variable, things run to completion without error.
>It's _really_ slow and chews _lots_ of memory still (because you are
>slicing the dataset the "wrong" way), but everything seems to be
>working correctly.
>
>         It's somewhat hard to fix the "slicing the wrong way" problem,
>because the library is building a list of all the chunks that will be
>affected by each I/O operation (so that we can do all the I/O on each
>chunk at once) and that has some memory issues when dealing with I/O
>operations that affect so many chunks at once right now.  Building a
>list of all the affected chunks is good for the parallel I/O case,
>but could be avoided in the serial I/O case, I think.  However, that
>would probably make the code difficult to maintain...  :-/
>
>         You could try adjusting the chunk cache size larger, which would
>probably help, if you make it large enough to hold all the chunks for
>the dataset.
>
>         Quincey
>
>
>----------------------------------------------------------------------
>This mailing list is for HDF software users discussion.
>To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
>To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>

--
Mike Folk   The HDF Group    http://hdfgroup.org     217.244.0647
1901 So. First St., Suite C-2, Champaign IL 61820




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Pytables-users] Writing to a dataset with 'wrong' chunksize

Francesc Altet
Hi Mike and others,

Sorry for the delay answering, but I was traveling past week.

Thanks for your explanation.  I understand what you both are saying, and
this reveals how important is choosing a correct chunkshape when you
want to get decent performance in HDF5 I/O.

PyTables initially tried to hide such 'low level' details to the user,
but after realising how important is this, we introduced
the 'chunkshape' parameter in dataset constructors in the 2.0 series.  
While PyTables still tries hard to avoid users to think about
chunkshape issues and automatically compute 'optimal' chunksizes, the
fact is that this only works well when the user wants to access their
data in the so-called 'C-order' (i.e. data is arranged in rows, not
columns).

However, users may have many valid reasons to choose another
arrangements than the C-order one.  So, in order to cope with this, I'm
afraid that the only solution will be to add a specific section in the
PyTables User's Guide in order to carefully explain this.  Your
explanations will definitely help to build a better guide on how to
choose the chunkshape that best fits the needs of the users.

Thanks!

A Thursday 06 December 2007, Mike Folk escrigu?:

> Fransesc et al:
> Just to elaborate  a little bit on Quincey's
> "slicing the wrong way" explanation.  (I hope I'm not just confusing
> matters.)
>
> If possible you want to design the shape of the
> chunk so that you get the most useful data with
> the fewest number of accesses.  If accesses are
> mostly contiguous elements along a certain
> dimension, you shape the chunk to contain the
> most elements along that dimension.  If accesses
> are random shapes and sizes, then it gets a
> little tricky -- we just generally recommend a
> square (cube, etc.), but that may not be as good
> as, say, a shape that has the same proportions as your dataset.
>
> So, for instance if your dataset is 3,000x6,000
> (3,000 rows, 6,000 columns) and you always access
> a single column, then each chunk should contain
> as much of a column as possible, given your best
> chunk size.  If we assume a good chunk size is
> 600 elements, then your chunks would all be
> 600x1, and accessing any column in its entirety
> would take 10 accesses.  Having each chunk be a
> part of a row (1x600) would give you the worst
> performance in this case, since you'd need to
> access 6,000 chunks to access a column.
>
> If accesses are unpredictable, perhaps a chunk
> size of 30x60 would be best, as your worst case
> performance (for reading a single column or row)
> would take 100 accesses.  (By worst case, I'm
> thinking of the case where you have to do the
> most accesses per useful data element.)
>
> In other cases, such as when you slice it one way
> you don't care about performance, but when you
> slice it another way you really do, would call
> for a chunk to be shaped accordingly.
>
> Mike
>
> At 11:01 AM 12/4/2007, Quincey Koziol wrote:
> >Hi Francesc,
> >
> >On Dec 3, 2007, at 11:21 AM, Francesc Altet wrote:
> >>A Monday 03 December 2007, Francesc Altet escrigu?:
> >>>Ups, I've ended with a similar program and send it to the
> >>>hdf-forum at hdfgroup.org list past Saturday.  I'm attaching my own
> >>>version (which is pretty similar to yours).  Sorry for not sending
> >>>you a copy of my previous message, because it could saved you some
> >>>work :-/
> >>
> >>Well, as Ivan pointed out, a couple of glitches slipped in my
> >> program. I'm attaching the correct version, but the result is the
> >> same, i.e. when N=600. I'm getting a segfault both under HDF5
> >> 1.6.5 and 1.8.0 beta5.
> >
> >         I was able to duplicate the segfault
> > with your program, but it was a
> >stack overflow and if you move the "data" array out of main() and
> >make it a global variable, things run to completion without error.
> >It's _really_ slow and chews _lots_ of memory still (because you are
> >slicing the dataset the "wrong" way), but everything seems to be
> >working correctly.
> >
> >         It's somewhat hard to fix the "slicing the wrong way"
> > problem, because the library is building a list of all the chunks
> > that will be affected by each I/O operation (so that we can do all
> > the I/O on each chunk at once) and that has some memory issues when
> > dealing with I/O operations that affect so many chunks at once
> > right now.  Building a list of all the affected chunks is good for
> > the parallel I/O case, but could be avoided in the serial I/O case,
> > I think.  However, that would probably make the code difficult to
> > maintain...  :-/
> >
> >         You could try adjusting the chunk cache size larger, which
> > would probably help, if you make it large enough to hold all the
> > chunks for the dataset.
> >
> >         Quincey
> >
> >
> >--------------------------------------------------------------------
> >-- This mailing list is for HDF software users discussion.
> >To subscribe to this list, send a message to
> > hdf-forum-subscribe at hdfgroup.org. To unsubscribe, send a message to
> > hdf-forum-unsubscribe at hdfgroup.org.
>
> --
> Mike Folk   The HDF Group    http://hdfgroup.org     217.244.0647
> 1901 So. First St., Suite C-2, Champaign IL 61820
>
>
> ---------------------------------------------------------------------
>- This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to
> hdf-forum-subscribe at hdfgroup.org. To unsubscribe, send a message to
> hdf-forum-unsubscribe at hdfgroup.org.



--
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Pytables-users] Writing to a dataset with 'wrong' chunksize

Dougherty, Matthew T.


This is particularly true in high resolution electron microscopy.
As the resolution improves sometimes it becomes necessary to change the order of axes to match xray datasets.
It would be preferable not to rewrite a very large dataset because you realize it was generated as a left handed coordinate system instead of right handed.
But the operative issue is the performance hit.


-----Original Message-----
From: Francesc Altet [mailto:faltet at carabos.com]
Sent: Wed 12/12/2007 4:13 AM
To: Mike Folk
Cc: hdf-forum at hdfgroup.org; Quincey Koziol; pytables-users at lists.sourceforge.net; Pauli Virtanen
Subject: Re: [Pytables-users] Writing to a dataset with 'wrong' chunksize

However, users may have many valid reasons to choose another
arrangements than the C-order one.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20071212/869c56d9/attachment.html>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Dougherty, Matthew T.
If I had a situation where I create a very large chunked dataset that had xyz ordering, and later find out that the preferred readout should be yzx;
then what would the best performance solution be:

1) recreate a new dataset with the yzx ordering
2) keep the old dataset and write software to reorganize the data
3) not worry about it because the performance hit is small because the data can be quickly reorganized as it is coming off storage into ram.
4) keep the old dataset, but coding changes deep in HDF will be needed to achieve good performance.

 
Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA
=========================================================================
=========================================================================




-----Original Message-----
From: Dougherty, Matthew T. [mailto:matthewd at bcm.edu]
Sent: Wed 12/12/2007 12:48 PM
To: Francesc Altet; Mike Folk
Cc: hdf-forum at hdfgroup.org; Quincey Koziol; pytables-users at lists.sourceforge.net; Pauli Virtanen
Subject: RE: [Pytables-users] Writing to a dataset with 'wrong' chunksize
 


This is particularly true in high resolution electron microscopy.
As the resolution improves sometimes it becomes necessary to change the order of axes to match xray datasets.
It would be preferable not to rewrite a very large dataset because you realize it was generated as a left handed coordinate system instead of right handed.
But the operative issue is the performance hit.


-----Original Message-----
From: Francesc Altet [mailto:faltet at carabos.com]
Sent: Wed 12/12/2007 4:13 AM
To: Mike Folk
Cc: hdf-forum at hdfgroup.org; Quincey Koziol; pytables-users at lists.sourceforge.net; Pauli Virtanen
Subject: Re: [Pytables-users] Writing to a dataset with 'wrong' chunksize

However, users may have many valid reasons to choose another
arrangements than the C-order one.  








-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20071220/2d75d223/attachment.html>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Francesc Altet
A Thursday 20 December 2007, Dougherty, Matthew T. escrigu?:

> If I had a situation where I create a very large chunked dataset that
> had xyz ordering, and later find out that the preferred readout
> should be yzx; then what would the best performance solution be:
>
> 1) recreate a new dataset with the yzx ordering
> 2) keep the old dataset and write software to reorganize the data
> 3) not worry about it because the performance hit is small because
> the data can be quickly reorganized as it is coming off storage into
> ram. 4) keep the old dataset, but coding changes deep in HDF will be
> needed to achieve good performance.

Well, I'd say that this largely depends on what kind of read performance
would you expect to obtain.  If, as you said, your dataset is very
large, then the chunks should be large as well, and reading a whole
chunk could be very costly in terms of disk I/O effort but also in
terms of CPU if your dataset is compressed.  So, if I were using
compression and wanted to obtain maximum performance, I'd choose option
1) with no discussion, as there is very little that the software can do
to avoid the necessary disk/CPU resources consumption that implies
an 'out-of-order' read.

This is in the case that you are using compression.  However,
experiments that I've carried some time ago tend to show that a chunk
has not necessarily to be read completely as long it is not compressed,
i.e. partial chunk reads are possible (Quincey, correct me if I'm
wrong).  In that case, perhaps options 3) and 4) could be feasible too.

Finally, I don't understand well the difference between 1) and 2), but
if the only difference is that 2) implies having two different datasets
on disk simultaneously, then 1) is clearly better, IMO.

My 2 cents,

--
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Quincey Koziol

On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:

> A Thursday 20 December 2007, Dougherty, Matthew T. escrigu?:
>> If I had a situation where I create a very large chunked dataset that
>> had xyz ordering, and later find out that the preferred readout
>> should be yzx; then what would the best performance solution be:
>>
>> 1) recreate a new dataset with the yzx ordering
>> 2) keep the old dataset and write software to reorganize the data
>> 3) not worry about it because the performance hit is small because
>> the data can be quickly reorganized as it is coming off storage into
>> ram. 4) keep the old dataset, but coding changes deep in HDF will be
>> needed to achieve good performance.
>
> Well, I'd say that this largely depends on what kind of read  
> performance
> would you expect to obtain.  If, as you said, your dataset is very
> large, then the chunks should be large as well, and reading a whole
> chunk could be very costly in terms of disk I/O effort but also in
> terms of CPU if your dataset is compressed.  So, if I were using
> compression and wanted to obtain maximum performance, I'd choose  
> option
> 1) with no discussion, as there is very little that the software  
> can do
> to avoid the necessary disk/CPU resources consumption that implies
> an 'out-of-order' read.
>
> This is in the case that you are using compression.  However,
> experiments that I've carried some time ago tend to show that a chunk
> has not necessarily to be read completely as long it is not  
> compressed,
> i.e. partial chunk reads are possible (Quincey, correct me if I'm
> wrong).  In that case, perhaps options 3) and 4) could be feasible  
> too.
>
> Finally, I don't understand well the difference between 1) and 2), but
> if the only difference is that 2) implies having two different  
> datasets
> on disk simultaneously, then 1) is clearly better, IMO.

        Frankly, it could be any of the options above, depending on your  
performance requirements, etc.  HDF5 currently (in the 1.6.x & 1.8.x  
branches) examines the file and memory dataspaces for an I/O  
operation to build up information about the portion of each chunk  
needed o perform the I/O access and then iterates through the  
affected chunks, transferring data between memory and disk as necessary.

        Obviously, if your I/O access "goes against the grain" and requires  
access to many chunks, it will require more memory to store the  
information about the chunks affected, as well as more I/O operations  
to access the data in the file.  If the chunk cache is large enough  
[possibly through the application tweaking it] and most/all of the  
chunk data can be held in memory, much of I/O access overhead will be  
amortized across multiple HDF5 dataset access operations and the  
against the grain access won't be so painful.

        Does that help?

                Quincey


> My 2 cents,
>
> --
>> 0,0<   Francesc Altet     http://www.carabos.com/
> V   V   C?rabos Coop. V.   Enjoy Data
>  "-"
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-
> subscribe at hdfgroup.org.
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Francesc Altet
A Thursday 20 December 2007, Quincey Koziol escrigu?:

> On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:
> > A Thursday 20 December 2007, Dougherty, Matthew T. escrigu?:
> >> If I had a situation where I create a very large chunked dataset
> >> that had xyz ordering, and later find out that the preferred
> >> readout should be yzx; then what would the best performance
> >> solution be:
> >>
> >> 1) recreate a new dataset with the yzx ordering
> >> 2) keep the old dataset and write software to reorganize the data
> >> 3) not worry about it because the performance hit is small because
> >> the data can be quickly reorganized as it is coming off storage
> >> into ram. 4) keep the old dataset, but coding changes deep in HDF
> >> will be needed to achieve good performance.
> >
> > Well, I'd say that this largely depends on what kind of read
> > performance
> > would you expect to obtain.  If, as you said, your dataset is very
> > large, then the chunks should be large as well, and reading a whole
> > chunk could be very costly in terms of disk I/O effort but also in
> > terms of CPU if your dataset is compressed.  So, if I were using
> > compression and wanted to obtain maximum performance, I'd choose
> > option
> > 1) with no discussion, as there is very little that the software
> > can do
> > to avoid the necessary disk/CPU resources consumption that implies
> > an 'out-of-order' read.
> >
> > This is in the case that you are using compression.  However,
> > experiments that I've carried some time ago tend to show that a
> > chunk has not necessarily to be read completely as long it is not
> > compressed,
> > i.e. partial chunk reads are possible (Quincey, correct me if I'm
> > wrong).  In that case, perhaps options 3) and 4) could be feasible
> > too.
> >
> > Finally, I don't understand well the difference between 1) and 2),
> > but if the only difference is that 2) implies having two different
> > datasets
> > on disk simultaneously, then 1) is clearly better, IMO.
>
> Frankly, it could be any of the options above, depending on your
> performance requirements, etc.  HDF5 currently (in the 1.6.x & 1.8.x
> branches) examines the file and memory dataspaces for an I/O
> operation to build up information about the portion of each chunk
> needed o perform the I/O access and then iterates through the
> affected chunks, transferring data between memory and disk as
> necessary.
>
> Obviously, if your I/O access "goes against the grain" and requires
> access to many chunks, it will require more memory to store the
> information about the chunks affected, as well as more I/O operations
> to access the data in the file.  If the chunk cache is large enough
> [possibly through the application tweaking it] and most/all of the
> chunk data can be held in memory, much of I/O access overhead will be
> amortized across multiple HDF5 dataset access operations and the
> against the grain access won't be so painful.

Yeah, I forgot about the fact that the chunk cache size is configurable
in forthcoming HDF5 1.8.0.  That's a very interesting option indeed!  
Do you know if there is any important drawback of pushing that cache
towards very large values (e.g. 1 GB)?

Cheers,

--
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Quincey Koziol
Hi Francesc,

On Dec 20, 2007, at 10:04 AM, Francesc Altet wrote:

> A Thursday 20 December 2007, Quincey Koziol escrigu?:
>> On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:
>>> A Thursday 20 December 2007, Dougherty, Matthew T. escrigu?:
>>>> If I had a situation where I create a very large chunked dataset
>>>> that had xyz ordering, and later find out that the preferred
>>>> readout should be yzx; then what would the best performance
>>>> solution be:
>>>>
>>>> 1) recreate a new dataset with the yzx ordering
>>>> 2) keep the old dataset and write software to reorganize the data
>>>> 3) not worry about it because the performance hit is small because
>>>> the data can be quickly reorganized as it is coming off storage
>>>> into ram. 4) keep the old dataset, but coding changes deep in HDF
>>>> will be needed to achieve good performance.
>>>
>>> Well, I'd say that this largely depends on what kind of read
>>> performance
>>> would you expect to obtain.  If, as you said, your dataset is very
>>> large, then the chunks should be large as well, and reading a whole
>>> chunk could be very costly in terms of disk I/O effort but also in
>>> terms of CPU if your dataset is compressed.  So, if I were using
>>> compression and wanted to obtain maximum performance, I'd choose
>>> option
>>> 1) with no discussion, as there is very little that the software
>>> can do
>>> to avoid the necessary disk/CPU resources consumption that implies
>>> an 'out-of-order' read.
>>>
>>> This is in the case that you are using compression.  However,
>>> experiments that I've carried some time ago tend to show that a
>>> chunk has not necessarily to be read completely as long it is not
>>> compressed,
>>> i.e. partial chunk reads are possible (Quincey, correct me if I'm
>>> wrong).  In that case, perhaps options 3) and 4) could be feasible
>>> too.
>>>
>>> Finally, I don't understand well the difference between 1) and 2),
>>> but if the only difference is that 2) implies having two different
>>> datasets
>>> on disk simultaneously, then 1) is clearly better, IMO.
>>
>> Frankly, it could be any of the options above, depending on your
>> performance requirements, etc.  HDF5 currently (in the 1.6.x & 1.8.x
>> branches) examines the file and memory dataspaces for an I/O
>> operation to build up information about the portion of each chunk
>> needed o perform the I/O access and then iterates through the
>> affected chunks, transferring data between memory and disk as
>> necessary.
>>
>> Obviously, if your I/O access "goes against the grain" and requires
>> access to many chunks, it will require more memory to store the
>> information about the chunks affected, as well as more I/O operations
>> to access the data in the file.  If the chunk cache is large enough
>> [possibly through the application tweaking it] and most/all of the
>> chunk data can be held in memory, much of I/O access overhead will be
>> amortized across multiple HDF5 dataset access operations and the
>> against the grain access won't be so painful.
>
> Yeah, I forgot about the fact that the chunk cache size is  
> configurable
> in forthcoming HDF5 1.8.0.  That's a very interesting option indeed!
> Do you know if there is any important drawback of pushing that cache
> towards very large values (e.g. 1 GB)?

        As long as the application has the memory, making the chunk cache  
larger doesn't have any real downside (from the HDF5 library's  
perspective).

        Quincey


> Cheers,
>
> --
>> 0,0<   Francesc Altet     http://www.carabos.com/
> V   V   C?rabos Coop. V.   Enjoy Data
>  "-"
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-
> subscribe at hdfgroup.org.
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Francesc Altet
A Thursday 20 December 2007, escrigu?reu:

> Hi Francesc,
>
> On Dec 20, 2007, at 10:04 AM, Francesc Altet wrote:
> > A Thursday 20 December 2007, Quincey Koziol escrigu?:
> >> On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:
> >>> A Thursday 20 December 2007, Dougherty, Matthew T. escrigu?:
> >>>> If I had a situation where I create a very large chunked dataset
> >>>> that had xyz ordering, and later find out that the preferred
> >>>> readout should be yzx; then what would the best performance
> >>>> solution be:
> >>>>
> >>>> 1) recreate a new dataset with the yzx ordering
> >>>> 2) keep the old dataset and write software to reorganize the
> >>>> data 3) not worry about it because the performance hit is small
> >>>> because the data can be quickly reorganized as it is coming off
> >>>> storage into ram. 4) keep the old dataset, but coding changes
> >>>> deep in HDF will be needed to achieve good performance.
> >>>
> >>> Well, I'd say that this largely depends on what kind of read
> >>> performance
> >>> would you expect to obtain.  If, as you said, your dataset is
> >>> very large, then the chunks should be large as well, and reading
> >>> a whole chunk could be very costly in terms of disk I/O effort
> >>> but also in terms of CPU if your dataset is compressed.  So, if I
> >>> were using compression and wanted to obtain maximum performance,
> >>> I'd choose option
> >>> 1) with no discussion, as there is very little that the software
> >>> can do
> >>> to avoid the necessary disk/CPU resources consumption that
> >>> implies an 'out-of-order' read.
> >>>
> >>> This is in the case that you are using compression.  However,
> >>> experiments that I've carried some time ago tend to show that a
> >>> chunk has not necessarily to be read completely as long it is not
> >>> compressed,
> >>> i.e. partial chunk reads are possible (Quincey, correct me if I'm
> >>> wrong).  In that case, perhaps options 3) and 4) could be
> >>> feasible too.
> >>>
> >>> Finally, I don't understand well the difference between 1) and
> >>> 2), but if the only difference is that 2) implies having two
> >>> different datasets
> >>> on disk simultaneously, then 1) is clearly better, IMO.
> >>
> >> Frankly, it could be any of the options above, depending on your
> >> performance requirements, etc.  HDF5 currently (in the 1.6.x &
> >> 1.8.x branches) examines the file and memory dataspaces for an I/O
> >> operation to build up information about the portion of each chunk
> >> needed o perform the I/O access and then iterates through the
> >> affected chunks, transferring data between memory and disk as
> >> necessary.
> >>
> >> Obviously, if your I/O access "goes against the grain" and
> >> requires access to many chunks, it will require more memory to
> >> store the information about the chunks affected, as well as more
> >> I/O operations to access the data in the file.  If the chunk cache
> >> is large enough [possibly through the application tweaking it] and
> >> most/all of the chunk data can be held in memory, much of I/O
> >> access overhead will be amortized across multiple HDF5 dataset
> >> access operations and the against the grain access won't be so
> >> painful.
> >
> > Yeah, I forgot about the fact that the chunk cache size is
> > configurable
> > in forthcoming HDF5 1.8.0.  That's a very interesting option
> > indeed! Do you know if there is any important drawback of pushing
> > that cache towards very large values (e.g. 1 GB)?
>
> As long as the application has the memory, making the chunk cache
> larger doesn't have any real downside (from the HDF5 library's
> perspective).

Well, it is clear that HDF5 1.8 series are bringing many exciting new
features :)

Cheers,

--
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Dougherty, Matthew T.
In reply to this post by Francesc Altet
#2: you are rewriting over the same dataset, one chunk at a time.
read the chunk, reorder, write the chunk back into a different chunk location.
the chunk sizes I would anticipate are 64^3 or 128^3.  So you would have to keep two chunks in ram;
one chunk that is being reordered, and the chunk that will be overwritten in storage.

if compression is used, then this complicates the matter.
it may be possible to not have two chunks.  read a chunk, reorder, write back to same chunk; modify the pointer to chunks instead.

 
Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA
=========================================================================
=========================================================================


Finally, I don't understand well the difference between 1) and 2), but
if the only difference is that 2) implies having two different datasets
on disk simultaneously, then 1) is clearly better, IMO.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20071220/228679d6/attachment.html>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Dougherty, Matthew T.
In reply to this post by Francesc Altet


not clear on this

1)  to build up information about the portion.
what information is built?

2) how does chunk cache work?  if I have a 1TB 3D image, chunks are 64^3, and I am looking at a ROI 512^3.
the ROI shifts interactively, so chunks are being accessed from storage and others being discarded from memory.
We skip the reordering problem for simplicity.  Does chunk cache contain references to all the possible chunks?
Like a mini-3D dataset with pointers to all chunk storage locations?


Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA
=========================================================================
=========================================================================

 HDF5 currently (in the 1.6.x & 1.8.x  
branches) examines the file and memory dataspaces for an I/O  
operation to build up information about the portion of each chunk  
needed o perform the I/O access

        Obviously, if your I/O access "goes against the grain" and requires  
access to many chunks, it will require more memory to store the  
information about the chunks affected, as well as more I/O operations  
to access the data in the file.  If the chunk cache is large enough  
[possibly through the application tweaking it] and most/all of the  
chunk data can be held in memory, much of I/O access overhead will be  
amortized across multiple HDF5 dataset access operations and the  
against the grain access won't be so painful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20071220/58967b07/attachment.html>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Quincey Koziol
Hi Matthew,

On Dec 20, 2007, at 12:43 PM, Dougherty, Matthew T. wrote:

>
>
> not clear on this
>
> 1)  to build up information about the portion.
> what information is built?
>

        Basically an internal representation of the elements to access on  
disk and in memory for each chunk.

> 2) how does chunk cache work?  if I have a 1TB 3D image, chunks are  
> 64^3, and I am looking at a ROI 512^3.
> the ROI shifts interactively, so chunks are being accessed from  
> storage and others being discarded from memory.
> We skip the reordering problem for simplicity.  Does chunk cache  
> contain references to all the possible chunks?
> Like a mini-3D dataset with pointers to all chunk storage locations?
>

        The chunk cache in the HDF5 library holds the actual "raw data"  
elements from the chunked dataset (i.e the ROI elements).  The chunks  
are located through another data structure that indexes them, but  
that information is put into the "metadata cache" in the HDF5  
library.  In the 1.8.x library, each chunked dataset has it's own  
chunk cache, but the metadata cache is global to all the metadata in  
the file.

        Quincey

> Matthew Dougherty
> 713-433-3849
> National Center for Macromolecular Imaging
> Baylor College of Medicine/Houston Texas USA
> ======================================================================
> ===
> ======================================================================
> ===
>
>  HDF5 currently (in the 1.6.x & 1.8.x
> branches) examines the file and memory dataspaces for an I/O
> operation to build up information about the portion of each chunk
> needed o perform the I/O access
>
>         Obviously, if your I/O access "goes against the grain" and  
> requires
> access to many chunks, it will require more memory to store the
> information about the chunks affected, as well as more I/O operations
> to access the data in the file.  If the chunk cache is large enough
> [possibly through the application tweaking it] and most/all of the
> chunk data can be held in memory, much of I/O access overhead will be
> amortized across multiple HDF5 dataset access operations and the
> against the grain access won't be so painful.


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Dougherty, Matthew T.


 
Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA
=========================================================================
=========================================================================

back to the reordering question, just one dataset per hdf file for simplicity.

This relates to option #4.

If metadata cache contains all the references to the chunks in storage;
then how difficult would it be to reorder the references to chunks?

what I am looking for is an api that allows me to arbitrarily reorder the axes.
I believe there is an api that does fortran ordering(default) and another for C ordering which is reverse on the axis sequence.

seems like this would be the area of code that could be modified.



-----Original Message-----
From: Quincey Koziol [mailto:koziol at hdfgroup.org]
Sent: Thu 12/20/2007 12:56 PM
To: Dougherty, Matthew T.
Cc: hdf-forum at hdfgroup.org
Subject: Re: hypthetical question on reordering axes
 

        Basically an internal representation of the elements to access on  
disk and in memory for each chunk.

        The chunk cache in the HDF5 library holds the actual "raw data"  
elements from the chunked dataset (i.e the ROI elements).  The chunks  
are located through another data structure that indexes them, but  
that information is put into the "metadata cache" in the HDF5  
library.  In the 1.8.x library, each chunked dataset has it's own  
chunk cache, but the metadata cache is global to all the metadata in  
the file.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20071220/c813d7a2/attachment.html>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hypthetical question on reordering axes

Quincey Koziol
Hi Matthew,

On Dec 20, 2007, at 1:11 PM, Dougherty, Matthew T. wrote:

> Matthew Dougherty
> 713-433-3849
> National Center for Macromolecular Imaging
> Baylor College of Medicine/Houston Texas USA
> ======================================================================
> ===
> ======================================================================
> ===
>
> back to the reordering question, just one dataset per hdf file for  
> simplicity.
>
> This relates to option #4.
>
> If metadata cache contains all the references to the chunks in  
> storage;
> then how difficult would it be to reorder the references to chunks?
>

        This really isn't going to help, since all the data in the chunks  
would have to be reordered as well.

> what I am looking for is an api that allows me to arbitrarily  
> reorder the axes.
> I believe there is an api that does fortran ordering(default) and  
> another for C ordering which is reverse on the axis sequence.
>

        We don't currently have an API for this.  We have designed the  
dataspace description to hold a "permutation vector" which could  
describe this, but it's currently not implemented.

        Quincey

> seems like this would be the area of code that could be modified.
>
>
>
> -----Original Message-----
> From: Quincey Koziol [mailto:koziol at hdfgroup.org]
> Sent: Thu 12/20/2007 12:56 PM
> To: Dougherty, Matthew T.
> Cc: hdf-forum at hdfgroup.org
> Subject: Re: hypthetical question on reordering axes
>
>
>         Basically an internal representation of the elements to  
> access on
> disk and in memory for each chunk.
>
>         The chunk cache in the HDF5 library holds the actual "raw  
> data"
> elements from the chunked dataset (i.e the ROI elements).  The chunks
> are located through another data structure that indexes them, but
> that information is put into the "metadata cache" in the HDF5
> library.  In the 1.8.x library, each chunked dataset has it's own
> chunk cache, but the metadata cache is global to all the metadata in
> the file.
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Loading...