[hdf-forum] reading in data trasposed

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Brock Palen
I have some data, that was created by a C program and thus all data  
is in rowMajor format.
We would like to read this into another C program that would then  
call LAPACK.  Because lapack for C is really a shim to fortran (which  
expects ColumnMajor) we would like to read the data in transposed.

Is there a simple way to do this?  Any tricks?  We have some Macros  
that let us access malloc()'ed memory in fortran style, but I can for  
the life of me figure out any way to use them with H5Dread() .

Any comments?


Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985




----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Dominik Szczerba-2
Welcome to the club!
There is no easy way because HDF5 does not care about this issue, which I very
much regret.
I resolve it by always adding an attribute to any arrays rank>1 how it is
stored. Then it is the responsibility of the client code to read it in
correctly.
And transposing in-place is not difficult, only will not allow a const
pointer, which is annoying.
regards,
- Dominik

On Wednesday 17 September 2008 09:35:40 pm Brock Palen wrote:

> I have some data, that was created by a C program and thus all data
> is in rowMajor format.
> We would like to read this into another C program that would then
> call LAPACK.  Because lapack for C is really a shim to fortran (which
> expects ColumnMajor) we would like to read the data in transposed.
>
> Is there a simple way to do this?  Any tricks?  We have some Macros
> that let us access malloc()'ed memory in fortran style, but I can for
> the life of me figure out any way to use them with H5Dread() .
>
> Any comments?
>
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to
> hdf-forum-subscribe at hdfgroup.org. To unsubscribe, send a message to
> hdf-forum-unsubscribe at hdfgroup.org.



--
Dominik Szczerba, Ph.D.
Biomedical Simulation Group
Computer Vision Lab CH-8092 Zurich
http://www.vision.ee.ethz.ch/~domi

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Brock Palen
Yes it's not that hard. But as a general rule, anything I write to do  
this in memory would not be as fast as a BLAS1 operation even.  For  
large matrices this has to be a memory bound operation.

Is there a standard way to do in place transpose that is high  
performance? Is there a way to do it with a blas2 call?

I was hoping that I could do it while reading from disk, because disk  
would be slower than memory anyway thus rearranging in memory while  
reading from disk would not be a performance hit.

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985



On Sep 17, 2008, at 3:54 PM, Dominik Szczerba wrote:

> Welcome to the club!
> There is no easy way because HDF5 does not care about this issue,  
> which I very
> much regret.
> I resolve it by always adding an attribute to any arrays rank>1 how  
> it is
> stored. Then it is the responsibility of the client code to read it in
> correctly.
> And transposing in-place is not difficult, only will not allow a const
> pointer, which is annoying.
> regards,
> - Dominik
>
> On Wednesday 17 September 2008 09:35:40 pm Brock Palen wrote:
>> I have some data, that was created by a C program and thus all data
>> is in rowMajor format.
>> We would like to read this into another C program that would then
>> call LAPACK.  Because lapack for C is really a shim to fortran (which
>> expects ColumnMajor) we would like to read the data in transposed.
>>
>> Is there a simple way to do this?  Any tricks?  We have some Macros
>> that let us access malloc()'ed memory in fortran style, but I can for
>> the life of me figure out any way to use them with H5Dread() .
>>
>> Any comments?
>>
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> -
>> This mailing list is for HDF software users discussion.
>> To subscribe to this list, send a message to
>> hdf-forum-subscribe at hdfgroup.org. To unsubscribe, send a message to
>> hdf-forum-unsubscribe at hdfgroup.org.
>
>
>
> --
> Dominik Szczerba, Ph.D.
> Biomedical Simulation Group
> Computer Vision Lab CH-8092 Zurich
> http://www.vision.ee.ethz.ch/~domi
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-
> subscribe at hdfgroup.org.
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Quincey Koziol
In reply to this post by Dominik Szczerba-2
Hi Dominik,

On Sep 17, 2008, at 2:54 PM, Dominik Szczerba wrote:

> Welcome to the club!
> There is no easy way because HDF5 does not care about this issue,  
> which I very
> much regret.

        Well, actually we do care about implementing this feature in HDF5,  
but it has never been a high enough priority by someone with money to  
get funded.  :-(

        Quincey

> I resolve it by always adding an attribute to any arrays rank>1 how  
> it is
> stored. Then it is the responsibility of the client code to read it in
> correctly.
> And transposing in-place is not difficult, only will not allow a const
> pointer, which is annoying.
> regards,
> - Dominik
>
> On Wednesday 17 September 2008 09:35:40 pm Brock Palen wrote:
>> I have some data, that was created by a C program and thus all data
>> is in rowMajor format.
>> We would like to read this into another C program that would then
>> call LAPACK.  Because lapack for C is really a shim to fortran (which
>> expects ColumnMajor) we would like to read the data in transposed.
>>
>> Is there a simple way to do this?  Any tricks?  We have some Macros
>> that let us access malloc()'ed memory in fortran style, but I can for
>> the life of me figure out any way to use them with H5Dread() .
>>
>> Any comments?
>>
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>>
>> ----------------------------------------------------------------------
>> This mailing list is for HDF software users discussion.
>> To subscribe to this list, send a message to
>> hdf-forum-subscribe at hdfgroup.org. To unsubscribe, send a message to
>> hdf-forum-unsubscribe at hdfgroup.org.
>
>
>
> --
> Dominik Szczerba, Ph.D.
> Biomedical Simulation Group
> Computer Vision Lab CH-8092 Zurich
> http://www.vision.ee.ethz.ch/~domi
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Andrew Cunningham-2
In reply to this post by Brock Palen
Just read your data "as is" in row-major format, then call LAPACK with  
'T' for transpose in the parameter list
On Sep 17, 2008, at 12:35 PM, Brock Palen wrote:

> I have some data, that was created by a C program and thus all data  
> is in rowMajor format.
> We would like to read this into another C program that would then  
> call LAPACK.  Because lapack for C is really a shim to fortran  
> (which expects ColumnMajor) we would like to read the data in  
> transposed.
>
> Is there a simple way to do this?  Any tricks?  We have some Macros  
> that let us access malloc()'ed memory in fortran style, but I can  
> for the life of me figure out any way to use them with H5Dread() .
>
> Any comments?

drew


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Andrew Cunningham-2
Hmm, seems to me we should call in a linear algebra guru....
dgesv is the combination of two calls..wild guess maybe dgetrs with  
'T' may work.

  //     Compute the LU factorization of A.

   dgetrf( n, n, a, lda, ipiv, info );
   if( info == 0 ) {

     //        Solve the system A*X = B, overwriting B with X.

     dgetrs( 'N'/* No transpose */, n, nrhs, a, lda, ipiv, b, ldb,
      info );
   }

On Sep 17, 2008, at 6:31 PM, Brock Palen wrote:

> The function we were looking at using (dgesv()) does not have a  
> transpose option.

Andrew

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20080917/a1a67e97/attachment.html>

Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Brock Palen
I saw that also.
I was looking more for a catch all way for reading data right in for  
C programmers for use by LAPACK, which has no (if poor) C interface.

macros like this don't work with H5read()

#define A(I,J) a[((J)-1)*lda+(I)-1]

Thanks I will check that out.

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985



On Sep 18, 2008, at 2:37 AM, Andrew Cunningham wrote:

> Hmm, seems to me we should call in a linear algebra guru....
> dgesv is the combination of two calls..wild guess maybe dgetrs with  
> 'T' may work.
>
>  //     Compute the LU factorization of A.
>
>   dgetrf( n, n, a, lda, ipiv, info );
>   if( info == 0 ) {
>
>     //        Solve the system A*X = B, overwriting B with X.
>
>     dgetrs( 'N'/* No transpose */, n, nrhs, a, lda, ipiv, b, ldb,
>      info );
>   }
>
> On Sep 17, 2008, at 6:31 PM, Brock Palen wrote:
>
>> The function we were looking at using (dgesv()) does not have a  
>> transpose option.
>
> Andrew
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] reading in data trasposed

Andrew Cunningham-2
One  solution might be for your users to use a C++ Matrix library that  
hides the actual implementation of the storage of the data so that  
while the data is stored column-major ( thus FORTRAN/LAPACK friendly)  
your C/C++ users see a familiar interface m(r,c)=val
e.g. Boost::uBLAS

On Sep 18, 2008, at 6:31 AM, Brock Palen wrote:

> I was looking more for a catch all way for reading data right in for  
> C programmers for use by LAPACK, which has no (if poor) C interface.

Andrew


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] unresolved external symbol

Dominik Szczerba-2
Hi,

I get the following linking error compiling my stuff with Visual Studio 2008
using hdf5 (binaries downloaded from the hdf5 download section). Can someone
point me out in the right direction please?

HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_UCHAR_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@@V?
$allocator at D@2@@std@@@Z)
HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_INT_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@@V?
$allocator at D@2@@std@@@Z)
HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_DOUBLE_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@@V?
$allocator at D@2@@std@@@Z)

with regards,
Dominik
--
Dominik Szczerba, Ph.D.
Biomedical Simulation Group
Computer Vision Lab CH-8092 Zurich
http://www.vision.ee.ethz.ch/~domi

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] unresolved external symbol

Lewandowski, George
This looks to me like a mismatch of debug/release settings in the
compiler, and/or a mismatch of multi-threaded/single-threaded.  Also, I
don't believe that HDF releases binaries built with VS2008, and that may
be another complicating factor.  I never attempt to link code  built
with different compiler versions, so I don't know if this will work.  

I would recommend building the binaries yourself.  That way you can
control all of the settings and make sure they match between the HDF5
libraries and your code.


George Lewandowski
(314)777-7890
Mail Code S270-2204
Building 270-E Level 2E Room 20E
P-8A

-----Original Message-----
From: Dominik Szczerba [mailto:domi at vision.ee.ethz.ch]
Sent: Thursday, October 16, 2008 5:20 PM
To: hdf-forum at hdfgroup.org
Subject: [hdf-forum] unresolved external symbol

Hi,

I get the following linking error compiling my stuff with Visual Studio
2008 using hdf5 (binaries downloaded from the hdf5 download section).
Can someone point me out in the right direction please?

HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_UCHAR_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@
@V?
$allocator at D@2@@std@@@Z)
HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_INT_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@
@V?
$allocator at D@2@@std@@@Z)
HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_DOUBLE_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@
@V?
$allocator at D@2@@std@@@Z)

with regards,
Dominik
--
Dominik Szczerba, Ph.D.
Biomedical Simulation Group
Computer Vision Lab CH-8092 Zurich
http://www.vision.ee.ethz.ch/~domi

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.




Reply | Threaded
Open this post in threaded view
|

[hdf-forum] unresolved external symbol

Kent Yang
George is right. The current release(1.8.1) is not tested with VS2008.
However, we will support vs2008 for 1.8.2 release.

Kent

-----Original Message-----
From: Lewandowski, George [mailto:george.lewandowski at boeing.com]
Sent: Monday, October 20, 2008 9:54 AM
To: Dominik Szczerba; hdf-forum at hdfgroup.org
Subject: RE: [hdf-forum] unresolved external symbol

This looks to me like a mismatch of debug/release settings in the
compiler, and/or a mismatch of multi-threaded/single-threaded.  Also, I
don't believe that HDF releases binaries built with VS2008, and that may
be another complicating factor.  I never attempt to link code  built
with different compiler versions, so I don't know if this will work.  

I would recommend building the binaries yourself.  That way you can
control all of the settings and make sure they match between the HDF5
libraries and your code.


George Lewandowski
(314)777-7890
Mail Code S270-2204
Building 270-E Level 2E Room 20E
P-8A

-----Original Message-----
From: Dominik Szczerba [mailto:domi at vision.ee.ethz.ch]
Sent: Thursday, October 16, 2008 5:20 PM
To: hdf-forum at hdfgroup.org
Subject: [hdf-forum] unresolved external symbol

Hi,

I get the following linking error compiling my stuff with Visual Studio
2008 using hdf5 (binaries downloaded from the hdf5 download section).
Can someone point me out in the right direction please?

HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_UCHAR_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@
@V?
$allocator at D@2@@std@@@Z)
HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_INT_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@
@V?
$allocator at D@2@@std@@@Z)
HDF5++.lib(HDF5Reader.obj) : error LNK2019: unresolved external symbol
_H5T_NATIVE_DOUBLE_g referenced in function "private: int __thiscall
HDF5::HDF5Reader::readData(class std::basic_string<char,struct
std::char_traits<char>,class std::allocator<char> > const &)"
(?readData at HDF5Reader@HDF5@@AAEHABV?$basic_string at DU?$char_traits at D@std@
@V?
$allocator at D@2@@std@@@Z)

with regards,
Dominik
--
Dominik Szczerba, Ph.D.
Biomedical Simulation Group
Computer Vision Lab CH-8092 Zurich
http://www.vision.ee.ethz.ch/~domi

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.



----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.