[hdf-forum] Memory leak in hdf5_hl PT routines?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] Memory leak in hdf5_hl PT routines?

Richard van Hees
Dear all,

I wrote a little C-program to experiment with the HDF5 packet table API.
I am specially interested in speed and according to the documentation
this should be a fast API. In my example program, I am interested to see
if the API slows down when I open many tables each in a different group,
appending many (> 10000) rows. (In real-life I would not open and close
the HDF5 file in a loop, but open a file with measurements, reading data
from it, open the HDF5 file and append this infomation to the tables.
But the HDF5 file would contain many groups and each groups several
tables (not all of the same size and shape)

I have compiled the code agains HDF5-1.8.1 on a linux platform
(dual-core - 2 GB RAM):
   # gcc -O3 -W -Wall -I/usr/local/include -L/usr/local/lib -lhdf5_hl
-lhdf5 -lz test_h5pt.c

When I write more than 5000 lines to each table, the program uses more
than 1 GB of memory! Even when I commented out the line where I append
data to the tables (H5PTappend), this problem still occurs. The problem
disappears when I also comment out the lines where I open and close the
tables (H5PTopen and H5PTclose in write_h5pt_data). I have checked the
source code of H5PTopen and H5PTclose, but the code looks fine to me.
Any suggestions?

Best regards, Richard van Hees
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_h5pt.c
Type: text/x-csrc
Size: 3697 bytes
Desc: not available
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20080902/67ae26e8/attachment.bin>
-------------- next part --------------
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.

Reply | Threaded
Open this post in threaded view
|

[hdf-forum] Memory leak in hdf5_hl PT routines?

Richard van Hees
Reply | Threaded
Open this post in threaded view
|

[hdf-forum] Memory leak in hdf5_hl PT routines? (with attachment)

Richard van Hees
In reply to this post by Richard van Hees
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20080903/c3232083/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch_H5PT.c
Type: text/x-csrc
Size: 1306 bytes
Desc: not available
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20080903/c3232083/attachment.bin>
-------------- next part --------------
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.

Reply | Threaded
Open this post in threaded view
|

[hdf-forum] Memory leak in hdf5_hl PT routines? (with attachment)

Elena Pourmal
Richard,

Thank you very much for the patch. I entered a bug report, and we will  
try to address it in our next release.

Elena

On Sep 3, 2008, at 4:56 AM, Richard van Hees wrote:

> Dear all,
>
> I found the memory leak in H5PTopen: a identifier for a copy of the  
> dataspace for a dataset  (= space_id) was not closed. Attached you  
> find the patch --- against hdf5-1.8.1-snap2/hl/src/H5PT.c ---  
> solving this issue.
>
> Best regards, Richard
>
> Richard van Hees wrote:
>>
>> Dear all,
>>
>> I wrote a little C-program to experiment with the HDF5 packet table  
>> API. I am specially interested in speed and according to the  
>> documentation this should be a fast API. In my example program, I  
>> am interested to see if the API slows down when I open many tables  
>> each in a different group, appending many (> 10000) rows. (In real-
>> life I would not open and close the HDF5 file in a loop, but open a  
>> file with measurements, reading data from it, open the HDF5 file  
>> and append this infomation to the tables. But the HDF5 file would  
>> contain many groups and each groups several tables (not all of the  
>> same size and shape)
>>
>> I have compiled the code agains HDF5-1.8.1 on a linux platform  
>> (dual-core - 2 GB RAM):
>>   # gcc -O3 -W -Wall -I/usr/local/include -L/usr/local/lib -
>> lhdf5_hl -lhdf5 -lz test_h5pt.c
>>
>> When I write more than 5000 lines to each table, the program uses  
>> more than 1 GB of memory! Even when I commented out the line where  
>> I append data to the tables (H5PTappend), this problem still  
>> occurs. The problem disappears when I also comment out the lines  
>> where I open and close the tables (H5PTopen and H5PTclose in  
>> write_h5pt_data). I have checked the source code of H5PTopen and  
>> H5PTclose, but the code looks fine to me. Any suggestions?
>>
>> Best regards, Richard van Hees
>> ----------------------------------------------------------------------
>> This mailing list is for HDF software users discussion.
>> To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org
>> .
>> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
> 233c233,234
> <   table = (htbl_t *)malloc(sizeof(htbl_t));
> ---
>>  if ( (table = (htbl_t *)malloc(sizeof(htbl_t))) == NULL )
>>       goto out;
> 237,239c238
> <       goto out;
> <   if(table->dset_id < 0)
> <     goto out;
> ---
>>       goto out;
> 243c242
> <     goto out;
> ---
>>       goto out;
> 247c246
> <     goto out;
> ---
>>       goto out;
> 250c249
> <     goto out;
> ---
>>       goto out;
> 254c253
> <     goto out;
> ---
>>       goto out;
> 258c257
> <     goto out;
> ---
>>       goto out;
> 260c259,260
> <     goto out;
> ---
>>       goto out;
>>  (void) H5Sclose( space_id );
> 264,267c264,265
> <   ret_value = H5Iregister(H5PT_ptable_id_type, table);
> <
> <   if(ret_value != H5I_INVALID_HID)
> <     H5PT_ptable_count++;
> ---
>>  if( (ret_value = H5Iregister(H5PT_ptable_id_type, table)) !=  
>> H5I_INVALID_HID)
>>       H5PT_ptable_count++;
> 269c267
> <     H5PT_close(table);
> ---
>>       H5PT_close(table);
> 276c274,275
> <   if(table)
> ---
>>  H5Sclose( space_id );
>>  if(table != NULL)
> 309c308
> <     goto out;
> ---
>>       goto out;
> 313c312
> <     goto out;
> ---
>>       goto out;
> 317c316
> <     goto out;
> ---
>>       goto out;
> 324c323
> <   if(table)
> ---
>>  if(table != NULL)
> 359c358
> <   if((table = H5Iremove_verify(table_id, H5PT_ptable_id_type))  
> ==NULL)
> ---
>>  if((table = H5Iremove_verify(table_id, H5PT_ptable_id_type)) ==  
>> NULL)
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20080903/2f38d935/attachment.html>