Quantcast

Add data to a file image in memory

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Add data to a file image in memory

Samer Afach-4

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best,
Samer Afach


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add data to a file image in memory

Landon Clipp
Hello Samer,

I think I get the gist of what you're asking. If you are writing to the object and then want to read from the updated file, you will need to call H5Fflush() in order to write your data buffers to the file on your disk. Modifying objects on your file does not actually immediately change the file itself. The changes are only made when the object and/or file is closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[hidden email]> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best,
Samer Afach


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add data to a file image in memory

Samer Afach-4

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but file images, which is basically the whole file stored in some buffer in memory. Imagine you use std::fstream::read() or the classic C FILE's fread() to read the whole file into memory, and then wanting to open it with the HDF5 library. This is why I'm using "H5LTopen_file_image()", which doesn't take a file name or file path, but takes a void* buffer, which is a pointer to the array where the file is stored in memory.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5 file object (or file handle), the one you're talking about. My question is: Now after we do the changes to the file under that object, and after we H5Fflush(), that buffer may be changed. My question is: How do we keep track of that buffer (and its size) to write it back to a file eventually (using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach


On 27.12.2016 12:19 AM, Landon Clipp wrote:
Hello Samer,

I think I get the gist of what you're asking. If you are writing to the object and then want to read from the updated file, you will need to call H5Fflush() in order to write your data buffers to the file on your disk. Modifying objects on your file does not actually immediately change the file itself. The changes are only made when the object and/or file is closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[hidden email]> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best, Samer Afach

_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add data to a file image in memory

Martijn Jasperse
Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the library makes a copy of the original buffer and operates on that. So once you've created the attribute you wanted and done a flush, you can get the file image back from two calls to H5LTopen_file_image as usual (first to get the size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely need to look into H5Pset_file_image_callbacks to handle when the file image is resized and hence when the buffer changes.

Cheers,
Martijn

On 27 December 2016 at 10:47, Samer Afach <[hidden email]> wrote:

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but file images, which is basically the whole file stored in some buffer in memory. Imagine you use std::fstream::read() or the classic C FILE's fread() to read the whole file into memory, and then wanting to open it with the HDF5 library. This is why I'm using "H5LTopen_file_image()", which doesn't take a file name or file path, but takes a void* buffer, which is a pointer to the array where the file is stored in memory.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5 file object (or file handle), the one you're talking about. My question is: Now after we do the changes to the file under that object, and after we H5Fflush(), that buffer may be changed. My question is: How do we keep track of that buffer (and its size) to write it back to a file eventually (using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach


On 27.12.2016 12:19 AM, Landon Clipp wrote:
Hello Samer,

I think I get the gist of what you're asking. If you are writing to the object and then want to read from the updated file, you will need to call H5Fflush() in order to write your data buffers to the file on your disk. Modifying objects on your file does not actually immediately change the file itself. The changes are only made when the object and/or file is closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[hidden email]> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best, Samer Afach

_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add data to a file image in memory

Samer Afach-4

Hi Martijn:

Thank you very much for your response. Since you split your answer into two parts, I also have to do the same and ask two questions about these parts, if you don't mind.

1. You said "you can get the file image back from two calls to H5LTopen_file_image" But I don't get what two calls you meant. Are you talking about the calls of "H5Pset_file_image_callbacks"?

2. I would love to do this as "in place", but there's a memory leak I reported in the HDF5 library that happens when opening images as in-place (ref. HDFFV-10019), and I'm waiting for feedback about that. But ignoring that memory leak, are you saying this is different from the first case (when it's not in-place)? And are you saying that to get the size of the image, the only way is to re-implement all the callback functions of H5Pset_file_image_callbacks?

Honestly that's a little scary that I'd love to see an example about this.

Thank you for all this information! You helped a lot :)

Best,
Samer


On 01.01.2017 11:16 PM, Martijn Jasperse wrote:
Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the library makes a copy of the original buffer and operates on that. So once you've created the attribute you wanted and done a flush, you can get the file image back from two calls to H5LTopen_file_image as usual (first to get the size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely need to look into H5Pset_file_image_callbacks to handle when the file image is resized and hence when the buffer changes.

Cheers,
Martijn

On 27 December 2016 at 10:47, Samer Afach <[hidden email]> wrote:

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but file images, which is basically the whole file stored in some buffer in memory. Imagine you use std::fstream::read() or the classic C FILE's fread() to read the whole file into memory, and then wanting to open it with the HDF5 library. This is why I'm using "H5LTopen_file_image()", which doesn't take a file name or file path, but takes a void* buffer, which is a pointer to the array where the file is stored in memory.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5 file object (or file handle), the one you're talking about. My question is: Now after we do the changes to the file under that object, and after we H5Fflush(), that buffer may be changed. My question is: How do we keep track of that buffer (and its size) to write it back to a file eventually (using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach


On 27.12.2016 12:19 AM, Landon Clipp wrote:
Hello Samer,

I think I get the gist of what you're asking. If you are writing to the object and then want to read from the updated file, you will need to call H5Fflush() in order to write your data buffers to the file on your disk. Modifying objects on your file does not actually immediately change the file itself. The changes are only made when the object and/or file is closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[hidden email]> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best, Samer Afach

_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add data to a file image in memory

Martijn Jasperse
Hi Samer,
Sorry I had a copy-paste malfunction there. I actually meant to use H5Fget_file_image twice instead. First using a NULL pointer to retrieve the "file size", then allocating some memory and calling it again to get the actual data.

I think the purpose of setting your own callbacks is that you can handle the realloc call so you know when the pointer gets changed. Because even if you use the "in-place" flag, adding attributes will likely cause the image to be expanded and hence require a new buffer. But having your own realloc lets you catch that. According to the manual, callback functions are allowed to be NULL to indicate the defaults should be used.

I did some work with file images a while back, but admittedly we didn't bother with in-place for our application as the copy overhead was relatively small. But maybe I can modify some old code to make an example.

Cheers,
Martijn


On 2 January 2017 at 13:28, Samer Afach <[hidden email]> wrote:

Hi Martijn:

Thank you very much for your response. Since you split your answer into two parts, I also have to do the same and ask two questions about these parts, if you don't mind.

1. You said "you can get the file image back from two calls to H5LTopen_file_image" But I don't get what two calls you meant. Are you talking about the calls of "H5Pset_file_image_callbacks"?

2. I would love to do this as "in place", but there's a memory leak I reported in the HDF5 library that happens when opening images as in-place (ref. HDFFV-10019), and I'm waiting for feedback about that. But ignoring that memory leak, are you saying this is different from the first case (when it's not in-place)? And are you saying that to get the size of the image, the only way is to re-implement all the callback functions of H5Pset_file_image_callbacks?

Honestly that's a little scary that I'd love to see an example about this.

Thank you for all this information! You helped a lot :)

Best,
Samer


On 01.01.2017 11:16 PM, Martijn Jasperse wrote:
Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the library makes a copy of the original buffer and operates on that. So once you've created the attribute you wanted and done a flush, you can get the file image back from two calls to H5LTopen_file_image as usual (first to get the size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely need to look into H5Pset_file_image_callbacks to handle when the file image is resized and hence when the buffer changes.

Cheers,
Martijn

On 27 December 2016 at 10:47, Samer Afach <[hidden email]> wrote:

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but file images, which is basically the whole file stored in some buffer in memory. Imagine you use std::fstream::read() or the classic C FILE's fread() to read the whole file into memory, and then wanting to open it with the HDF5 library. This is why I'm using "H5LTopen_file_image()", which doesn't take a file name or file path, but takes a void* buffer, which is a pointer to the array where the file is stored in memory.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5 file object (or file handle), the one you're talking about. My question is: Now after we do the changes to the file under that object, and after we H5Fflush(), that buffer may be changed. My question is: How do we keep track of that buffer (and its size) to write it back to a file eventually (using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach


On 27.12.2016 12:19 AM, Landon Clipp wrote:
Hello Samer,

I think I get the gist of what you're asking. If you are writing to the object and then want to read from the updated file, you will need to call H5Fflush() in order to write your data buffers to the file on your disk. Modifying objects on your file does not actually immediately change the file itself. The changes are only made when the object and/or file is closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[hidden email]> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best, Samer Afach

_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add data to a file image in memory

Samer Afach-4

Thanks for explaining! This was very helpful :)

Best,
Samer


On 02.01.2017 4:02 AM, Martijn Jasperse wrote:
Hi Samer,
Sorry I had a copy-paste malfunction there. I actually meant to use H5Fget_file_image twice instead. First using a NULL pointer to retrieve the "file size", then allocating some memory and calling it again to get the actual data.

I think the purpose of setting your own callbacks is that you can handle the realloc call so you know when the pointer gets changed. Because even if you use the "in-place" flag, adding attributes will likely cause the image to be expanded and hence require a new buffer. But having your own realloc lets you catch that. According to the manual, callback functions are allowed to be NULL to indicate the defaults should be used.

I did some work with file images a while back, but admittedly we didn't bother with in-place for our application as the copy overhead was relatively small. But maybe I can modify some old code to make an example.

Cheers,
Martijn


On 2 January 2017 at 13:28, Samer Afach <[hidden email]> wrote:

Hi Martijn:

Thank you very much for your response. Since you split your answer into two parts, I also have to do the same and ask two questions about these parts, if you don't mind.

1. You said "you can get the file image back from two calls to H5LTopen_file_image" But I don't get what two calls you meant. Are you talking about the calls of "H5Pset_file_image_callbacks"?

2. I would love to do this as "in place", but there's a memory leak I reported in the HDF5 library that happens when opening images as in-place (ref. HDFFV-10019), and I'm waiting for feedback about that. But ignoring that memory leak, are you saying this is different from the first case (when it's not in-place)? And are you saying that to get the size of the image, the only way is to re-implement all the callback functions of H5Pset_file_image_callbacks?

Honestly that's a little scary that I'd love to see an example about this.

Thank you for all this information! You helped a lot :)

Best,
Samer


On 01.01.2017 11:16 PM, Martijn Jasperse wrote:
Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the library makes a copy of the original buffer and operates on that. So once you've created the attribute you wanted and done a flush, you can get the file image back from two calls to H5LTopen_file_image as usual (first to get the size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely need to look into H5Pset_file_image_callbacks to handle when the file image is resized and hence when the buffer changes.

Cheers,
Martijn

On 27 December 2016 at 10:47, Samer Afach <[hidden email]> wrote:

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but file images, which is basically the whole file stored in some buffer in memory. Imagine you use std::fstream::read() or the classic C FILE's fread() to read the whole file into memory, and then wanting to open it with the HDF5 library. This is why I'm using "H5LTopen_file_image()", which doesn't take a file name or file path, but takes a void* buffer, which is a pointer to the array where the file is stored in memory.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5 file object (or file handle), the one you're talking about. My question is: Now after we do the changes to the file under that object, and after we H5Fflush(), that buffer may be changed. My question is: How do we keep track of that buffer (and its size) to write it back to a file eventually (using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach


On 27.12.2016 12:19 AM, Landon Clipp wrote:
Hello Samer,

I think I get the gist of what you're asking. If you are writing to the object and then want to read from the updated file, you will need to call H5Fflush() in order to write your data buffers to the file on your disk. Modifying objects on your file does not actually immediately change the file itself. The changes are only made when the object and/or file is closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[hidden email]> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and transfer them then through TCP/IP, so I read them into memory as `std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

My question is: Since I'm opening this as RW, how can I write an attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that can write an attribute to an object by its ID. Here it's:
void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{
    hid_t AttributeType = H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType, value.length());
    hid_t dataspaceHandle = H5Screate(H5S_SCALAR);
    hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle, AttributeType, &value.front());


    H5Aclose (AttrHandle);
    H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct image size and write it back to a file (or do anything else with it, for that matter)?

Thank you.

Best, Samer Afach

_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [hidden email] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...