Quantcast

Hdf5 for in memory circular buffer?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Hdf5 for in memory circular buffer?

davidsch
Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data.

best,

David Schneider
SLAC/LCLS
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hdf5 for in memory circular buffer?

Werner Benger
Hi,

  there is a virtual file driver that operates on a memory image of an
HDF5 file. It should be no problem to have this one also operate on
shared memory.

        Werner


On 10.12.2016 05:50, Schneider, David A. wrote:

> Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data.
>
> best,
>
> David Schneider
> SLAC/LCLS
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hdf5 for in memory circular buffer?

Steven Varga-2
Hello,

I am no expert by any means in your field, but I found that in HFT (high-frequency trading and similar fields) similar pattern often come up.

Boost circular/ring buffer can handle the storage which one process writes and many reads from; upon tail eviction/flushing the pocket one can channel it to the write process --in case of fault tolerance needed you can have many ring buffers on different hosts.

The above scheme can be done with various transportation mechanism ie: zeroMQ + protocol buffers/thrift. This is for production use.

For running experiments where being fail-safe has little or no benefit one can  access the HDF5 datasets locally on each cluster nodes; using  MPI + Grid Engine + some_software_task + HDF5_serial pattern.

If you find the above interesting, feel free to ask.

best,
steve

On Sun, Dec 11, 2016 at 7:51 AM, Werner Benger <[hidden email]> wrote:
Hi,

 there is a virtual file driver that operates on a memory image of an HDF5 file. It should be no problem to have this one also operate on shared memory.

       Werner


On 10.12.2016 05:50, Schneider, David A. wrote:
Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data.

best,

David Schneider
SLAC/LCLS
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019  Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: <a href="tel:%2B1%20225%20578%204809" value="+12255784809" target="_blank">+1 225 578 4809                        Fax.: <a href="tel:%2B1%20225%20578-5362" value="+12255785362" target="_blank">+1 225 578-5362


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...