Quantcast

h5serv and S3

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

h5serv and S3

Ray Courtney
Hiya,

We have a repository of HDF5 files stored in Amazon S3 buckets, and we'd like to use h5serv to work with these files.

I tried googling but didn't find any implementations of this.

Has anyone has any luck standing up h5serv in front of S3 storage instead of the local data directory?

Any chance of this becoming a future enhancement for the h5serv project?

Thanks,
RAY

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: h5serv and S3

JohnReadey

Hi Ray,

 

   H5serv doesn’t work directly with S3 storage since the HDF5 library is expecting a Posix file system.   Using something like s3fuse (https://github.com/s3fs-fuse/s3fs-fuse) may work for read-only access to HDF5 files stored as S3 objects, but I haven’t tried this.

 

    Anyway, as it happens, we’re currently working on a project for NASA to enable HDF Server to use object storage directly.  The goal is to enable the HDF api across mutii-PB datasets hosted on S3.  We’re about 6 months into the 2 year project, but it’s looking quite promising.

 

    I gave a talk at the summer ESIP meeting regarding this.  You can see the slides at: http://www.slideshare.net/HDFEOS/hdf-cloud-services.  The first part is a review of HDF Server progress to-date.  Slide 22 on talks about the object storage work.

 

 

Cheers,

John

 

From: Hdf-forum <[hidden email]> on behalf of Ray Courtney <[hidden email]>
Reply-To: HDF Users Discussion List <[hidden email]>
Date: Tuesday, February 14, 2017 at 12:39 PM
To: "[hidden email]" <[hidden email]>
Subject: [Hdf-forum] h5serv and S3

 

Hiya,

 

We have a repository of HDF5 files stored in Amazon S3 buckets, and we'd like to use h5serv to work with these files.

 

I tried googling but didn't find any implementations of this.

 

Has anyone has any luck standing up h5serv in front of S3 storage instead of the local data directory?

 

Any chance of this becoming a future enhancement for the h5serv project?

 

Thanks,

RAY


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Loading...