hdf5 parallel h5py

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

hdf5 parallel h5py

ISCaD GmbH

Dear all,

 

I work on an web application which should store and receive the data from an hdf5 file.

 

Because several people work with this file and long running processes I would like

to use the mpi4py, h5py and HDF5.

 

I work on debian linux stretch 64 Bit.

 

What’s the way to parallel h5py.

 

Thanks and regards

 

Friedhelm Matten

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: hdf5 parallel h5py

Nelson, Jarom

It doesn’t sound like parallel HDF5 is what you are wanting to do here. Parallel HDF5 is for an application where all applications are writing in a very coordinated manner. All processes need to write the same metadata to the file in “collective” calls to the library, i.e. each application makes the same calls using the same arguments in the same order when making calls that modify the file metadata (creating files, datasets or groups, writing attributes, etc.).

It sounds like you have separate applications that are executing in a somewhat independent manner. This will not work with parallel HDF5.

 

Using the serial library, I can think of at least one approach that might work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR) mode = to open a file. Using a SWMR file for each process, each process would open one file as the writer in SWMR mode, and open the files from all the other processes as read-only in SWMR mode.

 

http://docs.h5py.org/en/latest/swmr.html

 

Jarom

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of ISCaD GmbH
Sent: Thursday, August 3, 2017 1:19 AM
To: [hidden email]
Subject: [Hdf-forum] hdf5 parallel h5py

 

Dear all,

 

I work on an web application which should store and receive the data from an hdf5 file.

 

Because several people work with this file and long running processes I would like

to use the mpi4py, h5py and HDF5.

 

I work on debian linux stretch 64 Bit.

 

What’s the way to parallel h5py.

 

Thanks and regards

 

Friedhelm Matten

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: hdf5 parallel h5py

Thomas Caswell
I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts a server in front of your hdf5 file.  It serves as a single process owns it and serves as the serialization point which side-steps almost all of the multiple-client issues.  h5pyd (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an identical high-level API to h5py.

Tom

On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]> wrote:

It doesn’t sound like parallel HDF5 is what you are wanting to do here. Parallel HDF5 is for an application where all applications are writing in a very coordinated manner. All processes need to write the same metadata to the file in “collective” calls to the library, i.e. each application makes the same calls using the same arguments in the same order when making calls that modify the file metadata (creating files, datasets or groups, writing attributes, etc.).

It sounds like you have separate applications that are executing in a somewhat independent manner. This will not work with parallel HDF5.

 

Using the serial library, I can think of at least one approach that might work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR) mode = to open a file. Using a SWMR file for each process, each process would open one file as the writer in SWMR mode, and open the files from all the other processes as read-only in SWMR mode.

 

http://docs.h5py.org/en/latest/swmr.html

 

Jarom

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of ISCaD GmbH
Sent: Thursday, August 3, 2017 1:19 AM
To: [hidden email]
Subject: [Hdf-forum] hdf5 parallel h5py

 

Dear all,

 

I work on an web application which should store and receive the data from an hdf5 file.

 

Because several people work with this file and long running processes I would like

to use the mpi4py, h5py and HDF5.

 

I work on debian linux stretch 64 Bit.

 

What’s the way to parallel h5py.

 

Thanks and regards

 

Friedhelm Matten

 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: hdf5 parallel h5py

JohnReadey

Friedhelm,

  

   I’m not familiar with the specifics of your web app, but another possibility is to just have the app call h5serv directly.

 

   Anika @ NASA Goddard wrote a nice blog article on this approach: https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-hdf-server-implementation/.

 

John

 

From: Hdf-forum <[hidden email]> on behalf of Thomas Caswell <[hidden email]>
Reply-To: HDF Users Discussion List <[hidden email]>
Date: Saturday, August 5, 2017 at 9:48 AM
To: HDF Users Discussion List <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: [Hdf-forum] hdf5 parallel h5py

 

I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts a server in front of your hdf5 file.  It serves as a single process owns it and serves as the serialization point which side-steps almost all of the multiple-client issues.  h5pyd (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an identical high-level API to h5py.

 

Tom

 

On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]> wrote:

It doesn’t sound like parallel HDF5 is what you are wanting to do here. Parallel HDF5 is for an application where all applications are writing in a very coordinated manner. All processes need to write the same metadata to the file in “collective” calls to the library, i.e. each application makes the same calls using the same arguments in the same order when making calls that modify the file metadata (creating files, datasets or groups, writing attributes, etc.).

It sounds like you have separate applications that are executing in a somewhat independent manner. This will not work with parallel HDF5.

 

Using the serial library, I can think of at least one approach that might work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR) mode = to open a file. Using a SWMR file for each process, each process would open one file as the writer in SWMR mode, and open the files from all the other processes as read-only in SWMR mode.

 

http://docs.h5py.org/en/latest/swmr.html

 

Jarom

 

From: Hdf-forum [mailto:[hidden email]] On Behalf Of ISCaD GmbH
Sent: Thursday, August 3, 2017 1:19 AM
To: [hidden email]
Subject: [Hdf-forum] hdf5 parallel h5py

 

Dear all,

 

I work on an web application which should store and receive the data from an hdf5 file.

 

Because several people work with this file and long running processes I would like

to use the mpi4py, h5py and HDF5.

 

I work on debian linux stretch 64 Bit.

 

What’s the way to parallel h5py.

 

Thanks and regards

 

Friedhelm Matten

 

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: hdf5 parallel h5py

Jialin Liu
In reply to this post by ISCaD GmbH
Hi, 
The way to parallel h5py is first to compile h5py with parallel HDF5. If you are looking for examples of parallel H5py, take a look at the nersc webpage: http://www.nersc.gov/users/data-analytics/data-management/i-o-libraries/hdf5-2/h5py/

Best,
Jialin


On Mon, Aug 7, 2017 at 11:06 AM, <[hidden email]> wrote:
Send Hdf-forum mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: hdf5 parallel h5py (Thomas Caswell)
   2. Re: "File too large" error, seemingly related to MPI
      (Quincey Koziol)
   3. Re: VCS URI (Quincey Koziol)
   4. Re: hdf5 parallel h5py (John Readey)


----------------------------------------------------------------------

Message: 1
Date: Sat, 05 Aug 2017 16:48:23 +0000
From: Thomas Caswell <[hidden email]>
To: HDF Users Discussion List <[hidden email]>,
        "[hidden email]" <[hidden email]>
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts
a server in front of your hdf5 file.  It serves as a single process owns it
and serves as the serialization point which side-steps almost all of the
multiple-client issues.  h5pyd (https://github.com/HDFGroup/h5pyd) is a
client of h5serv which has an identical high-level API to h5py.

Tom

On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]> wrote:

> It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> Parallel HDF5 is for an application where all applications are writing in a
> very coordinated manner. All processes need to write the same metadata to
> the file in ?collective? calls to the library, i.e. each application makes
> the same calls using the same arguments in the same order when making calls
> that modify the file metadata (creating files, datasets or groups, writing
> attributes, etc.
> <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
>
> It sounds like you have separate applications that are executing in a
> somewhat independent manner. This will not work with parallel HDF5.
>
>
>
> Using the serial library, I can think of at least one approach that might
> work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR)
> mode = to open a file. Using a SWMR file for each process, each process
> would open one file as the writer in SWMR mode, and open the files from all
> the other processes as read-only in SWMR mode.
>
>
>
> http://docs.h5py.org/en/latest/swmr.html
>
>
>
> Jarom
>
>
>
> *From:* Hdf-forum [mailto:[hidden email]] *On
> Behalf Of *ISCaD GmbH
> *Sent:* Thursday, August 3, 2017 1:19 AM
> *To:* [hidden email]
> *Subject:* [Hdf-forum] hdf5 parallel h5py
>
>
>
> Dear all,
>
>
>
> I work on an web application which should store and receive the data from
> an hdf5 file.
>
>
>
> Because several people work with this file and long running processes I
> would like
>
> to use the mpi4py, h5py and HDF5.
>
>
>
> I work on debian linux stretch 64 Bit.
>
>
>
> What?s the way to parallel h5py.
>
>
>
> Thanks and regards
>
>
>
> Friedhelm Matten
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170805/4d84f060/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 7 Aug 2017 09:14:28 -0500
From: Quincey Koziol <[hidden email]>
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] "File too large" error, seemingly related to
        MPI
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=utf-8

Hi Frederic,
        Could you give us some more details about your file and the call(s) you are making to HDF5?   I can?t think of any reason that it would crash when creating a file like this, but something interesting could be going on?   :-)

        Quincey


> On Aug 7, 2017, at 5:28 AM, Frederic Perez <[hidden email]> wrote:
>
> Hi,
>
> While writing significant amount of data in parallel, I obtain the
> following error stack:
>
> HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66:
>  #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #001: H5Dint.c line 453 in H5D__create_named(): unable to create and
> link to dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #002: H5L.c line 1638 in H5L_link_object(): unable to create new
> link to object
>    major: Links
>    minor: Unable to initialize object
>  #003: H5L.c line 1882 in H5L_create_real(): can't insert link
>    major: Symbol table
>    minor: Unable to insert object
>  #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
>    major: Symbol table
>    minor: Object not found
>  #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
>    major: Symbol table
>    minor: Callback failed
>  #006: H5L.c line 1685 in H5L_link_cb(): unable to create object
>    major: Object header
>    minor: Unable to initialize object
>  #007: H5O.c line 3016 in H5O_obj_create(): unable to open object
>    major: Object header
>    minor: Can't open object
>  #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #009: H5Dint.c line 1060 in H5D__create(): can't update the metadata cache
>    major: Dataset
>    minor: Unable to initialize object
>  #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update
> layout/pline/efl header message
>    major: Dataset
>    minor: Unable to initialize object
>  #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to
> initialize storage
>    major: Dataset
>    minor: Unable to initialize object
>  #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to
> initialize dataset with fill value
>    major: Dataset
>    minor: Unable to initialize object
>  #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate
> all chunks of dataset
>    major: Dataset
>    minor: Unable to initialize object
>  #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to write
> raw data to file
>    major: Low-level I/O
>    minor: Write failed
>  #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable
> to write raw data to file
>    major: Low-level I/O
>    minor: Write failed
>  #016: H5Fio.c line 171 in H5F_block_write(): write through metadata
> accumulator failed
>    major: Low-level I/O
>    minor: Write failed
>  #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed
>    major: Low-level I/O
>    minor: Write failed
>  #018: H5FDint.c line 260 in H5FD_write(): driver write request failed
>    major: Virtual File Layer
>    minor: Write failed
>  #019: H5FDmpio.c line 1846 in H5FD_mpio_write(): MPI_File_write_at_all failed
>    major: Internal error (too specific to document in detail)
>    minor: Some MPI function failed
>  #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error ,
> error stack:
> ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large
>    major: Internal error (too specific to document in detail)
>    minor: MPI Error String
>
>
> It basically claims that I am creating a file too large. But I
> verified that the filesystem is capable of handling such a size. In my
> case, the file is around 4 TB when it crashes. Where could this
> problem come from? I thought HDF5 does not have a problem with very
> large files. Plus, I am dividing the file in several datasets, and the
> write operations work perfectly until, at some point, it crashes with
> the errors above.
>
> Could it be an issue with HDF5? Or could it be an MPI limitation? I am
> skeptic about the latter option: at the beginning, the program writes
> several datasets inside the file succesfully (all the datasets being
> the same size). If MPI was to blame, why wouldn't it crash at the
> first write?
>
> Thank you for your help.
> Fred
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5




------------------------------

Message: 3
Date: Mon, 7 Aug 2017 09:15:41 -0500
From: Quincey Koziol <[hidden email]>
To: HDF Users Discussion List <[hidden email]>
Subject: Re: [Hdf-forum] VCS URI
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="us-ascii"

Hi David,
        Sure, the git repo is here:  https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse>

                Quincey


> On Aug 7, 2017, at 5:27 AM, David Seifert <[hidden email]> wrote:
>
> Hi HDF5 team and users,
> is there any possibility for me to develop again the current VCS
> sources for adding pkgconfig + Meson support? This makes development
> and feature addition a lot easier, as there won't be any conflicts.
>
> Regards
> David
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/bebdf029/attachment-0001.html>

------------------------------

Message: 4
Date: Mon, 7 Aug 2017 15:05:28 +0000
From: John Readey <[hidden email]>
To: HDF Users Discussion List <[hidden email]>,
        "[hidden email]" <[hidden email]>
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Friedhelm,

   I?m not familiar with the specifics of your web app, but another possibility is to just have the app call h5serv directly.

   Anika @ NASA Goddard wrote a nice blog article on this approach: https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-hdf-server-implementation/.

John

From: Hdf-forum <[hidden email]> on behalf of Thomas Caswell <[hidden email]>
Reply-To: HDF Users Discussion List <[hidden email]>
Date: Saturday, August 5, 2017 at 9:48 AM
To: HDF Users Discussion List <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: [Hdf-forum] hdf5 parallel h5py

I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts a server in front of your hdf5 file.  It serves as a single process owns it and serves as the serialization point which side-steps almost all of the multiple-client issues.  h5pyd (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an identical high-level API to h5py.

Tom

On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]<mailto:[hidden email]>> wrote:
It doesn?t sound like parallel HDF5 is what you are wanting to do here. Parallel HDF5 is for an application where all applications are writing in a very coordinated manner. All processes need to write the same metadata to the file in ?collective? calls to the library, i.e. each application makes the same calls using the same arguments in the same order when making calls that modify the file metadata (creating files, datasets or groups, writing attributes, etc.<https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
It sounds like you have separate applications that are executing in a somewhat independent manner. This will not work with parallel HDF5.

Using the serial library, I can think of at least one approach that might work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR) mode = to open a file. Using a SWMR file for each process, each process would open one file as the writer in SWMR mode, and open the files from all the other processes as read-only in SWMR mode.

http://docs.h5py.org/en/latest/swmr.html

Jarom

From: Hdf-forum [mailto:[hidden email]<mailto:[hidden email]>] On Behalf Of ISCaD GmbH
Sent: Thursday, August 3, 2017 1:19 AM
To: [hidden email]<mailto:[hidden email]>
Subject: [Hdf-forum] hdf5 parallel h5py

Dear all,

I work on an web application which should store and receive the data from an hdf5 file.

Because several people work with this file and long running processes I would like
to use the mpi4py, h5py and HDF5.

I work on debian linux stretch 64 Bit.

What?s the way to parallel h5py.

Thanks and regards

Friedhelm Matten

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]<mailto:[hidden email]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/d8afdcac/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 98, Issue 6
****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: hdf5 parallel h5py

ISCaD GmbH
In reply to this post by ISCaD GmbH
Hi,

I look for SWMR and h5serv:

H5serv thanks good way but:

I read the h5serv release notes and notify:

High Performance usage ‐ the current release of h5serv serializes all requests, so would not be suitable for a demanding environment with a large number of clients and/or high throughput rates.

and

Variable length datatypes - Variable length datatypes are now supported.

This are remarkable issues for me.

SWMR sounds good but:

What I see, is in  early stage.
I need also parallel write for big data loads.


Thanks and regards


-----Ursprüngliche Nachricht-----
Von: Hdf-forum [mailto:[hidden email]] Im Auftrag von [hidden email]
Gesendet: Montag, 7. August 2017 17:13
An: [hidden email]
Betreff: Hdf-forum Digest, Vol 98, Issue 7

Send Hdf-forum mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific than "Re: Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: hdf5 parallel h5py (Jialin Liu)


----------------------------------------------------------------------

Message: 1
Date: Mon, 7 Aug 2017 11:12:23 -0400
From: Jialin Liu <[hidden email]>
To: [hidden email]
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi,
The way to parallel h5py is first to compile h5py with parallel HDF5. If you are looking for examples of parallel H5py, take a look at the nersc
webpage:
http://www.nersc.gov/users/data-analytics/data-management/i-o-libraries/hdf5-2/h5py/

Best,
Jialin


On Mon, Aug 7, 2017 at 11:06 AM, <[hidden email]>
wrote:

> Send Hdf-forum mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_
> lists.hdfgroup.org
>
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Hdf-forum digest..."
>
>
> Today's Topics:
>
>    1. Re: hdf5 parallel h5py (Thomas Caswell)
>    2. Re: "File too large" error, seemingly related to MPI
>       (Quincey Koziol)
>    3. Re: VCS URI (Quincey Koziol)
>    4. Re: hdf5 parallel h5py (John Readey)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 05 Aug 2017 16:48:23 +0000
> From: Thomas Caswell <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>,
>         "[hidden email]" <[hidden email]>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
> Message-ID:
>         <CAA48SF_LEzc5tRgMWgTwv03SBOoCvBcFeycP6
> [hidden email]>
> Content-Type: text/plain; charset="utf-8"
>
> I would also look at h5serv (https://github.com/HDFGroup/h5serv) which
> puts a server in front of your hdf5 file.  It serves as a single
> process owns it and serves as the serialization point which side-steps
> almost all of the multiple-client issues.  h5pyd
> (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an
> identical high-level API to h5py.
>
> Tom
>
> On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]> wrote:
>
> > It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> > Parallel HDF5 is for an application where all applications are
> > writing
> in a
> > very coordinated manner. All processes need to write the same
> > metadata to the file in ?collective? calls to the library, i.e. each
> > application
> makes
> > the same calls using the same arguments in the same order when
> > making
> calls
> > that modify the file metadata (creating files, datasets or groups,
> writing
> > attributes, etc.
> > <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
> >
> > It sounds like you have separate applications that are executing in
> > a somewhat independent manner. This will not work with parallel HDF5.
> >
> >
> >
> > Using the serial library, I can think of at least one approach that
> > might work well for you. HDF5 1.10 introduced a Single-writer
> > multi-reader
> (SWMR)
> > mode = to open a file. Using a SWMR file for each process, each
> > process would open one file as the writer in SWMR mode, and open the
> > files from
> all
> > the other processes as read-only in SWMR mode.
> >
> >
> >
> > http://docs.h5py.org/en/latest/swmr.html
> >
> >
> >
> > Jarom
> >
> >
> >
> > *From:* Hdf-forum [mailto:[hidden email]] *On
> > Behalf Of *ISCaD GmbH
> > *Sent:* Thursday, August 3, 2017 1:19 AM
> > *To:* [hidden email]
> > *Subject:* [Hdf-forum] hdf5 parallel h5py
> >
> >
> >
> > Dear all,
> >
> >
> >
> > I work on an web application which should store and receive the data
> > from an hdf5 file.
> >
> >
> >
> > Because several people work with this file and long running
> > processes I would like
> >
> > to use the mpi4py, h5py and HDF5.
> >
> >
> >
> > I work on debian linux stretch 64 Bit.
> >
> >
> >
> > What?s the way to parallel h5py.
> >
> >
> >
> > Thanks and regards
> >
> >
> >
> > Friedhelm Matten
> >
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
> -------------- next part -------------- An HTML attachment was
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170805/4d84f060/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 7 Aug 2017 09:14:28 -0500
> From: Quincey Koziol <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>
> Subject: Re: [Hdf-forum] "File too large" error, seemingly related to
>         MPI
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=utf-8
>
> Hi Frederic,
>         Could you give us some more details about your file and the
> call(s) you are making to HDF5?   I can?t think of any reason that it would
> crash when creating a file like this, but something interesting could be
> going on?   :-)
>
>         Quincey
>
>
> > On Aug 7, 2017, at 5:28 AM, Frederic Perez
> > <[hidden email]>
> wrote:
> >
> > Hi,
> >
> > While writing significant amount of data in parallel, I obtain the
> > following error stack:
> >
> > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66:
> >  #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #001: H5Dint.c line 453 in H5D__create_named(): unable to create
> > and link to dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #002: H5L.c line 1638 in H5L_link_object(): unable to create new
> > link to object
> >    major: Links
> >    minor: Unable to initialize object
> >  #003: H5L.c line 1882 in H5L_create_real(): can't insert link
> >    major: Symbol table
> >    minor: Unable to insert object
> >  #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path
> > traversal
> failed
> >    major: Symbol table
> >    minor: Object not found
> >  #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal
> > operator
> failed
> >    major: Symbol table
> >    minor: Callback failed
> >  #006: H5L.c line 1685 in H5L_link_cb(): unable to create object
> >    major: Object header
> >    minor: Unable to initialize object
> >  #007: H5O.c line 3016 in H5O_obj_create(): unable to open object
> >    major: Object header
> >    minor: Can't open object
> >  #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #009: H5Dint.c line 1060 in H5D__create(): can't update the
> > metadata
> cache
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update
> > layout/pline/efl header message
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to
> > initialize storage
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to
> > initialize dataset with fill value
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate
> > all chunks of dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to
> > write raw data to file
> >    major: Low-level I/O
> >    minor: Write failed
> >  #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable
> > to write raw data to file
> >    major: Low-level I/O
> >    minor: Write failed
> >  #016: H5Fio.c line 171 in H5F_block_write(): write through metadata
> > accumulator failed
> >    major: Low-level I/O
> >    minor: Write failed
> >  #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed
> >    major: Low-level I/O
> >    minor: Write failed
> >  #018: H5FDint.c line 260 in H5FD_write(): driver write request failed
> >    major: Virtual File Layer
> >    minor: Write failed
> >  #019: H5FDmpio.c line 1846 in H5FD_mpio_write():
> > MPI_File_write_at_all
> failed
> >    major: Internal error (too specific to document in detail)
> >    minor: Some MPI function failed
> >  #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error ,
> > error stack:
> > ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large
> >    major: Internal error (too specific to document in detail)
> >    minor: MPI Error String
> >
> >
> > It basically claims that I am creating a file too large. But I
> > verified that the filesystem is capable of handling such a size. In
> > my case, the file is around 4 TB when it crashes. Where could this
> > problem come from? I thought HDF5 does not have a problem with very
> > large files. Plus, I am dividing the file in several datasets, and
> > the write operations work perfectly until, at some point, it crashes
> > with the errors above.
> >
> > Could it be an issue with HDF5? Or could it be an MPI limitation? I
> > am skeptic about the latter option: at the beginning, the program
> > writes several datasets inside the file succesfully (all the
> > datasets being the same size). If MPI was to blame, why wouldn't it
> > crash at the first write?
> >
> > Thank you for your help.
> > Fred
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 7 Aug 2017 09:15:41 -0500
> From: Quincey Koziol <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>
> Subject: Re: [Hdf-forum] VCS URI
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi David,
>         Sure, the git repo is here:  https://bitbucket.hdfgroup.
> org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup.
> org/projects/HDFFV/repos/hdf5/browse>
>
>                 Quincey
>
>
> > On Aug 7, 2017, at 5:27 AM, David Seifert <[hidden email]> wrote:
> >
> > Hi HDF5 team and users,
> > is there any possibility for me to develop again the current VCS
> > sources for adding pkgconfig + Meson support? This makes development
> > and feature addition a lot easier, as there won't be any conflicts.
> >
> > Regards
> > David
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
>
> -------------- next part -------------- An HTML attachment was
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170807/bebdf029/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 7 Aug 2017 15:05:28 +0000
> From: John Readey <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>,
>         "[hidden email]" <[hidden email]>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset="utf-8"
>
> Friedhelm,
>
>    I?m not familiar with the specifics of your web app, but another
> possibility is to just have the app call h5serv directly.
>
>    Anika @ NASA Goddard wrote a nice blog article on this approach:
> https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-
> hdf-server-implementation/.
>
> John
>
> From: Hdf-forum <[hidden email]> on behalf of
> Thomas Caswell <[hidden email]>
> Reply-To: HDF Users Discussion List <[hidden email]>
> Date: Saturday, August 5, 2017 at 9:48 AM
> To: HDF Users Discussion List <[hidden email]>, "
> [hidden email]" <[hidden email]>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
>
> I would also look at h5serv (https://github.com/HDFGroup/h5serv) which
> puts a server in front of your hdf5 file.  It serves as a single
> process owns it and serves as the serialization point which side-steps
> almost all of the multiple-client issues.  h5pyd
> (https://github.com/HDFGroup/h5pyd)
> is a client of h5serv which has an identical high-level API to h5py.
>
> Tom
>
> On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]<mailto:
> [hidden email]>> wrote:
> It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> Parallel HDF5 is for an application where all applications are writing
> in a very coordinated manner. All processes need to write the same
> metadata to the file in ?collective? calls to the library, i.e. each
> application makes the same calls using the same arguments in the same
> order when making calls that modify the file metadata (creating files,
> datasets or groups, writing attributes,
> etc.<https://support.hdfgroup.org/HDF5/doc/RM/
> CollectiveCalls.html>).
> It sounds like you have separate applications that are executing in a
> somewhat independent manner. This will not work with parallel HDF5.
>
> Using the serial library, I can think of at least one approach that
> might work well for you. HDF5 1.10 introduced a Single-writer
> multi-reader (SWMR) mode = to open a file. Using a SWMR file for each
> process, each process would open one file as the writer in SWMR mode,
> and open the files from all the other processes as read-only in SWMR mode.
>
> http://docs.h5py.org/en/latest/swmr.html
>
> Jarom
>
> From: Hdf-forum
> [mailto:[hidden email]<mailto:hdf-
> [hidden email]>] On Behalf Of ISCaD GmbH
> Sent: Thursday, August 3, 2017 1:19 AM
> To: [hidden email]<mailto:[hidden email]>
> Subject: [Hdf-forum] hdf5 parallel h5py
>
> Dear all,
>
> I work on an web application which should store and receive the data
> from an hdf5 file.
>
> Because several people work with this file and long running processes
> I would like to use the mpi4py, h5py and HDF5.
>
> I work on debian linux stretch 64 Bit.
>
> What?s the way to parallel h5py.
>
> Thanks and regards
>
> Friedhelm Matten
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]<mailto:[hidden email]>
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
> -------------- next part -------------- An HTML attachment was
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170807/d8afdcac/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
>
>
> ------------------------------
>
> End of Hdf-forum Digest, Vol 98, Issue 6
> ****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/9e689fe5/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 98, Issue 7
****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: hdf5 parallel h5py

David Pearah

Friedhelm,

> H5serv thanks good way but: I read the h5serv release notes and notify:
> High Performance usage �\ the current release of h5serv
> serializes all requests, so would not be suitable for a demanding
> environment with a large number of clients and/or high throughput rates....

We've been actually working on a successor product to h5serv -- called HDF Cloud -- which we will be announcing in a couple weeks. HDF Cloud adds significant functionality beyond h5serv, including scalability for large numbers of clients, support for AWS S3 object storage, etc. We have several Beta clients in testing already.


> SWMR sounds good but:... What I see, is in  early stage. I need also parallel write for big data loads.

SWMR is fairly mature... there's always room for improvement but plenty of HDF users rely on this capability and can share their experience here.


Feel free to reach out to me or Dax if you would like to learn more. Cheers,

-- Dave Pearah


From: Hdf-forum <[hidden email]> on behalf of ISCaD GmbH <[hidden email]>
Sent: Tuesday, August 8, 2017 2:24:46 AM
To: [hidden email]
Subject: Re: [Hdf-forum] hdf5 parallel h5py
 
Hi,

I look for SWMR and h5serv:

H5serv thanks good way but:

I read the h5serv release notes and notify:

High Performance usage �\ the current release of h5serv serializes all requests, so would not be suitable for a demanding environment with a large number of clients and/or high throughput rates.

and

Variable length datatypes - Variable length datatypes are now supported.

This are remarkable issues for me.

SWMR sounds good but:

What I see, is in  early stage.
I need also parallel write for big data loads.


Thanks and regards


-----Ursprüngliche Nachricht-----
Von: Hdf-forum [[hidden email]] Im Auftrag von [hidden email]
Gesendet: Montag, 7. August 2017 17:13
An: [hidden email]
Betreff: Hdf-forum Digest, Vol 98, Issue 7

Send Hdf-forum mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific than "Re: Contents of Hdf-forum digest..."


Today's Topics:

   1. Re: hdf5 parallel h5py (Jialin Liu)


----------------------------------------------------------------------

Message: 1
Date: Mon, 7 Aug 2017 11:12:23 -0400
From: Jialin Liu <[hidden email]>
To: [hidden email]
Subject: Re: [Hdf-forum] hdf5 parallel h5py
Message-ID:
        <CAOWawpGNLZz38c0CnPv00RyctgAdW9bGWLc+[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi,
The way to parallel h5py is first to compile h5py with parallel HDF5. If you are looking for examples of parallel H5py, take a look at the nersc
webpage:
http://www.nersc.gov/users/data-analytics/data-management/i-o-libraries/hdf5-2/h5py/

Best,
Jialin


On Mon, Aug 7, 2017 at 11:06 AM, <[hidden email]>
wrote:

> Send Hdf-forum mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_
> lists.hdfgroup.org
>
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Hdf-forum digest..."
>
>
> Today's Topics:
>
>    1. Re: hdf5 parallel h5py (Thomas Caswell)
>    2. Re: "File too large" error, seemingly related to MPI
>       (Quincey Koziol)
>    3. Re: VCS URI (Quincey Koziol)
>    4. Re: hdf5 parallel h5py (John Readey)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 05 Aug 2017 16:48:23 +0000
> From: Thomas Caswell <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>,
>         "[hidden email]" <[hidden email]>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
> Message-ID:
>         <CAA48SF_LEzc5tRgMWgTwv03SBOoCvBcFeycP6
> [hidden email]>
> Content-Type: text/plain; charset="utf-8"
>
> I would also look at h5serv (https://github.com/HDFGroup/h5serv) which
> puts a server in front of your hdf5 file.  It serves as a single
> process owns it and serves as the serialization point which side-steps
> almost all of the multiple-client issues.  h5pyd
> (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an
> identical high-level API to h5py.
>
> Tom
>
> On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]> wrote:
>
> > It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> > Parallel HDF5 is for an application where all applications are
> > writing
> in a
> > very coordinated manner. All processes need to write the same
> > metadata to the file in ?collective? calls to the library, i.e. each
> > application
> makes
> > the same calls using the same arguments in the same order when
> > making
> calls
> > that modify the file metadata (creating files, datasets or groups,
> writing
> > attributes, etc.
> > <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>).
> >
> > It sounds like you have separate applications that are executing in
> > a somewhat independent manner. This will not work with parallel HDF5.
> >
> >
> >
> > Using the serial library, I can think of at least one approach that
> > might work well for you. HDF5 1.10 introduced a Single-writer
> > multi-reader
> (SWMR)
> > mode = to open a file. Using a SWMR file for each process, each
> > process would open one file as the writer in SWMR mode, and open the
> > files from
> all
> > the other processes as read-only in SWMR mode.
> >
> >
> >
> > http://docs.h5py.org/en/latest/swmr.html
> >
> >
> >
> > Jarom
> >
> >
> >
> > *From:* Hdf-forum [[hidden email]] *On
> > Behalf Of *ISCaD GmbH
> > *Sent:* Thursday, August 3, 2017 1:19 AM
> > *To:* [hidden email]
> > *Subject:* [Hdf-forum] hdf5 parallel h5py
> >
> >
> >
> > Dear all,
> >
> >
> >
> > I work on an web application which should store and receive the data
> > from an hdf5 file.
> >
> >
> >
> > Because several people work with this file and long running
> > processes I would like
> >
> > to use the mpi4py, h5py and HDF5.
> >
> >
> >
> > I work on debian linux stretch 64 Bit.
> >
> >
> >
> > What?s the way to parallel h5py.
> >
> >
> >
> > Thanks and regards
> >
> >
> >
> > Friedhelm Matten
> >
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
> -------------- next part -------------- An HTML attachment was
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170805/4d84f060/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 7 Aug 2017 09:14:28 -0500
> From: Quincey Koziol <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>
> Subject: Re: [Hdf-forum] "File too large" error, seemingly related to
>         MPI
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=utf-8
>
> Hi Frederic,
>         Could you give us some more details about your file and the
> call(s) you are making to HDF5?   I can?t think of any reason that it would
> crash when creating a file like this, but something interesting could be
> going on?   :-)
>
>         Quincey
>
>
> > On Aug 7, 2017, at 5:28 AM, Frederic Perez
> > <[hidden email]>
> wrote:
> >
> > Hi,
> >
> > While writing significant amount of data in parallel, I obtain the
> > following error stack:
> >
> > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66:
> >  #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #001: H5Dint.c line 453 in H5D__create_named(): unable to create
> > and link to dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #002: H5L.c line 1638 in H5L_link_object(): unable to create new
> > link to object
> >    major: Links
> >    minor: Unable to initialize object
> >  #003: H5L.c line 1882 in H5L_create_real(): can't insert link
> >    major: Symbol table
> >    minor: Unable to insert object
> >  #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path
> > traversal
> failed
> >    major: Symbol table
> >    minor: Object not found
> >  #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal
> > operator
> failed
> >    major: Symbol table
> >    minor: Callback failed
> >  #006: H5L.c line 1685 in H5L_link_cb(): unable to create object
> >    major: Object header
> >    minor: Unable to initialize object
> >  #007: H5O.c line 3016 in H5O_obj_create(): unable to open object
> >    major: Object header
> >    minor: Can't open object
> >  #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #009: H5Dint.c line 1060 in H5D__create(): can't update the
> > metadata
> cache
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update
> > layout/pline/efl header message
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to
> > initialize storage
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to
> > initialize dataset with fill value
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate
> > all chunks of dataset
> >    major: Dataset
> >    minor: Unable to initialize object
> >  #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to
> > write raw data to file
> >    major: Low-level I/O
> >    minor: Write failed
> >  #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable
> > to write raw data to file
> >    major: Low-level I/O
> >    minor: Write failed
> >  #016: H5Fio.c line 171 in H5F_block_write(): write through metadata
> > accumulator failed
> >    major: Low-level I/O
> >    minor: Write failed
> >  #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed
> >    major: Low-level I/O
> >    minor: Write failed
> >  #018: H5FDint.c line 260 in H5FD_write(): driver write request failed
> >    major: Virtual File Layer
> >    minor: Write failed
> >  #019: H5FDmpio.c line 1846 in H5FD_mpio_write():
> > MPI_File_write_at_all
> failed
> >    major: Internal error (too specific to document in detail)
> >    minor: Some MPI function failed
> >  #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error ,
> > error stack:
> > ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large
> >    major: Internal error (too specific to document in detail)
> >    minor: MPI Error String
> >
> >
> > It basically claims that I am creating a file too large. But I
> > verified that the filesystem is capable of handling such a size. In
> > my case, the file is around 4 TB when it crashes. Where could this
> > problem come from? I thought HDF5 does not have a problem with very
> > large files. Plus, I am dividing the file in several datasets, and
> > the write operations work perfectly until, at some point, it crashes
> > with the errors above.
> >
> > Could it be an issue with HDF5? Or could it be an MPI limitation? I
> > am skeptic about the latter option: at the beginning, the program
> > writes several datasets inside the file succesfully (all the
> > datasets being the same size). If MPI was to blame, why wouldn't it
> > crash at the first write?
> >
> > Thank you for your help.
> > Fred
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 7 Aug 2017 09:15:41 -0500
> From: Quincey Koziol <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>
> Subject: Re: [Hdf-forum] VCS URI
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi David,
>         Sure, the git repo is here:  https://bitbucket.hdfgroup.
> org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup.
> org/projects/HDFFV/repos/hdf5/browse>
>
>                 Quincey
>
>
> > On Aug 7, 2017, at 5:27 AM, David Seifert <[hidden email]> wrote:
> >
> > Hi HDF5 team and users,
> > is there any possibility for me to develop again the current VCS
> > sources for adding pkgconfig + Meson support? This makes development
> > and feature addition a lot easier, as there won't be any conflicts.
> >
> > Regards
> > David
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > [hidden email]
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.
> > org
> > Twitter: https://twitter.com/hdf5
>
> -------------- next part -------------- An HTML attachment was
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170807/bebdf029/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 7 Aug 2017 15:05:28 +0000
> From: John Readey <[hidden email]>
> To: HDF Users Discussion List <[hidden email]>,
>         "[hidden email]" <[hidden email]>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset="utf-8"
>
> Friedhelm,
>
>    I?m not familiar with the specifics of your web app, but another
> possibility is to just have the app call h5serv directly.
>
>    Anika @ NASA Goddard wrote a nice blog article on this approach:
> https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-
> hdf-server-implementation/.
>
> John
>
> From: Hdf-forum <[hidden email]> on behalf of
> Thomas Caswell <[hidden email]>
> Reply-To: HDF Users Discussion List <[hidden email]>
> Date: Saturday, August 5, 2017 at 9:48 AM
> To: HDF Users Discussion List <[hidden email]>, "
> [hidden email]" <[hidden email]>
> Subject: Re: [Hdf-forum] hdf5 parallel h5py
>
> I would also look at h5serv (https://github.com/HDFGroup/h5serv) which
> puts a server in front of your hdf5 file.  It serves as a single
> process owns it and serves as the serialization point which side-steps
> almost all of the multiple-client issues.  h5pyd
> (https://github.com/HDFGroup/h5pyd)
> is a client of h5serv which has an identical high-level API to h5py.
>
> Tom
>
> On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[hidden email]<mailto:
> [hidden email]>> wrote:
> It doesn?t sound like parallel HDF5 is what you are wanting to do here.
> Parallel HDF5 is for an application where all applications are writing
> in a very coordinated manner. All processes need to write the same
> metadata to the file in ?collective? calls to the library, i.e. each
> application makes the same calls using the same arguments in the same
> order when making calls that modify the file metadata (creating files,
> datasets or groups, writing attributes,
> etc.<https://support.hdfgroup.org/HDF5/doc/RM/
> CollectiveCalls.html>).
> It sounds like you have separate applications that are executing in a
> somewhat independent manner. This will not work with parallel HDF5.
>
> Using the serial library, I can think of at least one approach that
> might work well for you. HDF5 1.10 introduced a Single-writer
> multi-reader (SWMR) mode = to open a file. Using a SWMR file for each
> process, each process would open one file as the writer in SWMR mode,
> and open the files from all the other processes as read-only in SWMR mode.
>
> http://docs.h5py.org/en/latest/swmr.html
>
> Jarom
>
> From: Hdf-forum
> [mailto:[hidden email]<mailto:hdf-
> [hidden email]>] On Behalf Of ISCaD GmbH
> Sent: Thursday, August 3, 2017 1:19 AM
> To: [hidden email]<mailto:[hidden email]>
> Subject: [Hdf-forum] hdf5 parallel h5py
>
> Dear all,
>
> I work on an web application which should store and receive the data
> from an hdf5 file.
>
> Because several people work with this file and long running processes
> I would like to use the mpi4py, h5py and HDF5.
>
> I work on debian linux stretch 64 Bit.
>
> What?s the way to parallel h5py.
>
> Thanks and regards
>
> Friedhelm Matten
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]<mailto:[hidden email]>
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
> -------------- next part -------------- An HTML attachment was
> scrubbed...
> URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
> hdfgroup.org/attachments/20170807/d8afdcac/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
>
>
> ------------------------------
>
> End of Hdf-forum Digest, Vol 98, Issue 6
> ****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170807/9e689fe5/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


------------------------------

End of Hdf-forum Digest, Vol 98, Issue 7
****************************************


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5