Question about domain name in h5 server

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about domain name in h5 server

Hui, Yawei

Hi,

 

  I’m a new user of h5 server. On my test server, I installed and ran the h5serv docker container. From a remote browser, I could get response by sending http://128.XXX.XXX.XXX:5000/  to the server. However, when I tried to use curl-based request, such as “curl --header "Host: dp.h5_store.hdfgroup.org" http://128.XXX. XXX. XXX:5000/”, an error “tornado.web.HTTPError: HTTP 404: Not Found” occurred. The h5 files I want the server to serve is under “/data/h5_store”. Because the h5 server is installed on a machine without domain name (e.g. hdfgroup.org), how could I set up a valid HTTP header so that the curl command could be used to send the request?

 

  Thanks.

 

  • Yawei

 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 

Computer Science Research Group

Computer Science and Mathematics Division

Oak Ridge National Laboratory

PO BOX 2008 MS6164

OAK RIDGE TN 37831-6164

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: Question about domain name in h5 server

JohnReadey

Hi Yawei,

 

   For curl, always use the dns name of the server the container is running on, i.e. the http://128… address you have. 

 

   To address files, the pseudo-DNS name is created by taking the “domain” config value (in h5serv/server/config.py), and combining with the reverse path of the file relative to the data directory.  There’s another config value: “datapath” that gives the location to the data directory.  So in your case you should set this to: /data/h5_store.  After this (and restarting the server), if you have “myfile.h5” in the h5_store directory, the h5serv host path to that file would be: myfile.hdfgroup.org.

 

   If you’ve setup a DNS entry for your server, say h5serv.ornl.gov, then you can update the domain config value to match.  In  that case you’d have myfile.h5serv.ornl.gov.  The various config entries are documented here: http://h5serv.readthedocs.io/en/latest/Installation/ServerSetup.html#server-configuration.

 

   Hope this helps!  Configuration is always the challenging aspect.  You can email me directly (jreadey AT hdfgroup DOT org) if that doesn’t work.

 

Cheers,

John

 

From: Hdf-forum <[hidden email]> on behalf of "Hui, Yawei" <[hidden email]>
Reply-To: HDF Users Discussion List <[hidden email]>
Date: Tuesday, October 24, 2017 at 9:20 AM
To: "[hidden email]" <[hidden email]>
Subject: [Hdf-forum] Question about domain name in h5 server

 

Hi,

 

  I’m a new user of h5 server. On my test server, I installed and ran the h5serv docker container. From a remote browser, I could get response by sending http://128.XXX.XXX.XXX:5000/  to the server. However, when I tried to use curl-based request, such as “curl --header "Host: dp.h5_store.hdfgroup.org" http://128.XXX. XXX. XXX:5000/”, an error “tornado.web.HTTPError: HTTP 404: Not Found” occurred. The h5 files I want the server to serve is under “/data/h5_store”. Because the h5 server is installed on a machine without domain name (e.g. hdfgroup.org), how could I set up a valid HTTP header so that the curl command could be used to send the request?

 

  Thanks.

 

-          Yawei

 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 

Computer Science Research Group

Computer Science and Mathematics Division

Oak Ridge National Laboratory

PO BOX 2008 MS6164

OAK RIDGE TN 37831-6164

 


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

HDF5 library bug or feature?

Rafal Lichwala
Hi All,

I've just discovered that when I create a dataset with one dimension
which has a size = 0, then I cannot properly open such dataset -
H5Dopen() returns -1.
But when I create similar dataset with one dimension which has a size at
least 1 (even without writing data to such dataset), then I can properly
open such dataset.

Is this a bug or a feature in the HDF5 library?

I think I should be able to open 0-size dimension dataset, just to
create an attribute inside such dataset - and that was my intension - no
data inside (thus dimension size = 0), but write some attributes to such
dataset...

Best regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Gerd Heber
Rafal, this might be a bug. However, if you don't intend to store
data there's a cheaper and maybe less contorted option: You can
create a dataset with a dataspace of class H5S_NULL.
Think of that as a dataset whose dataspace is an empty set.
(Likewise you'd use H5S_SCALAR for a singleton.)

Best, G.

-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of Rafal Lichwala
Sent: Thursday, October 26, 2017 3:00 AM
To: [hidden email]
Subject: [Hdf-forum] HDF5 library bug or feature?

Hi All,

I've just discovered that when I create a dataset with one dimension which has a size = 0, then I cannot properly open such dataset -
H5Dopen() returns -1.
But when I create similar dataset with one dimension which has a size at least 1 (even without writing data to such dataset), then I can properly open such dataset.

Is this a bug or a feature in the HDF5 library?

I think I should be able to open 0-size dimension dataset, just to create an attribute inside such dataset - and that was my intension - no data inside (thus dimension size = 0), but write some attributes to such dataset...

Best regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Rafal Lichwala
Hi Gerd,

Thank you for an answer!

> Rafal, this might be a bug.

Is there any way to record/track this issue?

> However, if you don't intend to store
> data there's a cheaper and maybe less contorted option: You can
> create a dataset with a dataspace of class H5S_NULL.

Good point - I will remember about that possibility in the future -
thank you.
For now I've just moved all attributes level up - to the parent group -
works fine.

Best regards,
Rafal


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Gerd Heber
>> Rafal, this might be a bug.

> Is there any way to record/track this issue?

Not sure. Helpdesk will check. (It's still early morning here...)

G.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Andrey Paramonov
In reply to this post by Gerd Heber
Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:
> Rafal, this might be a bug. However, if you don't intend to store
> data there's a cheaper and maybe less contorted option: You can
> create a dataset with a dataspace of class H5S_NULL.
> Think of that as a dataset whose dataspace is an empty set.
> (Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider
XML attributes as example). I tend to use empty groups which only serve
as container of attributes. What do you think: would empty datasets
(H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Gerd Heber
Andrey, the experts should chime in on the storage trade-offs.
(I imagine the null dataset is stored with compact layout, but in
newer versions of the file format an empty group is very "compact"
as well.) At the conceptual level, the two are very different and
that might tip the balance in either direction.
A group leaves the door open for other descendants.
A dataset is a terminal node and, despite being empty, is nevertheless
typed. (A null dataset has an element type!) Of course, you could
"type" a group with additional attributes, but it would be a very
different kind of variable.

This might be a case where there're no right or wrong choices
as far as HDF5 is concerned, and users will have to decide what
makes it right for them.

Best, G.


-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of ?????? ?????????
Sent: Thursday, October 26, 2017 6:52 AM
To: [hidden email]
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:
> Rafal, this might be a bug. However, if you don't intend to store data
> there's a cheaper and maybe less contorted option: You can create a
> dataset with a dataspace of class H5S_NULL.
> Think of that as a dataset whose dataspace is an empty set.
> (Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider XML attributes as example). I tend to use empty groups which only serve as container of attributes. What do you think: would empty datasets
(H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Ger van Diepen
A choice consideration might be that the number of attributes has a limit.

I think HDF5 should support empty N-dim datasets. Being empty also conveys information.
Note that in any language it would be strange if you could not have an empty sequence.

Ger

>>> Gerd Heber <[hidden email]> 10/26/17 2:27 PM >>>
Andrey, the experts should chime in on the storage trade-offs.
(I imagine the null dataset is stored with compact layout, but in
newer versions of the file format an empty group is very "compact"
as well.) At the conceptual level, the two are very different and
that might tip the balance in either direction.
A group leaves the door open for other descendants.
A dataset is a terminal node and, despite being empty, is nevertheless
typed. (A null dataset has an element type!) Of course, you could
"type" a group with additional attributes, but it would be a very
different kind of variable.

This might be a case where there're no right or wrong choices
as far as HDF5 is concerned, and users will have to decide what
makes it right for them.

Best, G.


-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of ?????? ?????????
Sent: Thursday, October 26, 2017 6:52 AM
To: [hidden email]
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:
> Rafal, this might be a bug. However, if you don't intend to store data
> there's a cheaper and maybe less contorted option: You can create a
> dataset with a dataspace of class H5S_NULL.
> Think of that as a dataset whose dataspace is an empty set.
> (Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider XML attributes as example). I tend to use empty groups which only serve as container of attributes. What do you think: would empty datasets
(H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

bljones
In reply to this post by Gerd Heber

Hi Rafal,

I am able to create a dataset with zero dimensions, close it, and re-open. (See attached).
Can you send us an example of what you are trying that fails?
Please send it to the helpdesk: [hidden email]

Thanks!
-Barbara
[hidden email]

-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of Gerd Heber
Sent: Thursday, October 26, 2017 6:49 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

>> Rafal, this might be a bug.

> Is there any way to record/track this issue?

Not sure. Helpdesk will check. (It's still early morning here...)

G.

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

rafal.c (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Andrey Paramonov
In reply to this post by Ger van Diepen
26.10.2017 18:40, Ger van Diepen пишет:
> A choice consideration might be that the number of attributes has a limit.

Hmm, do I understand correctly that this limit is different for a group
and (an empty) dataset?

Best wishes,
Andrey Paramonov

> I think HDF5 should support empty N-dim datasets. Being empty also
> conveys information.
> Note that in any language it would be strange if you could not have an
> empty sequence.
>
> Ger
>
>>>> Gerd Heber <[hidden email]> 10/26/17 2:27 PM >>>
> Andrey, the experts should chime in on the storage trade-offs.
> (I imagine the null dataset is stored with compact layout, but in
> newer versions of the file format an empty group is very "compact"
> as well.) At the conceptual level, the two are very different and
> that might tip the balance in either direction.
> A group leaves the door open for other descendants.
> A dataset is a terminal node and, despite being empty, is nevertheless
> typed. (A null dataset has an element type!) Of course, you could
> "type" a group with additional attributes, but it would be a very
> different kind of variable.
>
> This might be a case where there're no right or wrong choices
> as far as HDF5 is concerned, and users will have to decide what
> makes it right for them.
>
> Best, G.
>
>
> -----Original Message-----
> From: Hdf-forum [mailto:[hidden email]] On Behalf
> Of ?????? ?????????
> Sent: Thursday, October 26, 2017 6:52 AM
> To: [hidden email]
> Subject: Re: [Hdf-forum] HDF5 library bug or feature?
>
> Hi Gerd!
>
> 26.10.2017 14:31, Gerd Heber пишет:
>  > Rafal, this might be a bug. However, if you don't intend to store data
>  > there's a cheaper and maybe less contorted option: You can create a
>  > dataset with a dataspace of class H5S_NULL.
>  > Think of that as a dataset whose dataspace is an empty set.
>  > (Likewise you'd use H5S_SCALAR for a singleton.)
>
> A common desire is to store non-uniform sparse data in HDF5 (consider
> XML attributes as example). I tend to use empty groups which only serve
> as container of attributes. What do you think: would empty datasets
> (H5S_NULL) with attributes be more efficient (introduce less overhead)?
>
> Best wishes,
> Andrey Paramonov
>
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

Rafal Lichwala
In reply to this post by bljones
Hi Barbara,

Thank you for an answer!
Yes, your code also works fine on my side, BUT...
when I try to set some custom H5P_FILE_ACCESS by call:

H5Pset_fapl_mpio(plist_id, comm, info)

(so running this example under OpenMPI - even for 1 process only)
it DOES NOT work any more.
In that case (see attached slightly modified your code) I've got the
following output:

2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fcreate: OK
2: H5Screate_simple: OK
2: H5Dcreate2: OK
2: H5Dclose: OK
2: H5Sclose: OK
2: H5Pclose: OK
2: H5Fclose: OK
2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fopen: OK
2: H5Dopen2 Failed!
2: H5Dget_space Failed!
2: H5Dclose Failed!
2: H5Sclose Failed!
2: H5Pclose: OK
2: H5Fclose: OK

HDF5 library also reports some error messages in that case:
        unable to open dataset
        unable to initialize file storage
        unable to mark dataspace as dirty
        unable to update layout info
        unable to write object header message
        unable to modify constant message

The most interesting thing is (which possibly means it's a bug), that
when in the same code I just change dims definition to:

    dims[0] = 1;
    dims[1] = 1;

everything WORKS FINE now! :-)

Best regards,
Rafal



W dniu 2017-10-26 o 17:53, Barbara Jones pisze:

>
> Hi Rafal,
>
> I am able to create a dataset with zero dimensions, close it, and re-open. (See attached).
> Can you send us an example of what you are trying that fails?
> Please send it to the helpdesk: [hidden email]
>
> Thanks!
> -Barbara
> [hidden email]
>
> -----Original Message-----
> From: Hdf-forum [mailto:[hidden email]] On Behalf Of Gerd Heber
> Sent: Thursday, October 26, 2017 6:49 AM
> To: HDF Users Discussion List
> Subject: Re: [Hdf-forum] HDF5 library bug or feature?
>
>>> Rafal, this might be a bug.
>
>> Is there any way to record/track this issue?
>
> Not sure. Helpdesk will check. (It's still early morning here...)
>
> G.
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

test.cpp (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 library bug or feature?

bljones
Hi Rafal,

Ahh! I found a bug report for this same issue (HDFFV-10001).
I added your information to the bug report.

Thanks!
-Barbara
[hidden email]

-----Original Message-----
From: Hdf-forum [mailto:[hidden email]] On Behalf Of Rafal Lichwala
Sent: Friday, October 27, 2017 4:06 AM
To: [hidden email]
Cc: HDF Helpdesk
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Barbara,

Thank you for an answer!
Yes, your code also works fine on my side, BUT...
when I try to set some custom H5P_FILE_ACCESS by call:

H5Pset_fapl_mpio(plist_id, comm, info)

(so running this example under OpenMPI - even for 1 process only) it DOES NOT work any more.
In that case (see attached slightly modified your code) I've got the following output:

2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fcreate: OK
2: H5Screate_simple: OK
2: H5Dcreate2: OK
2: H5Dclose: OK
2: H5Sclose: OK
2: H5Pclose: OK
2: H5Fclose: OK
2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fopen: OK
2: H5Dopen2 Failed!
2: H5Dget_space Failed!
2: H5Dclose Failed!
2: H5Sclose Failed!
2: H5Pclose: OK
2: H5Fclose: OK

HDF5 library also reports some error messages in that case:
        unable to open dataset
        unable to initialize file storage
        unable to mark dataspace as dirty
        unable to update layout info
        unable to write object header message
        unable to modify constant message

The most interesting thing is (which possibly means it's a bug), that when in the same code I just change dims definition to:

    dims[0] = 1;
    dims[1] = 1;

everything WORKS FINE now! :-)

Best regards,
Rafal



W dniu 2017-10-26 o 17:53, Barbara Jones pisze:

>
> Hi Rafal,
>
> I am able to create a dataset with zero dimensions, close it, and re-open. (See attached).
> Can you send us an example of what you are trying that fails?
> Please send it to the helpdesk: [hidden email]
>
> Thanks!
> -Barbara
> [hidden email]
>
> -----Original Message-----
> From: Hdf-forum [mailto:[hidden email]] On
> Behalf Of Gerd Heber
> Sent: Thursday, October 26, 2017 6:49 AM
> To: HDF Users Discussion List
> Subject: Re: [Hdf-forum] HDF5 library bug or feature?
>
>>> Rafal, this might be a bug.
>
>> Is there any way to record/track this issue?
>
> Not sure. Helpdesk will check. (It's still early morning here...)
>
> G.
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [hidden email]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
>

_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5