Re: Handling incomplete columns for integer datatypes
Daniel, unless "magic values" are accompanied by unambiguous metadata the
NaN/fill-value approach is risky and limits portability.
A better approach might be to maintain a "shadowing" dataset of a suitable
bitfield type whose values would indicate the validity of a dataset element
or fields in a compound, etc. By using compression on the shadowing dataset
the storage overhead should be negligible. Of course, if the shadowed dataset
ever gets updated and elements or fields change their status between
'valid' and 'N/A', both datasets must be updated and kept in sync.
Depending on how elaborate you want this to be, you could decorate the shadowed dataset
with a "MASK" attribute whose value is an object reference to the shadowing dataset.
Alternatively, if the shadowed dataset is linked to exactly one group and there is no potential for
name conflicts, you could have a convention that lets you derive the name of the shadowing dataset
from the link name of the shadowed dataset.
From: Hdf-forum <[hidden email]> on behalf of Daniel Rimmelspacher <[hidden email]>
Sent: Thursday, January 12, 2017 6:15:32 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Handling incomplete columns for integer datatypes
I encounter some conceptual problems, when trying to store incomplete columns for integer datatypes.