The DAP4 specification, the TDS, and the Hyrax do not agree on endianness

[ndp-opendap]

TDS implementation of DAP4 appears to be incorrectly setting the endianness in the chunk type fields in data response.

It appears the specification has conflicting information about how to determine endianness.

In the DMR attributes there is an optional DAP Attribute _DAP4_Little_Endian that can be used to assert the endianness of the response. Yet, each Chunk header contains a chunk-type that also can assert the endianness (see Chunk Type Encoding table).

Which also states the:

"The Endian flag must be set only in the first Data chunk. It applies to the whole response. If set in any subsequent chunk type, it will be ignored."

And the TDS is setting the endianness in the ChunkType of the very first chink (The one with the DMR)

This is kind of a mess - the specification has too many ways to determine this, and even then it's not crystal clear how it should be accomplished.

I think Hyrax, the TDS, and the DAP4 specification all need an endianness tuneup.

[Mikejmnez]

TDS implementation of DAP4 appears to be incorrectly setting the endianness in the chunk type fields in data response.

From our last meeting we found that endianness in the TDS was being set correctly.

And the TDS is setting the endianness in the ChunkType of the very first chunk (The one with the DMR)

This is true, and endianness is never again defined in the DAP response for the TDS. With Hyrax, endianness is defined also ChunkType of the very first chunk (the one containing the DMR), but endianness is again defined for any subsequent data stream.

A follow up question is:

Is endianness supposed to be homogeneous, or can a dataset have different variables some with differing endianness?

Right now, for simplicity, pydap only reads the chunktype of the first chunk (the one containing the DMR) and sets the endianness of the entire dataset once, the TDS approach, and ignores any subsequent endianness declaration in the rest of the dap response. Depending on the answer to the question above, pydap will adapt.

[DennisHeimbigner]

Endianness is supposed to be homogeneous.
We could in theory set it on the variable level, but if memory serves,
we (the DAP4 committee) decided against it.
In any case, setting the endianness on the chunk level would be incorrect since there is no
guarantee that the chunk boundary and the variable boundary coincide.

but endianness is again defined for any subsequent data stream.
As for this, the endianness of the dmr is meaningless because it is XML text.
So setting endianness on the initial chunk inherently implies that it is being
set on any attached data stream.

[DennisHeimbigner]

To follow up, this part of the spec:

| *2* | The data in this response is encoded using Big-Endian (i.e. network byte order) | The data in this response is encoded using Little-Endian

strongly implies that the endian bit in the first chunk is controlling. But we should make it expliciit.

[ndp-opendap]

To follow up, this part of the spec:

| *2* | The data in this response is encoded using Big-Endian (i.e. network byte order) | The data in this response is encoded using Little-Endian

strongly implies that the endian bit in the first chunk is controlling. But we should make it expliciit.

That was my understanding as well. The fact that Hyrax transmits the endianness with every single chunk is a bug in libdap