Support speed and NMEA heading from Simrad .raw files

[gavinmacaulay]

This PR addresses issue #1346

• Add speed over ground to the EchoData object (obtained from NMEA RMC and VTG messages),
• Add heading data to the EchoData object for EK60 files and EK80 files (if heading is not available from MRU datagrams). Obtained from NMEA HDT messages,
• Add tests for the new code. The test for EK80 heading from HDT uses a .raw file (D20260613-T230914.zip) that needs to be added to the test assets in an ek80/heading directory and is used by test_convert_ek80::test_parse_NMEA_heading .

Notes:

• An existing comment in convert/set_groups_base.py for a function this PR modifies says "move this to be part of parser as it is not a 'set' operation". This has not been done as it would muddy the changes in this PR,
• The new speed over ground and heading variables have their own time coordinates,
• The Provide vessel speed over ground when available #1346 issue was only about speed, but while adding that I noticed that heading was missing for raw files without MRU datagrams so included that.

[gavinmacaulay]

The unit tests look to all be failing due to not finding the extra .raw file that needs to be added to the test assets.

[LOCEANlloydizard]

Hi @gavinmacaulay, thx for the PR! i'll had the data first thing tomorrow and will then re-trigger the set of test! cheers!

[gavinmacaulay]

@LOCEANlloydizard - just pushed a new set of changes that should address most of the test failures not related to that new data file

[LOCEANlloydizard]

@gavinmacaulay i've added the new test data file as a separate release asset (ek80_heading.zip) in #1704
I did it this way because ek80.zip is already quite large (~1 GB), so I thought it would be cleaner to avoid rebuilding and re-uploading it every time we need to add a new test file..

So the PR needs small changes:

• add an ek80_heading_path fixture in the EK80 test file

@pytest.fixture
def ek80_heading_path(test_path):
    return test_path["EK80_HEADING"]

• update test_parse_NMEA_heading to use ek80_heading_path

raw_file=ek80_heading_path / "heading" / "D20260613-T230914.raw"

let me know if something doesnt work!

[LOCEANlloydizard]

@gavinmacaulay I've merged #1704 into main, so you can(/need) pull the latest changes! cheers!

[gavinmacaulay]

@LOCEANlloydizard Thanks for adding the extra raw file. Tests under Windows now all pass but those under Ubuntu don't.

It looks to be because when ek80_heading.zip is unzipped the raw file ends up in ek80_heading\heading under Windows (as expected) but in ek80/heading under Ubuntu (not expected) and the test fails because it can't find the raw file.

I suspect (am not 100% sure) that the directory structure in the .zip file needs to be ek80_heading/heading/D...raw instead of ek80/heading/D...raw.

[gavinmacaulay]

Similar problem still:) On windows the raw file ends up in ek80_heading/ but on ubuntu it is in ek80_heading/heading/

[LOCEANlloydizard]

@gavinmacaulay i've updated the ZIP file to match the actual test path (ek80_heading_path / "D20260613-T230914.raw")! Yes, I ran into this issue before between Windows and Ubuntu... and I think the flattening function may need a bit more investigation. In any case, it passes CI now!

By the way, is this PR ready for review, or is it still a draft? Thanks!

[gavinmacaulay]

@LOCEANlloydizard Ready for review now (was just being careful and waiting for all checks to properly pass).

[LOCEANlloydizard]

did not review the code, just putting this as a note for later: how are duplicate timestamps handled for these new variables? e.g. #1627

[gavinmacaulay]

Duplicate timestamps are removed in set_groups_base.py:_extract_selected_NMEA() (lines 227-233). This applies to the existing lat/lon parsing as well as the added heading and speed parsing. The first of any duplicated timestamps are removed.

I've just now noticed #1478 (comment) and note that removing duplicate timestamps for lat/lon resolves that problem in a different, perhaps unwanted, way. And a similar problem may occur with the speed, which currently reads from RMC and VTG NMEA messages.

[LOCEANlloydizard]

@gavinmacaulay could I bother you to check/fill in these fields based on what you know (particularly the source)? we’re trying to document the different test assets we use (see #1684). I added the information I could infer from the PR (i'll add it then to the other PR1684):

ek80_heading.zip:
documented_checksum: 141b9a482af22483097de6f928db4d7b0494d4f7a153f88cffd24c070e11fadb
instrument: EK80
description: EK80 raw data containing NMEA HDT messages, used to test heading parsing when heading is unavailable from MRU datagrams.
source: null
contributor: gavinmacaulay
references: []
notes: Used by test_convert_ek80::test_parse_NMEA_heading
files: {D20260613-T230914.raw}

[gavinmacaulay]

Extra data and edits:

ek80_heading.zip:
documented_checksum: 141b9a482af22483097de6f928db4d7b0494d4f7a153f88cffd24c070e11fadb
instrument: ES80
description: ES80 raw data containing NMEA HDT messages used to test heading parsing when heading is unavailable from MRU datagrams.
source: Collected during a sphere calibration of a 38 kHz ES80 on a fishing vessel (name deliberately omitted)
contributor: gavinmacaulay
references: []
notes: Used by test_convert_ek80::test_parse_NMEA_heading files: {D20260613-T230914.raw}

[ctuguinay]

@gavinmacaulay Let me know whenever this is ready for review and I can take another look at it. Thanks!

[gavinmacaulay]

Thanks - it's ready now

[ctuguinay]

Thanks for this PR Gavin! I just have a few small comments and a potentially bigger comment that I noticed about the time{num} coordinate names.

I was wondering what the appropriate naming convention should be now that Echopype will be including times from more NMEA variables. Also the additional variables that can be added with update_platform can potentially get us to really high numbers, but this is rarely used. I think it may potentially be useful to rename the platform times to be descriptive like {NMEA_SENTENCE}_heading_time. Let me know what y'all think too @leewujung @LOCEANlloydizard.

This could make it easier if we want to add location, heading, and speed from multiple NMEA sentences or alongside MRU, although I don't personally know of any cases where we'll need for example both GGA and GGL or both RMC and VGT. It may also be easier cognitively to just have the purpose of each time in the variable name.

[ctuguinay]

Do you know of any cases where there is heading received from both MRU and NMEA? I'm wondering if you can just save those as separate variables.

[ctuguinay]

And I guess the same applies to the NMEA speed sentences

[gavinmacaulay]

I see many vessel-based systems setup to receive both MRU0 datagrams and heading (from a separate gyrocompass via the HDT message), although I found none in the echopype test dataset and is why I added one 😄 My thinking was that from the user's point of view they shouldn't need to (initially) think too much about where the heading came from and just have it be in a heading variable.

For the NMEA speed, many vessels have it in both RMC and VTG messages, but then some vessels don't have RMC and come don't have VTG and that's a detail that user's don't really need to worry about if they both end up in a speed_over_ground variable.

[gavinmacaulay]

The time variable naming convention of time{int} is recommended in the SONAR-netCDF4 convention, although the way that xarray and netCDF4 variables list the relevant coordinate dimensions means that the time variable name could be anything, especially if users write their code to take advantage of that rather than hard-coding coordinate variable names.

One of the reasons for having generic time variable names was so that they could be used for multiple data types (e.g., RMC messages provide lat, long, speed, and course) and for other types of echosounders that store only per-ping interpolated lat, long, heading, speed, etc, in their files.

I think these sorts of choices also determine how much of the underlying messiness of the NMEA datastream is exposed to the users. Consolidating multiple sources of useful data into consistently-named variables (heading instead of MRU_heading and HDT_heading; speed_over_ground instead of RMC_speed, VTG_speed, MRU_speed) makes the use of echopype data structures simpler and for most users would probably be preferred.

One thing I didn't do with the speed or heading (but could readily do) was add a matching sentence_type variable, as is done for latitude and longitude. Having that would allow users to filter on the source of the heading and speed (or the full NMEA messages are also available in the EchoData object for independent extraction).