Add the processing of MBARI Long Range AUV data

[MBARIMike]

The development of this code base was informed by the long history of Matlab processing of Dorado-class AUV data and the STOQS processing of LRAUV data. Now it's time to bring the LRAUV data processing under the fold of this project. Here are a few reasons why:

• The STOQS processing uses Pandas series, auv-python uses the higher level Xarray library
• The existing LRAUV data processing is tied to the STOQS project, it makes more sense to have it here
• We'd like to compute the plankton proxies for the LRAUV data as well, it's easier to add that capability here
• We need to inter-compare Dorado-class and LRAUV (Tethys-class) data and using the same processing will help with that

This is a significant task which will require development on a long-lived branch.

[MBARIMike]

One thing different about the source of LRAUV data is that it is in NetCDF 4 format vs. Dorado's binary log file format. The STOQS processing uses the Python netCDF4 library to read data from groups in the LRAUV source files from the file system. Here's an exploration of different ways to read the source netCDF using xarray.

Reading a group (/CTD_Seabird) directly from the file system

dev-user@58f0a747367a:/app$ python
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> ds = xr.open_dataset(
    "/Volumes/LRAUV/pontus/missionlogs/2025/20250107_20250123/20250109T065618/202501090656_202501090656.nc4", 
    group="/CTD_Seabird")
>>> ds
<xarray.Dataset> Size: 4kB
Dimensions:                                            (loadControl_time: 0,
                                                        uart_time: 0,
                                                        baud_time: 0,
                                                        loadAtStartup_time: 0,
                                                        simulateHardware_time: 0,
                                                        maxPressBound_time: 0,
                                                        ...
                                                        bin_median_sea_water_temperature_time: 7,
                                                        bin_mean_sea_water_temperature_time: 7,
                                                        bin_standard_deviation_sea_water_temperature_time: 7,
                                                        bin_median_sea_water_salinity_time: 7,
                                                        bin_mean_sea_water_salinity_time: 7,
                                                        bin_standard_deviation_sea_water_salinity_time: 7)
Coordinates: (12/40)
  * loadControl_time                                   (loadControl_time) datetime64[ns] 0B ...
  * uart_time                                          (uart_time) datetime64[ns] 0B ...
  * baud_time                                          (baud_time) datetime64[ns] 0B ...
  * loadAtStartup_time                                 (loadAtStartup_time) datetime64[ns] 0B ...
  * simulateHardware_time                              (simulateHardware_time) datetime64[ns] 0B ...
  * maxPressBound_time                                 (maxPressBound_time) datetime64[ns] 0B ...
    ...                                                 ...
  * bin_median_sea_water_temperature_time              (bin_median_sea_water_temperature_time) datetime64[ns] 56B ...
  * bin_mean_sea_water_temperature_time                (bin_mean_sea_water_temperature_time) datetime64[ns] 56B ...
  * bin_standard_deviation_sea_water_temperature_time  (bin_standard_deviation_sea_water_temperature_time) datetime64[ns] 56B ...
  * bin_median_sea_water_salinity_time                 (bin_median_sea_water_salinity_time) datetime64[ns] 56B ...
  * bin_mean_sea_water_salinity_time                   (bin_mean_sea_water_salinity_time) datetime64[ns] 56B ...
  * bin_standard_deviation_sea_water_salinity_time     (bin_standard_deviation_sea_water_salinity_time) datetime64[ns] 56B ...
Data variables: (12/38)
    loadControl                                        (loadControl_time) <U1 0B ...
    uart                                               (uart_time) <U1 0B ...
    baud                                               (baud_time) float64 0B ...
    loadAtStartup                                      (loadAtStartup_time) uint8 0B ...
    simulateHardware                                   (simulateHardware_time) uint8 0B ...
    maxPressBound                                      (maxPressBound_time) float32 0B ...
    ...                                                 ...
    bin_median_sea_water_temperature                   (bin_median_sea_water_temperature_time) float32 28B ...
    bin_mean_sea_water_temperature                     (bin_mean_sea_water_temperature_time) float32 28B ...
    bin_standard_deviation_sea_water_temperature       (bin_standard_deviation_sea_water_temperature_time) float32 28B ...
    bin_median_sea_water_salinity                      (bin_median_sea_water_salinity_time) float32 28B ...
    bin_mean_sea_water_salinity                        (bin_mean_sea_water_salinity_time) float32 28B ...
    bin_standard_deviation_sea_water_salinity          (bin_standard_deviation_sea_water_salinity_time) float32 28B ...
>>> ds["sea_water_temperature"]
<xarray.DataArray 'sea_water_temperature' (sea_water_temperature_time: 33)> Size: 132B
array([10.092188, 10.092188, 10.092188, 10.092188, 10.092188, 10.092188,
       10.092188, 10.092188, 10.1     , 10.092188, 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     , 10.1     , 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     , 10.1     , 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     , 10.1     , 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     ], dtype=float32)
Coordinates:
  * sea_water_temperature_time  (sea_water_temperature_time) datetime64[ns] 264B ...
Attributes:
    units:          degC
    standard_name:  sea_water_temperature
>>> ds["sea_water_temperature"].values
array([10.092188, 10.092188, 10.092188, 10.092188, 10.092188, 10.092188,
       10.092188, 10.092188, 10.1     , 10.092188, 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     , 10.1     , 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     , 10.1     , 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     , 10.1     , 10.1     , 10.1     ,
       10.1     , 10.1     , 10.1     ], dtype=float32)
>>> 

Using the Hyrax server to serve NetCDF4 files containing groups

Setup server on elvis2 following these instructions with a site.conf file and started with:

docker_user@elvis2:~$ docker run --rm -h hyrax81 -p 81:8080 --volume ~/tmp/data:/usr/share/hyrax/data --volume ~/site.conf:/etc/bes/site.conf --name=hyrax81 opendap/hyrax:latest

For testing, some LRAUV files were copied to the tmp/data directory in docker_user's home directory. A site.conf file (from the instructions) was also put into the home directory so that netCDF4 files with groups can be served. Navigating to one of the LRAUV nc4 files shows that the groups appear at the bottom of this page: http://elvis2.shore.mbari.org:81/opendap/hyrax/data/lrauv/pontus/missionlogs/2025/20250107_20250123/20250109T065618/202501090656_202501090656.nc4.dmr.html. This is useful for exploring the data from a web browser, whereas the existing legacy Hyrax server on elvis cannot view the member variables of netCDF4 groups. Here's method I used to view the data for member variable /CTD_Seabird/sea_water_temperature. Navigate to the directory containing the .nc4 file: http://elvis2.shore.mbari.org:81/opendap/hyrax/data/lrauv/pontus/missionlogs/2025/20250107_20250123/20250109T065618/ and click on the viewers link in the rightmost column. Select the DAP2 Service and check the variable you want, scroll back to the top of the page, and click the "Get as ASCII" button. This gives the URL: http://elvis2.shore.mbari.org:81/opendap/hyrax/data/lrauv/pontus/missionlogs/2025/20250107_20250123/20250109T065618/202501090656_202501090656.nc4.ascii?/CTD_Seabird/sea_water_temperature%5B0:1:32%5D which shows the data values:

Dataset: 202501090656_202501090656.nc4
/CTD_Seabird/sea_water_temperature, 10.0922, 10.0922, 10.0922, 10.0922, 10.0922, 10.0922, 10.0922, 10.0922, 10.1, 10.0922, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1, 10.1

Reading from Hyrax into xarray

Xarray's open_datatree() method should be able to read these .nc4 data into Python preserving the hierarchical structure, but errors happen for both file and Hyrax access:

>>> ds = xr.open_datatree("/Volumes/LRAUV/pontus/missionlogs/2025/20250107_20250123/20250109T065618/202501090656_202501090656.nc4")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/datatree.py", line 166, in check_alignment
    align(node_ds, parent_ds, join="exact", copy=False)
  File "/usr/local/lib/python3.10/dist-packages/xarray/structure/alignment.py", line 939, in align
    aligner.align()
  File "/usr/local/lib/python3.10/dist-packages/xarray/structure/alignment.py", line 631, in align
    self.align_indexes()
  File "/usr/local/lib/python3.10/dist-packages/xarray/structure/alignment.py", line 452, in align_indexes
    raise AlignmentError(
xarray.structure.alignment.AlignmentError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensions): 'depth_time' ('depth_time',)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/xarray/backends/api.py", line 1132, in open_datatree
    backend_tree = backend.open_datatree(
  File "/usr/local/lib/python3.10/dist-packages/xarray/backends/netCDF4_.py", line 733, in open_datatree
    return datatree_from_dict_with_io_cleanup(groups_dict)
  File "/usr/local/lib/python3.10/dist-packages/xarray/backends/common.py", line 235, in datatree_from_dict_with_io_cleanup
    tree = DataTree.from_dict(groups_dict)
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/datatree.py", line 1219, in from_dict
    obj._set_item(
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/treenode.py", line 650, in _set_item
    current_node._set(name, item)
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/datatree.py", line 965, in _set
    new_node._set_parent(new_parent=self, child_name=key)
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/treenode.py", line 115, in _set_parent
    self._attach(new_parent, child_name)
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/treenode.py", line 152, in _attach
    self._pre_attach(parent, child_name)
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/datatree.py", line 549, in _pre_attach
    check_alignment(path, node_ds, parent_ds, self.children)
  File "/usr/local/lib/python3.10/dist-packages/xarray/core/datatree.py", line 170, in check_alignment
    raise ValueError(
ValueError: group '/Dock' is not aligned with its parents:

I suspect that differences between shared coordinate/dimension names in the root and groups are causing this alignment problem. So it seems that xarray can read specific groups as Datasets, but using xarray for anything else with the LRAUV .nc4 files is problematic. The netCDF4 python package will need to be used for those operations.

[MBARIMike]

So the bottom line is that we'll use a combination of xarray and netCDF4 to open and operate on the LRAUV .nc4 data. In production the software will need write access to the LRAUV share, so we'll have file access anyway and do not need web access (via Hyrax) to the data.

[MBARIMike]

On Monday 6 October 2025 the elvis-services docker stack took over the dods.mbari.org and search.mbari.org official opendap and thredds data services. Therefore this is now the official permanent URL for the sample LRAUV .nc4 data:

https://dods.mbari.org/opendap/data/lrauv/pontus/missionlogs/2025/20250107_20250123/20250109T065618/202501090656_202501090656.nc4.dmr.html

Though this doesn't really apply as we'll need to read .nc4 files from the filesystem rather than from opendap.

[MBARIMike]

Started long-lived LRAUV development branch. As I get more frustrated with dealing with data quality edge cases in the STOQS lrauvNc4ToNetcdf.py code, I'd like to get started on this sooner rather than later.

[MBARIMike]

Frustrations with QCing some of the brizo data from the October 2025 CANON deployments have caused me to accelerate moving the LRAUV processing to this code base. PRs are coming...

[MBARIMike]

Had a long conversation with Claude Sonnet 4 about improving all the different directory Paths that need to be set for the different classes of vehicles. Original data are copied from different locations for dorado, i2map, and LRAUV data. They are copied to a local directory Path (that also varies), and then copied to archive locations that are different. If we can tease out some commonalities and set some patterns for Path construction we should be able to build a FileManager or VehicleConfig class for central management and remove existing Path settings that are sprinkled throughout the code base. This will be needed for reusing the align.py, resample.py, and archive.py steps for the LRAUV data.

[MBARIMike]

I'm rethinking the VehicleConfig/FileManager approach... It adds some complication and requires rewriting a lot of Dorado processing code that currently works. Instead, I'm prototyping the idea of using "mission" to identify Dorado data and "log_file" to identify LRAUV data and using if/elif to branch the processing in appropriate places.

[MBARIMike]

Now that the base level QC sub-issue has been solved we can move forward on the combine.py script.