Fqpr¶

class HSTB.kluster.fqpr_generation.Fqpr(multibeam=None, motion_latency=0.0, address=None, show_progress=True, parallel_write=True, debug=False)¶

Bases: ZarrBackend

Fully qualified ping record: contains all records built from the raw MBES file and supporting data files. Built around the BatchRead engine which supplies the multibeam data conversion.

Fqpr processing is built using the method detailed in “Application of Surface Sound Speed Measurements in Post-processing for Multi-Sector Multibeam Echosounders” by J.D. Beaudoin and John Hughes Clarke

Processing consists of five main steps:
Fqpr.read_from_source - run xarray_conversion to get xarray Datasets for ping/attitude/navigation records
Fqpr.get_orientation_vectors - Build transmit/receive unit vectors rotated by attitude and mounting angle
Fqpr.get_beam_pointing_vectors - Correct sonar relative beam angles by orientation to get corrected
beam pointing vectors and azimuths
Fqpr.sv_correct - Use the corrected beam vectors, travel time and sound velocity profile to ray trace the beams
Fqpr.georef_xyz - Using pyproj, transform the vessel relative offsets to georeferenced xyz

See fqpr_convenience.convert_multibeam, process_multibeam and perform_all_processing for example use.

Parameters

multibeam (Optional[BatchRead]) – instance of xarray_conversion BatchRead class
motion_latency (float) – optional motion latency adjustment
address (Optional[str]) – passed to dask_find_or_start_client to setup dask cluster
show_progress (bool) – If true, uses dask.distributed.progress. Disabled for GUI, as it generates too much text
parallel_write (bool) – if True, will write in parallel to disk

Attributes Summary

`has_sbet`	True if an SBET has been imported into this FQPR instance
`input_datum`	The basic input datum of the converted multibeam data.
`last_operation_date`	Get the datetime of the last operation performed on this fqpr instance
`max_time`	Get the nicely formatted time in UTC for the end time of this fqpr object
`min_time`	Get the nicely formatted time in UTC for the start time of this fqpr object
`multibeam_extension`	The file extension of the multibeam file(s) used in this fqpr object
`number_of_heads`	Get the number of sonar heads for the sonar in this FQPR instance
`number_of_pings`	Get the number of pings for the sonar in this FQPR instance
`sbet_navigation`	Return the sbet navigation for the first sonar head.
`sonar_model`	Get the sonar type from the ping record
`status`	Get the processing status of the Fqpr
`total_distance_meters`

Methods Summary

`calc_max_var`([varname])	For given variable, return the maximum value found across all sectors
`calc_min_var`([varname])	For given variable, return the minimum value found across all sectors
`calculate_total_uncertainty`([subset_time, ...])	Use the tpu module to calculate total horizontal uncertainty and total vertical uncertainty for each sounding.
`close`([close_dask])	Must forcibly close the logging handlers to allow the data written to disk to be moved or deleted.
`construct_crs`([epsg, datum, projected, vert_ref])	Build pyproj crs from several different options, used with georef_across_along_depth.
`copy`()	Return a copy of this Fqpr instance.
`determine_altitude_corr`(alt, raw_att, ...)	We use the nav as provided by the POSMV.
`determine_induced_heave`(ra, hve, raw_att, ...)	From Kongsberg datagram doc: Note that heave is displayed and logged as positive downwards (the sign is changed) including roll and pitch induced lever arm translation to the system’s transmit transducer.
`export_dataset`(dataset_name, dest_path)	Run the export module to export each variable in the given dataset to one csv, writing to the provided path, see export.export_dataset_to_csv
`export_lines_to_file`([linenames, ...])	Run the export module to export only the data belonging to the given lines to file, see export.export_lines_to_file
`export_pings_to_file`([output_directory, ...])	Run the export module to export point cloud relevant data to file, see export.export_pings_to_file
`export_soundings_to_file`(datablock[, ...])	Run the export module to export given soundings to file, see export.export_soundings_to_file
`export_tracklines_to_file`([linenames, ...])	Run the export module to export the navigation to vector file, see export.export_tracklines_to_file
`export_variable`(dataset_name, var_name, ...)	Run the export module to export the given variable to csv, writing to the provided path, see export.export_variable_to_csv
`fix_indices`()	Resolve any issues with the time index that might have come up during conversion.
`generate_starter_orientation_vectors`([txrx, ...])	Take in identifiers to find the correct xyzrph entry, and use the heading value to figure out if the transmitter/receiver (tx/rx) is oriented backwards.
`georef_xyz`([subset_time, prefer_pp_nav, ...])	Use the raw attitude/navigation to transform the vessel relative along/across/down offsets to georeferenced soundings.
`get_beam_pointing_vectors`([subset_time, ...])	Beam pointing vector is the beam specific vector that arises from the intersection of the tx ping and rx cone of sensitivity.
`get_cluster_params`()	Attempt to figure out what the chunk size and number of chunks at a time parameters should be given the dims of the dataset.
`get_orientation_vectors`([subset_time, ...])	Using attitude angles, mounting angles, build the tx/rx vectors that represent the orientation of the tx/rx at time of transmit/receive.
`get_variable_by_filter`(var_name[, ...])	ping_filter is set upon selecting points in 2d/3d in Kluster.
`import_post_processed_navigation`(navfiles[, ...])	Load from post processed navigation files (currently just SBET and SMRMSG) to get lat/lon/altitude as well as 3d error for further processing.
`import_sound_velocity_files`(src[, ...])	Load to self.cast_files the file paths to the sv casts of interest.
`initial_att_interpolation`()	We provide as an optional step in self.get_orientation_vectors (or run separately) the ability to interpolate the raw attitude and navigation to the ping record times and save these records to disk.
`initialize_intermediate_data`(sec_ident, ky)	self.intermediate_dat is the storage for all the futures generated by the main processes (get_orientation_vectors, get_beam_pointing_vectors, etc.).
`initialize_log`()	Initialize the fqpr logger using the multibeam logfile attribute.
`interp_to_ping_record`(sources[, attributes])	Take in a dataset that is not at ping time (raw navigation, attitude, etc.) and interpolate it to ping time and save it to the raw ping datasets.
`intersects`(min_y, max_y, min_x, max_x[, ...])	Check if the provided extents intersect with this fqpr instance.
`is_processed`([in_depth])	Kluster maintains two records for processing status.
`line_attributes`(line_name)	Attributes by line are added after conversion to the ping attribution.
`line_is_processed`(line_name)	If line is processed, the TVU will not be all NaN in the middle of the line.
`overwrite_raw_navigation`(navfiles, ...[, ...])	Load from raw navigation files (currently just POS MV .000) to get lat/lon/altitude.
`process_backscatter`([subset_time, ...])	Correct the raw reflectivity for system and environmental effects, leaving the backscatter target strength.
`read_from_source`([build_offsets, skip_dask])	Activate rawdat object's appropriate read class
`remove_post_processed_navigation`()	import_post_processed_navigation will write navigation and navigation related attributes to the Fqpr instance.
`remove_profile`(profile_name)	Sound velocity profiles are stored in the Fqpr datastore as attributes with the 'profile_timestamp' format, ex: 'profile_1503411780'.
`restore_subset`()	Restores the original data if subset_by_time has been run.
`return_additional_xyz_offsets`(ra, prefixes, ...)	Apply tx to reference point offset to beams.
`return_all_profiles`()	convenience for xarray_conversion.BatchRead.return_all_profiles
`return_applicable_casts`([method])	When we check for sound velocity correct actions, we look to see if any new sv profiles imported into the fqpr instance are applicable, by running the chosen method (default is cast nearest in time to the ping chunk).
`return_cast_dict`()	Return a dictionary object combining the profile data and the attribution for each cast
`return_cast_idx_nearestindistance`(idx_by_chunk)	Need to find the cast associated with each chunk of data.
`return_cast_idx_nearestindistance_fourhours`(...)	Need to find the cast associated with each chunk of data.
`return_cast_idx_nearestintime`(idx_by_chunk)	Need to find the cast associated with each chunk of data.
`return_cast_idx_nearestintime_fourhours`(...)	Need to find the cast associated with each chunk of data.
`return_chunk_indices`(idx_mask, pings_per_chunk)	Use self.get_cluster_params to figure out how big the chunks should be according to the cluster memory capacity and pass that number in here as pings_per_chunk.
`return_line_dict`([line_names, ping_times])	Return all the lines with associated start and stop times for all sectors in the fqpr dataset.
`return_line_time`(line_name)	Return the start and end time for the given line name
`return_line_xyzrph`(line_name)	Return only the relevant xyzrph (kluster vessel config data) entries for the given line name.
`return_lines_for_times`(times)	Given the 1d array of times (utc seconds), return a same size object array with the string value of the line file name that matches the time.
`return_navigation`([start_time, end_time, ...])	Return the navigation from the multibeam data for the first sonar head.
`return_next_action`([new_vertical_reference, ...])	Determine the next action to take, building the arguments for the fqpr_convenience.process_multibeam function.
`return_next_unprocessed_line`()	Return the next unprocessed line in this container, see line_is_processed
`return_processing_dashboard`()	Return the necessary data for a dashboard like view of this fqpr instance.
`return_rounded_frequency`()	Returns the frequency rounded to match the freq settings commonly given with sonar manufacturer settings.
`return_runtime_idx_nearestintime`(idx_by_chunk)	Find the runtime parameter that is nearest to each chunk in the provided chunk arrays
`return_soundings_in_polygon`(polygon[, ...])	Using provided coordinates (in either horizontal_crs projected or geographic coordinates), return the soundings and sounding attributes for all soundings within the coordinates, see subset module.
`return_total_pings`([min_time, max_time])	Get the total ping count, optionally within the provided mintime maxtime range
`return_total_soundings`([min_time, max_time])	Return the number of soundings in all systems within this fqpr instance, optionally within the provided mintime maxtime range
`return_unique_mode`()	Finds the unique mode entries in raw_ping Datasets.
`run_filter`(filtername, *args[, ...])	Run the filter module with the provided filtername, will match the filename of the filter python file.
`set_filter_by_polygon`(polygon[, geographic])	Alternative way to set the ping_filter attribute which can be used with set_variable_by_filter get_variable_by_filter, see subset module.
`set_variable_by_filter`([var_name, newval, ...])	ping_filter is set upon selecting points in 2d/3d in Kluster.
`set_vertical_reference`(vert_ref)	Set the Fqpr instance vertical reference.
`subset_by_lines`(line_names)	Use subset module to trim the fqpr instance to the given lines
`subset_by_time`([mintime, maxtime])	Use subset module to trim the fqpr instance to the given time range
`subset_by_time_and_beam`(subset_time, subset_beam)	Use subset module to subset by time,beam provided, returns a 1d boolean mask for each sonar head
`subset_by_times`(time_segments)	Use subset module to trim the fqpr instance to the given time ranges
`subset_variables`(variable_selection[, ...])	Take specific variable names and return just those variables in a new xarray dataset, see subset module.
`subset_variables_by_line`(variable_selection)	Apply subset_variables to get the data split up into lines for the variable_selection provided, see subset module
`sv_correct`([add_cast_files, ...])	Apply sv cast/surface sound speed to raytrace.
`write_attribute_to_ping_records`(attr_dict)	Convenience method that allows you to write the provided attribute dictionary to each ping dataset and change the currently loaded instance as well
`write_intermediate_futs_to_zarr`(mode, ...[, ...])	Flush some of the intermediate data that was mapped to the cluster (and lives in futures objects) to disk, puts it in the multibeam, as the time dimension should be the same.

Attributes Documentation

has_sbet¶

True if an SBET has been imported into this FQPR instance

Returns: If SBET has been imported, return True
Return type: bool

input_datum¶: The basic input datum of the converted multibeam data. Will be ignored in processing if an sbet_datum exists, as sbet navigation and altitude are used by default if they exist unless you explicitly request non-sbet processing.

last_operation_date¶

Get the datetime of the last operation performed on this fqpr instance

Returns: datetime object of the last operation performed on this fqpr instance
Return type: datetime

max_time¶

Get the nicely formatted time in UTC for the end time of this fqpr object

Returns: the formatted string representation of the maximum time of this dataset in UTC
Return type: str

min_time¶

Get the nicely formatted time in UTC for the start time of this fqpr object

Returns: the formatted string representation of the minimum time of this dataset in UTC
Return type: str

multibeam_extension¶: The file extension of the multibeam file(s) used in this fqpr object

number_of_heads¶

Get the number of sonar heads for the sonar in this FQPR instance

Returns: number of sonar heads
Return type: int

number_of_pings¶

Get the number of pings for the sonar in this FQPR instance

Returns: number of sonar heads
Return type: int

sbet_navigation¶: Return the sbet navigation for the first sonar head. Can assume that all sonar heads have basically the same navigation

sonar_model¶

Get the sonar type from the ping record

Returns: the sonar model string
Return type: str

status¶

Get the processing status of the Fqpr

Returns: the processing status of the Fqpr object
Return type: str

total_distance_meters¶

Methods Documentation

calc_max_var(varname='depthoffset')¶

For given variable, return the maximum value found across all sectors

Parameters: varname (str) – name of the variable you are interested in
Returns: maximum value across all sectors
Return type: float

calc_min_var(varname='depthoffset')¶

For given variable, return the minimum value found across all sectors

Parameters: varname (str) – name of the variable you are interested in
Returns: minimum value across all sectors
Return type: float

calculate_total_uncertainty(subset_time=None, dump_data=True)¶

Use the tpu module to calculate total horizontal uncertainty and total vertical uncertainty for each sounding. See tpu.py for more information

To process only a section of the dataset, use subset_time.
ex: subset_time=[1531317999, 1531321000] means only process times that are from 1531317999 to 1531321000
ex: subset_time=[[1531317999, 1531318885], [1531318886, 1531321000]] means only process times that are
from either 1531317999 to 1531318885 or 1531318886 to 1531321000

Parameters

subset_time (Optional[list]) – List of unix timestamps in seconds, used as ranges for times that you want to process.
dump_data (bool) – if True dump the futures to the multibeam datastore. Set this to false for an entirely in memory workflow

close(close_dask=True)¶: Must forcibly close the logging handlers to allow the data written to disk to be moved or deleted.

construct_crs(epsg=None, datum='WGS84', projected=True, vert_ref=None)¶

Build pyproj crs from several different options, used with georef_across_along_depth.

Optionally set the vertical reference as well, using set_vertical_reference. This isn’t tied to the pyproj instance, so it can be done separately.

Options include: - epsg mode: set epsg to string identifier - geographic mode: set ellips to string identifier and projected to False - projected mode: set ellips to sting identifier and projected to True. Will autodetermine zone

Parameters

epsg (Optional[str]) – optional, epsg code
datum (str) – datum identifier i.e. ‘WGS84’ or ‘NAD83’
projected (bool) – if True uses utm zone projected coordinates
vert_ref (Optional[str]) – vertical reference for the survey, one of [‘ellipse’, ‘waterline’, ‘NOAA MLLW’, ‘NOAA MHW’, ‘Aviso MLLW’]

Returns

If true, the CRS was successfully constructed and was different from the original

Return type

bool

copy()¶

Return a copy of this Fqpr instance. The xarray datasets will be distinct, so you can subset them without affecting this instance.

Returns: copy of the current Fqpr object
Return type: Fqpr

determine_altitude_corr(alt, raw_att, tx_tstmp_idx, prefixes, timestmp)¶

We use the nav as provided by the POSMV. This will be at the reference point designated by the POSMV. As we assume that your RP is either the TX or the IMU, if there is a lever arm between TX and RP, there is induced heave in the altitude equal to the attitude-rotated TX lever arm.

Generate that time series attitude adjustment and add it to the altitude record.

Parameters

alt (DataArray) – altitude at ping time
raw_att (Dataset) – raw attitude Dataset including roll, pitch, yaw
tx_tstmp_idx (DataArray) – ping time index
prefixes (str) – prefix identifier for the tx/rx, will vary for dual head systems
timestmp (str) – timestamp for the appropriate xyzrph record

Returns

navigation at ping time (latitude, longitude, altitude) with altitude correction

Return type

xr.Dataset

determine_induced_heave(ra, hve, raw_att, tx_tstmp_idx, prefixes, timestmp)¶

From Kongsberg datagram doc: Note that heave is displayed and logged as positive downwards (the sign is changed) including roll and pitch induced lever arm translation to the system’s transmit transducer.

Here we use the primary to secondary lever arm to build induced heave seen at the secondary system. As heave is reported at the tx of the primary system. This will return all zeros for induced heave in instances where:

system is not a dual head system
system is the primary system of a dual head system

Parameters

ra (Dataset) – raw_ping dataset for this sector/freq/serial number
hve (DataArray) – heave record at ping time
raw_att (Dataset) – raw attitude Dataset including roll, pitch, yaw
tx_tstmp_idx (DataArray) – ping time index
prefixes (str) – prefix identifier for the tx/rx, will vary for dual head systems
timestmp (str) – timestamp for the appropriate xyzrph record

Returns

induced heave (z) value for each ping time

Return type

xr.DataArray

export_dataset(dataset_name, dest_path)¶: Run the export module to export each variable in the given dataset to one csv, writing to the provided path, see export.export_dataset_to_csv

export_lines_to_file(linenames=None, output_directory=None, file_format='csv', csv_delimiter=' ', filter_by_detection=True, format_type='xyz', z_pos_down=True, export_by_identifiers=True)¶: Run the export module to export only the data belonging to the given lines to file, see export.export_lines_to_file

export_pings_to_file(output_directory=None, file_format='csv', csv_delimiter=' ', filter_by_detection=True, format_type='xyz', z_pos_down=True, export_by_identifiers=True)¶: Run the export module to export point cloud relevant data to file, see export.export_pings_to_file

export_soundings_to_file(datablock, output_directory=None, file_format='csv', csv_delimiter=' ', filter_by_detection=True, format_type='xyz', z_pos_down=True)¶: Run the export module to export given soundings to file, see export.export_soundings_to_file

export_tracklines_to_file(linenames=None, output_file=None, file_format='GPKG')¶: Run the export module to export the navigation to vector file, see export.export_tracklines_to_file

export_variable(dataset_name, var_name, dest_path, reduce_method=None, zero_centered=False)¶: Run the export module to export the given variable to csv, writing to the provided path, see export.export_variable_to_csv

fix_indices()¶: Resolve any issues with the time index that might have come up during conversion. This method will be occasionally run when issues arise with the index not being monotonic increasing or containing duplicate values.

generate_starter_orientation_vectors(txrx=None, tstmp=None)¶

Take in identifiers to find the correct xyzrph entry, and use the heading value to figure out if the transmitter/receiver (tx/rx) is oriented backwards. Otherwise return ideal vectors for representation of the tx/rx.

Parameters

txrx (Optional[list]) – transmit/receive identifiers for xyzrph dict ([‘tx’, ‘rx’])
tstmp (Optional[float]) – timestamp for the appropriate xyzrph entry

georef_xyz(subset_time=None, prefer_pp_nav=True, dump_data=True, vdatum_directory=None)¶

Use the raw attitude/navigation to transform the vessel relative along/across/down offsets to georeferenced soundings. Will support transformation to geographic and projected coordinate systems and with a vertical reference that you select.

If uncertainty is included in the source data, will calculate the unc based on depth.

First does a forward transformation using the geoid provided in horizontal_crs Then does a transformation from geographic to projected, if that is included in horizontal_crs

Uses pyproj to do all transformations. User must run self.construct_crs first to establish the destination datum and ellipsoid.

Sends the data and calculations to the cluster, receive futures objects back. Use the dump_data/delete_futs to interact with the futures object.

To process only a section of the dataset, use subset_time.
ex: subset_time=[1531317999, 1531321000] means only process times that are from 1531317999 to 1531321000
ex: subset_time=[[1531317999, 1531318885], [1531318886, 1531321000]] means only process times that are
from either 1531317999 to 1531318885 or 1531318886 to 1531321000

Parameters

subset_time (Optional[list]) – List of unix timestamps in seconds, used as ranges for times that you want to process.
prefer_pp_nav (bool) – if True will use post-processed navigation/height (SBET)
dump_data (bool) – if True dump the futures to the multibeam datastore. Set this to false for an entirely in memory workflow
vdatum_directory (Optional[str]) – if ‘NOAA MLLW’ ‘NOAA MHW’ is the vertical reference, a path to the vdatum directory is required here

get_beam_pointing_vectors(subset_time=None, dump_data=True)¶

Beam pointing vector is the beam specific vector that arises from the intersection of the tx ping and rx cone of sensitivity. Points at that area. Is in the geographic coordinate system, built using the tx/rx at time of ping/receive. Sends the data and calculations to the cluster, receive futures objects back. Use the dump_data/delete_futs to interact with the futures object.

To process only a section of the dataset, use subset_time.
ex: subset_time=[1531317999, 1531321000] means only process times that are from 1531317999 to 1531321000
ex: subset_time=[[1531317999, 1531318885], [1531318886, 1531321000]] means only process times that are
from either 1531317999 to 1531318885 or 1531318886 to 1531321000

Parameters

subset_time (Optional[list]) – List of unix timestamps in seconds, used as ranges for times that you want to process.
dump_data (bool) – if True dump the futures to the multibeam datastore. Set this to false for an entirely in memory workflow

get_cluster_params()¶

Attempt to figure out what the chunk size and number of chunks at a time parameters should be given the dims of the dataset. It’s pretty rough, definitely needs something more sophisticated, but this serves as a place holder.

Basically uses the avg number of beams per ping and the worker memory size to get the chunk sizes (in time)

Returns

int – number of pings in each chunk
int – number of chunks to run at once

get_orientation_vectors(subset_time=None, dump_data=True, initial_interp=False)¶

Using attitude angles, mounting angles, build the tx/rx vectors that represent the orientation of the tx/rx at time of transmit/receive. Sends the data and calculations to the cluster, receive futures objects back. Use the dump_data/delete_futs to interact with the futures object.

To process only a section of the dataset, use subset_time.
ex: subset_time=[1531317999, 1531321000] means only process times that are from 1531317999 to 1531321000
ex: subset_time=[[1531317999, 1531318885], [1531318886, 1531321000]] means only process times that are
from either 1531317999 to 1531318885 or 1531318886 to 1531321000

Parameters

subset_time (Optional[list]) – List of unix timestamps in seconds, used as ranges for times that you want to process.
dump_data (bool) – if True dump the tx/rx vectors to the multibeam datastore. Set this to false for an entirely in memory workflow
initial_interp (bool) – if True, will interpolate attitude to the ping record and store in the raw_ping datasets. This is not mandatory for processing, but useful for other kluster functions post processing.

get_variable_by_filter(var_name, selected_index=None, by_sonar_head=False)¶

ping_filter is set upon selecting points in 2d/3d in Kluster. See return_soundings_in_polygon. Here we can take those points and get one of the variables individually. This is going to be faster than running return_soundings_in_polygon again and is kind of an added feature for just getting one other variable.

Optionally, you can include a selected_index that is a list of flattened indices to points in the ping_filter that you want to super-select, see subset module.

import_post_processed_navigation(navfiles, errorfiles=None, logfiles=None, weekstart_year=None, weekstart_week=None, override_datum=None, override_grid=None, override_zone=None, override_ellipsoid=None, max_gap_length=1.0, overwrite=False)¶

Load from post processed navigation files (currently just SBET and SMRMSG) to get lat/lon/altitude as well as 3d error for further processing. Will save as variables/attributes within the ping record for the nearest data point to each ping time.

Parameters

navfiles (list) – list of postprocessed navigation file paths
errorfiles (Optional[list]) – list of postprocessed error file paths. If provided, must be same number as nav files
logfiles (Optional[list]) – list of export log file paths associated with navfiles. If provided, must be same number as nav files
weekstart_year (Optional[int]) – if you aren’t providing a logfile, must provide the year of the sbet here
weekstart_week (Optional[int]) – if you aren’t providing a logfile, must provide the week of the sbet here
override_datum (Optional[str]) – provide a string datum identifier if you want to override what is read from the log or you don’t have a log, ex: ‘NAD83 (2011)’
override_grid (Optional[str]) – provide a string grid identifier if you want to override what is read from the log or you don’t have a log, ex: ‘Universal Transverse Mercator’
override_zone (Optional[str]) –

provide a string zone identifier if you want to override what is read from the log or you don’t have a log,
ex: ‘UTM North 20 (66W to 60W)’
override_ellipsoid (Optional[str]) – provide a string ellipsoid identifier if you want to override what is read from the log or you don’t have a log, ex: ‘GRS80’
max_gap_length (float) – maximum allowable gap in the sbet in seconds, excluding gaps found in raw navigation
overwrite (bool) – if True, will include files that are already in the navigation dataset as valid

import_sound_velocity_files(src, cast_selection_method='nearest_in_time')¶

Load to self.cast_files the file paths to the sv casts of interest.

Parameters

src (Union[str, list]) – either a list of files to include or the path to a directory containing sv files (only supporting .svp currently)
cast_selection_method (str) – method used to determine the cast appropriate for each data chunk. Used here to determine whether or not this new cast(s) will require reprocessing, i.e. they are selected by one or more chunks of this dataset.

initial_att_interpolation()¶: We provide as an optional step in self.get_orientation_vectors (or run separately) the ability to interpolate the raw attitude and navigation to the ping record times and save these records to disk. Otherwise, each time attitude/navigation is needed by the processing module, it will be interpolated then.

initialize_intermediate_data(sec_ident, ky)¶

self.intermediate_dat is the storage for all the futures generated by the main processes (get_orientation_vectors, get_beam_pointing_vectors, etc.). It is organized by sector identifier/process key.

This method will initialize the storage for a new sector identifier/process key.

Parameters

sec_ident (str) – raw_ping sector identifier, ex: ‘40107_1_320000’
ky (str) – process key, one of ‘orientation’, ‘bpv’, etc.

initialize_log()¶: Initialize the fqpr logger using the multibeam logfile attribute.

self.logfile is the path to the text log that the logging module uses

self.logger is the logging.Logger object

interp_to_ping_record(sources, attributes=None)¶

Take in a dataset that is not at ping time (raw navigation, attitude, etc.) and interpolate it to ping time and save it to the raw ping datasets.

Parameters

sources (Union[Dataset, list]) – one or more datasets that you want to interpolate and save to the raw ping datastores
attributes (Optional[dict]) – optional attributes to write to the zarr datastore

intersects(min_y, max_y, min_x, max_x, geographic=True)¶

Check if the provided extents intersect with this fqpr instance. Requires georeferencing has been performed

Parameters

min_y (float) – minimum northing/latitude of extents
max_y (float) – maximum northing/latitude of extents
min_x (float) – minimum easting/longitude of extents
max_x (float) – maximum easting/longitude of extents
geographic (bool) – if True, autotransforms to projected, if False, uses the northing/easting

Returns

True if the extents provided intersect with the fqpr instance, False if they do not

Return type

bool

is_processed(in_depth=False)¶

Kluster maintains two records for processing status. current_processing_status is a scalar attribute used by the intelligence engine to max processing decisions. processing_status is a sounding variable that records the integer processing status for each sounding.

The is_processed check will see if this fqpr instance has achieved max_processing_status. in_depth will use the processing_status variable, checking each sounding attribute to compare against the max_processing_status. Otherwise, we just check the current_processing_status number, which is much faster

Parameters: in_depth (bool) – if True, will use the more expensive check to ensure each sounding is fully processed
Returns: if True, this fqpr is fully processed
Return type: bool

line_attributes(line_name)¶

Attributes by line are added after conversion to the ping attribution. This is a shortcut for returning the attribution for a line

Parameters: line_name (str) – name of the line file, ex: 0634_20180711_142125.all
Returns: list of line attributes, [start time, end time, start latitude, start longitude, end latitude, end longitude, line azimuth]
Return type: list

line_is_processed(line_name)¶

If line is processed, the TVU will not be all NaN in the middle of the line. This method will check that and return whether or the given line is processed. We use TVU because

Parameters: line_name (str) – name of the line you want to check, ex: ‘0648_20180711_151142.all’
Returns: None if line is not found, False if line is not processed, True if line is processed.
Return type: bool

overwrite_raw_navigation(navfiles, weekstart_year, weekstart_week, overwrite=False)¶

Load from raw navigation files (currently just POS MV .000) to get lat/lon/altitude. Will overwrite the original raw navigation zarr rootgroup, so you can compare pos mv to sbet.

No interpolation is done, but it will slice the incoming data to the time extents of the raw navigation and identify time gaps larger than the provided max_gap_length in seconds.

Parameters

navfiles (list) – list of postprocessed navigation file paths
weekstart_year (int) – must provide the year of the pos mv file here
weekstart_week (int) – must provide the week of the pos mv file here
overwrite (bool) – if True, will include files that are already in the navigation dataset as valid

process_backscatter(subset_time=None, dump_data=True, fixed_gain_corrected=True, tvg_corrected=True, transmission_loss_corrected=True, area_corrected=True)¶

Correct the raw reflectivity for system and environmental effects, leaving the backscatter target strength. Some parameters are sonar manufacturer specific, see backscatter module.

Parameters

subset_time (Optional[list]) – List of unix timestamps in seconds, used as ranges for times that you want to process.
dump_data (bool) – if True dump the futures to the multibeam datastore. Set this to false for an entirely in memory workflow
fixed_gain_corrected (bool) – remove the fixed gain from the raw reflectivity, if any
tvg_corrected (bool) – correct for time varying gain
transmission_loss_corrected (bool) – correct for transmission loss
area_corrected (bool) – correct for an approximation of the effect of the insonified area

read_from_source(build_offsets=True, skip_dask=False)¶

Activate rawdat object’s appropriate read class

Parameters

build_offsets (bool) – if this is set, also build the xyzrph attribute, which is mandatory for processing later in Kluster. Make it optional so that when processing chunks of files, we can just run it once at the end after read()
skip_dask (bool) – if False, will skip creating the dask client

remove_post_processed_navigation()¶: import_post_processed_navigation will write navigation and navigation related attributes to the Fqpr instance. This method will remove all variables and attributes related to post processed navigation. If the current processing status of this Fqpr is greater than or equal to georeference, this method will also write a new current processing status informing the user/intelligence module to restart processing at georeferencing.

remove_profile(profile_name)¶

Sound velocity profiles are stored in the Fqpr datastore as attributes with the ‘profile_timestamp’ format, ex: ‘profile_1503411780’. Here we take a profile name that is of that format, and remove the matching profile from the Fqpr attribution, both the loaded data and the data written to disk.

Parameters: profile_name (str) – name of the profile with the ‘profile_timestamp’ format, ex: ‘profile_1503411780’

restore_subset()¶: Restores the original data if subset_by_time has been run.

return_additional_xyz_offsets(ra, prefixes, timestmp, idx_by_chunk)¶

Apply tx to reference point offset to beams.

All the kongsberg sonars have additional offsets in the installation parameters document listed as the difference between the measured center of the transducer and the phase center of the transducer. Here we get those values for the provided system (we’ve previously stored them in the xyzrph data)

Parameters

ra (Dataset) – xarray dataset for the rawping dataset we are working with
prefixes (str) – prefix identifier for the tx/rx, will vary for dual head systems
timestmp (str) – timestamp for the appropriate xyzrph record
idx_by_chunk (list) – list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping

Returns

[float, additional x offset, float, additional y offset, float, additional z offset]

Return type

list

return_all_profiles()¶: convenience for xarray_conversion.BatchRead.return_all_profiles

return_applicable_casts(method='nearest_in_time')¶

When we check for sound velocity correct actions, we look to see if any new sv profiles imported into the fqpr instance are applicable, by running the chosen method (default is cast nearest in time to the ping chunk). If new profiles are applicable, we need to re-sv correct. Use this method to find the applicable sound velocity casts.

Parameters: method – string identifier for the cast selection method, default is nearest in time to the ping chunk
Returns: list of profile names for all casts that would be used if we sound velocity correct using the provided method
Return type: list

return_cast_dict()¶

Return a dictionary object combining the profile data and the attribution for each cast

Returns: dictionary of all the data for each profile, key is profile attribute name, ex: ‘profile_1495563079’
Return type: dict

return_cast_idx_nearestindistance(idx_by_chunk, silent=False)¶

Need to find the cast associated with each chunk of data. Currently we just take the average chunk time and find the closest cast in terms of distance. We also need the index of the chunk in the original size dataset, as we built the casts based on the original size soundvelocity dataarray.

Parameters

idx_by_chunk (list) – list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping
silent (bool) – if True, will not print out messages

Returns

list of lists, each sub-list is [xarray Datarray with times/indices for the chunk, integer index of the cast that applies to that chunk]

Return type

data

return_cast_idx_nearestindistance_fourhours(idx_by_chunk, silent=False)¶

Need to find the cast associated with each chunk of data. Currently we just take the average chunk time and find the closest cast in terms of distance. We also need the index of the chunk in the original size dataset, as we built the casts based on the original size soundvelocity dataarray.

Only retain the cast if it is within four hours.

Parameters

idx_by_chunk (list) – list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping
silent (bool) – if True, will not print out messages

Returns

list of lists, each sub-list is [xarray Datarray with times/indices for the chunk, integer index of the cast that applies to that chunk]

Return type

data

return_cast_idx_nearestintime(idx_by_chunk, silent=False)¶

Need to find the cast associated with each chunk of data. Currently we just take the average chunk time and find the closest cast time, and assign that cast. We also need the index of the chunk in the original size dataset, as we built the casts based on the original size soundvelocity dataarray.

Parameters

idx_by_chunk (list) – list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping
silent (bool) – if True, will not print out messages

Returns

list of lists, each sub-list is [xarray Datarray with times/indices for the chunk, integer index of the cast that applies to that chunk]

Return type

data

return_cast_idx_nearestintime_fourhours(idx_by_chunk, silent=False)¶

Need to find the cast associated with each chunk of data. Currently we just take the average chunk time and find the closest cast time, and assign that cast. We also need the index of the chunk in the original size dataset, as we built the casts based on the original size soundvelocity dataarray.

This method will only retain the cast if it is within four hours, otherwise, you will get a None for that chunk

Parameters

idx_by_chunk (list) – list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping
silent (bool) – if True, will not print out messages

Returns

list of lists, each sub-list is [xarray Datarray with times/indices for the chunk, integer index of the cast that applies to that chunk]

Return type

data

return_chunk_indices(idx_mask, pings_per_chunk)¶

Use self.get_cluster_params to figure out how big the chunks should be according to the cluster memory capacity and pass that number in here as pings_per_chunk. Use pings_per_chunk to divide the idx (boolean mask of applicable pings with time of ping as a coordinate)

Idx_of_chunk values are dependent on the mask. The total lengths will be equivalent, with the values counting from zero to length of mask. This lets us use it to index the data later on. Idx_of_chunk time is the ping time associated with the data we are going to be pulling later.

Parameters

idx_mask (DataArray) – the applicable_index generated from return_system_time_indexed_array
pings_per_chunk (int) – number of pings in each worker chunk

Returns

list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping

Return type

list

return_line_dict(line_names=None, ping_times=None)¶

Return all the lines with associated start and stop times for all sectors in the fqpr dataset.

If line_names is provide, only return line data for those lines. If ping_times is provided, trim all lines or drop lines that are not within the ping_times tuple (starttime in utc seconds, endtime in utc seconds)

Parameters

line_names (Union[str, list, None]) – if provided, only returns data for the line(s), otherwise, returns data for all lines
ping_times (Optional[tuple]) – time to select the dataset by, must be a tuple of (min time, max time) in utc seconds. If None, will use the full min/max time of the dataset

Returns

dictionary of names/start and stop times for all lines, ex: {‘0022_20190716_232128_S250.all’: [1563319288.304, 1563319774.876]}

Return type

dict

return_line_time(line_name)¶

Return the start and end time for the given line name

Parameters

line_name (str) – file name for the multibeam file, ex: 0000_testhis.all

Returns

float – start time in utc seconds for the line
float – end time in utc seconds for the line

return_line_xyzrph(line_name)¶

Return only the relevant xyzrph (kluster vessel config data) entries for the given line name.

Parameters: line_name (str) – file name of the multibeam line
Returns: xyzrph trimmed to only the relevant entries for the line
Return type: dict

return_lines_for_times(times)¶

Given the 1d array of times (utc seconds), return a same size object array with the string value of the line file name that matches the time.

Parameters: times (array) – 1d numpy array of times in utc seconds
Returns: 1d object array of the string file name for the multibeam file that encompasses each time
Return type: np.array

return_navigation(start_time=None, end_time=None, nav_source='raw')¶

Return the navigation from the multibeam data for the first sonar head. Can assume that all sonar heads have basically the same navigation. If sbet navigation exists, return that instead, renaming the sbet variables so that existing methods work.

Parameters

start_time (Optional[float]) – if provided will allow you to only return navigation after this time. Selects the nearest time value to the one provided.
end_time (Optional[float]) – if provided will allow you to only return navigation before this time. Selects the nearest time value to the one provided.
nav_source (str) – one of [‘raw’, ‘processed’] if you want to specify the navigation source to be the raw multibeam data or the processed sbet

Returns

latitude/longitude/altitude pulled from the navigation part of the ping record

Return type

xr.Dataset

return_next_action(new_vertical_reference=None, new_coordinate_system=None, new_offsets=False, new_angles=False, new_tpu=False, new_input_datum=None, new_waterline=False, process_mode='normal', cast_selection_method='nearest_in_time')¶

Determine the next action to take, building the arguments for the fqpr_convenience.process_multibeam function. Uses the processing status, which is updated as a process is completed at a sounding level.

0 = conversion 1 = orientation 2 = beam vectors 3 = sound velocity 4 = georeference 5 = tpu

Used in fqpr_intelligence in generating processing actions to take as data is converted/updated.

Needs some more sophistication with time ranges (i.e. navigation was added, but only for xxxxxx.xx-xxxxxxxx.xx time range, only process this segment)

Parameters

new_vertical_reference (Optional[str]) – If the user sets a new vertical reference that does not match the existing one, this will trigger a processing action starting at georeferencing
new_coordinate_system (Optional[CRS]) – If the user sets a new coordinate system that does not match the existing one, this will trigger a processing action starting at georeferencing
new_offsets (bool) – True if new offsets have been set, requires processing starting at sound velocity correction
new_angles (bool) – True if new mounting angles have been set, requires the full processing stack to be run
new_tpu (bool) – True if new tpu values have been set, requires compute TPU to run
new_input_datum (Optional[str]) – None, if there is no change to the input datum requested, otherwise this is the new input datum we need to set, should trigger a new processing action starting at georeferencing
new_waterline (bool) – True if a new waterline value has been set, requires processing starting at sound velocity correction
process_mode (str) – one of the following process modes: - normal = generate the next processing action using the current_processing_status attribute as normal - reprocess = perform a full reprocess of the dataset ignoring the current_processing_status - convert_only = only convert incoming data, return no processing actions - concatenate = process line by line if there is no processed data for that line
cast_selection_method (str) – the method used to select the cast that goes with each chunk of the dataset, one of [‘nearest_in_time’, ‘nearest_in_time_four_hours’, ‘nearest_in_distance’, ‘nearest_in_distance_four_hours’]

Returns

list – list of processing arguments to feed fqpr_convenience.process_multibeam
dict – dict of processing keyword arguments to feed fqpr_convenience.process_multibeam

return_next_unprocessed_line()¶

Return the next unprocessed line in this container, see line_is_processed

Returns: line name for the next unprocessed line
Return type: str

return_processing_dashboard()¶

Return the necessary data for a dashboard like view of this fqpr instance. Currently we are concerned with the total multibeam files associated with instance, and the processing status of each sector at a sounding level.

The returned dict object looks something like this:

{‘sounding_status’: {‘40072_0_260000’: {‘converted’: 0, ‘orientation’: 0, ‘beamvector’: 0, ‘soundvelocity’: 0,

‘georeference’: 0, ‘tpu’: 7536046},

‘40072_0_290000’: {‘converted’: 0, ‘orientation’: 0, ‘beamvector’: 0, ‘soundvelocity’: 0,

‘georeference’: 0, ‘tpu’: 7536046}, …

‘last_run’: {‘40072_0_260000’: {‘_conversion_complete’: ‘Tue Nov 24 12:42:41 2020’, ‘_compute_orientation_complete’: ‘Tue Nov 24 12:44:21 2020’,

‘_compute_beam_vectors_complete’: ‘Tue Nov 24 12:46:20 2020’, ‘_sound_velocity_correct_complete’: ‘Tue Nov 24 12:48:25 2020’,

‘_georeference_soundings_complete’: ‘Tue Nov 24 12:50:04 2020’, ‘_total_uncertainty_complete’: ‘Tue Nov 24 12:51:55 2020’},

‘40072_0_290000’: {‘_conversion_complete’: ‘Tue Nov 24 12:42:41 2020’, ‘_compute_orientation_complete’: ‘Tue Nov 24 12:44:40 2020’,

‘_compute_beam_vectors_complete’: ‘Tue Nov 24 12:46:40 2020’, ‘_sound_velocity_correct_complete’: ‘Tue Nov 24 12:48:39 2020’,

‘_georeference_soundings_complete’: ‘Tue Nov 24 12:50:21 2020’, ‘_total_uncertainty_complete’: ‘Tue Nov 24 12:52:14 2020’}, …

‘multibeam_files’: {‘0000_202003_S222_EM2040.all’: [1584426535.491, 1584426638.015], ‘0001_202003_S222_EM2040.all’: [1584427154.74, 1584427341.396],

‘0002_202003_S222_EM2040.all’: [1584427786.983, 1584427894.186], ‘0003_202003_S222_EM2040.all’: [1584428272.65, 1584428465.862], …

Returns: processing status at the sector level
Return type: dict

return_rounded_frequency()¶

Returns the frequency rounded to match the freq settings commonly given with sonar manufacturer settings. If you have entries like [270000, 290000, 310000, 330000], it returns [300000]. If its something like [69000, 71000] it returns [70000].

Returns: array of rounded frequencies
Return type: np.array

return_runtime_idx_nearestintime(idx_by_chunk)¶

Find the runtime parameter that is nearest to each chunk in the provided chunk arrays

Parameters: idx_by_chunk (list) – list of xarray Datarrays, values are the integer indexes of the pings to use, coords are the time of ping
Returns: list of dict objects for the runtime parameters that is nearest to each chunk
Return type: list

return_soundings_in_polygon(polygon, geographic=True, variable_selection=('head', 'x', 'y', 'z', 'tvu', 'detectioninfo', 'time', 'beam'), isolate_head=None)¶: Using provided coordinates (in either horizontal_crs projected or geographic coordinates), return the soundings and sounding attributes for all soundings within the coordinates, see subset module. Also sets the ping_filter attribute which can be used with set_variable_by_filter get_variable_by_filter

return_total_pings(min_time=None, max_time=None)¶

Get the total ping count, optionally within the provided mintime maxtime range

Parameters

min_time (Optional[float]) – the minimum time desired from the raw_ping dataset
max_time (Optional[float]) – the maximum time desired from the raw_ping dataset

Returns

total number of pings for this dataset

Return type

int

return_total_soundings(min_time=None, max_time=None)¶

Return the number of soundings in all systems within this fqpr instance, optionally within the provided mintime maxtime range

Parameters

min_time (Optional[float]) – the minimum time desired from the raw_ping dataset
max_time (Optional[float]) – the maximum time desired from the raw_ping dataset

Returns

total number of soundings in the dataset

Return type

int

return_unique_mode()¶

Finds the unique mode entries in raw_ping Datasets. If there is more than one unique mode, return them in order of most often found.

Returns: array of mode settings
Return type: np.array

run_filter(filtername, *args, selected_index=None, save_to_disk=True, **kwargs)¶

Run the filter module with the provided filtername, will match the filename of the filter python file.

Parameters

filtername (str) – name of the file that you want to load
selected_index (Optional[list]) – optional list of 1d boolean arrays representing the flattened index of those values to retain. Used mainly in Points View filtering, where you have a (time,beam) space but only want to retain the beams shown in Points View.
save_to_disk (bool) – if True, will save the new sounding status to disk

set_filter_by_polygon(polygon, geographic=True)¶: Alternative way to set the ping_filter attribute which can be used with set_variable_by_filter get_variable_by_filter, see subset module.

set_variable_by_filter(var_name='detectioninfo', newval=2, selected_index=None)¶: ping_filter is set upon selecting points in 2d/3d in Kluster. See return_soundings_in_polygon. Here we can take those points and set one of the variables with new data. Optionally, you can include a selected_index that is a list of flattened indices to points in the ping_filter that you want to super-select, see subset module.

set_vertical_reference(vert_ref)¶

Set the Fqpr instance vertical reference. This will feed into the georef and calculate tpu processes.

If the new vert_ref conflicts with an existing written vert_ref, issue a warning.

Parameters: vert_ref (str) – vertical reference for the survey, one of [‘ellipse’, ‘waterline’, ‘NOAA MLLW’, ‘NOAA MHW’, ‘Aviso MLLW’]

subset_by_lines(line_names)¶: Use subset module to trim the fqpr instance to the given lines

subset_by_time(mintime=None, maxtime=None)¶: Use subset module to trim the fqpr instance to the given time range

subset_by_time_and_beam(subset_time, subset_beam)¶: Use subset module to subset by time,beam provided, returns a 1d boolean mask for each sonar head

subset_by_times(time_segments)¶: Use subset module to trim the fqpr instance to the given time ranges

subset_variables(variable_selection, ping_times=None, skip_subset_by_time=False, filter_by_detection=False)¶: Take specific variable names and return just those variables in a new xarray dataset, see subset module.

subset_variables_by_line(variable_selection, line_names=None, ping_times=None, filter_by_detection=False)¶: Apply subset_variables to get the data split up into lines for the variable_selection provided, see subset module

sv_correct(add_cast_files=None, cast_selection_method='nearest_in_time', subset_time=None, dump_data=True)¶

Apply sv cast/surface sound speed to raytrace. Generates xyz for each beam. Currently only supports nearest-in-time for selecting the cast for each chunk. Sends the data and calculations to the cluster, receive futures objects back. Use the dump_data/delete_futs to interact with the futures object.

To process only a section of the dataset, use subset_time.
ex: subset_time=[1531317999, 1531321000] means only process times that are from 1531317999 to 1531321000
ex: subset_time=[[1531317999, 1531318885], [1531318886, 1531321000]] means only process times that are
from either 1531317999 to 1531318885 or 1531318886 to 1531321000

Parameters

add_cast_files (Union[str, list, None]) – either a list of files to include or the path to a directory containing files. These are in addition to the casts in the ping dataset.
cast_selection_method (str) – the method used to select the cast that goes with each chunk of the dataset, one of [‘nearest_in_time’, ‘nearest_in_time_four_hours’, ‘nearest_in_distance’, ‘nearest_in_distance_four_hours’]
subset_time (Optional[list]) – List of unix timestamps in seconds, used as ranges for times that you want to process.
dump_data (bool) – if True dump the futures to the multibeam datastore. Set this to false for an entirely in memory workflow

write_attribute_to_ping_records(attr_dict)¶

Convenience method that allows you to write the provided attribute dictionary to each ping dataset and change the currently loaded instance as well

Parameters: attr_dict (dict) – dictionary of attributes that you want stored in the ping datasets

write_intermediate_futs_to_zarr(mode, sys_ident, timestmp, skip_dask=False)¶

Flush some of the intermediate data that was mapped to the cluster (and lives in futures objects) to disk, puts it in the multibeam, as the time dimension should be the same. Mode allows for selecting the output from one of the main processes for writing.

Parameters

mode (str) – one of [‘orientation’, ‘bpv’, sv_corr’, ‘georef’, ‘tpu’, ‘backscatter’]
sys_ident (str) – the multibeam system identifier attribute, used as a key to find the intermediate data
timestmp (str) – timestamp of the installation parameters instance used
skip_dask (bool) – if True will not use the dask.distributed client to submit tasks, will run locally instead

Fqpr¶

kluster

Navigation

Related Topics