FqprProject

class HSTB.kluster.fqpr_project.FqprProject(project_path=None, **kwargs)

Bases: LoggerClass

The FqprProject class contains all the fqpr_generated.Fqpr objects and has methods for interacting with multiple of these objects as one big project.

Each fqpr_generated.Fqpr object is like a container of lines, it can either be a single processed line, a bunch of lines for one day, a week’s worth of lines, etc. If you want to find which container has which line, or find all lines within a specific area, the FqprProject should be able to do this.

proj_data contains the paths to the top level folders for two Fqpr generated objects
C:/data_dir/EM2040/convert1
C:/data_dir/EM2040/convert1/attitude.zarr
C:/data_dir/EM2040/convert1/soundings.zarr
C:/data_dir/EM2040/convert1/navigation.zarr
C:/data_dir/EM2040/convert1/ping_40107_0_260000.zarr
C:/data_dir/EM2040/convert1/ping_40107_1_320000.zarr
C:/data_dir/EM2040/convert1/ping_40107_2_290000.zarr
C:/data_dir/EM2040/convert1/logfile_094418.txt
proj_data = [r”C:/data_dir/convert1”, r”C:/data_dir/convert2”]
fqp = FqprProject()
for pd in proj_data:
fqp.add_fqpr(pd, skip_dask=True)

Methods Summary

absolute_path_from_relative(pth)

see path_relative_to_project, will convert the returned relative path from that method to an absolute file path

add_fqpr(pth[, skip_dask])

Add a new Fqpr object to this project.

add_surface(pth)

Add a new Bathygrid object to the project, either by loading from file or by directly adding a Bathygrid object provided

add_vessel_file([vessel_file_path, ...])

Attach a new or existing vessel file to this project.

build_visualizations(pth, visualization_type)

Take the provided project path and create visualizations of that project

close()

close project and clear all data.

get_dask_client()

Project is the holder of the Dask client object.

get_fqpr_by_serial_number(...[, same_day_as])

Find the fqpr instance that matches the provided serial number.

get_fqprs_by_paths(fqpr_paths[, line_dict, ...])

Return a list of the paths and Fqpr instances associated with each provided path.

new_project_from_directory(directory_path)

Take in a path to a directory where we want to build a new project.

open_project(projfile[, skip_dask])

Open a project from file.

path_relative_to_project(pth)

Return the relative path for the provided pth from the project file

refresh_fqpr_attribution(pth[, relative_path])

regenerate_fqpr_lines(pth)

After adding a new Fqpr object, we want to get the line information from the attributes so that we can quickly access how many lines are in a project, and the time boundaries of these lines.

remove_fqpr(pth[, relative_path])

Remove an attached Fqpr instance from the project by path to Fqpr converted folder

remove_surface(pth[, relative_path])

Remove an attached Bathygrid instance from the project by path to Fqpr converted folder

retrieve_data_for_time_segments(systems, ...)

For the manual patch test, we retrieve the Fqpr object and time segments that are currently displayed in the points view and use that data to populate the patch test widget.

return_fqpr_instances()

Get all loaded fqpr instances

return_fqpr_paths()

Get the absolute paths to all loaded fqpr instances

return_line_navigation(line[, override_source])

For given line name, return the latitude/longitude from the ping record

return_line_owner(line)

Return the Fqpr instance that contains the provided multibeam line

return_lines_in_box(min_lat, max_lat, ...)

With the given latitude/longitude boundaries, return the lines that are completely within these boundaries

return_project_folder()

Return the project folder, the folder that contains the project file

return_project_lines([proj, relative_path])

Return the lines associated with the provided Fqpr path (proj) or all projects/lines

return_sorted_line_list()

Return all lines in the project sorted by name

return_soundings_in_polygon(polygon)

With the given latitude/longitude polygon, return the soundings that are within the boundaries.

return_surface(surface_name[, relative_path])

Return the bathygrid instance for the surface that matches the given surface path

return_surface_containers(surface_name[, ...])

Project has loaded surface and fqpr instances.

return_surface_paths()

Get the absolute paths to all loaded surface instances

return_vessel_file()

Return the VesselFile instance for this project's vessel_file path

run_auto_patch_test(line_pairs)

save_project()

Save the current FqprProject instance to file.

set_settings(settings)

Set the project settings with the provided dictionary.

sort_lines_patch_test_pairs(line_list)

Take the provided list of linenames and sort them into pairs for the patch test tool.

update_surface(pth, surf[, relative_path])

Update an attached Bathygrid instance with a new instance, to avoid a costly close/re-open

Methods Documentation

absolute_path_from_relative(pth)

see path_relative_to_project, will convert the returned relative path from that method to an absolute file path

Parameters

pth (str) – relative file path from the directory containing the project file

Returns

absolute file path

Return type

str

add_fqpr(pth, skip_dask=False)

Add a new Fqpr object to this project. If skip_dask is True, will auto start a new dask LocalCluster

Parameters

  • pth (Union[str, Fqpr]) – path to the top level folder for the Fqpr project or the already loaded Fqpr instance itself

  • skip_dask (bool) – if True will skip auto starting a dask LocalCluster

Returns

  • str – project entry in the dictionary, will be the relative path to the kluster data store from the project file

  • bool – False if the fqpr was already in the project, True if added

add_surface(pth)

Add a new Bathygrid object to the project, either by loading from file or by directly adding a Bathygrid object provided

Parameters

pth (Union[str, BathyGrid]) – path to surface file or existing Bathygrid object

add_vessel_file(vessel_file_path=None, update_with_project=True)

Attach a new or existing vessel file to this project. Optionally populate it with the found offsets and angles in the existing fqpr instances in the project

Parameters

  • vessel_file_path (Optional[str]) – path to the new or existing vessel file

  • update_with_project (bool) – if True, will update the vessel file with the offsets and angles of all the fqpr instances in the project

build_visualizations(pth, visualization_type)

Take the provided project path and create visualizations of that project

Parameters

  • pth (str) – path to the Fqpr object

  • visualization_type (str) – one of ‘orientation’, ‘beam_vectors’, ‘corrected_beam_vectors’

close()

close project and clear all data. have to close the fqpr instances with the fqpr close method.

get_dask_client()

Project is the holder of the Dask client object. Use this method to return the current Client. Client is currently setup with kluster_main.start_dask_client or kluster_main.open_dask_dashboard. If skip_dask is set, will not automatically start a client. If the client does not exist, we set it here and then set the client to the Fqpr and BatchRead instance

get_fqpr_by_serial_number(primary_serial_number, secondary_serial_number, same_day_as=None)

Find the fqpr instance that matches the provided serial number. Should just be one instance in a project with the same serial number, if there are more, that is going to be a problem.

Parameters

  • primary_serial_number (int) – primary serial number for the system you want to find, primary serial number will just be the serial number of the port system for dual head kongsberg

  • secondary_serial_number (int) – secondary serial number for the system you want to find, this will be zero if not dual head, otherwise it is the serial number of the starboard head

  • same_day_as (Optional[datetime]) – optional, if provided wil only return an Fqpr instance if it is on the same day as the provided datetime object

Returns

  • str – folder path to the fqpr instance

  • Fqpr – fqpr instance that matches the serial numbers provided

get_fqprs_by_paths(fqpr_paths, line_dict=None, relative_path=True, allow_different_offsets=True, concatenate=True, raise_exception=False)

Return a list of the paths and Fqpr instances associated with each provided path. Path can be relative to the project or an absolute path.

If a line_dict is provided, we will subset each Fqpr object to just the data associated with the lines in the line_dict. If you provide multiple fqpr_paths, it will take the data for the lines across all fqpr_paths provided, and merge them into one Fqpr object. So if you provide 2 fqpr_paths, and a line_dict that includes 2 lines from each fqpr object, the end result is a single Fqpr object with the data for all four lines across the two original Fqpr instances.

multibeam_files attribute the in the returned fqpr object is adjusted for the actual lines in the Fqpr returned.

Parameters

  • fqpr_paths (list) – list of relative/absolute paths to the Fqpr objects we want

  • line_dict (Optional[dict]) – dict of {fqpr_path: [line1, line2, …]} for the desired lines

  • relative_path (bool) – if True, provided paths are relative to the project, otherwise they are absolute paths

  • allow_different_offsets (bool) – if True, and concatenate is True, will concatenate Fqpr instances that have different offsets/angles. Otherwise will reject.

  • concatenate (bool) – if True, will concatenate the fqprs into one object, only possible if the sonar serial numbers match

  • raise_exception (bool) – if True, will raise an exception instead of handling the error

Returns

  • list – absolute paths to the fqprs queried

  • list – list of loaded fqpr instances

new_project_from_directory(directory_path)

Take in a path to a directory where we want to build a new project. This can be an empty project (if the directory provided is empty) or a populated project with the converted data in the provided directory.

Parameters

directory_path (str) – Path to a directory that is either empty or has converted data in it

open_project(projfile, skip_dask=False)

Open a project from file. See save_project for how to generate this file.

Parameters

  • projfile (str) – path to the project file

  • skip_dask (bool) – if True, will not autostart a dask client. client is necessary for conversion/processing

path_relative_to_project(pth)

Return the relative path for the provided pth from the project file

Parameters

pth (str) – absolute file path

Returns

relative file path from the project file

Return type

str

refresh_fqpr_attribution(pth, relative_path=False)
regenerate_fqpr_lines(pth)

After adding a new Fqpr object, we want to get the line information from the attributes so that we can quickly access how many lines are in a project, and the time boundaries of these lines.

Parameters

pth (str) – path to the Fqpr object

remove_fqpr(pth, relative_path=False)

Remove an attached Fqpr instance from the project by path to Fqpr converted folder

Parameters

  • pth (str) – path to the top level folder for the Fqpr project

  • relative_path (bool) – if True, pth is a relative path (relative to self.path)

remove_surface(pth, relative_path=False)

Remove an attached Bathygrid instance from the project by path to Fqpr converted folder

Parameters

  • pth (str) – path to the surface file

  • relative_path (bool) – if True, pth is a relative path (relative to self.path)

retrieve_data_for_time_segments(systems, time_segments)

For the manual patch test, we retrieve the Fqpr object and time segments that are currently displayed in the points view and use that data to populate the patch test widget. Here we take the returned system name and time segment data from the points view to get the corresponding data from the project.

Parameters

  • systems (list) – list of the system name for each segment

  • time_segments (list) – list of lists for the start time/end time for the line in utc seconds

Returns

list of lists for each segment containing the Fqpr object, the serial number, the time segments, the xyzrph dict for that Fqpr, the system identifier, the head index, and the vesselfile name.

Return type

list

return_fqpr_instances()

Get all loaded fqpr instances

Returns

list of fqpr_generation.Fqpr objects

Return type

list

return_fqpr_paths()

Get the absolute paths to all loaded fqpr instances

Returns

list of str paths to all fqpr instances

Return type

list

return_line_navigation(line, override_source=None)

For given line name, return the latitude/longitude from the ping record

Parameters

  • line (str) – line name

  • override_source (Optional[str]) – optional, one of [‘raw’, ‘processed’] if you want to specify the navigation source to be the raw multibeam data or the processed sbet

Returns

  • np.array – latitude values (geographic) downsampled in degrees

  • np.array – longitude values (geographic) downsampled in degrees

return_line_owner(line)

Return the Fqpr instance that contains the provided multibeam line

Parameters

line (str) – line name

Returns

None if you can’t find a line owner, else the fqpr_generation.Fqpr object associated with the line

Return type

Fqpr

return_lines_in_box(min_lat, max_lat, min_lon, max_lon)

With the given latitude/longitude boundaries, return the lines that are completely within these boundaries

Parameters

  • min_lat (float) – float, minimum latitude in degrees

  • max_lat (float) – float, maximum latitude in degrees

  • min_lon (float) – float, minimum longitude in degrees

  • max_lon (float) – float, maximum longitude in degrees

Returns

line names that fall within the box

Return type

list

return_project_folder()

Return the project folder, the folder that contains the project file

Returns

either None (if the project hasn’t been set up yet) or the folder containing the kluster_project.json file

Return type

str

return_project_lines(proj=None, relative_path=True)

Return the lines associated with the provided Fqpr path (proj) or all projects/lines

Parameters

  • proj (Optional[str]) – optional, str, Fqpr path if you only want lines associated with that project

  • relative_path (bool) – if True, proj is a relative path (relative to self.path)

Returns

all line names in the project or just the line names associated with proj

Return type

dict

return_sorted_line_list()

Return all lines in the project sorted by name

Returns

sorted list of line names

Return type

dict

return_soundings_in_polygon(polygon)

With the given latitude/longitude polygon, return the soundings that are within the boundaries. Use the Fqpr horizontal_crs recorded EPSG to do the transformation to northing/easting, and then query all the x, y to get the soundings.

If full swath is used return the whole swaths that are within the bounds.

Parameters

polygon (ndarray) – (N, 2) array of points that make up the selection polygon, (longitude, latitude) in degrees

Returns

dict where keys are the fqpr instance name, values are the sounding values as 1d arrays

Return type

dict

return_surface(surface_name, relative_path=True)

Return the bathygrid instance for the surface that matches the given surface path

Parameters

  • surface_name (str) – path to the surface on disk

  • relative_path (bool) – if True, will assume the given name is a relative path to the project file

Returns

surface instance that matches the given surface name

Return type

BathyGrid.Bgrid

return_surface_containers(surface_name, relative_path=True)

Project has loaded surface and fqpr instances. This method will return the names of the existing fqpr instances in the surface and a list of the fqpr instances in the project that are not in the surface yet.

Fqpr instances marked with an asterisk are those that need to be updated in the surface. The surface soundings for that instance are out of date relative to the last operation performed on the fqpr instance.

Parameters

  • surface_name (str) – path to the surface, either relative to the project or absolute path

  • relative_path (bool) – if True, surface_name is a relative path

Returns

  • dict – dict of the fqpr instance names/line names that are in the surface, with an asterisk at the end if the surface version of the fqpr instance soundings is out of date

  • dict – dict of fqpr instances/line names that are in the project and not in the surface

return_surface_paths()

Get the absolute paths to all loaded surface instances

Returns

list of str paths to all surface instances

Return type

list

return_vessel_file()

Return the VesselFile instance for this project’s vessel_file path

Returns

Instance of VesselFile for the vessel_file attribute path. If self.vessel_file is not set, this returns None

Return type

VesselFile

run_auto_patch_test(line_pairs)
save_project()

Save the current FqprProject instance to file. Use open_project to reload this instance.

set_settings(settings)

Set the project settings with the provided dictionary. Pull out fqpr specific settings like whether or not to enable parallel write as well

Parameters

settings (dict) – dictionary from the Qsettings store, see kluster_main._load_previously_used_settings

sort_lines_patch_test_pairs(line_list)

Take the provided list of linenames and sort them into pairs for the patch test tool. Each pair consists of two lines that are reciprocal and start/end in the same place.

Parameters

line_list (list) – list of line names that we want to sort

Returns

  • list – list of lists of line names in pairs

  • dict – line dict containing the start position, end position and azimuth of each line

update_surface(pth, surf, relative_path=False)

Update an attached Bathygrid instance with a new instance, to avoid a costly close/re-open

Parameters

  • pth (str) – path to the surface file

  • surf (BathyGrid) – new bathygrid instance

  • relative_path (bool) – if True, pth is a relative path (relative to self.path)