FqprIntel

class HSTB.kluster.fqpr_intelligence.FqprIntel(project=None, parent=None, **kwargs)

Bases: LoggerClass

Kluster intelligence module.

Take in a file, gather the file level information, determine where in the project it needs to go (based on things like serial number and time of file). The project contains converted data by system/sector/frequency etc.

From there determine how to proceed. Is the user ready to convert and the incoming file is a multibeam file? Go ahead and convert the line to the appropriate Fqpr instance within the project (appending to or overwriting the existing data). Does the user want to go ahead and process all the way to georeference on adding new lines? Run the full processing for that line.

If the incoming file is an SBET, run import ppnav for all applicable data. Re-run georeference after.

If the incoming file is an SVP, run svcorrect on the cast on the nearest in time data that would apply.

Attributes Summary

has_actions

Property used to identify that there are actions in the intel module that are ready to be executed

Methods Summary

add_file(infile[, silent])	Starting point for FqprIntel, adding a file to the class which then adds it to one of the intel objects.
bind_to_action_update(callback)	Pass in a method as callback, method will be triggered on actions being updated
clear()	Clear all data in the intelligence modules and the attached action container.
execute_action([idx])	Execute the next action in the action container (default index is 0).
match_multibeam_files_to_project()	Match multibeam files to one of the fqpr_instances in the project.
match_navigation_files()	POSPac data comes in as three separate files: SBET file (the nav/altitude data), the SMRMSG file (the associated uncertainties) and the export log (text file generated when you export an sbet)
match_navigation_files_to_project()	Difficult to match processed navigation files to the fqpr instance.
match_svp_files_to_project()	This match is dead simple.
regenerate_actions([also_update_conversion, ...])	Regenerate all the actions related to exising fqpr instances in the project.
remove_file(infile)	Remove this file from any intelligence modules
return_intel_modules()	Return a list of all intel modules
set_auto_processing_mode([process_mode])	Set the intel auto processing mode to one of the available options.
set_designated_surface(surf_name)	Set a new designated surface by providing the path to a bathygrid folder here.
set_settings(settings)	Set new settings for the FqprIntel object, triggers a regenerate actions action, to see if there are new actions related to the new settings.
start_folder_monitor(folderpath[, is_recursive])	Create a new DirectoryMonitor object for the provided folder path.
stop_folder_monitor(folderpath)	Stop and remove the monitor object for the given folderpath
update_actions_client()	Trigger updating the dask distributed client instance in all actions
update_from_project([project_updated])	Called from FqprProject._bind_to_project_updated.
update_intel_for_action_results(action_type)	After a new action, we need to rematch the files, especially if there are new converted fqpr instances in the project.
update_matches()	Every time a file is successfully added or removed from the intelligence module, we need to update the relevant Actions, to reflect the new files.

Attributes Documentation

has_actions

Property used to identify that there are actions in the intel module that are ready to be executed

Methods Documentation

add_file(infile, silent=True)

Starting point for FqprIntel, adding a file to the class which then adds it to one of the intel objects.

We check to see if the file is in one of the approved file extension lists, or check in a more comprehensive way (see sbet.is_sbet) before adding.

Parameters

• infile (str) – full file path to the new file

• silent (bool) – if silent, will not print messag on failing to add

Returns

• str – the updated_type that matches this file

• OrderedDict – attributes associated with one of the gather_xxxx functions

• bool – True if a new project was created or loaded

bind_to_action_update(callback)

Pass in a method as callback, method will be triggered on actions being updated

Used in kluster_main to update the actions gui

Parameters

callback (function) – method that is run on setting newfile

clear()

Clear all data in the intelligence modules and the attached action container. Triggered on closing the project in kluster_main.

execute_action(idx=0)

Execute the next action in the action container (default index is 0). Actions are sorted by priority, as they are added to the container. Conversion is always the highest priority. If a multibeam action is run here, we have to rebuild the multibeam/project matches.

All actions return the new fqpr instance, so we overwrite the project Fqpr instance reference with this new one.

match_multibeam_files_to_project()

Match multibeam files to one of the fqpr_instances in the project. Assign the path to the matching fqpr_instance to the multibeam_intel matching_fqpr dictionary. If there is no match, leave it blank.

If there is a project, additionally group the multibeam lines by either the fqpr instance they should be converted to or the serial number identifier that we will use to create a new fqpr instance for these files. Use the serial number group when the new multibeam files do not go with an existing fqpr instance (as the new serial numbers do not match any existing fqpr instance serial numbers).

match_navigation_files()

POSPac data comes in as three separate files: SBET file (the nav/altitude data), the SMRMSG file (the associated uncertainties) and the export log (text file generated when you export an sbet)

We need to determine which of these files go together. We do that by finding the sbet file that matches each log file and the sbet that matches each error file. The matching sbet path is saved as an attribute in the self.naverror_intel and self.navlog_intel instances. See self.naverror_intel.matching_sbet

match_navigation_files_to_project()

Difficult to match processed navigation files to the fqpr instance. POSPac SBET files are in weekly seconds, so it’s not a matter of just getting the closest time match. Also the SBET can be much longer than the multibeam line(s), so we can’t really use position either.

We currently use:

• Only use nav files that have matching error and log files. Technically we support importing nav files that do not have log files (you have to tell me the coord system and start date) but we want to at least start off by telling the user that they have to provide a log file.

• Check weekly seconds and fqpr instance weekly seconds to see if sbet is on the same day.

• Check the navfilepath for fqpr instance identifiers like serial number and model number

• (TODO) look at getting a time/position from the multibeam and see if that is in the SBET with a close position

match_svp_files_to_project()

This match is dead simple. If the provided svp file is not in the project, we add it.

regenerate_actions(also_update_conversion=False, reprocess_fqpr=None)

Regenerate all the actions related to exising fqpr instances in the project. Everytime an fqpr instance is removed or added to the project, we run this method.

Parameters

• also_update_conversion (bool) – if True, will also regenerate conversion actions. Generally only needed post conversion when new multibeam files are added during conversion, and you need to clear out the file that just converted.

• reprocess_fqpr (Optional[str]) – optional, the relative path (from project) for an fqpr instance, triggers full reprocessing for that instance, should only be used in emergency

remove_file(infile)

Remove this file from any intelligence modules

Parameters

infile (str) – full file path to the file

Returns

• str – the updated_type that matches this file

• int – unique id as integer for the file removed

return_intel_modules()

Return a list of all intel modules

Returns

list of all intel modules

Return type

list

set_auto_processing_mode(process_mode='normal')

Set the intel auto processing mode to one of the available options.

Parameters

process_mode (str) – One of the following process modes: normal=generate the next processing action using the current_processing_status attribute as normal, convert_only=only convert incoming data, return no processing actions, concatenate=process line by line if there is no processed data for that line

set_designated_surface(surf_name)

Set a new designated surface by providing the path to a bathygrid folder here. The designated surface will be automatically updated when new processed fqpr containers are added to the project.

Parameters

surf_name (str) – path to the bathygrid folder that you want to set as the designated surface

set_settings(settings)

Set new settings for the FqprIntel object, triggers a regenerate actions action, to see if there are new actions related to the new settings. For instance, changing to vert_ref=ellipse would then trigger new georeferencing actions to process using the new vertical reference.

Parameters

settings (dict) – dictionary of processing settings

start_folder_monitor(folderpath, is_recursive=True)

Create a new DirectoryMonitor object for the provided folder path. Automatically start the monitoring and store it as an attribute in the monitors dictionary

Parameters

• folderpath – absolute folder path to the directory we want to monitor

• is_recursive – if True, search subdirectories as well

stop_folder_monitor(folderpath)

Stop and remove the monitor object for the given folderpath

Parameters

folderpath – absolute folder path to the directory we want to monitor

update_actions_client()

Trigger updating the dask distributed client instance in all actions

update_from_project(project_updated=True)

Called from FqprProject._bind_to_project_updated. Whenever a fqpr instance is added or removed from the fqpr project, this method is called. Updates the processing actions.

Parameters

project_updated (bool) – if True, the project has been updated. Basically always True

update_intel_for_action_results(action_type)

After a new action, we need to rematch the files, especially if there are new converted fqpr instances in the project.

Parameters

action_type (str) – the action type of the action that was executed, action type is an attribute of the FqprAction

update_matches()

Every time a file is successfully added or removed from the intelligence module, we need to update the relevant Actions, to reflect the new files.