dask_helpers module¶

HSTB.kluster.dask_helpers Module¶

`client_needs_restart`(client)	Having issues with memory leaks, even though workers have no assigned tasks.
`dask_close_localcluster`()	Retrieve and close the LocalCluster if it exists, otherwise pass
`dask_find_or_start_client`([address, ...])	Either start or return Dask client in local/networked cluster mode.
`determine_optimal_chunks`(client, beams_per_ping)	A very rudimentary placeholder-esque way to determine the chunk size and number of chunks for an array to process in memory.
`get_max_cluster_allocated_memory`(client)	Retrieve the max memory across all workers in the cluster added together
`get_number_of_workers`(client)	Retrieve the total number of workers from the dask cluster
`split_array_by_number_of_workers`(client, ...)	In order to operate on an array in a parallelized way, we need to split the array into equal chunks to pass to each worker.

Provides synchronization using file locks via the fasteners package.