determine_optimal_chunks

HSTB.kluster.dask_helpers.determine_optimal_chunks(client, beams_per_ping, safety_margin=0.75, chunks_per_worker=2, mem_per_beam=1.5e-05)

A very rudimentary placeholder-esque way to determine the chunk size and number of chunks for an array to process in memory. Too many chunks/Too big chunks and you run out of memory. Too few and you aren’t utilizing the resources adequately. Here we scale mainly based off of the number of workers and the amount of memory available in the cluster.

Parameters

• client (Client) – dask distributed client

• beams_per_ping (float) – avg number of beams per ping

• safety_margin (float) – made up number to ensure we don’t expect 100% of the memory to be available

• chunks_per_worker (int) – determines the number of chunks to build and process

• mem_per_beam (float) – metric I came up with looking at processes run, approx amount of memory used per beam in the svcorr process.

Returns

• int – length in time dimension of each chunk

• int – total number of chunks to process