split_array_by_number_of_workers

HSTB.kluster.dask_helpers.split_array_by_number_of_workers(client, dataarray, max_len=None)

In order to operate on an array in a parallelized way, we need to split the array into equal chunks to pass to each worker. Here we do that by just dividing by the number of workers.

Optional parameter is to restrict the size of the chunks by an int max_len. This of course only applies if the chunks were going to be larger than max_len anyway.

Drop empty if the length of the array is greater than the number of workers.

Parameters

• client (Client) – dask distributed client

• dataarray (DataArray) – one dimensional array

• max_len (Optional[int]) – max number of values per chunk, if None, ignored

Returns

• list – list of numpy arrays representing chunks of the original array

• list – list of numpy arrays representing indexes of new values from original array