my_open_mfdataset

HSTB.kluster.xarray_helpers.my_open_mfdataset(paths, chnks=None, concat_dim='time', compat='no_conflicts', data_vars='all', coords='different', join='outer')

Trying to address the limitations of the existing xr.open_mfdataset function. This is my modification using the existing function and tweaking to resolve the issues i’ve found.

(see https://github.com/pydata/xarray/blob/master/xarray/backends/api.py)

Current issues with open_mfdataset (1/8/2020):
1. open_mfdataset only uses the attrs from the first nc file
2. open_mfdataset will not run with parallel=True or with the distributed.LocalCluster running
3. open_mfdataset infers time order from position. (I could just sort outside of the function, but i kinda
like it this way anyway. Also a re-indexing would probably resolve this.)

Only resolved item = 1 so far. See https://github.com/pydata/xarray/issues/3684

Parameters

  • paths (list) – list of file paths to existing netcdf stores

  • chnks (Optional[dict]) – if provided, used to load dataset into chunks

  • concat_dim (str) – dimension to concatenate along

  • compat (str) – String indicating how to compare non-concatenated variables of the same name for potential conflicts

  • data_vars (str) – which variables will be concatenated

  • coords (str) – which coordinate variables will be concatenated

  • join (str) – String indicating how to combine differing indexes (excluding dim) in objects

Returns

attributes, variables, dimensions of combined netCDF files. Returns dask arrays, compute to access local numpy array.

Return type

xr.Dataset