Repository navigation
Allow windowed array to hold a wider window - #3061
jemmajeffree wants to merge 10 commits into
Conversation
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…nto big_memory_window
for more information, see https://pre-commit.ci
|
For what it's worth, this is how I've been defining a fieldset chunks = {'lat':-1,'lon':-1,'time':24}
ds_fset = parcels.convert.mom6_to_sgrid(
fields={"U": xr.open_mfdataset(uv_files,engine='netcdf4',chunks=chunks,
)["SSU"].sel(time=slice('2009-01-01','2009-07-05')),
"V": xr.open_mfdataset(uv_files,engine='netcdf4',chunks=chunks,
)["SSV"].sel(time=slice('2009-01-01','2009-07-05')),
# Engine really shouldn't need to be specified, but it was throwing errors without it
},
coords=xr.load_dataset(coords_file,engine='netcdf4'),
)
fieldset = parcels.FieldSet.from_sgrid_conventions(ds_fset).to_windowed_arrays(time_chunk=24*28)The rest of my testing setup was pretty much following the MOM6 tutorial |
for more information, see https://pre-commit.ci
|
Thanks so much for this contribution, @jemmajeffree! And of course also all the other context and Issues. Really nice to see someone engaging with our code so much! I see this PR is still in Draft - do you want a review of it? Or would you prefer to keep working on it? |
|
Please review it! It's working for my use-case, so I'm not planning on making changes without feedback I just left it as a draft because I think it might be failing a handful of tests and I don't know if it generalises to all use cases, or if it aligns with how you would like the codebase to fit together |
|
OK, I'll then Approve the workflows to run and will aks @wyatt-fluidnumerics or @fluidnumericsJoe to review this code, as they developed the WIndowedArrays() backend so have more insight into how this PR would work there |
wyatt-fluidnumerics
left a comment
There was a problem hiding this comment.
Hi @jemmajeffree , thanks for this PR and the associated issue, its especially cool how close to numpy performance you were able to get out of windowed arrays. A couple thoughts/questions which relate to your specific parcels use case, and then to this PR:
- Are you using a
dask.distributed.Client? I think that the speedup you are experiencing may be coming from the larger chunk reads rather than parallelized file reading. The netCDF backend for xarray has a per-process lock that prevents multiple threads within a process from reading from disk at the same time.
xarray/backends/netCDF4_.py
try:
with self.datastore.lock:
original_array = self.get_array(needs_lock=False)
array = getitem(original_array, key)The dask default for multi-core jobs is just to use multiple threads, because of the lock your reading speed with 1 vs 28 cores would be the same. In order to actually parallelize the file reading you'd need to set up a dask.distributed.Client and prescribe multiple single threaded processes by calling something like dask.distribured.Client(n_workers=28, threads_per_worker=1) before running the code for your parcels simulation.
- I am also wondering if you have tried changing the
max_cache_bytesin.to_chunk_cached_arrays(). With your chunking scheme (i.e. only chunking in time),ChunkCachedArraysshould (to my knowledge) perform essentially the same as this modified version ofWindowedArraysby pulling in large, multi step chunks of model data all at once. However, the default formax_cache_bytesis 600 MB, which would definitely not be enough to hold multiple model time levels, whenmax_cache_bytesfor a variable is exceededChunkCachedArraysdoesn't save the data to the cache at all, which would of course be very slow. - I am concerned this could introduce a bug in the interaction between
self._max/max_levelsandtime_chunk. With these changes, the caching process loads an entire chunk worth of time levels on every read, however, when running intoself._maxcap, it still removes individual time levels. I worry that this could lead to weird behavior where not all of the time levels from a chunk are removed from the cache, and then the chunk is reloaded, leading to duplicate time levels in the cache. - On the point of generalization, I think it depends on how these changes compare to using
ChunkCachedArraywith comparably large chunks being loaded at once. If it turns out that this is significantly faster than aChunkCachedArraywith chunking of{'lat':-1,'lon':-1,'time':24*28}then I think it is definitely worth getting into the code base for other users, and I'd be happy to help work on a solution to themax_bytes/time_chunkproblem.
|
Thanks!
|
|
Awesome! Good point that you would loose the speedup from the dask client with
|
|
I tested the same setup (2000 particles distributed randomly through space and time for 100 days) with both chunk cached arrays and my suggested windowed arrays backend. Lists of times are from repeat tests
Taking the two setups in bold above, trying instead releasing 2000 particles at the start of the simulation rather than randomly distributed through time, I get:
Slight improvement in some situations, not universal. I think it'd be possible to persuade max_levels to coordinate with time_chunks (e.g. enforcing that it needed to be an integer multiple of time_chunks, and throwing out timechunks together), but if there isn't any speed-up to be gained then probably not worth introducing the complication. Looking back at how I started down this rabbit hole; the "squeezing performance" docs implied that windowed arrays were going to be a more intelligent way of loading only some data when particles covered the whole domain. It looks like you've got the cached chunked arrays backend working well enough that it's actually fine in that scenario, as long as the max_cache_bytes reflects the system memory limitations and there's no dask client running. caveats would be:
Let me know your thoughts |
|
I'm curious, what kind of file system is holding the NetCDF files for the benchmarks referenced in the discussion here ? Is this lustre, beegfs, vast, something else ?? |
|
lustre I'd be curious to know if people get different comparisons on different systems |
Description
Changes
class WindowedArrayto grab as many timesteps as requested bytime_chunk, to allow for dask parallelisation of file reading alongside fast numpy interpolation.See details in #3060
Checklist
mainfor normal development,v3-supportfor v3 support)No AI was used in the development/writing of this code