Lazy dask plotting

[fmfreeze]

I love holoviz tools now for years.
As I increasingly have to deal with “BigData” (larger than memory data), please demystify and dewind my head in one core thing around datashader and dask I thought both together are capable of:

Can datashader plot data from a huge (let’s say 500GB) lazy dask array without the need of coding logic around loading chunks and .compute() them?

Trying to datashade a 5GB lazy dask array, it takes much, much, much longer to plot than call .compute() and then datashading it.

The data is stored in (multiple) hdf5 files, dask chunks are aligned with hdf5 chunks.

[philippjfr]

Loading data from disk will always take a while there’s nothing datashader can do about that (although there’s probably some work we could do to optimize that, e.g. by computing the plot ranges efficiently). That said if you do have the memory available or a cluster to distribute to you can still leverage dask by using .persist() instead of .compute(). That will load the data into memory but still allow datashader to parallelize over each chunk.

[fmfreeze]

Does that also mean, either .persist() or .compute() should be used?
Just passing the lazy dask array works, but seems to be very inefficient.

[ahuang11]

Some related posts

Larger than memory plotting: how to set initial slice to plot

Okay, I think this works import h5py import numpy as np import panel as pn import xarray as xr import holoviews as hv import dask.array as da from holoviews.operation.datashader import rasterize, shade hv.extension('bokeh', logo=False) f = h5py.File('data.h5', "r") full_data = da.from_array(f['data'], chunks=(60*60, 2401), name=False) da = xr.DataArray( full_data, dims=("x", "y"), coords={"x": np.arange(full_data.shape[0]), "y": np.arange(full_data.shape[1])}, ) class BigSpectrogr…

How to plot a huge image out of memory?

It loads the entire dataset before doing anything (and crashes), so I was wondering if it could just skip every 10 at farthest zoom (e.g. is there a decimate function for datashader outside of hv)? import numpy as np import dask.array as da import xarray as xr from datashader import transfer_functions as tf def sample(range_=(0.0,2.4)): xs = da.arange(700000, dtype=np.float32) ys = da.arange(280000, dtype=np.float32) return xr.DataArray( da.random.random((len(ys), len(xs)),…

Also, perhaps converting to zarr format can help

[fmfreeze]

Thanks, I don’t use a distributed cluster, so I’m stuck on a local machine.
Remains the question:
Why is it uncomparably MUCH fast to use datashade(da.compute()) instead of just using datashade(da) on the same amount of data, even when it easily fits into memory?

[philippjfr]

@fmfreeze

Why is it uncomparably MUCH fast to use datashade(da.compute()) instead of just using datashade(da) on the same amount of data, even when it easily fits into memory?

Are you using da.persist()? Because otherwise you are still forcing it to load data into memory (multiple times) during the datashading step, which is not a fair comparison.