Interactive and dynamically created dict of widgets to xarray sel

[StuckDuckF]

I have a Select that chooses the data variable from an xarray dataset and I have written this little loop to identify and create the widgets necessary to select the data needed for a QuadMesh :

widget_box = pn.WidgetBox('# WidgetBox')
sel_dict = {}

for (i,d) in enumerate(xds[var_name].dims):
    for (j,c) in enumerate(xds[var_name].coords):
        if d == c and (d != 'lon' and d != 'lat'):
            if xds[var_name].coords[c].dtype == np.dtype('<M8[ns]'):
                time_list = []
                for i, el in np.ndenumerate(xds.P0.coords['time'].values):
                    time_list.append(np.datetime_as_string(el, timezone='UTC'))
                time_slider = pnw.DiscreteSlider(options=time_list)
                sel_dict[d] = time_slider
                widget_box.append(time_slider)
            else:
                sel_dict[d] = pnw.DiscreteSlider(options=xds[var_name].coords[c].values.tolist())
                widget_box.append(sel_dict[d])

At the moment if I simply add .value to the Sliders I get a dict that correctly selects when passed to sel but has static values. I would like to retain the interactivity of code like xds.ES.interactive.sel(time=pnw.DiscreteSlider,pres1=pnw.DiscreteSlider).hvplot() which renders the interactive widgets and the plot (which is linked to the widgets). I tried passing a dict of the widgets themselves and other variations but I can not seem to get it. The workflow should be :

1. Use a Select to choose data variable to plot
2. Use the value of the Select in my little loop to create the widgets necessary.
3. Render the widgets and the plot bith linked and interactive.

I am not sure if I am using the right APIs within Panel, so please let me know if there is a more Panel way of doing what I need to do.

Oh and here is what my xarray.Dataset looks like. As you can see different variables have different dimensions hence the need to dynamically creating their widgets.

[Marc]

Hi @StuckDuckF

Looks like a useful example. Would it be possible for you to provide a minimum, reproducible example including a link to the dataset?

This would make it so much easier to try to help. Thanks

[StuckDuckF]

Yes sir! Also if I manage to get it working I would like to submit a PR to the public doc so other people can use it as I’ve seen similar questions around the internet.

Here are the links to the 4 example files.
file 1, file 2, file 3, file 4

You can install fstd2nc using pip install fstd2nc.

import hvplot.xarray
import xarray as xr
import panel as pn
import panel.widgets as pnw
from glob import glob
import fstd2nc
import cartopy
import cartopy.crs as ccrs
import param
from panel.interact import interact
import hvplot
import numpy as np
import geoviews as gv
from pathlib import Path
import collections # Weird dependency has not updated to the new Python where the Callble is in abc.Callable
collections.Callable = collections.abc.Callable

pn.extension()

glob_path = Path(r"C:\Users\spart\Documents\Anaconda-Work-Dir")
file_list = [str(pp) for pp in glob_path.glob("2022*")]
file_list = [pp.split('\\')[-1] for pp in file_list]
file_list

file_path = r"C:\Users\spart\Documents\Anaconda-Work-Dir\{}".format(file_list[3])
xds = fstd2nc.Buffer(file_path).to_xarray()
xds

select_file = pnw.Select(name="File", options=file_list)
select_field = pnw.Select(name="Field", options=list(xds.data_vars))
cmap_sel = pnw.Select(name="Colormap", options=['cool','hot','jet','viridis','brg','rainbow'])
rasterize_toggle = pnw.Toggle(name='Rasterize', button_type='success')

@pn.depends(select_file,select_field) 
def load_file(select_file,select_field):
    file_path = r"C:\Users\spart\Documents\Anaconda-Work-Dir\{}".format(select_file)
    xds = fstd2nc.Buffer(file_path).to_xarray()
    sel_dict = {}
    for (i,d) in enumerate(xds[select_field].dims):
        for (j,c) in enumerate(xds[select_field].coords):
            if d == c and (d != 'lon' and d != 'lat'):
                if xds[select_field].coords[c].dtype == np.dtype('<M8[ns]'):
                    time_list = []
                    for i, el in np.ndenumerate(xds[select_field].coords['time'].values):
                        time_list.append(np.datetime_as_string(el, timezone='UTC'))
                    time_slider = pnw.DiscreteSlider(options=time_list)
                    sel_dict[d] = time_slider
                else:
                    sel_dict[d] = pnw.DiscreteSlider(options=xds[select_field].coords[c].values.tolist())
    output = xds[select_field].interactive.sel(**sel_dict).hvplot(kind='quadmesh', rasterize=True, data_aspect=1, frame_height=800,cmap='jet', crs=ccrs.PlateCarree(), projection=ccrs.PlateCarree(), project=True, geo=True, coastline=True, global_extent=True)
    return output

pn.Row(pn.Column(select_file,select_field,cmap_sel,rasterize_toggle),pn.Column(load_file))

This works well except for the connection between the file and field select and the plot (for some reason the @pn.depends() does not hook them as I understand they should.

Also I still need to have a loaded file in order to populate my select_field, but Ideally I would have it done inside the definition somehow and it would be populated with the default file (first in select_file) and then the user can load another one using the select_file widget.

holoviews 1.15.0
hvplot 0.8.0
panel 0.13.1
geoviews 1.9.5

[Marc]

Hi @StuckDuckF

The example is almost there. It is still not minimum or reproducible though and will take too much of my private time to get started with.

You need to give the code a second iteration so that it can run end to end by it self.

• Add code to automatically download the files and move them to a location that can be used. And please make the files much smaller as having to download several GBs is not an option for me.
• Replace the reference to your own personal folders with some general reference where the downloaded files are located.

You can save the smaller files on Github to get direct links. Seems like Google drive does not provide that.

Thanks.

[StuckDuckF]

@Marc Thank you again for the feedback reduced the files to only 4 data_vars and converted them to .nc files which can easily be opened with xarray.open_dataset(PATH_TO_FILE). They are downloaded using gdown in the script but can also be found here ( File 1, File 2, File 1 and File 1 ). I have included a python script which should work now and only download 200MB (unless you uncomment all 4 files) and a Google Colab Notebook which I could not get to show the interactive Panel objects even if I followed your example in another Discourse Question. Thank you very much for you help I am trying to solve it as well and am experimenting with Param as well now, but I would really appreciate some help in getting this working.

import gdown
import hvplot.xarray
import xarray as xr
import panel as pn
import panel.widgets as pnw
from glob import glob
import cartopy
import cartopy.crs as ccrs
import param
from panel.interact import interact
import hvplot
import numpy as np
from pathlib import Path
import collections 
collections.Callable = collections.abc.Callable

# Download at least 2 example files
gdown.download(id='1vI5xH-kmA4OkoqJJz9keze-vN3cYCIf4', output='2022070700_231_example.nc', quiet=False) # 90 MB only
# gdown.download(id='1aQKSMFoUgoGPisA-VnbkIxa3w6YaLrkm', output='2022070700_234_example.nc', quiet=False) # 220 MB Omitted by default
gdown.download(id='1RnWmfiW3U7VYS4Qoqo7_uFdeAIXAajye', output='2022070700_237_example.nc', quiet=False) # 90 MB only
# gdown.download(id='17JqTO_W62oN3HbBhoHZ9GS4aQOO59oL4', output='2022070700_240_example.nc', quiet=False) # 220 MB Omitted by default

glob_path = Path.cwd()
file_list = [str(pp) for pp in glob_path.glob("2022*")]
file_list = [pp.split('\\')[-1] for pp in file_list]

xds = xr.open_dataset(file_list[0])

select_file = pnw.Select(name="File", options=file_list)
select_field = pnw.Select(name="Field", options=list(xds.data_vars))
cmap_sel = pnw.Select(name="Colormap", options=['cool','hot','jet','viridis','brg','rainbow'])
rasterize_toggle = pnw.Toggle(name='Rasterize', button_type='success')

@pn.depends(select_file,select_field) 
def load_file(select_file,select_field):
    xds = xr.open_dataset(select_file)
    sel_dict = {}
    for (i,d) in enumerate(xds[select_field].dims):
        for (j,c) in enumerate(xds[select_field].coords):
            if d == c and (d != 'lon' and d != 'lat'):
                if xds[select_field].coords[c].dtype == np.dtype('<M8[ns]'):
                    time_list = []
                    for i, el in np.ndenumerate(xds[select_field].coords['time'].values):
                        time_list.append(np.datetime_as_string(el, timezone='UTC'))
                    time_slider = pnw.DiscreteSlider(options=time_list)
                    sel_dict[d] = time_slider
                else:
                    sel_dict[d] = pnw.DiscreteSlider(options=xds[select_field].coords[c].values.tolist())
    return xds[select_field].interactive.sel(**sel_dict).hvplot(kind='quadmesh', rasterize=True, data_aspect=1, frame_height=800,cmap='jet', crs=ccrs.PlateCarree(), projection=ccrs.PlateCarree(), project=True, geo=True, coastline=True, global_extent=True)

pn.Row(pn.Card(select_file,select_field),load_file)

[Marc]

Hi @StuckDuckF

I’ve tried to work with the code. I’ve refactored it a bit to make it easier for me to work with.

import collections
from pathlib import Path

import cartopy.crs as ccrs
import gdown
import hvplot
import hvplot.xarray
import numpy as np
import panel as pn
import panel.widgets as pnw
import xarray as xr
import pathlib

collections.Callable = collections.abc.Callable

CONFIG = {
    "2022070700_231_example.nc": "1vI5xH-kmA4OkoqJJz9keze-vN3cYCIf4",
    "2022070700_237_example.nc": "1RnWmfiW3U7VYS4Qoqo7_uFdeAIXAajye",
}

for file, file_id in CONFIG.items():
    if not pathlib.Path(file).exists():
        gdown.download(id=file_id, output=file, quiet=False)

file_list = [pathlib.Path(file) for file in CONFIG]

def open_dataset(file):
    key = f"file {file}"
    if not key in pn.state.cache:
        pn.state.cache[key]=xr.open_dataset(file)
    return pn.state.cache[key]

def get_sub_dataset(file, field):
    xds = open_dataset(file)
    return xds[field]

def get_options(file, field):
    options = {}
    xds_sub = get_sub_dataset(file, field)
    for (_, d) in enumerate(xds_sub.dims):
        for (_, c) in enumerate(xds_sub.coords):
            if d == c and (d != "lon" and d != "lat"):
                if xds_sub.coords[c].dtype == np.dtype("<M8[ns]"):
                    time_list = []
                    for _, el in np.ndenumerate(
                        xds_sub.coords["time"].values
                    ):
                        time_list.append(np.datetime_as_string(el, timezone="UTC"))
                    options[d] = time_list
                else:
                    options[d] = xds_sub.coords[c].values.tolist()
    return options

def get_widgets(file, field):
    options=get_options(file, field)
    print("options", file,field,options)
    return {key: pnw.DiscreteSlider(options=value) for key, value in options.items()}

xds = open_dataset(file_list[0])
select_file = pnw.Select(name="File", options=file_list)
select_field = pnw.Select(name="Field", value="HU", options=list(xds.data_vars))

# I just want to see the options
def warm():
    for file in file_list:
        xds = open_dataset(file)
        for field in xds.data_vars:
            options = get_options(file, field)
            print(file, field, options)
warm()

@pn.depends(select_file, select_field)
def load_file(file, field):
    xds_sub = get_sub_dataset(file,field)
    widgets = get_widgets(file, field)
    
    return (
        xds_sub
        .interactive.sel(**widgets)
        .hvplot(
            kind="quadmesh",
            rasterize=True,
            data_aspect=1,
            frame_height=800,
            cmap="jet",
            crs=ccrs.PlateCarree(),
            projection=ccrs.PlateCarree(),
            project=True,
            geo=True,
            coastline=True,
            global_extent=True,
        )
    )


pn.Row(pn.Card(select_file, select_field), load_file).servable()

For me it now works like below for me

For me it seems to work as expected. Could you please try to describe the issue again? Thanks.

[StuckDuckF]

Thank you again @Marc . The issue is to be able to select a file which then populates the widgets which are used to manipulate the interactive hvplot component. The widgets are created dynamically based on the selected file but I am unable to hook onto file select to recreate the plot and repopulate the options of the widgets. I do not want to load in the files every time one of the widgets is changed (except of course the file selector widget).

[Marc]

Hi @StuckDuckF .

Do you mean you would like to avoid reloading the file? If that is the case my code already avoids this using pn.state.cache. But as the dataset is large I don’t know if it will lead to a memory issue.

[StuckDuckF]

Yes Mr. @Marc . I will be working with a directory of 1000s of files and I would like to load them only if the user chooses them and the cache function you’ve implemented is awesome but I am afraid of keeping GBs of files in cache. Also do you have any suggestions in terms of performance of hvplot? I have read that it is better to recreate the plot completely on widget change rather than having a interactive DataSet for large files.

[Marc]

Hi @StuckDuckF

Regarding performance its really hard to say because it depends on so many things. Its much to recommend something when a specific issue has been identified.

But caching is normally as very good way to speed up applications. In you case you might implement a caching mechanism that stores only the X latest files in memory.

[Marc]

Hi @StuckDuckF

Could you share a larger version of the below CONFIG?

CONFIG = {
    "2022070700_231_example.nc": "1vI5xH-kmA4OkoqJJz9keze-vN3cYCIf4",
    "2022070700_237_example.nc": "1RnWmfiW3U7VYS4Qoqo7_uFdeAIXAajye",
}

It would be quite interesting to try out with more files.

[StuckDuckF]

Thank you Mr. @Marc . I will share the 4 files I have already. As for performance I will try to see how to maybe cache the last 3 files max. Also I am looking into how to show a loading spiner for loading files, generating plots and on interaction (as you’ve experienced playing with global data sometimes takes a couple of seconds to rasterize).

[StuckDuckF]

Here is a larger list of files Mr. @Marc , but I still do not see the plot being remade on file select nor on data variable select.

CONFIG = {
    "2022070700_231_example.nc": "1vI5xH-kmA4OkoqJJz9keze-vN3cYCIf4",
    "2022070700_234_example.nc": "1aQKSMFoUgoGPisA-VnbkIxa3w6YaLrkm",
    "2022070700_237_example.nc": "1RnWmfiW3U7VYS4Qoqo7_uFdeAIXAajye",
    "2022070700_240_example.nc": "17JqTO_W62oN3HbBhoHZ9GS4aQOO59oL4",
}

I see the options change and being “logged” in my JupyterLab but not on the plot (I tried .show() to see if it will be interactive but that did not work)

[Marc]

Ahh. Now I start to understand your issue.

I had this issue tornado.websocket.WebSocketClosedError with xarray app · Issue #3743 · holoviz/panel (github.com) when running it on the Panel server. I did not see it as related to your issue as you where running in Notebook. But now I think it is.

Please report this bug. You can add it to the bug I’ve reported and explain it does not work in notebook either.

[StuckDuckF]

Thank you so much sir. I thought I was a moron or a bad programmer for not getting this to work. If I manage to solve this or get any help I will update this tread.

[Marc]

For now @StuckDuckF you can work around the issue by not using hvplot.interactive. Instead something like the below should do it.

import collections
from pathlib import Path

import cartopy.crs as ccrs
import gdown
import hvplot
import hvplot.xarray
import numpy as np
import panel as pn
import panel.widgets as pnw
import xarray as xr
import pathlib

collections.Callable = collections.abc.Callable

pn.extension(throttled=True, template="fast")
pn.state.template.param.update(site="Awesome Panel", title="XArray data app with dynamically created widgets")

CONFIG = {
    "2022070700_231_example.nc": "1vI5xH-kmA4OkoqJJz9keze-vN3cYCIf4",
    "2022070700_234_example.nc": "1aQKSMFoUgoGPisA-VnbkIxa3w6YaLrkm",
    "2022070700_237_example.nc": "1RnWmfiW3U7VYS4Qoqo7_uFdeAIXAajye",
    "2022070700_240_example.nc": "17JqTO_W62oN3HbBhoHZ9GS4aQOO59oL4",
}

for file, file_id in CONFIG.items():
    if not pathlib.Path(file).exists():
        gdown.download(id=file_id, output=file, quiet=False)

file_list = [pathlib.Path(file) for file in CONFIG]

# Data Transformations

def open_dataset(file) -> xr.Dataset:
    key = f"file {file}"
    if not key in pn.state.cache:
        pn.state.cache[key]=xr.open_dataset(file)
    return pn.state.cache[key]

def get_sub_dataset(file, field):
    xds = open_dataset(file)
    return xds[field]

def get_options(file, field):
    options = {}
    xds_sub = get_sub_dataset(file, field)
    for (_, d) in enumerate(xds_sub.dims):
        for (_, c) in enumerate(xds_sub.coords):
            if d == c and (d != "lon" and d != "lat"):
                if xds_sub.coords[c].dtype == np.dtype("<M8[ns]"):
                    time_list = []
                    for _, el in np.ndenumerate(
                        xds_sub.coords["time"].values
                    ):
                        time_list.append(np.datetime_as_string(el, timezone="UTC"))
                    options[d] = time_list
                else:
                    options[d] = xds_sub.coords[c].values.tolist()
    return options

def plot(xds_sub, **kwargs):
    return (xds_sub
        .sel(**kwargs)
        .hvplot(
            kind="quadmesh",
            rasterize=True,
            data_aspect=1,
            frame_height=800,
            cmap="jet",
            crs=ccrs.PlateCarree(),
            projection=ccrs.PlateCarree(),
            project=True,
            geo=True,
            coastline=True,
            global_extent=True,
        )
    )

# App Components

def main_component():
    xds = open_dataset(file_list[0])

    select_file = pnw.Select(name="File", options=file_list).servable(target="sidebar")
    select_field = pnw.Select(name="Field", value="HU", options=list(xds.data_vars)).servable(target="sidebar")

    return pn.Row(
        pn.bind(sub_component, file=select_file, field=select_field),
    )

def sub_component(file, field):
    xds_sub = get_sub_dataset(file=file, field=field)
    widgets = get_widgets(file=file, field=field)
    
    return pn.Column(
        pn.Row(*widgets.values()),
        pn.panel(pn.bind(plot, xds_sub, **widgets)),
    )

def get_widgets(file, field):
    options=get_options(file, field)
    print("options", file,field,options)
    return {key: pnw.DiscreteSlider(options=value, name=key) for key, value in options.items()}

    

main_component().servable()

To improve the speed of the app you could look into using caching.
To improve the user experience you might try to get the plot to size responsively.

[StuckDuckF]

Wow !!! Mr. @Marc that looks great sir. Thank you so much! Running the code above only gives me the main component and not the app you showed in your video and I am not sure how to structure it as an app, would you mind editing your solution so that on .show() it shows the app from the video sir? I think this is great but I wonder if I should document it or just wait until the issue is resolved and rewrite everything for the 1.0.0 release as to not confuse people.