Button to refresh loaded Xarray.Dataset from multiple NetCDF files

Hi all,

New Panel user here. I’m working on a project to create a dashboard to serve frequently refresh radar files. I got the dashboard setup, yet still trying to solve the issue: how to keep the data refresh while the app keeps running.

I found this post last year: Streams from Xarray?

A few options I considered:
(1) use .zarr and try to stream it, there seems to be more progress on that.
(2) use xarray.open_mfdataset periodically to load in the same dataset based on a refreshed file list using glob
(3) use widget.button.on_click to force refreshing the Dataset

I am leaning toward the third option at this point, as it seems the most simple and this project is very urgent. Would love to hear your thoughts. Much appreciated!

A few things I’ve tried already:
(1) I started with this example
and did:

def refresh_ds(event):
    # close the currently opened file
    today = date.today().strftime("%Y%m%d")
    files = sorted(glob.glob(f'2021*04.nc'))[-2:]
    ds = xr.open_mfdataset(files)

I thought the ds I opened earlier in the script will get updated but seemed not after the button is clicked. Seems like the change of ds inside the function is not open to outside the function?

(2) So I tried returning ds from the function. However, this appears only returns None. And the graph function only sees the empty ds when starting the app (I think it’s empty because there is no click yet).

def refresh_ds(event):
    # close the currently opened file
    today = date.today().strftime("%Y%m%d")
    files = sorted(glob.glob(f'2021*04.nc'))[-2:]
    ds = xr.open_mfdataset(files)
    return ds
ds = refresh.on_click(refresh_ds)

A few questions I have at this point:
(1) Is the button.on_click() capable of returning object?
(2) Where to place the refresh.on_click(refresh_ds) to get the function I want?
(3) What would you do to serve data to this app? Right now we have a ever-growing list of netcdf files, and I only want to serve say the last 1hr of data.

Some of my codes for a better context:

# Multi timestep, angle data:
ds = xr.open_mfdataset('2021*.nc')

### CREATE Widgets
vars_list = ["Reflectivity", "SpecificDifferentialPhase", 'CorrectedReflectivity','DifferentialReflectivity', 'CorrectedDifferentialReflectivity', 'CrossPolCorrelation', 'Velocity']
var_select = pn.widgets.Select(
    options=vars_list, value=vars_list[0], name="Variable")
angle_list = list(ds['scan_angle'].values)
angle_select = pn.widgets.Select(options=angle_list, value=angle_list[0], name="Scan Angle")
timeoptions = list(np.arange(-5, 0, 1)) # last 5 timesteps
player = pn.widgets.DiscretePlayer(name="Loop Control", options=timeoptions, value=-1, loop_policy='loop', align='center')
refresh = pn.widgets.Button(name='Refresh Data', button_type='primary')

# Function to reload ds
def refresh_ds(event):
    # close the currently opened file
    today = date.today().strftime("%Y%m%d")
    files = sorted(glob.glob(f'2021*04.nc'))[-2:]
    ds = xr.open_mfdataset(files)
    return ds
dss = refresh.on_click(refresh_ds)

@pn.depends(var_select.param.value, angle_select.param.value, player.param.value)
def graph(selected_var, angle, timestep): 
    # get timestamp
    timestamp = ds['time'][timestep].astype(int)
    ns = 1e-9
    dt = datetime.utcfromtimestamp(timestamp * ns)
    ts = dt.strftime("%Y/%m/%d %H:%M:%S")

    data = ds[selected_var].sel(scan_angle=angle).isel(time=timestep).load() 
    ### part of graph function is hidden.

## Create Dashboard from template
dark_material = pn.template.MaterialTemplate(title='Skyler', theme='dark')


I think you may need a class so that you can utilize self.ds and then I think you need to manually trigger “graph” inside refresh ds.

I created an introduction to Streaming with Panel that lays out the basic challenges and solutions when creating streaming panel apps. I hope it can be used as inspiration.

Panel - Starting a Stream of Data - Showcase - HoloViz Discourse

For number 3, you could check the creation time and filter your list of displayed (or used/loaded) files.

I would agree with @ahuang11: using a class will allow you keep track of the state. Otherwise, your ds will just vanish into memory nirvana.

If you post your full code, maybe the community can have a look and give you some hints.

1 Like

Thank you! I managed to package my code into a class, and the refresh button now works as intended! Here is the full code.

from datetime import datetime, date
import glob

import geoviews as gv
import holoviews as hv
from holoviews.operation.datashader import rasterize
import numpy as np
import panel as pn
import param
from shapely.geometry import Point
import xarray as xr

#ds = xr.open_dataset('/usr/src/app/test_raytheon.nc')
# ns = 1e-9 # number of seconds in a nanosecond
# delta = timedelta(days=1, minutes=15)
# latest = datetime.utcfromtimestamp(ds['time'].values[0].astype(int) * ns)
# start = latest - delta
# date_slider = pn.widgets.DateSlider(name='Timestamp', start=start, end=latest, value=latest)

today = date.today().strftime("%Y%m%d")
files = sorted(glob.glob(f'/data/{today}*.nc'))
#files = sorted(glob.glob(f'2021*.nc'))
ds = xr.open_mfdataset(files)

var_list = ["Reflectivity", "SpecificDifferentialPhase", 'CorrectedReflectivity','DifferentialReflectivity', 'CorrectedDifferentialReflectivity', 'CrossPolCorrelation', 'Velocity']

angle_list = list(ds['scan_angle'].values)

# Package things into class:
class SkylerApp(param.Parameterized):
    # param depends on
    var_selected = param.ObjectSelector(default='Reflectivity', objects=var_list)
    scan_angle = param.ObjectSelector(default=2.0, objects=angle_list)
    timestep = param.ObjectSelector(default = -1, objects=list(np.arange(-70, 0))) #last 30min

    def __init__(self, ds, **params):
        self._ds = ds
        # Refresh data using a button
        self.refresh = pn.widgets.Button(name='Refresh Data', button_type='primary')
        def update_data(event):
            today = date.today().strftime("%Y%m%d")
            files = sorted(glob.glob(f'/data/{today}.nc')) # 500 second
            self._ds = xr.open_mfdataset(files)
            print(f'File list refreshed {files[-5:]}')
        self.base_map() # create basemap

    def base_map(self):
        tiles = gv.tile_sources.StamenTonerBackground().apply.opts()
        # site
        ntc = gv.Text(-73.82, 40.7, 'National Tennis Center').opts(text_font_style='bold')
        site = gv.Points([(-73.8492017,40.7498823)]).opts(color='red', line_color='black', line_width=0.5, size=5)
        site_shapely = Point(-73.8492017,40.7498823)
        first_circle = gv.Shape(site_shapely.buffer(0.0573)).opts(line_color='red', line_dash='dashed', fill_alpha=0,line_width=2, alpha=0.5) #3mi
        second_circle = gv.Shape(site_shapely.buffer(0.0955)).opts(line_color='blue', line_dash='dashed',  fill_alpha=0, line_width=2, alpha=0.5) #5mi
        third_circle = gv.Shape(site_shapely.buffer(0.191)).opts(line_color='green', line_dash='dashed',  fill_alpha=0, line_width=2, alpha=0.5) #10mi
        cirle_anno1 = gv.Text(-73.8492017, 40.807, '3mi').opts(text_alpha=0.6, text_font_size='12px')
        cirle_anno2 = gv.Text(-73.8492017, 40.846, '5mi').opts(text_alpha=0.6, text_font_size='12px')
        cirle_anno3 = gv.Text(-73.8492017, 40.941, '10mi').opts(text_alpha=0.6, text_font_size='12px')
        out = tiles * site * ntc * first_circle * second_circle * third_circle * cirle_anno1 * cirle_anno2 * cirle_anno3
        self._base = out

    @param.depends('var_selected', 'scan_angle', 'timestep')
    def make_map(self):
        ### TODO how to handle the dependency
        ### https://panel.holoviz.org/user_guide/Param.html
        timestamp = self._ds['time'][self.timestep].astype(int)
        ns = 1e-9
        dt = datetime.utcfromtimestamp(timestamp * ns)
        ts = dt.strftime("%Y/%m/%d %H:%M:%S")
        # Need to solve this part
        data = self._ds[self.var_selected].sel(scan_angle=self.scan_angle).isel(time=self.timestep).load() 
        NWSVelocity = [
        NWSReflectivity = [
        if self.var_selected in ['Reflectivity', 'CorrectedReflectivity','DifferentialReflectivity', 'CorrectedDifferentialReflectivity']:
            color_map = NWSReflectivity
        elif self.var_selected in ['Velocity']:
            color_map = NWSVelocity
            color_map = 'gist_ncar'
        data = hv.Dataset(data, vdims=self.var_selected)
        graph = data.to(gv.Image, ["lon", "lat"])
        graph = (
            rasterize(graph).opts(title=ts, height=550, width=800, colorbar=True, cnorm='eq_hist', cmap=color_map, alpha=0.5, tools=['hover'])
        return graph * self._base

skyler = SkylerApp(ds)
## Create Dashboard from template
dark_material = pn.template.MaterialTemplate(title='US Open Skyler', theme='dark')
# dark_material.sidebar.append(skyler.var_selected)
# dark_material.sidebar.append(skyler.angle)
    pn.Row(pn.widgets.DiscretePlayer.from_param(skyler.param['timestep'], loop_policy='loop', interval=2000))
1 Like

Could you share a screenshot or gif @fischcheng . Really curious to see what you have built.

Here it is, still working on the dynamic refresh, right now it’s just keep blinking.