Hi all,
New Panel user here. I’m working on a project to create a dashboard to serve frequently refresh radar files. I got the dashboard setup, yet still trying to solve the issue: how to keep the data refresh while the app keeps running.
I found this post last year: Streams from Xarray?
A few options I considered:
(1) use .zarr and try to stream it, there seems to be more progress on that.
(2) use xarray.open_mfdataset periodically to load in the same dataset based on a refreshed file list using glob
(3) use widget.button.on_click to force refreshing the Dataset
I am leaning toward the third option at this point, as it seems the most simple and this project is very urgent. Would love to hear your thoughts. Much appreciated!
A few things I’ve tried already:
(1) I started with this example
https://panel.holoviz.org/reference/widgets/Button.html
and did:
def refresh_ds(event):
# close the currently opened file
today = date.today().strftime("%Y%m%d")
files = sorted(glob.glob(f'2021*04.nc'))[-2:]
ds = xr.open_mfdataset(files)
refresh.on_click(refresh_ds)
I thought the ds I opened earlier in the script will get updated but seemed not after the button is clicked. Seems like the change of ds inside the function is not open to outside the function?
(2) So I tried returning ds from the function. However, this appears only returns None. And the graph function only sees the empty ds when starting the app (I think it’s empty because there is no click yet).
def refresh_ds(event):
# close the currently opened file
today = date.today().strftime("%Y%m%d")
files = sorted(glob.glob(f'2021*04.nc'))[-2:]
ds = xr.open_mfdataset(files)
return ds
ds = refresh.on_click(refresh_ds)
A few questions I have at this point:
(1) Is the button.on_click() capable of returning object?
(2) Where to place the refresh.on_click(refresh_ds) to get the function I want?
(3) What would you do to serve data to this app? Right now we have a ever-growing list of netcdf files, and I only want to serve say the last 1hr of data.
Some of my codes for a better context:
# Multi timestep, angle data:
ds = xr.open_mfdataset('2021*.nc')
### CREATE Widgets
vars_list = ["Reflectivity", "SpecificDifferentialPhase", 'CorrectedReflectivity','DifferentialReflectivity', 'CorrectedDifferentialReflectivity', 'CrossPolCorrelation', 'Velocity']
var_select = pn.widgets.Select(
options=vars_list, value=vars_list[0], name="Variable")
angle_list = list(ds['scan_angle'].values)
angle_select = pn.widgets.Select(options=angle_list, value=angle_list[0], name="Scan Angle")
timeoptions = list(np.arange(-5, 0, 1)) # last 5 timesteps
player = pn.widgets.DiscretePlayer(name="Loop Control", options=timeoptions, value=-1, loop_policy='loop', align='center')
refresh = pn.widgets.Button(name='Refresh Data', button_type='primary')
# Function to reload ds
def refresh_ds(event):
# close the currently opened file
today = date.today().strftime("%Y%m%d")
files = sorted(glob.glob(f'2021*04.nc'))[-2:]
ds = xr.open_mfdataset(files)
return ds
dss = refresh.on_click(refresh_ds)
@pn.depends(var_select.param.value, angle_select.param.value, player.param.value)
def graph(selected_var, angle, timestep):
# get timestamp
print(dss)
timestamp = ds['time'][timestep].astype(int)
ns = 1e-9
dt = datetime.utcfromtimestamp(timestamp * ns)
ts = dt.strftime("%Y/%m/%d %H:%M:%S")
data = ds[selected_var].sel(scan_angle=angle).isel(time=timestep).load()
### part of graph function is hidden.
## Create Dashboard from template
dark_material = pn.template.MaterialTemplate(title='Skyler', theme='dark')
dark_material.sidebar.append(var_select)
dark_material.sidebar.append(angle_select)
dark_material.sidebar.append(refresh)
dark_material.main.append(
pn.Row(
pn.Card(graph)
)
)
dark_material.main.append(
pn.Row(player)
)
dark_material.servable()
