How to create a Parameterized Dashboard with seperation between data transforms and data views

Marc · December 17, 2019, 4:19am

Hi

I was trying to create a Parameterized Dashboard with a Select, a parent and a child plot. The parent and child plot both depending on the selection. The child plot also depending on the range RangeXY of the parent plot. I simply could not get it working in a meaningfull way as it turns out I simply don’t understand the behavior of the @param.depends annotation.

So I’ve tried to create a smaller example below to hopefully get some help.

import param
columns=["a", "b", "c", "d"]

class Dashboard(param.Parameterized):

    categories = param.Selector(objects=columns)
    data = param.String(columns[0])
    
    @param.depends('categories')
    def set_data(self):
        self.data=self.categories
    
    @param.depends("categories")
    def view_categories(self):
        return "Category: " + self.categories
    
    @param.depends("data")
    def view_plot(self):
        return "Plot: " + self.data
    
dashboard = Dashboard()

pn.Column(dashboard.param.categories, dashboard.param.data, dashboard.view_categories, dashboard.view_plot)

When using this Dashboard I would expect all params and views to change when I change categories using the drop down selection. But it does not.

If instead I add dashboard.set_data to the pn.Column then everything get’s reactive.

pn.Column(dashboard.param.categories, dashboard.param.data, dashboard.view_categories, dashboard.view_plot, dashboard.set_data)

I just don’t understand why I need to add set_data to the pn.Column in order for it to get triggered? It does not display anything and I don’t wan’t it to.

To me that is counter intuitive. And it actually took me a long time to figure out. I tried all sorts of things to make it work before I found out this was nescessary.

Marc · December 17, 2019, 5:05am

And this is another example that I really don’t understand why does not work

import param
import panel as pn
pn.extension()
columns=["a", "b", "c", "d"]

class Dashboard(param.Parameterized):

    categories = param.Selector(objects=columns)
    scatter_df = param.String(columns[0])
    bar_df = param.String("bar" + columns[0])
    
    @param.depends('categories')
    def set_scatter_df(self):
        self.scatter_df=self.categories
    
    @param.depends("scatter_df")
    def view_scatter_plot(self):
        self.bar_df = "dummy "+columns[0]
        
        return "Plot: " + self.scatter_df
    
    @param.depends("bar_df")
    def view_bar_chart(self):
        return "Bar: " + self.bar_df
    
    def view(self):
        return pn.Column(dashboard.param.categories, dashboard.view_scatter_plot, dashboard.view_bar_chart, dashboard.set_scatter_df)
    
dashboard = Dashboard()
Dashboard().view()

As I set bar_df in the view_scatter_plot function which we can see is run, I would expect it to change, but it does not.

Marc · December 17, 2019, 5:29am

In order to understand why I think the above is a good idea is that i’m trying to test the claims of https://www.sicara.ai/blog/2018-01-30-bokeh-dash-best-dashboard-framework-python.

So I need the scatter_plot and bar_chart to respond to the user changing the categories. I have implemented that.

I’m also trying to have the bar_chart respond to the rangeXY changes of the scatter_plot. I cannot get the combination of changes to the categories and the rangeXY working.

My code can be found here https://github.com/MarcSkovMadsen/awesome-panel/blob/master/src/pages/gallery/kickstarter_dashboard/main.py

Maybe the idea of seperating the data transformations and the data views is just not a good idea. The purpose of seperating the two is that have the have the Dataframes as param.DataFrames would make it possible to show the filtered data sets in a reactive way if needed and also offer download buttons of the (filtered) data sets to the user.

Marc · December 17, 2019, 5:48am

I also found the MultiSelect could be improved so I’ve filed a feature request.

philippjfr · December 17, 2019, 12:12pm

I think the only concept you are missing is that param.depends by itself only declares a dependency which can be read out by another library to set up an actual callback. For methods that only have side-effects and aren’t passed to another library you will want to set watch=True, so in your first example all you have to change is:

@param.depends('categories', watch=True)
def set_data(self):
    self.data=self.categories

In your second example:

@param.depends('categories', watch=True)
def set_scatter_df(self):
    self.scatter_df=self.categories

Hope that makes sense!

Marc · December 17, 2019, 1:19pm

Aaahhhhhh

That makes a lot of sense.

I had not understood that. Makes sense.

SandervandenOord · December 17, 2019, 6:15pm

It also took me a long time to get that it’s about watch=True.
This page contains documentation on it:
https://panel.pyviz.org/user_guide/Param.html

It feels sort of hidden in the documentation, but at the same time it’s crucial for a parameterized class.

I created a SO-question + answer a while ago on it:
https://stackoverflow.com/questions/57870870/how-do-i-automatically-update-a-dropdown-selection-widget-when-another-selection

Marc · December 18, 2019, 2:39am

Ok. So both examples I can get working.

The second one after some “bug fixes”. This is the version of the second example that works

import param
import panel as pn
pn.extension()
columns=["a", "b", "c", "d"]

class Dashboard(param.Parameterized):

    categories = param.Selector(objects=columns)
    scatter_df = param.String(columns[0])
    bar_df = param.String("bar" + columns[0])
    
    @param.depends('categories', watch=True)
    def set_scatter_df(self):
        self.scatter_df=self.categories
    
    @param.depends("scatter_df")
    def view_scatter_plot(self):
        self.bar_df = "dummy "+self.categories
        display("scatter")
        return "Plot: " + self.scatter_df
    
    @param.depends("scatter_df")
    def view_bar_chart(self):
        return "Bar: " + self.bar_df
    
    def view(self):
        return pn.Column(dashboard.param.categories, dashboard.view_scatter_plot, dashboard.view_bar_chart)
    
dashboard = Dashboard()
dashboard.view()

But for some reason I still cannot get the large example working. I will keep trying.

Marc · December 18, 2019, 3:01am

Another thing I learned is that the @pn.depends annotated function does not raise exceptions in Panel.

So now I can see that it was also one of my challenges. There is an error in a function. I don’t get any information about it. So I did not investigate that.

@philippjfr. I’ve tried to look at the Panel and Param documentation but I cannot find any information on how to turn on error messages. How do I do that?

philippjfr · December 18, 2019, 3:09am

Is this in the notebook or a deployed app? The deployed app should definitely be logging the error messages in the notebook they get logged to the browser console. The only case that I’m aware of that suppresses errors was when distributed is imported, they used to set the bokeh log level suppressing error messages (see this issue) but I fixed that.

Marc · December 18, 2019, 4:11am

When I wrote it was notebook.

Marc · December 18, 2019, 4:49am

And @philippjfr. Thanks for the help.

Now I think I’ve got the final missing link.

I wan’t to be able to react to changes in the rangeXY of a scatter plot. It just confused me a lot that

In hvplot.pandas if I have multiple series with different colors it’s actually not a scatter plot but a Ndoverlay.
And rangeXY works for a scatter plot but not for Ndoverlay

Works

# Works because scatter is holoviews.element.chart.Scatter
import hvplot.pandas
import panel as pn
import pandas as pd
import holoviews as hv

pn.extension()

data = [
    (1,2, "a"),
    (3,4, "b"),
]
df = pd.DataFrame(data, columns=["x","y", "z"])

scatter = df.hvplot.scatter(x="x", y="y")

rangexy = hv.streams.RangeXY(source=scatter)

@pn.depends(rangexy.param.x_range, rangexy.param.y_range)
def view(x_range, y_range):
    return str((x_range, y_range))
    
pn.Column(scatter, view, sizing_mode="stretch_width")

Does not work

# Does not work because scatter is holoviews.core.overlay.NdOverlay
import hvplot.pandas
import panel as pn
import pandas as pd
import holoviews as hv

pn.extension()

data = [
    (1,2, "a"),
    (3,4, "b"),
]
df = pd.DataFrame(data, columns=["x","y", "z"])

scatter = df.hvplot.scatter(x="x", y="y", by="z") # CHANGE!

rangexy = hv.streams.RangeXY(source=scatter)

@pn.depends(rangexy.param.x_range, rangexy.param.y_range)
def view(x_range, y_range):
    return str((x_range, y_range))
    
pn.Column(scatter, view, sizing_mode="stretch_width")

Works - Problem Solved

# Works because we use scatter.last for rangeXy and its a holoviews.element.chart.Scatter
import hvplot.pandas
import panel as pn
import pandas as pd
import holoviews as hv

pn.extension()

data = [
    (1,2, "a"),
    (3,4, "b"),
]
df = pd.DataFrame(data, columns=["x","y", "z"])

scatter = df.hvplot.scatter(x="x", y="y", by="z")

rangexy = hv.streams.RangeXY(source=scatter.last) # CHANGE!

@pn.depends(rangexy.param.x_range, rangexy.param.y_range)
def view(x_range, y_range):
    return str((x_range, y_range))
    
pn.Column(scatter, view, sizing_mode="stretch_width")

@philippjfr. Would it be an idea to add a feature request for rangeXY supporting Ndoverlays?

Marc · December 18, 2019, 7:07am

Ups @philippjfr One more thing I really don’t get. I really don’t understand why

These 3 display of the scatter plot are independent and
rangeXY only works on the last two display of the scatter plot. rangeXY does not work on the first display??

I’m running this in a notebook

CELL 1

import param
import panel as pn
import holoviews as hv
import pandas as pd
import hvplot.pandas
pn.extension()

Cell 2

data = [
    (1,2, "a"),
    (3,4, "b"),
]
df = pd.DataFrame(data, columns=["x","y", "z"])

scatter = df.hvplot.scatter(x="x", y="y", by="z")
scatter

Cell 3

rangexy = hv.streams.RangeXY(source=scatter.last)
@pn.depends(rangexy.param.x_range, rangexy.param.y_range)
def view(x_range, y_range):
    return str((x_range, y_range))
pn.Column(rangexy, scatter, view, sizing_mode="stretch_width")

Cell 4

scatter

xavArtley · December 19, 2019, 2:48pm

Your scatter object is an holoviews element and the plots are bokeh elements.
Bokeh element are uniques in a document.
So the first time you display scatter ( cell 2) it creates a bokeh plot without the link which is created only on the third cell.

philippjfr · December 19, 2019, 9:49pm

That is correct, a stream must be attached before the plot is displayed.