Overlaying Holoviews scatter plots output by parametrized class

I am working on creating a GUI dashboard interface for an existing project. I’ve written a parameterized class that works well to add widgets and return a holoviews scatter plot. I have two instances created from my parameterized class, so two sets of widgets for control and two scatterplots. This works well, but I would like to overlay the scatterplot outputs.

Here is a screenshot of the version that works:

This is what I would like to create, but the plot overlay does not update.
Screen Shot 2019-12-24 at 3.41.35 PM

The second screenshot was generated from a function that takes the two instances of the parameterized class as arguments.

I’m clearly missing something here. I like the code reuse that I am getting from the parameterized class, but I’m not seeing how to have a reactive dashboard that ties together two instances of the class.

If anyone could point me in the right direction, I would very much appreciate the help!

If you could post a tiny runnable version of the code you have so far, using synthetic data (see the hv.Scatter docs for examples), we can probably show you how to do what you want.

@jbednar, here is a runnable condensed version of the code I am working with. I have a vague notion that the parameters should possibly be pushed down from the CapdataGui class into the CapData class, but the CapData class shown here is a simplified version of code that I have in my existing project (pvcaptest). I’m trying to understand panel better and do some prototyping before I start doing any refactoring of the pvcaptest code to make it more compatible with panel.

import pandas as pd
import numpy as np
import holoviews as hv
import panel as pn
import param

hv.extension('bokeh')
pn.extension() # not sure if this is needed?

class CapData(object):
    def __init__(self):
        super(CapData, self).__init__()
        self.df = pd.DataFrame()
        self.df_flt = None
    
    def filter_irradiance(self, low, high, column_name):
        flt_str = '@low <= ' + column_name + ' <= @high'
        ix = self.df.query(flt_str).index
        self.df_flt = self.df.loc[ix, :]
    
    def reset_flt(self):
        self.df_flt = self.df.copy()
    
    def scatter(self):
        return hv.Scatter(self.df_flt, 'poa', 'power').opts(width=400, xlim=(0, 1200), ylim=(0, 1250))

# create synthetic data
x = np.arange(0, 1200, 10)
y = x
y2 = x + 50

# create two instances of the CapData class using synthetic data
cd = CapData()
cd.df = pd.DataFrame({'poa':x, 'power':y})
cd.df_flt = cd.df.copy()

cd2 = CapData()
cd2.df = pd.DataFrame({'poa':x, 'power':y2})
cd2.df_flt = cd2.df.copy()

class CapdataGui(param.Parameterized):
    # class wrapping the CapData class to add parameters
    irr_range = param.Range(default=(0,1200), bounds=(0, 1200))

    def __init__(self, capdata, **param):
        super().__init__(**param)
        self.capdata = capdata
    
    @param.depends('irr_range', watch=True)
    def filter_irr(self):
        self.capdata.reset_flt()
        self.capdata.filter_irradiance(*self.irr_range, 'poa')
    
    @param.depends('irr_range', watch=True)
    def view_scatter(self):
        scatter = self.capdata.scatter().relabel(self.name)
        return scatter

gui_1 = CapdataGui(cd, name='gui_1')
gui_2 = CapdataGui(cd2, name='gui_2')

pn.Row(pn.Column(gui_1.param, gui_2.param), pn.Column(gui_1.view_scatter, gui_2.view_scatter))

# overlay = gui_1.view_scatter * gui_2.view_scatter
# Returns:
# TypeError: unsupported operand type(s) for *: 'method' and 'method'

overlay = gui_1.view_scatter() * gui_2.view_scatter() # overlay is not reactive
pn.Row(pn.Column(gui_1.param, gui_2.param), overlay)

Thanks! The simplest answer to your question is that although you can’t overlay methods as you were attempting, you can easily make a HoloViews DynamicMap from a method, and DynamicMaps can be overlaid (see #3 below). But there are some other issues to clear up as well:

  1. Normally, one won’t use Param’s “watch=True” mechanism on a method that returns a value. “watch=True” indicates that Param should re-run that method whenever the indicated parameter changes, and when Param runs it, it won’t have any idea what to do with the returned value; it will just disappear into the void. So if you use watch=True on a method, the body of the method would need to have something that works by side effects, not by returning something. In this case, view_scatter shouldn’t be marked watch=True; there’s no reason Param should re-run that method and then throw away the value when the indicated parameter changes.

  2. Unlike Param, Panel Rows and Columns do know what to do with the output of view_scatter, which is why you can pass the view_scatter method object directly to a Panel Row or Column and see it update when its dependencies change. When given such a method object, Panel evaluates it once to have something to display, and then later re-evaluates it and re-displays the result whenever the dependent parameters change. Given that you want to return a value here, you also want to provide the view_scatter object to Panel, not the evaluated result view_scatter().

  3. As you discovered, gui_1.view_scatter() * gui_2.view_scatter() successfully creates an overlay because the result of those calls is something overlayable (a HoloViews Element in both cases), but you can’t simply overlay a method on a method; Python methods don’t have any notion of “overlaying”. Panel also doesn’t provide any support for overlaying; it only knows how to lay things out alongside each other, not on top of each other. So how do you do it? Well, a HoloViews DynamicMap can dynamically construct anything you want, with any dependencies you need, so you could write a function that overlays the two objects and use that function as the callback for a DynamicMap that you display. However, in this case I think it would be tricky to write a single DynamicMap that would respect the dependencies declared in the separate classes used here. Luckily, individual HoloViews DynamicMaps can be overlaid on each other, so you can simply turn each of the methods into a DynamicMap and then overlay those. HoloViews will print a warning if you stop there, as it doesn’t want DynamicMaps nested inside of Overlay objects, but you call .collate() on the final result to turn it inside out with a single DynamicMap at the top level, as HoloViews expects things to be organized. I.e.:

overlay = hv.Overlay([hv.DynamicMap(gui_1.view_scatter), hv.DynamicMap(gui_2.view_scatter)]).collate()
pn.Row(pn.Column(gui_1.param, gui_2.param), overlay)

I think this achieves the result that you are looking for.

  1. As you suspected, I don’t think you need to have a Capdatagui class in the first place; it says that it is “wrapping the CapData class to add parameters”, but classes don’t become GUI classes just because they have Parameters (e.g. most of our code is Parameterized but only a few things are ever shown in a GUI). Normally one would just add Parameters to the class that actually uses the value of those parameters, i.e. CapData in this case. If you don’t control CapData (e.g. if it’s imported from somewhere else) then it can make sense to have a wrapper, but otherwise it’s much simpler to keep parameters tightly linked to the code that they will be controlling, i.e. in CapData in this case. We made sure Param had no external dependencies so that it would be simple to include Parameterized objects in any codebase, letting you prepare that code for GUI usage without ever committing to any particular GUI library.

The updated runnable example is below (though I didn’t merge CapData and Capdatagui since I wasn’t sure if that was appropriate for you).

import pandas as pd
import numpy as np
import holoviews as hv
import panel as pn
import param

hv.extension('bokeh')

class CapData(object):
    def __init__(self):
        super(CapData, self).__init__()
        self.df = pd.DataFrame()
        self.df_flt = None
    
    def filter_irradiance(self, low, high, column_name):
        flt_str = '@low <= ' + column_name + ' <= @high'
        ix = self.df.query(flt_str).index
        self.df_flt = self.df.loc[ix, :]
    
    def reset_flt(self):
        self.df_flt = self.df.copy()
    
    def scatter(self):
        return hv.Scatter(self.df_flt, 'poa', 'power').opts(width=400, xlim=(0, 1200), ylim=(0, 1250))

# create synthetic data
x = np.arange(0, 1200, 10)
y = x
y2 = x + 50

# create two instances of the CapData class using synthetic data
cd = CapData()
cd.df = pd.DataFrame({'poa':x, 'power':y})
cd.df_flt = cd.df.copy()

cd2 = CapData()
cd2.df = pd.DataFrame({'poa':x, 'power':y2})
cd2.df_flt = cd2.df.copy()

class CapdataGui(param.Parameterized):
    # class wrapping the CapData class to add parameters
    irr_range = param.Range(default=(0,1200), bounds=(0, 1200))

    def __init__(self, capdata, **param):
        super().__init__(**param)
        self.capdata = capdata
    
    @param.depends('irr_range', watch=True)
    def filter_irr(self):
        self.capdata.reset_flt()
        self.capdata.filter_irradiance(*self.irr_range, 'poa')
    
    @param.depends('irr_range')
    def view_scatter(self):
        scatter = self.capdata.scatter().relabel(self.name)
        return scatter

gui_1 = CapdataGui(cd,  name='gui_1')
gui_2 = CapdataGui(cd2, name='gui_2')

def olay(*args):
    widgets = pn.Column(*[a.param for a in args])
    overlay = hv.Overlay([hv.DynamicMap(a.view_scatter) for a in args]).collate()
    return pn.Row(widgets, overlay)

olay(gui_1, gui_2)

Thank you! This solved my immediate issue with the overlay and the additional detail on the other points is very helpful. I do control the project where the CapData class is defined, so I will likely push the parameters into it at some point.

Thanks again for this response and also for the work that you and others at Anaconda put into the entire holoviz stack!

2 Likes

This is a nice example. Thank you.

I struggle how to declare a param similar to the irr_range, but where the bounds depend on the dataframe passed into the class __init__ .

In this example, it would be something like

class CapdataGui(param.Parameterized):

irr_range = param.Range(default=(capdata.df.index[0], capdata.df.index[30]), bounds=(capdata.df.index[0], capdata.df.index[-1])

How can this be declared if df is only really created in __init__?

you can use self.param.irr_range.bounds = (min, max)
in your init

1 Like

But then how does it know of which column in the dataframe? Is it always the index?

Also, is this in the param documentation somewhere? I have searched it but cannot recall reading this.

It’s our eternal shame that there aren’t any docs. We do have funding to develop those and hopefully they’ll start appearing over the next 2 months.

2 Likes

The more traction Panel gets the more of a need there is for good documentation of param. Its the foundation on which we stand :slight_smile:

I’m looking forward to expanded param docs! Thanks for the update.