Panel / holoviews / decimate event processing not working (possible race condition)

Panel
, , , ,
1

I’m using Panel with a paramerterized class to display a set of time series Curves as holoviews Layout.
The number of datapoints per Curve may be quite larege (e.g. 5_000_000 and up). Therefore I use a decimate function to reduce datapoints for the browser (e.g. 4000 per Curve or less).

Example: 9 Curves are shown as Layout three plots with 3 overlayed curves.

Observations:

  • When the number of curves ~< 5, then zooming, panning, etc. works smoothly, and the busy indicator on the panel page gets to an idle state normally
  • When a larger number of curves is displayed, then zooming does not work anymore and the x_range gets reset to the original limits and only the zoomed-in part of the curve is shown on the original x_range (the bokeh server sends the update message to reset the x_range back to its initial setting). In addition the busy indicator does not go back to idle anymore and the cpu load is continueing to stay on a high level. The app is very unresponsive. This only goes back to normal when a full page refresh is done opening a new websocket.
  • When I take out the decimate function and only display 5000 points, for example, everything works without any issue.
  • When I run the app in a jupyter notebook, then the same problem does not occure, in the sense that zooming / panning works on the overlay that is selected. Zooming / panning works also for the other overlays in the complete layout, but the decimate update is not happing without mouse interaction on the other overlays (pan or zoom)

Are there any known issues when using decimate in a configuration together with panel/bokeh server?
Any idea how this problem can be resolved?

stack used:
Panel Server running on following versions:
Python 3.10.7, Panel: 0.14.1, Bokeh: 2.4.3, Param: 1.12.2

Any advice is appreciated
Franz

2

Hi @zieherf

Can you share a minimal, reproducible example (MRE)?

In general, a complete script that can be copied/pasted and immediately run, as-is, with no modifications. This is much more useful than snippets.

1 Like
3

Hi @Hoxbro
I tried to get an example together. “Layout 3 Curve” should work fine, “Layout 5 Curves” and “Layout 8 Curves” shows the reset range when zooming / panning.
The spinner is less busy as in the case I have observed, but is also visible for the latter two cases.


import random

import numpy as np
import holoviews as hv
import panel as pn
import param
from holoviews.operation import decimate
from bokeh.models import CrosshairTool

hv.extension("bokeh")
pn.extension(template="fast")


def redim_range(curve, xrange={}):
    """Re-adjust the xrange. You have to specify a 
    dictionary in the form {dimension: (from, to)}.
    @param xrange:  {x-dimension: (from, to)}
    """
    c = curve
    if xrange and list(xrange.values())[0]:
        c = c.redim.range(**xrange)
    return c


def plotChannel(x, y, xlabel, ylabel, xrange):
    """Create curve plot for the settings given using decimate_min_max to reduce data.
    @param channel: channel name for the y-range settings
    @param x: x data
    @param y: y data
    @param xlabel: label for the x-data
    @param xrange: (xmin, xmax) tuple for the xrange
    """
    return decimate(
        redim_range(hv.Curve({"x": x, "y": y}, ("x", xlabel), ("y", ylabel), label=ylabel).opts(ylabel='temperature [°C]'), {'x': xrange})
    )


@pn.cache
def getMeasurement(channel):
    no_samples = 100_000
    return {
        "x": np.linspace(0, 10, no_samples),
        channel: 100*random.random()*np.random.rand(no_samples)
    }


def plotLayout(layout, width=800, height=250):

    # make crosshair object
    linked_crosshair = CrosshairTool(dimensions="height")

    # make a hook to manipulate bokeh figure properties
    def hook_crosshair(plot, element):
        plot.state.add_tools(linked_crosshair)

    curve_opts = dict(
        width=width,
        height=height,
        line_width=1.5,
        tools=["hover", "undo", "redo"],
        show_grid=True,
        framewise=True,
        fontsize={'labels': 12, 'title': 14, 'legend': 9, 'xticks': 10, 'yticks': 10},
        bgcolor="#fff",
        xformatter='%f',
        hooks=[hook_crosshair],
        title=""
    )

    plot_layout = []    

    # build list of curves
    for i in range(len(layout)):
            
        curve_objs = []
        for channel in layout[i]:
            data = getMeasurement(channel)
            curve_objs.append(plotChannel(data["x"], data[channel], 'time [s]', channel, (-1, 11)))

        overlay = hv.Overlay(curve_objs).collate()

        # add curve with specific options
        plot_layout.append(overlay)


    return (
        hv.Layout(plot_layout)
        .opts(hv.opts.Curve(**curve_opts))
        .cols(1)
    )


def App(app_config, name="Measurement explorer"):
    """A holoviz panel application to explore  measurement data.
    
    @param app_config: TP configuration dictionary. 
    @param name: a name that shall appear on the application.
    """

    layout_names = list(app_config['layouts'].keys())
    
    class ChannelPlot(param.Parameterized):

        layout_name = param.Selector(objects=layout_names)
        action = param.Action(lambda x: x.param.trigger('action'), label='Plot')

        @param.depends('layout_name')
        def view(self):
            layout = app_config['layouts'][self.layout_name]
            return pn.Column(plotLayout(layout))
    
    channelPlot = ChannelPlot(name=name)
    title = pn.pane.HTML(f"<h1>{name}</h1>")

    app = pn.Column(
        title,
        pn.Row(
            pn.panel(channelPlot.param, widgets={"action": {"button_type": "primary"}}),
            channelPlot.view
        )
    )

    return pn.panel(app)


# App configuration
app_config = {
    "layouts": {
        "Layout 3 curves": [
            ("y1", "y2"),
            ("y3",)
        ],
        "Layout 5 curves": [
            ("y1", "y2", "y3"),
            ("y4", "y5")
        ],
        "Layout 8 curves": [
            ("y1", "y2", "y3"),
            ("y4", "y5", "y6"),
            ("y7", "y8")
        ]
    }
}

app = App(app_config, name="Measurement")
app.servable(title="Data explorer")
4

Can you try to reduce your example down to the essential parts?

1 Like
5

Another shot with a more simple code.
Panning/zooming works for 3 curves.
Panning/zooming fails for 5 curves.


import random

import numpy as np
import holoviews as hv
import panel as pn
import param
from holoviews.operation import decimate

hv.extension("bokeh")
pn.extension(template="fast")


@pn.cache
def getMeasurement(channel):
    no_samples = 1000_000
    return {
        "x": np.linspace(0, 10, no_samples),
        channel: 100*random.random()*np.random.rand(no_samples)
    }


def plot_layout(layout_name, width=800, height=250):

    curve_opts = dict(
        width=width,
        height=height,
        line_width=1.5,
        tools=["hover", "undo", "redo"],
        show_grid=True,
        framewise=True,
        fontsize={'labels': 12, 'title': 14, 'legend': 9, 'xticks': 10, 'yticks': 10},
        bgcolor="#fff",
        xformatter='%f',
        title=""
    )

    overlay_objs = []    

    layout = app_config['layouts'][layout_name]

    # build list of curves
    for i in range(len(layout)):
            
        curve_objs = []
        for channel in layout[i]:

            data = getMeasurement(channel)
            x = data["x"]
            y = data[channel]

            curve = decimate(
                hv.Curve(
                    {"x": x, "y": y}, 
                    ("x", 'time [s]'), 
                    ("y", channel), 
                    label=channel
                )
                .opts(ylabel='temperature [°C]')
                .redim.range(x=(-1, 11))
            )

            curve_objs.append(curve)

        # add curve with specific options
        overlay_objs.append(
            hv.Overlay(curve_objs).collate()
        )


    return (
        hv.Layout(overlay_objs)
        .opts(hv.opts.Curve(**curve_opts))
        .cols(1)
    )


def App(app_config):
    layout_names = list(app_config['layouts'].keys())
    layout_name = pn.widgets.Select(name="Layout name", options=layout_names)

    # bind events
    return pn.Row(
        pn.WidgetBox(layout_name),
        pn.bind(plot_layout, layout_name)
    )


# App configuration
app_config = {
    "layouts": {
        "Layout 3 curves": [
            ("y1", "y2"),
            ("y3",)
        ],
        "Layout 5 curves": [
            ("y1", "y2", "y3"),
            ("y4", "y5")
        ]
    }
}

app = App(app_config)
app.servable(title="Data explorer")
6

Found the issue about resetting the x_range when zooming or panning. framewise=True is causing a reset of the x_range in case of larger holoviews layouts. The app still keeps being busy when zooming on larger holoviews layouts, showing many 2022-12-04 16:06:18,329 Patching attribute 'tags' of Figure(id='1151', ...) with [] statements in the trace output.

7

I’m still not 100 % sure I follow the problem, but glad you have found a solution. :slight_smile:

When looking at it, I noticed that the code you provided did not sync the resolution between the different plots. This seems to be related to the use of different kdims. I have made a small example of it:

Different kdims, different labels:

Same kdims, different labels:

8

Hi @Hoxbro, thanks for looking into it.
Sorry for uploading a too complicated example.
The intention was always to only link the x-axis.

The problem I’m still having is that when I have a larger number of curves, and then using the reset action button on the toolbar, it causes the app to be busy for a “long” time.

Can you please run the attached code with panel serve app.py --log-level trace and
observe the busy indicator and the terminal output. Choosing 4 panels, 2 curves each on the app.

It starts to do a lot of messaging on the websocket when the layout gets larger.


import numpy as np
import holoviews as hv
import panel as pn
import param
from holoviews.operation import decimate

hv.extension("bokeh")
pn.extension(template="fast")

no_points = 100_000

curve_opts = dict(
    width=600,
    height=200,
    line_width=1.5,
    tools=["hover", "undo", "redo"],
    show_grid=True,
    bgcolor="#fff",
    xformatter='%f',
    title=""
)


def App():


    class Plotter(param.Parameterized):

        no_panels = param.Selector(objects=[1,2,3,4,5], default=1)
        curves_per_panel = param.Selector(objects=[1,2,3,4,5], default=1)


        @param.depends('no_panels', 'curves_per_panel')
        def view(self):
            t = np.linspace(0, 10, no_points)
            return (
                hv.Layout(
                    [
                        hv.Overlay(
                            [decimate(hv.Curve({'t': t, 'y': np.random.rand(no_points)}, 't', ('y', f"y{p}-{c}"), label=f"y{p}-{c}")) for c in range(self.curves_per_panel)]
                        ).collate() for p in range(self.no_panels)
                    ]
                ).opts(hv.opts.Curve(**curve_opts)).cols(1)
            )


    plotter = Plotter()
    return pn.panel(
        pn.Row(
            pn.panel(plotter.param),
            pn.panel(plotter.view)
        )
    )


app = App()
app.servable()
9

Case 1:
The problem is best seen when increasing the number of plots with one curve on each plot.
Setting number of panels to 4 and no of curves to 1 slows down the app significantly.

Case 2:
When I have 1 plot with 4 curves on it, the performance degradation is not happening.

10

I think a video of it could be helpful. Another thing, what version of holoviews are you using?

11

Holoviews: 1.15.2
Below is a video showing the effect. When you use 5 plots the performance becomes even worse.

12

13

I appreciate your patience with me. This is definitely a bug! When I run the following I can move one plot one time and then it freezes. Can you open an issue here?

import holoviews as hv
import numpy as np
import panel as pn
from holoviews.operation import decimate

hv.extension("bokeh")

x = np.random.rand(1000)
curves = [decimate(hv.Curve(x)) for _ in range(8)]
plot = hv.Layout(curves)

pn.panel(plot).servable()
14

Issue is open.
holoviews layout in combination with “decimate” not responsive when served with panel · Issue #5546 · holoviz/holoviews (github.com)

2 Likes