Programmatically Change Selected Data in HeatMap

HoloViews
1

This is certainly related to can-i-change-which-points-are-selected-programmatically, but the solution there is insufficient.

In my application I generate a heatmap indicating the status of many tasks, with metadata in the tooltips. I want users to be able to click the cell and highlight the corresponding row in a Tabulator widget from Panel. All of this works as expected using hv.streams.Selection1D

I also want users to be able to click rows in the table and have the corresponding cells in the heatmap selected - simply the inverse of the above. This what I am struggling to get working. I want to be able to programmatically tell the heatmap what cells have been selected, and get the same alpha applied to the unselected data.

Below is what I have tried so far as MRE, but I am not sure how to get this to work as needed. In addition to what is below, I have also tried updating the selection_stream, but this does not propagate the selection back to the heatmap, only to the table.

I am hoping there is an easy way to do this, as I feel like I have overcomplicated everything. Any help is much appreciated!

import holoviews as hv
import numpy as np
import pandas as pd
import panel as pn

hv.extension('bokeh')
pn.extension('tabulator')

# setup data
dim = 3
bounds=(-0.5, dim-0.5)
xx, yy = np.meshgrid(range(dim), range(dim))
df = pd.DataFrame({'x': xx.flatten(), 'y': yy.flatten(), 'z': np.random.uniform(size=dim*dim)})

# setup table and heatmap
tb = pn.widgets.Tabulator(df, selectable=True, disabled=True)
heatmap = hv.HeatMap(df, kdims=['x', 'y'], vdims=['z']).opts(tools=['tap', 'box_select'], aspect=1, responsive=True, xlim=bounds, ylim=bounds).redim.range(z=(0, 1.0))

# setup selection stream
selection_stream = hv.streams.Selection1D(source=heatmap)

# callback to update the table selection. need to transpose index
def update_table(index):
    tb.selection = [(i % dim) * dim + i // dim for i in index]
    return tb

# callback to update the heatmap. this is where the problems are
def update_heatmap(index):
    # if no index, return entire heatmap
    if not index:
        return heatmap
    # return a slice of the heatmap using the given indices
    # problems:
    # - need to reset the bounds and z-range
    # - rendered cells are too large until their row/column forces them into a normal size
    # - unselected cells simply are not shown, since we filtered them out
    # - cannot deselect cells
    return heatmap.iloc[index].opts(xlim=bounds, ylim=bounds).redim.range(z=(0, 1.0))

# create simple layout with binds
pn.Column(
    pn.bind(update_table, selection_stream.param.index),
    pn.bind(update_heatmap, tb.param.selection)
).servable()
2

I tried but could not get it working. I’ve opened a feature request here.

1 Like
3

I appreciate you trying to figure it out, and for opening the feature request! I ended working around it by basically just embedding another column in my data to indicate if it was selected, and then used that column as a dim assigned to alpha parameter. There is a bit of rearranging what I had, but it is not too bad. Having a better built-in solution would certainly be great though!

import holoviews as hv
import numpy as np
import pandas as pd
import panel as pn

hv.extension('bokeh')
pn.extension('tabulator')

# setup data
dim = 3
bounds=(-0.5, dim-0.5)
xx, yy = np.meshgrid(range(dim), range(dim))

# added the column 's' to use as the selection to manipulate the alpha
df = pd.DataFrame({'x': xx.flatten(), 'y': yy.flatten(), 'z': np.random.uniform(size=dim*dim), 's': np.ones(shape=dim*dim)})

# callback to update the table selection. need to transpose index
def create_heatmap(data):
    if data:
        df.loc[data, 's'] = 1.0
        df.loc[df.index.difference(data), 's'] = 0.2
    else:
        df.loc[:, 's'] = 1.0
    heatmap = hv.HeatMap(
        df,
        kdims=['x', 'y'],
        vdims=['z', 's']).opts(
            tools=['tap', 'box_select'],
            aspect=1,
            responsive=True,
            xlim=bounds,
            ylim=bounds,
            alpha=hv.dim('s')
        ).redim.range(z=(0, 1.0))
    return heatmap

# update callback
def update_selection(*events):
    for event in events:
        if event.name == 'index':
            # transpose
            indices = [(i % dim) * dim + i // dim for i in event.new]
            tb.selection = indices
            # short circuit to avoid infinite recursion since the line
            # right above will retrigger this method
            return
        pipe.send(event.new)

# setup table
tb = pn.widgets.Tabulator(df, selectable=True, disabled=True)

# create pipe, and use a stream for the heatmap dynamic map,
# and use the dmap as the source for the Selection1D
pipe = hv.streams.Pipe(data=[])
dmap = hv.DynamicMap(create_heatmap, streams=[pipe])
selection_stream = hv.streams.Selection1D(source=dmap)

# set up watchers to trigger the selection update
tb_watcher = tb.param.watch(update_selection, ['selection'], onlychanged=True)
heatmap_watcher = selection_stream.param.watch(update_selection, ['index'], onlychanged=True)

pn.Column(
    tb,
    dmap
).servable()
1 Like
4

Example of the above code working as expected

Animation

1 Like
5

Congrats on finding a good workaround. And thanks for sharing with the community.

6

Extending the above to larger dataframes; and looking to apply filter to the dataframe to what is selected (this bit is fine), but then editing a value within that dataframe…

import holoviews as hv
import numpy as np
import pandas as pd
import panel as pn

hv.extension('bokeh')
pn.extension('tabulator')

# setup data
dim = 341
bounds=(-0.5, dim-0.5)
xx, yy = np.meshgrid(range(dim), range(dim))

# added the column 's' to use as the selection to manipulate the alpha
df = pd.DataFrame({'x': xx.flatten(), 
                   'y': yy.flatten(), 
                   'z': np.random.uniform(size=dim*dim), 
                   's': np.ones(shape=dim*dim)})

selected_indices = pn.widgets.ArrayInput(name="selected_index")

# callback to update the table selection. need to transpose index
def create_heatmap(data):
    if data:
        df.loc[data, 's'] = 1.0
        df.loc[df.index.difference(data), 's'] = 0.2
    else:
        df.loc[:, 's'] = 1.0
    heatmap = hv.HeatMap(
        df,
        kdims=['x', 'y'],
        vdims=['z', 's']).opts(
            tools=['tap', 'box_select'],
            aspect=1,
            responsive=True,
            xlim=bounds,
            ylim=bounds,
            alpha=hv.dim('s')
        ).redim.range(z=(0, 1.0))
    return heatmap

# update callback
def update_selection(*events):
    for event in events:
        if event.name == 'index':
            # transpose
            indices = [(i % dim) * dim + i // dim for i in event.new]
            tb.selection = indices
            
            if indices:
                selected_indices.value = indices
            else:
                selected_indices.value = []
            # short circuit to avoid infinite recursion since the line
            # right above will retrigger this method
            return
        pipe.send(event.new)

# setup table
tabulator_formatters = {
    'z': {'type': 'progress', 'max': 1},
    }

tabulator_editors = {
    'x': None,
    'y': None,
    'z': {'type': 'number', 'min': 0, 'max': 1, 'step': 0.01},
    's': None,
    }

tb = pn.widgets.Tabulator(df, 
                          selectable=True, 
                          disabled=False,
                          layout="fit_columns",
                          editors=tabulator_editors,
                          formatters=tabulator_formatters,
                          initial_page_size=40,
                          )

tb.on_edit(
    lambda e: print(
        f"Updated col: {e.column}, row: {e.row}, old: {e.old}, new: {e.value}"))

def df_filter(df, indices):
    if not indices:
        return df
    return df.loc[df.index.isin(indices)]

# create pipe, and use a stream for the heatmap dynamic map,
# and use the dmap as the source for the Selection1D
pipe = hv.streams.Pipe(data=[])
dmap_heatmap = hv.DynamicMap(create_heatmap, streams=[pipe])
selection_stream = hv.streams.Selection1D(source=dmap_heatmap)

# set up watchers to trigger the selection update
tb_watcher = tb.param.watch(
    update_selection, 
    ['selection'], 
    onlychanged=True,
    )
heatmap_watcher = selection_stream.param.watch(
    update_selection, 
    ['index'], 
    onlychanged=True,
    )

tb.add_filter(pn.bind(df_filter, indices=selected_indices))

pn.serve(
    pn.Row(
        tb,
        dmap_heatmap)
)

It runs, the panel allows selection via the sidebar dataframe and highlights. Select by the image and it filters the dataframe. So far, so good.

Attempt to modify the dataframe from a dataframe selection, no problem.

Attempt to modify the filtered dataframe (from image selection), while the dataframe acknowledges the update, there are a load of warnings fired in the console:

ERROR:tornado.application:Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x00000246C6656090>>, <Task finished name='Task-1836' coro=<ServerSession.with_document_locked() done, defined at ~\.conda\envs\flask\Lib\site-packages\bokeh\server\session.py:77> exception=IndexError('index 82613 is out of bounds for axis 0 with size 1')>)
Traceback (most recent call last):
  File "~\.conda\envs\flask\Lib\site-packages\tornado\ioloop.py", line 750, in _run_callback
    ret = callback()
          ^^^^^^^^^^
  File "~\.conda\envs\flask\Lib\site-packages\tornado\ioloop.py", line 774, in _discard_future_result
    future.result()
  File "~\.conda\envs\flask\Lib\site-packages\bokeh\server\session.py", line 98, in _needs_document_lock_wrapper
    result = await result
             ^^^^^^^^^^^^
  File "~\.conda\envs\flask\Lib\site-packages\panel\reactive.py", line 508, in _change_coroutine
    state._handle_exception(e)
  File "~\.conda\envs\flask\Lib\site-packages\panel\io\state.py", line 468, in _handle_exception
    raise exception
  File "~\.conda\envs\flask\Lib\site-packages\panel\reactive.py", line 506, in _change_coroutine
    self._change_event(doc)
  File "~\.conda\envs\flask\Lib\site-packages\panel\reactive.py", line 524, in _change_event
    self._process_events(events)
  File "~\.conda\envs\flask\Lib\site-packages\panel\widgets\tables.py", line 1329, in _process_events
    return super()._process_events(events)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~\.conda\envs\flask\Lib\site-packages\panel\reactive.py", line 1422, in _process_events
    self._process_data(events.pop('data'))
  File "~\.conda\envs\flask\Lib\site-packages\panel\widgets\tables.py", line 1419, in _process_data
    return super()._process_data(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~\.conda\envs\flask\Lib\site-packages\panel\reactive.py", line 1410, in _process_data
    self.value = old_data
    ^^^^^^^^^^
  File "~\.conda\envs\flask\Lib\site-packages\param\parameterized.py", line 528, in _f
    instance_param.__set__(obj, val)
  File "~\.conda\envs\flask\Lib\site-packages\param\parameterized.py", line 530, in _f
    return f(self, obj, val)
           ^^^^^^^^^^^^^^^^^
  File "~\.conda\envs\flask\Lib\site-packages\param\parameterized.py", line 1553, in __set__
    obj.param._call_watcher(watcher, event)
  File "~\.conda\envs\flask\Lib\site-packages\param\parameterized.py", line 2526, in _call_watcher
    self_._execute_watcher(watcher, (event,))
  File "~\.conda\envs\flask\Lib\site-packages\param\parameterized.py", line 2506, in _execute_watcher
    watcher.fn(*args, **kwargs)
  File "~\.conda\envs\flask\Lib\site-packages\panel\widgets\tables.py", line 141, in _reset_selection
    idx = event.old.index[sel]
          ~~~~~~~~~~~~~~~^^^^^
  File "~\AppData\Roaming\Python\Python312\site-packages\pandas\core\indexes\base.py", line 5389, in __getitem__
    return getitem(key)
           ^^^^^^^^^^^^
IndexError: index 82613 is out of bounds for axis 0 with size 1
Updated col: z, row: 82613, old: 0.47128191914644113, new: 0.49

The .onedit callback occurs after the edit has been performed, otherwise resetting the filter then reapplying would be an option. Anyone know of a way of getting myself into the middle of the Tabulator update?

(Besides the sledgehammer approach of modifying panel.model.tabulator.TableEditEvent ?)

Thanks!

7

Ah, I’m using panel 1.5.4 - I see 1.6.1 addresses #7712, and it may have an effect on behaviour. Will update and try it.

edit: No change on 1.6.1. Same.

8

Perhaps this:

Might be a holoviews index transposed thing.