Two way link between a widget and a linked stream

I am trying to create something where I have an image, whose xaxis limits can be controlled by a slider widget. Additionally, I also want to have the option to be able to pan/zoom on the image using the toolbar, and want the slider widget to take values in response to this (so that the widget and the axis on the plot are always in sync).

I managed to make an example that “works”, but not well enough to be used. The example code is attached below.

The plot can be moved around using the widget, and the widget does respond if the plot is panned using the tools. However, when using pan, the widget responds, this makes the xlim respond, and the small lag puts the image back to some intermediate place (hard to explain but you will see what I mean if you try to pan the image the code generates) — not allowing a smooth interaction (making it unusable).

Does anyone have any suggestions about how to fix this problem? Which part of the documentation should I consult?

import xarray as xr
import hvplot.xarray  # noqa
import panel as pn
from holoviews import streams
pn.extension()

air_ds = xr.tutorial.open_dataset('air_temperature').load()
air = air_ds.air

image = air_ds.air.sel(time='2013-01-01T00:00:00', method='nearest').hvplot() # setup an image

# Create widget and connect it to xlims
Xslide = pn.widgets.RangeSlider(start=float(air_ds.lon.min().data), 
                                end=float(air_ds.lon.max().data), 
                                step=0.1)
@pn.depends(Xslide)
def set_xlim(Xlims):
    return image.opts(xlim=Xlims).opts(width=500)

# Create linked stream and make the widget values respond to it
rangex = streams.RangeX(source=image)
def subscriber(x_range):
    Xslide.value = x_range
rangex.add_subscriber(subscriber)

pn.Row(Xslide, set_xlim)