How to keep color limits steady with widgets

I am trying to make a plot where I can use a widget to control the xlim, and also have an adjoint hist to set the color limits. The problem right now is that if I set a certain color limit using the hist and then change the xlim using the widget, the colorlimit gets reset. Is there some simple option like framewise or something to stop the colorlimit from being reset? Is there some part in the documentation that I should look at?

Here is some example code:

import xarray as xr
import hvplot.xarray  # noqa
import panel as pn

air_ds = xr.tutorial.open_dataset('air_temperature').load()

image = air_ds.air.sel(time='2013-01-01T00:00:00', method='nearest').hvplot() # setup an image

# Setup a panel slider to set the xlim
Xslide = pn.widgets.RangeSlider(start=float(air_ds.lon.min().data), 

def set_xlim(Xlims):
    return image.opts(xlim=Xlims).opts(width=500).hist()

pn.Row(Xslide, set_xlim)

This questions is also related to:Inconsistent behavior of colormap for Adjoint layout with hist