A DynamicMap has special code for figuring out in-place minimal updates, precisely to avoid having to re-render the whole plot as you see in this case. I don’t know of any direct equivalent to that for pure bokeh plots, but I’d guess there’s some way to use a Bokeh JS link (JavaScript callbacks — Bokeh 2.4.2 Documentation) to wire the widget to the position of that marker.