You could separate the hover from the image and put it on the side like
(Instead of a tap stream, you can make it a continuous follow Pointerxy)
http://holoviews.org/user_guide/Custom_Interactivity.html
But if you do want a single hover tooltip, it’s an underlying bokeh issue I think, or you can try writing your own javascript (no idea how though :D, https://docs.bokeh.org/en/latest/docs/user_guide/tools.html#custom-tooltip)