Annotator doesn't work with hv.Raster

I’m trying to convert the Annotator example for an image:

img = hv.Image(np.load('../assets/twophoton.npz')['Calcium'][..., 0])
cells = hv.Points([]).opts(width=500, height=500, responsive=False, padding=0)

hv.annotate(img * cells, annotations=['Label'], name="Cell Annotator")

to use hv.Raster instead of hv.Image, because I need the annotations in numpy/pixel coordinates.
But trying to do so, the image/raster doesn’t show. I think it might have something to do with the y-axis being inverted? I tried to use invert_yaxis=True as an opts to Raster, but to no avail. Indeed the annotate axis of the annotator inverted the y-axis as well, when I did that.
If annotation is not supported with raster, what’s my workaround, to get the coordinates in numpy index space?

thanks!
Michael

I found that doing

cells = hv.Points([]).opts(width=500, height=500, responsive=False, padding=0, invert_yaxis=True)
hv.annotate(img * cells, annotations=['Label'], name="Cell Annotator")

does work. However, I now actually need it in the opposite direction. My image data should not have the y-axis inverted (so I would use invert_yaxis=True on the raster to prevent that inversion), but then I cannot find a way to have the annotater working.