Bi-directional syncing requires a bit of setup. Have you looked at the annotators which are set up to do exactly this:
from holoviews.annotators import PointAnnotator
df = pd.DataFrame(dict(x=[1, 1.5], y=[-1, 2], label=['A','B']))
PointAnnotator(df)