Scatter - color mapping for "by"

Hey,
I am running -

df.hvplot.scatter('time','value',by='category',cmap={'cat1':'red','cat2':'blue'})

This doesn’t work with by, only with c='category' but then I am unable to mute categories by pressing on the legend
Any ideas?
Thanks.