Worth noting that there’s now a helper for this in the form of apply_when:
import holoviews as hv
from holoviews.operation.datashader import rasterize
from holoviews.operation import apply_when
points = hv.Points(np.random.randn(100000, 2))
apply_when(points, operation=rasterize, predicate=lambda x: len(x) > 5000)