Programmatically update Annotators

I’m following the example of creating RectangleAnnotator with BoxEdit tool according to the documentation:

boxes = hv.Rectangles([(0, 0, 1, 1), (1.5, 1.5, 2.5, 2.5)])
box_annotator = hv.annotate.instance()
layout = box_annotator(boxes.opts(width=800, height=400, responsive=False), annotations=['Label'])

This works for interactive user input.
How can I programmatically modify the collected rectangles list? I.e. add/remove rectangles from code without dropping user drawn ones?