Access current keys of DynamicMap

is there a way to get the current keys (or items) of a DynamicMap?
I have a scatter = df.hvplot.scatter(..., groupby(['col1', 'col2'])and want to get the data for points selected using the box_select tool. Without the groupby, I can get the points using sel = streams.Selection1D(source=scatter), then use sel.index. Unfortunately I noticed that the returned index does not match.
One workaround I found was to to get the data for the selected points using
subset =[scatter.last_key].dframe().iloc[sel.index]
Unfortunately, the keys and items are ordered alphabetically as far as I can see, while I’d need the latest/current selected groupby/key.
Is there either a way to get the current keys or another way to get the currently selected points, or is this only possible for a simple scatter plot without grouping?

While I didn’t find a proper solution, a workaround is to do data down-selection and plotting in panel, using a callback to load/create a dataframe with the subset of data instead of the use of groupby.
This still won’t allow for row or col options to organize y categorical columns though.
I tried to use the relatively new feature of linked brushing (which didn’t even exist when I looked for something similar the last time). While this is an awesome addition, it also seems not to work with plots using row, col, or groupby to split data as far as I could tell.