The key to HoloViews themes is understanding bokeh themes https://docs.bokeh.org/en/latest/docs/reference/themes.html.
Template like the MaterialTemplate provide a custom bokeh theme. You will need to override/ update that theme or provide a custom theme for the HoloViews pane.
See https://github.com/holoviz/panel/blob/master/panel/template/material/init.py
and https://github.com/holoviz/panel/blob/master/panel/pane/holoviews.py
