Some attributes get lost when using vtk (pyvista) in panel

The following snippet creates some vtk plane by using pyvista:

import panel as pn
import numpy as np
import pyvista as pv
pn.extension('vtk')

plot = pv.Plotter()
plot.show_bounds(xlabel='x (m)', ylabel='y (m)', zlabel='z (m)')
plot.set_background(color="#553311")
plot.show_axes()
x, y = np.meshgrid(np.linspace(-5, 5, 20),
                   np.linspace(-5, 5, 20))
z = np.cos(x + y)
mesh = pv.StructuredGrid(x, y, z)
mesh.point_arrays['value'] = (x + np.cos(y) ** 2).flatten('F')
plot.add_mesh(mesh, scalars='value')

If I follow this up with

plot.show()

i get the vtk pop-up and everything works as expected.

Now I want to use this in a panel pane, so I replace the show command by

vtkpan = pn.panel(plot.ren_win, sizing_mode='stretch_both')
pn.serve(vtkpan)

It does run without errors, but the background color control is gone, the little orientation widget is lost, and the axes are removed. Why? and how can I get the panel pane be equal to the vtk pop-up window?