What is the best way to make custom validation and transform before validation

Yes probably. Here it goes:
Param with overridden set is not reflected in panel widget - Panel - HoloViz Discourse