Param and command line tools

Serker · February 10, 2025, 12:38pm

Hi, I’m currently trying to figure out the best way to combine command line tools (argparse, jsonargparse, …) with param.

Goal

A simple format to provide parameter definitions for scripts, preferably from within the script. For example as shown by the dataclass below.

Note that some of the scripts are really short (<50 lines of code), so I’m looking for something that can be very lightweight in the way of providing parameter information (so not jsonschema).

It then needs to be easy to provide a command line interface and a panel interface (using param) from that definition. Furthermore, that definition (not the instantiated with the parameters) needs to be serializable. Dataclasses seem ideal as they can be used with jsonargparse and are very lightweight, but they seem non trivial to convert to param and non trivial to serialize (the dataclass definition).

@dataclass
class Args:
    folder: Path = Path(".")
    """Folder on which the program searches for the txt files"""
    bin_size: int | Literal["auto"] = "auto"
    """Size of the bins"""

    bin_agg: Literal["max", "mean", "median", "min"] = "max"
    glob_pattern: str = "**/*.txt"
    out_folder: Path | Literal["__input_folder_path__"] = "__input_folder_path__"
    show: bool = False
    overwrite: Literal["ask_user", "yes", "no"] = "ask_user"
    files: List[Tuple[Path, Path]] = field(default_factory=lambda: [("f1", "f2")])

Any ideas ?

ahuang11 · February 10, 2025, 6:11pm

Perhaps pydantic?

We have a translator in Lumen from param/pydantic.

github.com

holoviz/lumen/blob/c2404e16d2e1c4e8abd1a32af0447b6de01f0a33/lumen/ai/translate.py

from __future__ import annotations

import datetime
import inspect
import re
import warnings

from collections.abc import Callable
from inspect import Signature, signature
from types import FunctionType
from typing import (
    Any, Literal, TypeVar, Union,
)

import param

from griffe import Docstring, DocstringSectionKind
from pydantic import BaseModel, ConfigDict, create_model
from pydantic._internal import _typing_extra
from pydantic.fields import FieldInfo, PydanticUndefined

This file has been truncated. show original

Then

Perhaps

Serker · February 11, 2025, 4:18pm

So after looking at pydantic, it seems that:

Defining pydantic models is lightweight
I can export the model to json_schema and I can reload the model using datamodel_code_generator (not tested yet)
I can use pydantic_argparse to generate a CLI (not tested)
I can use the translate code from lumen to generate a param (not tested) that I should be able to display using panel to provide a web interface (not tested for param generated by the lumen translator)

I’ll test all these steps on simple examples and describe the results. Of course, I’m still open to other ideas.

ahuang11 · February 11, 2025, 6:43pm

The translate code ideally would be in param instead of lumen.