How to make the pn.state.schedule_task non-blocking?

Panel
1

How to make pn.state.schedule_task non-blocking? I am using it for cache updating tasks and each task takes roughly 3 minutes. However, while a task is running, the app freezes and I am unable to interact with it. Any suggestions?

1 Like
2

Hi @neonpurple

Welcome to the community :+1:

It depends on what you are trying to achieve. Running things in threads or processes might help you. Running things using Dask might help you. Implementing the function as an async function might also help you.

Below you see a proof of concept using async background processes.

task1.py

import time

if __name__=="__main__":
    time.sleep(5)
    # Here you could do whatever and save the result(s) to stdout, file(s), database or a cache
    print("task 1 result: abcd")

tasks.py

import asyncio
from datetime import datetime

import panel as pn
import param


class TaskStatus(param.Parameterized):
    task1_log = param.String()

status = TaskStatus()

pn.state.cache["value"]=0

async def run_task1():
    status.task1_log += f"{datetime.now():%Y-%m-%d %H:%M:%S} Task 1 started\n"
    proc = await asyncio.create_subprocess_exec(
       'python','task1.py',
       stdout=asyncio.subprocess.PIPE,
       stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()
    if stdout:
        status.task1_log += f"{stdout.decode('utf8')}"
    if stderr:
        status.task1_log += f"{stderr.decode('utf8')}"
    status.task1_log += f"{datetime.now():%Y-%m-%d %H:%M:%S} Task 1 finished\n"
    # Here you could read the task results from stdout, file(s), database or cache
    # and set the cache
    pn.state.cache["value"]+=1


pn.state.schedule_task(name="update-cache", callback=run_task1, period="10s")

app.py

import panel as pn
from tasks import status

pn.extension(sizing_mode="stretch_width", template="fast", theme="dark")
pn.state.template.site="Panel"
pn.state.template.title="Non blocking Background tasks using async"

slider = pn.widgets.IntSlider(value=5, start=0, end=10)

pn.Column(
    "# Slider",
    slider, slider.param.value,
    "# Task Log",
    pn.widgets.TextAreaInput.from_param(status.param.task1_log, height=500),
    """
    Global Tasks are scheduled using `pn.state.schedule_task(name="update-cache", callback=run_task1, period="10s")`
    """
).servable()

panel serve app.py

schedule-background-tasks

I prefer an approach like this where long running task are separated from the app and saves the result to stdout, files, a database or a cache like diskcache. The app then has to do a minimum amount of work to read the results.

You might even schedule the tasks outside of your app and then just use pn.state.schedule_task to check if there are new results and if yes read+update the cache.

1 Like
3

Wow. This is so cool. It would also be so difficult in literally any other framework!

Thinking about all the enterprise use cases for @metaperl …

1 Like