Python Panel dashboard causing BufferError and RuntimeErrors

I have struggled for some time to create a data streaming interface using Panel.

Essentially I have approximately 20 named python objects that I monitor and read the spectral output from.

I want to have a dashboard displaying this in the form of 20 plots which must continuously overwrite themselves as the spectral output must be displayed over the same x-range (channels).

The dashboard runs fine for some time and then I either get:

a) RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes


b) BufferError: Existing exports of data: object cannot be re-sized {PYTHON_ENV_PATH}/lib/python3.6/site-packages/bokeh/document/ RuntimeWarning: coroutine 'WSHandler.send_message' was never awaited gc.collect()

I’ve drafted up a MRE as follows:

import numpy as np
import pandas as pd
import hvplot.streamz
import numpy as np
import panel as pn
from streamz.dataframe import PeriodicDataFrame

#object from which data is collected:
class data_gen:
    def __init__(self,name,size=1024,sets=4): = name
        self.size = size
        self.sets = sets

    def get_data(self):
        return np.random.randn(self.sets,self.size)
#Have a dictionary of items with name:
data_dict = {
    "a" : data_gen("a"),
    "b" : data_gen("b"),
    "c" : data_gen("c"),
    "d" : data_gen("d"),
    "e" : data_gen("e"),
    "f" : data_gen("f"),

#Generate dataframe
def name_dataFrame(**kwargs):
    dct = {}
    for name,dg in data_dict.items():
        d = dg.get_data()
        sets, size = d.shape
        t_dict ={}
        for i in range(sets):
            t_dict[i] = {
                c : d[i,c] for c in range(size)
        t_df = pd.DataFrame(t_dict).transpose()
        dct[name] = t_df
    df = pd.concat(dct).transpose()
    return df

#Have it be streamed
df = PeriodicDataFrame(name_dataFrame, interval='10s')

#Compose panel layout
pn_realtime = pn.Column("# Data Dashboard")
for name in data_dict:
            (pn.Row(f"""##Name: {name}""")))
                df[name].hvplot.line(backlog=1024, width = 600, height=500, xlabel="n", ylabel="f(n)", grid=True)


My set up is:

# Name                    Version                   Build  Channel
panel                     0.12.1             pyhd3eb1b0_0
hvplot                    0.7.3              pyhd3eb1b0_1
pandas                    1.1.5            py36ha9443f7_0
streamz                   0.6.3              pyhd3eb1b0_0
Python 3.6.13 :: Anaconda, Inc.
Ubuntu 20.04.3 LTS (Focal Fossa)

I’m pretty new to dashboard design (and pandas for that matter) so I wouldn’t be surprised if there were a vastly simpler way to do what I am attempting to do.
My suspicion is that the appending of Panel objects is causing memory buffers to overfill and garbage collection cannot handle it. If so, what can I do?

Running this MRE on my beefier Windows machine with python 3.9.7 did not seem to crash, but perhaps that is simply because I’ve not run it for long enough?

I’ve also set ylims on the hvplot and that seemed to stop crashes from occurring (again maybe I did not run it for long enough), but due to the nature of my application, I cannot have static ylims.

I appreciate your time and input.


Try to create a new environment with python 3.9 and see if the issue still happens.

Hi, I thought the exact same thing and have just done this with the following versions:
Python 3.9.12 , panel 0.13 and pandas 1.4.3,
but still get the above mentioned errors.

Can you give me the output of conda info and conda list for the environment?

conda info:

    active env location : /home/cosmic/anaconda3/envs/dashboardenv
            shell level : 3
       user config file : /home/cosmic/.condarc
 populated config files : 
          conda version : 4.10.1
    conda-build version : 3.21.4
         python version :
       virtual packages : __linux=5.4.0=0
       base environment : /home/cosmic/anaconda3  (writable)
      conda av data dir : /home/cosmic/anaconda3/etc/conda
  conda av metadata url :
           channel URLs :
          package cache : /home/cosmic/anaconda3/pkgs
       envs directories : /home/cosmic/anaconda3/envs
               platform : linux-64
             user-agent : conda/4.10.1 requests/2.25.1 CPython/3.8.8 Linux/5.4.0-109-generic ubuntu/20.04.3 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False```

conda list:

# packages in environment at /home/cosmic/anaconda3/envs/dashboardenv:
# Name                    Version                   Build  Channel
```_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
aniso8601                 9.0.1                    pypi_0    pypi
argon2-cffi               21.3.0             pyhd3eb1b0_0  
argon2-cffi-bindings      21.2.0           py39h7f8727e_0  
asttokens                 2.0.5              pyhd3eb1b0_0  
async-timeout             4.0.2                    pypi_0    pypi
attrs                     21.4.0             pyhd3eb1b0_0  
backcall                  0.2.0              pyhd3eb1b0_0  
beautifulsoup4            4.11.1           py39h06a4308_0  
blas                      1.0                         mkl  
bleach                    4.1.0              pyhd3eb1b0_0  
bokeh                     2.4.2            py39h06a4308_1  
bottleneck                1.3.5            py39h7deecbd_0  
brotli                    1.0.9                he6710b0_2  
brotlipy                  0.7.0           py39h27cfd23_1003  
ca-certificates           2022.4.26            h06a4308_0  
certifi                   2022.6.15        py39h06a4308_0  
cffi                      1.15.0           py39hd667e15_1  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.3                    pypi_0    pypi
colorcet                  3.0.0            py39h06a4308_0  
cosmic                    0.2.0                    pypi_0    pypi
cosmic-f-remote           0.0.2-91fadc0b-dirty          pypi_0    pypi
cryptography              37.0.1           py39h9ce1e76_0  
cycler                    0.11.0             pyhd3eb1b0_0  
debugpy                   1.5.1            py39h295c915_0  
decorator                 5.1.1              pyhd3eb1b0_0  
defusedxml                0.7.1              pyhd3eb1b0_0  
deprecated                1.2.13                   pypi_0    pypi
entrypoints               0.4              py39h06a4308_0  
executing                 0.8.3              pyhd3eb1b0_0  
flask                     2.1.2                    pypi_0    pypi
flask-restful             0.3.9                    pypi_0    pypi
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.11.0               h70c0345_0  
giflib                    5.2.1                h7b6447c_0  
heapdict                  1.0.1              pyhd3eb1b0_0  
holoviews                 1.14.8             pyhd3eb1b0_0  
hvplot                    0.7.3              pyhd3eb1b0_1  
idna                      3.3                pyhd3eb1b0_0  
importlib-metadata        4.11.3           py39h06a4308_0  
intel-openmp              2021.4.0          h06a4308_3561  
ipykernel                 6.9.1            py39h06a4308_0  
ipython                   8.3.0            py39h06a4308_0  
ipython_genutils          0.2.0              pyhd3eb1b0_1  
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.1           py39h06a4308_1  
jinja2                    3.0.3              pyhd3eb1b0_0  
jpeg                      9e                   h7f8727e_0  
jsonschema                4.4.0            py39h06a4308_0  
jupyter_client            7.2.2            py39h06a4308_0  
jupyter_core              4.10.0           py39h06a4308_0  
jupyterlab_pygments       0.1.2                      py_0  
kiwisolver                1.4.2            py39h295c915_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.18               h7b6447c_0  
libstdcxx-ng              11.2.0               h1234567_1  
libtiff                   4.2.0                h2818925_1  
libwebp                   1.2.2                h55f646e_0  
libwebp-base              1.2.2                h7f8727e_0  
lz4-c                     1.9.3                h295c915_1  
markdown                  3.3.4            py39h06a4308_0  
markupsafe                2.1.1            py39h7f8727e_0  
matplotlib-base           3.5.1            py39ha18d171_1  
matplotlib-inline         0.1.2              pyhd3eb1b0_2  
mistune                   0.8.4           py39h27cfd23_1000  
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py39h7f8727e_0  
mkl_fft                   1.3.1            py39hd3c417c_0  
mkl_random                1.2.2            py39h51133e4_0  
munkres                   1.1.4                      py_0  
nbclient                  0.5.13           py39h06a4308_0  
nbconvert                 6.4.4            py39h06a4308_0  
nbformat                  5.3.0            py39h06a4308_0  
ncurses                   6.3                  h5eee18b_3  
nest-asyncio              1.5.5            py39h06a4308_0  
notebook                  6.4.11           py39h06a4308_0  
numexpr                   2.8.3            py39h807cd23_0  
numpy                     1.22.3           py39he7a7128_0  
numpy-base                1.22.3           py39hf524024_0  
openssl                   1.1.1q               h7f8727e_0  
packaging                 21.3               pyhd3eb1b0_0  
pandas                    1.4.3            py39h6a678d5_0  
pandocfilters             1.5.0              pyhd3eb1b0_0  
panel                     0.13.0           py39h06a4308_0  
param                     1.12.0             pyhd3eb1b0_0  
parso                     0.8.3              pyhd3eb1b0_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pillow                    9.0.1            py39h22f2fdc_0  
pip                       22.1.2           py39h06a4308_0  
prometheus_client         0.13.1             pyhd3eb1b0_0  
prompt-toolkit            3.0.20             pyhd3eb1b0_0  
ptyprocess                0.7.0              pyhd3eb1b0_2  
pure_eval                 0.2.2              pyhd3eb1b0_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyct                      0.4.6            py39h06a4308_0  
pygments                  2.11.2             pyhd3eb1b0_0  
pyopenssl                 22.0.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pyrsistent                0.18.0           py39heee7806_0  
pysocks                   1.7.1            py39h06a4308_0  
python                    3.9.12               h12debd9_1  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-fastjsonschema     2.15.1             pyhd3eb1b0_0  
pytz                      2022.1           py39h06a4308_0  
pyviz_comms               2.0.2              pyhd3eb1b0_0  
pyyaml                    6.0              py39h7f8727e_1  
pyzmq                     22.3.0           py39h295c915_2  
readline                  8.1.2                h7f8727e_1  
redis                     4.3.4                    pypi_0    pypi
remoteobjects             1.9.4                    pypi_0    pypi
requests                  2.28.0           py39h06a4308_0  
send2trash                1.8.0              pyhd3eb1b0_1  
setuptools                61.2.0           py39h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
soupsieve                 2.3.1              pyhd3eb1b0_0  
sqlite                    3.38.5               hc218d9a_0  
stack_data                0.2.0              pyhd3eb1b0_0  
streamz                   0.6.3              pyhd3eb1b0_0  
terminado                 0.13.1           py39h06a4308_0  
testpath                  0.6.0            py39h06a4308_0  
tk                        8.6.12               h1ccaba5_0  
toolz                     0.11.2             pyhd3eb1b0_0  
tornado                   6.1              py39h27cfd23_0  
tqdm                      4.64.0           py39h06a4308_0  
traitlets                 5.1.1              pyhd3eb1b0_0  
typing-extensions         4.1.1                hd3eb1b0_0  
typing_extensions         4.1.1              pyh06a4308_0  
tzdata                    2022a                hda174b7_0  
urllib3                   1.26.9           py39h06a4308_0  
wcwidth                   0.2.5              pyhd3eb1b0_0  
webencodings              0.5.1            py39h06a4308_1  
werkzeug                  2.1.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.5                h7f8727e_1  
yaml                      0.2.5                h7b6447c_0  
zeromq                    4.3.4                h2531618_0  
zict                      2.1.0            py39h06a4308_0  
zipp                      3.8.0            py39h06a4308_0  
zlib                      1.2.12               h7f8727e_2  
zstd                      1.5.2                ha4553b6_0```

Try running conda update --all -c conda-forge and see if that work.

It crashes after a little time usually so just waiting to see.

Thanks, actually seems stable.

Given the difficulty I’ve had with this, am I doing this whole thing in a sensible way? I don’t like that I have to write numpy arrays to DataFrames so that I can plot it but I saw no other way.

Perhaps you could point me to a tutorial I’ve not seen yet?

Thanks again.

Good to hear that it is stable.

Maybe you can get some inspiration from this example:

Excellent thank you.