Tornado Stream Error


I have a small program that retrieves netcdf files, performs some analysis, and displays a plot based upon user settings. A simplified version is:

import os

import as ccrs
import cmocean
import hvplot.xarray
import pandas as pd
import panel as pn
import xarray as xr

import socket

# This part controls which names are displayed
experiments = pn.widgets.Select(
    name="Experiments", options=list(df["simulation name"])
# For the control experiments
controls = pn.widgets.Select(
    options=["Absolute Values"] + list(df["simulation name"]),
times = pn.widgets.RadioButtonGroup(
    name="Time Period",
    options=["climmean", "DJF", "MAM", "JJA", "SON"],
)  # , "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"])
variables = pn.widgets.RadioButtonGroup(
    name="Variables", value=["temp2"], options=["temp2", "tsurf", "aprt"]

def make_plot(var, exp, ctrl, time):
   # Do work to get variable for experiment, control, and time
   ds_exp = xr.open_dataset("/some/path")
   ds_ctrl = xr.open_dataset("/some/other_path")
   anom_ds = getattr(exp_ds, var) - getattr(ctrl_ds, var)
   return anom_ds.hvplot.quadmesh(coastline=True, projection=ccrs.PlateCarree(), title=f"{exp} - {ctrl} ({var}, {time})",).opts({"Image": dict(colorbar_position="bottom", color_levels=20)})

def text_callback(target, event):
    target.object = "Showing experiments from the following: "+", ".join(

def expid_filter_callback(target, event):
    experiments = list(df[df["model and resolution"].isin(]['simulation name'])
    target.options = experiments

text = pn.pane.Markdown("Showing experiments from the following: "), callbacks={"value": expid_filter_callback}), callbacks={"value": text_callback}), callbacks={"value": expid_filter_callback})

app = pn.Column(
    pn.Row("**Model:**", models),
        pn.Row("**Experiment:** ", experiments),
        pn.Row("**Control:** ", controls),
        pn.Row("**Time Average:** ", times),
        pn.Row("**Variable:** ", variables),


Here’s what it actually looks like:

Basically, I have a set of simulations, select an experiment, a control, a time average, and a variable, and then would like to see the anomaly between the two. However, I currently run into the following error when changing what is suppose to be used as experiment:

future: <Task finished coro=<WebSocketProtocol13.write_message.<locals>.wrapper() done, defined at /opt/miniconda3/lib/python3.7/site-packages/tornado/> exception=WebSocketClosedError()>
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/", line 1104, in wrapper
    await fut
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/", line 1106, in wrapper
    raise WebSocketClosedError()
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7ff1999b1350>>, <Task finished coro=<_needs_document_lock.<locals>._needs_document_lock_wrapper() done, defined at /opt/miniconda3/lib/python3.7/site-packages/bokeh/server/> exception=WebSocketClosedError()>)
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/", line 743, in _run_callback
    ret = callback()
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/", line 767, in _discard_future_result
  File "/opt/miniconda3/lib/python3.7/site-packages/bokeh/server/", line 71, in _needs_document_lock_wrapper
    result = await result
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/", line 191, in wrapper
    result = func(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/", line 516, in _change_coroutine
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/", line 526, in _change_event
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/", line 512, in _process_events
  File "/opt/miniconda3/lib/python3.7/site-packages/param/", line 1365, in set_param
  File "/opt/miniconda3/lib/python3.7/site-packages/param/", line 1480, in _batch_call_watchers
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/", line 700, in _replace_pane
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/pane/", line 368, in _update_inner
    self._pane.object = new_object
  File "/opt/miniconda3/lib/python3.7/site-packages/param/", line 296, in _f
    return f(self, obj, val)
  File "/opt/miniconda3/lib/python3.7/site-packages/param/", line 861, in __set__
    obj.param._call_watcher(watcher, event)
  File "/opt/miniconda3/lib/python3.7/site-packages/param/", line 1456, in _call_watcher
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/pane/", line 188, in _update_pane
    self._update_object(ref, doc, root, parent, comm)
  File "/opt/miniconda3/lib/python3.7/", line 119, in __exit__
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/io/", line 91, in unlocked
    WebSocketHandler.write_message(socket, msg.content_json)
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/", line 339, in write_message
    raise WebSocketClosedError()
/opt/miniconda3/lib/python3.7/site-packages/bokeh/document/ RuntimeWarning: coroutine 'WSHandler.send_message' was never awaited

Any hints would be great, I’m rather stuck!

I may have found a (possible) cause. I’m using NGINX as a web server, and it might be closing off the web socket if the “do work” part of that function takes too long.

OK, I just told my web server to keep the socket alive for 7 days (likely overkill), but in case anyone else runs into this, here is how to configure NGINX:

Assume you run your application like this:

$ panel serve --port=6000

And you have NGINX configured like this:

        location /my_app {
                proxy_pass http://localhost:6000;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $host:$server_port;
                proxy_buffering off;
                proxy_connect_timeout 7d; 
                proxy_send_timeout 7d; 
                proxy_read_timeout 7d; 

                access_log  /var/log/nginx/server_name.my_app.access.log;
                error_log   /var/log/nginx/server_name.my_app.error.log debug;

These three lines:

                proxy_connect_timeout 7d; 
                proxy_send_timeout 7d; 
                proxy_read_timeout 7d; 

seem to do the trick.

Here, a more helpful error message would have been nice … or a book so I can start understanding the web better :frowning:

1 Like

I think you should give more context and if possible a reproducible copy/pastable example.
When you say that you are using nginx, does it mean that your code works on your local machine but not when deployed?

I get this error when stress-testing my app, which also sits behind NGINX. So, would make sense that it’s a timeout issue.

Will try your solution, thanks!

1 Like