Tornado Stream Error

Panel
1

Hello,

I have a small program that retrieves netcdf files, performs some analysis, and displays a plot based upon user settings. A simplified version is:

import os

import cartopy.crs as ccrs
import cmocean
import hvplot.xarray
import pandas as pd
import panel as pn
import xarray as xr

import socket

# This part controls which names are displayed
experiments = pn.widgets.Select(
    name="Experiments", options=list(df["simulation name"])
)
# For the control experiments
controls = pn.widgets.Select(
    name="Controls",
    options=["Absolute Values"] + list(df["simulation name"]),
)
times = pn.widgets.RadioButtonGroup(
    name="Time Period",
    value=["climmean"],
    options=["climmean", "DJF", "MAM", "JJA", "SON"],
)  # , "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"])
variables = pn.widgets.RadioButtonGroup(
    name="Variables", value=["temp2"], options=["temp2", "tsurf", "aprt"]
)

@pn.depends(
    variables.param.value,
    experiments.param.value,
    controls.param.value,
    times.param.value,
)
def make_plot(var, exp, ctrl, time):
   # Do work to get variable for experiment, control, and time
   ds_exp = xr.open_dataset("/some/path")
   ds_ctrl = xr.open_dataset("/some/other_path")
   anom_ds = getattr(exp_ds, var) - getattr(ctrl_ds, var)
   return anom_ds.hvplot.quadmesh(coastline=True, projection=ccrs.PlateCarree(), title=f"{exp} - {ctrl} ({var}, {time})",).opts({"Image": dict(colorbar_position="bottom", color_levels=20)})

def text_callback(target, event):
    target.object = "Showing experiments from the following: "+", ".join(event.new)


def expid_filter_callback(target, event):
    experiments = list(df[df["model and resolution"].isin(event.new)]['simulation name'])
    target.options = experiments



text = pn.pane.Markdown("Showing experiments from the following: ") 
models.link(experiments, callbacks={"value": expid_filter_callback})
models.link(text, callbacks={"value": text_callback})
models.link(controls, callbacks={"value": expid_filter_callback})


app = pn.Column(
    pn.Row("**Model:**", models),
    pn.Row(
        text,
        pn.Column(
        pn.Row("**Experiment:** ", experiments),
        pn.Row("**Control:** ", controls),
        pn.Row("**Time Average:** ", times),
        pn.Row("**Variable:** ", variables),
        ),
        make_plot,
    )
).servable()

app

Here’s what it actually looks like:

Screen Shot 2020-10-21 at 6.40.35 AM
Screen Shot 2020-10-21 at 6.40.35 AM1221×677 142 KB

Basically, I have a set of simulations, select an experiment, a control, a time average, and a variable, and then would like to see the anomaly between the two. However, I currently run into the following error when changing what is suppose to be used as experiment:

future: <Task finished coro=<WebSocketProtocol13.write_message.<locals>.wrapper() done, defined at /opt/miniconda3/lib/python3.7/site-packages/tornado/websocket.py:1102> exception=WebSocketClosedError()>
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/websocket.py", line 1104, in wrapper
    await fut
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/websocket.py", line 1106, in wrapper
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7ff1999b1350>>, <Task finished coro=<_needs_document_lock.<locals>._needs_document_lock_wrapper() done, defined at /opt/miniconda3/lib/python3.7/site-packages/bokeh/server/session.py:51> exception=WebSocketClosedError()>)
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/opt/miniconda3/lib/python3.7/site-packages/bokeh/server/session.py", line 71, in _needs_document_lock_wrapper
    result = await result
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 191, in wrapper
    result = func(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/viewable.py", line 516, in _change_coroutine
    self._change_event(doc)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/viewable.py", line 526, in _change_event
    self._process_events(events)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/viewable.py", line 512, in _process_events
    self.param.set_param(**self._process_property_change(events))
  File "/opt/miniconda3/lib/python3.7/site-packages/param/parameterized.py", line 1365, in set_param
    self_._batch_call_watchers()
  File "/opt/miniconda3/lib/python3.7/site-packages/param/parameterized.py", line 1480, in _batch_call_watchers
    watcher.fn(*events)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/param.py", line 700, in _replace_pane
    self._update_inner(new_object)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/pane/base.py", line 368, in _update_inner
    self._pane.object = new_object
  File "/opt/miniconda3/lib/python3.7/site-packages/param/parameterized.py", line 296, in _f
    return f(self, obj, val)
  File "/opt/miniconda3/lib/python3.7/site-packages/param/parameterized.py", line 861, in __set__
    obj.param._call_watcher(watcher, event)
  File "/opt/miniconda3/lib/python3.7/site-packages/param/parameterized.py", line 1456, in _call_watcher
    watcher.fn(event)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/pane/base.py", line 188, in _update_pane
    self._update_object(ref, doc, root, parent, comm)
  File "/opt/miniconda3/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/opt/miniconda3/lib/python3.7/site-packages/panel/io/server.py", line 91, in unlocked
    WebSocketHandler.write_message(socket, msg.content_json)
  File "/opt/miniconda3/lib/python3.7/site-packages/tornado/websocket.py", line 339, in write_message
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
/opt/miniconda3/lib/python3.7/site-packages/bokeh/document/document.py:499: RuntimeWarning: coroutine 'WSHandler.send_message' was never awaited
  gc.collect()

Any hints would be great, I’m rather stuck!

2

I may have found a (possible) cause. I’m using NGINX as a web server, and it might be closing off the web socket if the “do work” part of that function takes too long.

3

OK, I just told my web server to keep the socket alive for 7 days (likely overkill), but in case anyone else runs into this, here is how to configure NGINX:

Assume you run your application like this:

$ panel serve my_app.py --port=6000 --allow-websocket-origin=server_name.com

And you have NGINX configured like this:

        location /my_app {
                proxy_pass http://localhost:6000;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $host:$server_port;
                proxy_buffering off;
                proxy_connect_timeout 7d; 
                proxy_send_timeout 7d; 
                proxy_read_timeout 7d; 

                access_log  /var/log/nginx/server_name.my_app.access.log;
                error_log   /var/log/nginx/server_name.my_app.error.log debug;
        }

These three lines:

                proxy_connect_timeout 7d; 
                proxy_send_timeout 7d; 
                proxy_read_timeout 7d;

seem to do the trick.

Here, a more helpful error message would have been nice … or a book so I can start understanding the web better :frowning:

1 Like
4

I think you should give more context and if possible a reproducible copy/pastable example.
When you say that you are using nginx, does it mean that your code works on your local machine but not when deployed?

5

I get this error when stress-testing my app, which also sits behind NGINX. So, would make sense that it’s a timeout issue.

Will try your solution, thanks!

2 Likes