Panel/Holoviews/Bokeh app memory leaks, looking for general best practices

SteveAKopias · June 7, 2021, 12:20am

This is a followup for previous threads:

I started this thread with the hope that the long runtime and the memory leak in my app are connected and by finding the source of the runtime, I would eliminate the memory leak.
When that didn’t work out, I reprioritized, realizing I can live together with the slow runtime, but I definitely need to resolve the memory leak issue, so that’s when I started this thread on the Bokeh Discourse.

Since that I did not see the error message mentioned there, but I’m still struggling with multiple memory leaks, and also, having the feeling that I must be doing something terribly wrong. So I just got a sense that my question is more about the recommended way of developing Holoviews/Panel applications rather than being a technical issue with the Bokeh server, so I decided to create a new thread for this here, hoping @philippjfr @jbednar @Marc could share their two cents on the issue.

What I’m trying to do is not rocket science. My script is 6000 lines total with all the css, html, js code and configuration variables. This is a bit too big just to present it as an MRE, but as far as webapps go, this is by far the smallest I ever worked on. I use python 3.9.2, I don’t have threads or anything weird in the code. I don’t even use @contexts other than @pn.depends'. The main logic of the script is pretty simple. It has practically two main workflows:

Reads in a CSV of locations with pd.read_csv(), filters them based on Panel widgets and displays them as a) Holoviews Points if the user is zoomed in enough; b) as a Datashader categorical aggregate otherwise.
Based on Panel widgets it reads in some Tiff files with rioxarray and displays them as Datashader regrid.

Both of these go onto a Holoviews tilemap and are configurable by the Panel widgets through either @pn.depends or .apply(some_function, some_panel_widget), then both the map and the widgets get loaded into a template and finally the template is displayed by .servable().

And no matter what I do, the script uses an increasing amount of memory on every page load with no plateau. If I separate the CSV and TIFF workflows into two separate scripts, both of them will have the same problem. If I eliminate both of these workflows and only leave the basic configuration variables, Panel widgets and templates in place, that still has a memory leak.

I’m not looking for a solution for this specific problem. My main issue is that I’m poking around in the dark here. My script ends with a full_content.servable(). After that line I don’t have any control over the variables and that’s where they should be freed up. I don’t get any “hey, these variables could not be freed” error. I don’t have any information on what uses these variables. I don’t even know what the Bokeh server does with them. Yeah, I can use tracemalloc to see the most heavily memory-intensive areas of the script, but wouldn’t tell me anything about the source of the leaks (I did not see any sings of those leaks I found later).

And to be honest, there are some alarms ringing in my head. While being pretty new with the Bokeh+Holoviz ecosystem, I have a bit of experience with developing web applications, and I’m definitely missing something here. I’m pretty sure this is not how professional development of a web application should look like. Like I can’t imagine that the best way of developing, the way experienced agencies are developing is putting together an app in weeks and then spending further weeks poking around in the dark.

Based on my research (I’ve read everything I could find) there is surprisingly few information on memory leaks in python, and also basically no tools fit for monitoring commercial level projects and providing detailed information on memory usage. That (and these articles stating it) makes me think that these issues are extremely rare and developers usually shouldn’t care about this.

On the other hand, my experience is that basically no matter what I do, whichever way I cut my project, it bleeds memory everywhere, originally creating the feeling that this is a pretty common issue. Which wouldn’t be a huge pain if memory management would be the developers’ responsibility with the necessary tools available (as it is in some languages like C++), but again, it doesn’t look like it is. So I’m stuck in this weird limbo, where I try to modify my script line by line, running it, then refreshing a few dozen times (every refresh of course takes ~10s) to see if anything changed, but even if it did, I don’t really have a clear idea what went wrong and how to solve it.

Like the first memory leak was related to an opts(hooks=...) part coming from this comment of Phillip, that allowed for me to change the actual background-color of a Holoviews Overlay. It took literally a week to discover that in the N-th really dumbed-down version of the code. If that line was there, there was a heavy memory leak, as soon as I removed it, the dummy script worked like a charm. (I’m talking about either staying between 700-900MB even if constantly refreshed by window.location.reload() for hours and consuming about (400+N*80) MB memory where N = number of refreshes and steadily climbing up to any gigabytes, so the difference is pretty striking.) I use hooks in multiple places and they usually don’t cause any problems, but this time the combined = (tiles * layer1 * layer2).opts(hooks=...) wasn’t working out. So I did the same in my original script removing the same opts(), and nothing changed. The reason for that being, I assume, that there are definitely more memory leaks.

The second example I found is a @pn.depends() function that returns some widgets responsible for configuring the currently visible TIFF layers embedded in Cells and Rows. The function depends on the values of three widgets. If I remove the 3rd one (a DiscreteSlider) from the @pn.depends(...) part, from the function parameters and remove every line that would use it, my dumbed down script runs without any leaks. If then I put it back into the @pn.depends() part and into the function parameters but still not using the variable anywhere in the function, than the script will have a memory leak again. (About the same difference as described above.) The most fascinating experience. If I replace the DiscreteSlider with an IntSlider or a TextInput, nothing changes, exactly the same leak happens or not. Also, if I remove the widget from the same function in my original app, accepting the loss of functionality, it doesn’t even help. The most likely reason for that, I’m thinking, that my dumbed down script only referenced this widget in that one function, but the full script uses it in multiple places, so I remove it from everywhere, just to see that it doesn’t change anything. Maybe it helped, but there’s still another leak so I just don’t see the improvement? Most probably, but I’m just guessing.

So you get the picture. I can not search for memory leaks by commenting out a few bits as there are multiple leaks, and If I leave even one in, I will not realize I just commented out another one. So I have to create really dumbed down dummy scripts and building them up again line by line, spending multiple minutes with every refresh. And even that falls apart when I find a bit that without any question causes a memory leak, and I 1) have no idea why; 2) don’t know how what to do to eliminate the leak but keep the functionality; 3) even if I completely delete that part, it is possible that some other parts not present in the current dumbed down version will trigger the same issue.

Now my working theory as always is that I must be doing something wrong if this keeps happening, but anyways, sooner or later I will figure this out for this project. But what’s more interesting for me is if this is what I should expect every time I create something? I’m not surprised at me creating a few bugs using new tools, I’ve did that every time I switched to new languages or modules, but this is the first time I don’t see a clear path to learn to quickly resolve them. I assume, however, this is not how it is intended to work, and supposedly @philippjfr and @Marc and many others did create or regularly create professional commercially viable applications either for real or as an example. I’m really curious of your experience and want to learn what you do differently.

jbednar · June 7, 2021, 2:01am

That doesn’t sound like very much fun! We have a monitoring app covering everything we’ve deployed at examples.pyviz.org, and on Thursday we noticed that three of our recently re-deployed apps show increasing memory usage over time. For those, eventually the container gets restarted, but we still need to trace what the source of it is. Maybe we could have you pair along with @philippjfr or @jstevens and swap hints and approaches, then write up a guide of some sort.

Marc · June 7, 2021, 3:08am

Hi @SteveAKopias

I feel your pain. I have not been experiencing or fighting memory leaks. But your usage of the HoloViz ecosystem is also more advanced than my normal usage. My (initial) problems have been around using the Panel layouts for everything instead of (custom) Templates. That made my apps slow.

Would it be possible to share you code in a repository with specific instructions on how to run it and where you experience the memory leaks?

Without something that can be reproduced its almost impossible to track down.

SteveAKopias · June 7, 2021, 3:26pm

@jbednar I very much would be interested in any knowledge-sharing. I’m not sure we both have the same issue as my memory leak looks pretty intense and hard to miss, but maybe just because my live server has pretty limited resources I was paying more attention. But in any case, I’m very interested in any kind of knowledge sharing as without being confident that I will be able to quickly resolve future issues like this, it’s hard for me to imagine being able to use these tools on a recurring bases regardless how much I love them otherwise.

@Marc Well, I did not want to share the code originally, as 1) solving this specific issue is obviously my responsibility and it would be unrealistic to expect anybody else to jump into a 6000 line code just for the fun of it; and 2) the conceptual problem is not with this specific script but my inability to reliably debug these kind of problems, so even if somebody would solve this to me, I would still be wary to start a new project.

That being said, it’s no secret either, so I created a github repo: https://github.com/ka-steve/mwi_demo if anybody wants to play with it.

It is just a slightly dumbed down version. The original script picks a few of about 10K TIFF files based on selected dataset, subset, datetime, resolution and other widget values. I injected a small dummy script at the beginning of these functions to highjack them and load the same elevation TIFF no matter what was requested. This means the script now needs only this one TIFF, but if you select any other layer than the elevation, the resulting image most probably will be useless, but that shouldn’t matter.
Other than this, the script only needs a waterpoints csv, which is also included. If you only download the script, it will download both files automatically.
There is also an environment.yml included to be able to replicate the exact environment. The versions matter too, for example older matplotlib versions do not use the cmoption colormaps and thus the script would throw an error when Holoviews tries to use them.
I run the script by running python -m panel serve app_demo.py --allow-websocket-origin=localhost:5006 in the activated environment, and then the site will be available at localhost:5006/app_demo
Almost at the end there is a commented out block that only displays the interactive Panel/Holoviews content. After that comes a full templated solution, but that does not have any problems with it, so it can be easier to remove that and use just the shorter version if you’re only after the memory leaks.
Also, both display blocks contain a commented out window.location.reload() JS line. By enabling that, the site will keep reloading and using increasing amount of memory. Like right now I’m running the demo script to be sure it works the same way and it’s at 16GB after 45 reloads.
There is a huge block in the script with a custom JS tooltip solution. This is a workaround for the fact that to the best of my knowledge, it’s impossible to display one tooltip for multiple overlaid images. So what I do is use the custom JS tooltips for every layer to export their values to a global JS variable (and hide them with CSS display: none) and then use a single transparent layer to display every previously exported variable in a single tooltip. I’ve tested it, this does not cause any memory leaks, so you can ignore the whole block. (Matter of fact, you can even delete all the tooltip functions and replace them with a simple tools=['hover'] wherever they are referenced, thus losing the meaningful tooltips but getting a shorter script.)

What I found out so far:

The waterpoint workflow does not have a memory leak. The loading → filtering → displaying flow works fine in itself.
The custom tooltips and the whole template part work fine.
The spatial workflow (loading and displaying the tiffs) works fine in itself, until no widgets are implemented, so at least rioxarray (the module I use to read the tiffs) is not a problem.
As I mentioned before, there was an .opts(hook=) part that did cause a memory leak when only displaying the waterpoint workflow, and that went away as soon as I commented that out, but you can still find it in the code right after the combined = block.
Currently I’m investigating the pn_dates widget, that’s the one I also mentioned above. If I comment out all the hd.regrid(...) lines from combined, and comment out get_category_widgets in the widgets = block, then the memory usage stops at around 800MB. If I put back get_category_widgets here, but remove any references from its function definition to pn_dates, then it’s still good. If I then put back the pn_dates lines into the pn.depends() block and into the function parameters, but still leaving every line that would use it commented out, that again produces a memory leak. And mind you, when I removed all widgets in another experiment and hardcoded the default variables into the get_image_*** functions, they did not cause a memory leak either. And also, it did not matter when I replaced the pn_dates DiscreteSlider with an IntSlider just as a test (and calculated the necessary ‘YYYY-MM’ strings by using the value of the IntSlider as “months since 2010-01”), or even when I simply replaced it with a TextInput, so the problem is not that having a DiscreteSlider with 130 string options would be too much.

That’s all I know for now.

_jm · June 7, 2021, 6:10pm

@SteveAKopias

Thanks for this thread.

I have developed panel/bokeh server data analysis and visualization apps, which I am deploying them to Heroku (an online platform as a service).

The affordable hosting options limit app use to 512MB, so I have similar resource constraints that you cite.

Moreover, Heroku has a metrics dashboard that shows memory usage traces versus time (total, swap, RSS), and I see growing memory consumption as each new session is created, and the app holds onto it after the session closes via user log-out. I think this is analogous to your observation – assuming that your terminology page reload corresponds to a new server session being created.

My current working theory is that there are some tricky references happening at a low level such that python’s default memory management and/or the garbage collector, which I have tried to manually invoke via gc.collect() do not clean up things nicely when a session is no longer open.

Bokeh does have lifecycle hooks, and I am experimenting there with mechanisms to clean up variables and do forced garbage collection via the on_session_destroyed() mechanism. Unfortunately, I am at the stage of trying to bound the problem and come up with a procedure that makes sense conceptually and doesn’t just happen to work.

On a related note, I did notice looking at the panel source code that it calls this hook – if its registered – as part of servable() or server_doc() so that has some bearing / constraints on the implementation.

Thanks again for sharing your experiences and I hope to benefit from the findings of the root cause analysis cited earlier in this discussion for panel’s example apps.

SteveAKopias · June 7, 2021, 6:56pm

Hi,

It’s very interesting to hear about another experience, thanks! (Tbh, just hearing that other than the memory leak, it is feasible to host these apps on Heroku in practice is a valuable information me. I did not dare to try that seeing my initial memory consumption, so right now I’m on a full blown ubuntu AWS EC2 server, but in case this issue gets resolved, I will try Heroku too.)

Relating to your question: I assume we talk about the same thing, but not sure. I did not dive into the Bokeh server code yet, so so I don’t know anything about sessions being created or not. All I do is start my script with panel serve, refresh the localhost:5006/appname url in a browser multiple times and then I can see that on every browser reload the memory consumption keeps growing. Most probably a new session is created and that’s where something goes pear-shaped.

May I ask what kind of elements and application logic do you use?

For example I use in my code:

Holoviews Points() and through holoviews.operation.datashader hd.datashade and hd.aggregate to display points;
Holoviews Image() to display Tiff files loaded by rioxarray, the images being put through hd.regrid();
these elements being overlaid on a tilemap in a combined = (tilemap * image * points) way;
while all of these are controlled by pn.widgets. widgets used either in @pn.depends(widget_value = widget.param.value) decorated functions or through some_element.apply(function_name, widget_value = widget) way;
the backend for everything is Bokeh.

Maybe if we would have a few summaries like these (assuming if we already know of 3 cases like this, there could be other people affected who could chip in), we could see some common patterns that could help us pinpoint the specific area.

My experience so far is not necessarily helpful in this, as there is only one memory leak I successfully eliminated by commenting out the hooks= option, but assuming that feature is rarely used and reaches into the internal workings of the Bokeh backend itself, I’m not surprised it had issues. And that still left me with some other leak, most probably rooted in a more common functionality. I have a hunch that it could be related to either the Panel widgets/panes or to the underlying Params functionality (that I don’t use directly anywhere) because enabling-disabling areas that are almost entirely built out of Panel elements make it appear-disappear. But I’ve had many false hunches during this bughunt, and seeing how the logic of Holoviews is that “the developer just creates a recipe that will be implemented if anything request that element”, it’s pretty hard be sure in whether the leak is in the part you actually commented out, or if it is still in the live code it’s just not getting executed as nothing made that code run anymore… Anyways, that’s why I think it would be interesting to pool our usecases and look for common elements. Like if you are using Panel and Params at all, those can be the culprits, but if you don’t then either we have multiple similar issues, or it may worth to look into some common elements…

And also an update, just as an experiment I’ve kept my script running with automatic reload and in about 2h it reached 48GB memory consumption on my local machine, so yeah, it does not seem to be stopping. And btw, previously I’ve tried spreading some gc.collect() all over my code and even manually del-ing every variable, and of course, those did not do anything, as I didn’t expect them, but had to try.

SteveAKopias · June 7, 2021, 7:03pm

My last comments just made me realize that this would be so much easier if we would get any kind of logs of the elements being processed. Like right now even though I create a points= hv.Points(...) element or anything like that it does not actually gets created if then nothing references it. Which is awesome, but in an app with multiple dependent functions, with multiple chunks being commented out for debugging purposes, it’s pretty hard to track which element did actually get generated and which gets regenerated when I change the value of a Panel widget. Also, I assume, something similar could happen with the Panel widgets, only existing in theory until something references them, so it would be nice to know when they get accessed, as for example my problem right now seems to be related to a specific widget being used or not.

_jm · June 7, 2021, 7:17pm

Certainly. My code architecture is very bokeh-centric with panel as a higher-level wrapper. I do not currently leverage parameter, the @pn.depends decorator, nor any Holoviews functionality.

Philosophically, I leverage panel when I need capabilities not available in bokeh. It was nice that I started in bokeh a few years back without any exposure to panel, and could easily extend things via panel without having to refactor anything that was already developed.

It has been remarkably smooth; the only current issue I have is with the gauge indicator based on how I have to sequentially build up my app at startup.

Specific examples of panel features leveraged in my apps include (1) panel’s layouts (Column and Row) because they have list-like features not in their bokeh counterparts that allow me to append/remove/pop/clear if the app to dynamically include or exclude different plots in a user’s interactive analysis; (2) panel’s tabs again because of the list-like attributes and also greater control over the tabs like where to place them (to a side, above, etc); (3) widgets like the gauge indicator available in panel but not bokeh; (4) panel’s busy-state indicator/spinner.

_jm · June 8, 2021, 10:58am

A thought based on dissecting a few observations in your original detailed summary, specifically this point about the linear dependence on the number of times that you’re refreshing/reloading

and this regarding the refresh frequency.

If I correctly interpret how you’re running things, the bokeh server is started once but each refresh might correspond to a new session / client connection.

The server has session expiration options which are relevant if that’s the case. They are listed in the panel server user’s guide here. And documented in more detail in the bokeh reference document here. See the Session Expiration Options section of the bokeh reference document.

The bokeh reference indicates that the default lifetime of unused sessions is 15 seconds; see the --unused-session-lifetime option.

If you’re refreshes occur every ~10 seconds and a new session is created with each, your app can be accumulating sessions as time progresses; by way of a simplified example in 30 seconds you have three new sessions but only two oldest unused ones will have expired for a net of one additional session hanging around.

The server won’t clean up any memory associated with the sessions until it actually classifies them as unused so that could (hopefully) account for why you’re seeing such severe/pronounced memory leaks. (My memory issues appear to be much slower and much more subtle).

In any event, it should be straight forward to test the above hypothesis by configuring the option and or adjusting the refresh rate of your tests. Apologies in advance if its a dead end, but it’s worth a try.

khannaum · June 9, 2021, 8:08am

Dear all,

i am reading this thread for many days.

I experienced the same memory leaks in my code using (pandas to hvplot/holoviews/) embedded in panel object

I changed to (vaex->vaex.groupby->pandas to hvplot/holoviews/) embbeded in panel object

but after every embed panel i place a time.sleep(5) for 5 seconds so that any vectorisation,visulization etc completed in panel object completed before assigning new values in panel object

khannaum · June 9, 2021, 8:14am

e.g
#nur_cal_line=
wan_eth_traf_rx_Node_err_line=wan_eth_traf_rx_stat_data_region.hvplot.line(shared_axes=False,ylabel=‘Packets(in thousands)’,title=‘Node Error Packet rate’,x=‘Month’,y=[‘Node_error_rate’],by=‘Region’,height=500,width=600,rot=70,sort_date=True).opts(legend_position=‘top’,toolbar=‘above’)
.opts(toolbar=None,xaxis=‘bottom’,tools=[HoverTool(tooltips=[(‘Region’, ‘@{Region}’),(‘Month’, ‘@{Month}{%Y-%m}’),(‘Node_Error_Packet_rate’, “@Node_error_rate{1}”)], formatters={’@{Month}’: ‘datetime’})])
time.sleep(0.5)
wan_eth_traf_rx_link_err_line=wan_eth_traf_rx_stat_data_region.hvplot.line(shared_axes=False,ylabel=‘Packets(in thousands)’,title=‘Link Packet Error rate’,x=‘Month’,y=[‘link_error_rate’],by=‘Region’,height=500,width=600,rot=70,sort_date=True).opts(legend_position=‘top’,toolbar=‘above’)
.opts(toolbar=None,xaxis=‘bottom’,tools=[HoverTool(tooltips=[(‘Region’, ‘@{Region}’),(‘Month’, ‘@{Month}{%Y-%m}’),(‘Link_Error_Packet_rate’, “@link_error_rate{1}”)], formatters={’@{Month}’: ‘datetime’})])

time.sleep(0.5)

wan_eth_traf_rx_Node_discard_pack_line=wan_eth_traf_rx_stat_data_region.hvplot.line(shared_axes=False,ylabel='Packets(in thousands)',title='Node Discarded Packet rate',x='Month',y=['Node_Discarded_Packet_rate'],by='Region',height=500,width=600,rot=70,sort_date=True).opts(legend_position='top',toolbar='above')\
.opts(toolbar=None,xaxis='bottom',tools=[HoverTool(tooltips=[('MBU', '@{MBU}'),('Month', '@{Month}{%Y-%m}'),('Node_Discarded_Packet_rate', "@Node_Discarded_Packet_rate{1}")], formatters={'@{Month}': 'datetime'})])
time.sleep(0.5)  
wan_eth_traf_rx_link_discard_pack_line=wan_eth_traf_rx_stat_data_region.hvplot.line(shared_axes=False,ylabel='Packets(in thousands)',title='Link Discarded Packet rate',x='Month',y=['link_Discarded_Packet_rate'],by='Region',height=500,width=600,rot=70,sort_date=True).opts(legend_position='top',toolbar='above')\
.opts(toolbar=None,xaxis='bottom',tools=[HoverTool(tooltips=[('MBU', '@{MBU}'),('Month', '@{Month}{%Y-%m}'),('link_Discarded_Packet_rate', "@link_Discarded_Packet_rate{1}")], formatters={'@{Month}': 'datetime'})])
                
time.sleep(0.5)

line_graphs=(wan_eth_traf_rx_Node_err_line + wan_eth_traf_rx_link_err_line+wan_eth_traf_rx_Node_discard_pack_line+wan_eth_traf_rx_link_discard_pack_line).cols(2)
time.sleep(0.5)

tab_dash=pn.Tabs(
(‘Error_Discarded Packets’,tab_wan_ethernet_traffic ),
(‘Congestion’, tab_wan_ethernet_cong),
(‘Transport KPIs with Availability and Microwave Severe Errors’, ‘Comming’),
(‘Fiber/SPO Utilization with CRC Alignment errors’, ‘Comming’),
dynamic=True
)
time.sleep(0.5)

SteveAKopias · June 10, 2021, 11:37pm

@_jm Thanks a lot for the session lifetime idea! I’ve played around with the values, and now I get why the memory previously increased in a sawtooth manner (continuously increasing with regular minor drops in it): there were always some old sessions hanging around, and when they got cleaned up, that freed up some memory. So now I experimented with decreasing the session parameters, and it made the memory increase smoother as the old sessions are immediately cleaned up, so no more dips - but unfortunately the memory leak still stayed. Which makes sense as on the long run it does not really matter if a session lives for 20 more seconds without anything to do or not, it was cleaned up previously and is cleaned up now, just a bit sooner. So whatever remains is something that could not be cleaned up previously and could not be cleaned up now either. But it’s great that at least I understand this mechanic a bit better.

Also interesting that seemingly there are not a lot of common element in our codes. It’s just Panel Columns and Rows (that are pretty basic and would be a big surprise if they would cause this) and just the fact that we both are using Panel Widgets, although not even the same ones. Mighty curious. If you are okay with that, I would be happy to look at your code (even if you only want to share it in private, you can find me at attila.steve.kopias@gmail.com or at https://github.com/ka-steve) to see if a fresh eye can come up with something. Other than dealing with this issue to close this project, I’m currently between jobs/projects, so have some free time to throw around. (But of course only if you’re comfortable with it.)

SteveAKopias · June 10, 2021, 11:46pm

Hi @khannaum,
Thanks for the code example. It’s not entirely clear for me that do you still experience memory leak with this code, or putting in time.sleep() solved it for you?
As far as I understand, sleep does not do much in this situation, as in the code you only define a recipe for the plot and it will only get actually created when the server actually want to display line_graphs for the first time and to do that, it has to create all its elements too, but I can be gravely mistaken as the under the hood operations of the server are still somewhat of a mystery for me.

SteveAKopias · June 10, 2021, 11:50pm

@philippjfr If your time allows, could you share your experiences with the memory leaks in the example projects @jbednar mentioned? It would be be a pretty important data point to know that either 1) you too are poking around in the dark currently; 2) you know what’s happening and it’s on your side of the code; 3) or confirming that the holoviz tools in fact run perfectly for all the demos, so it’s a 100% there is something wrong in our codes.
Thanks!

_jm · June 11, 2021, 12:08am

Hi @SteveAKopias

Thanks for the offer to look over the code; it is sincerely appreciated. Unfortunately, the work is for a closed project that cannot be shared.

The app I am focusing on has session to session variability in the memory use. But things mostly get cleaned up after sessions are acknowledged as unused and then closed.

What I am facing is a slow leak – something on the order of tens of kilobytes in discrete increments every 30 minutes or so. This appears to be fairly periodic although not completely deterministic. And it happens when the session is just idling, i.e. it’s open and there’s a keep-alive mechanism between the server and client, but I’m not actively doing anything interactive.

Regarding the session lifetime properties, there’s also a check-unused-session parameter that works in tandem with the unused-session-lifetime property. It’s basically the resolution on how often the system performs such checks to characterize all the sessions it knows about. So, this can also influence how long it takes an old session to actually get cleaned up based on when the checks are made relative to when it actually stops being used.

One additional thought… have you enabled higher-level logging in the bokeh server. If you set things to debug level, for example, you can get regular printouts of how many sessions it thinks are knocking around and their state.

2021-06-10 18:14:15,614 [pid 64036] 0 clients connected
2021-06-10 18:14:15,615 [pid 64036]   /server has 0 sessions with 0 unused

You can set this via an environment variable as follows (apologies if I am stating the obvious).

export BOKEH_PY_LOG_LEVEL=DEBUG

Arifin · June 18, 2021, 8:49am

Dear all,

I recently realized the memory problem when I deploy my apps. The memory utilization of my app keeps going up and then hang my server.

My app utilizing the panel plotly pane and using its hovering functions to get it’s index from the dataframe, pass it to other widgets by using pn.depends in order to update the objects of other widgets.

To debug it, I started to use --mem-log-frequency to print out the memory usage the apps. Some points that I found:

If new sessions is opened, the memory is increasing a bit.
When I hovering over the scatter plot, the memory keep increasing.
When I closed the browser tab, the numbers of documents, uncollected Documents and uncollected Models are reducing and the uncollected Sessions become 0. However, the memory is not going back to the baseline.
Furthermore, this kind of error also shown when I closed the tab: Module <module 'bokeh_app_2cc5b245240b4a44b4c714144b1685d1' from 'my_apps.py'> has extra unexpected referrers! This could indicate a serious memory leak. Extra referrers: [<cell at 0x7fd66ebf6310: module object at 0x7fd67015c2f0>]
If nothing happened, the memory usage is kept same.

ahuang11 · June 22, 2021, 4:44pm

is your scatter plot powered by datashader by any chance? I noticed the same bokeh app message when I use datashader

Arifin · June 23, 2021, 12:12am

Plotly Pane was used in my case, since my data is under 10^3 order and I need to represent each points on the plot.

ahuang11 · June 29, 2021, 4:00pm

I remember I made an app a while back that also used pd.read_pickle and when I hosted it on Heroku, after some time, I would run into a memory error and the app would crash. To remedy that, rather than reading the entire dataset with pd.read_pickle everytime, I swapped it out with a sqlite3 database + calls Support wildcards · ahuang11/historname@6c27436 · GitHub and I don’t think I experienced memory problems after, but I stopped hosting the app eventually.

sunny · July 27, 2021, 9:13am

Hi there, I’m facing the same “unexpected referrers!” error. I can start my app in Spyder with .show() without error but in the terminal using panel serve I get the mysterious message and the app does not run. Before updating Panel and Geoviews and bokeh (new version 2.3.3) it runs in the terminal like charme. But I did not deployed the app yet. Therefore I porobably didn’t notice the memory leak.