This is a followup for previous threads:
I started this thread with the hope that the long runtime and the memory leak in my app are connected and by finding the source of the runtime, I would eliminate the memory leak.
When that didn’t work out, I reprioritized, realizing I can live together with the slow runtime, but I definitely need to resolve the memory leak issue, so that’s when I started this thread on the Bokeh Discourse.
Since that I did not see the error message mentioned there, but I’m still struggling with multiple memory leaks, and also, having the feeling that I must be doing something terribly wrong. So I just got a sense that my question is more about the recommended way of developing Holoviews/Panel applications rather than being a technical issue with the Bokeh server, so I decided to create a new thread for this here, hoping @philippjfr @jbednar @Marc could share their two cents on the issue.
What I’m trying to do is not rocket science. My script is 6000 lines total with all the css, html, js code and configuration variables. This is a bit too big just to present it as an MRE, but as far as webapps go, this is by far the smallest I ever worked on. I use python 3.9.2, I don’t have threads or anything weird in the code. I don’t even use
@contexts other than
@pn.depends'. The main logic of the script is pretty simple. It has practically two main workflows:
- Reads in a CSV of locations with
pd.read_csv(), filters them based on Panel widgets and displays them as a) Holoviews Points if the user is zoomed in enough; b) as a Datashader categorical aggregate otherwise.
- Based on Panel widgets it reads in some Tiff files with rioxarray and displays them as Datashader regrid.
Both of these go onto a Holoviews tilemap and are configurable by the Panel widgets through either
.apply(some_function, some_panel_widget), then both the map and the widgets get loaded into a template and finally the template is displayed by
And no matter what I do, the script uses an increasing amount of memory on every page load with no plateau. If I separate the CSV and TIFF workflows into two separate scripts, both of them will have the same problem. If I eliminate both of these workflows and only leave the basic configuration variables, Panel widgets and templates in place, that still has a memory leak.
I’m not looking for a solution for this specific problem. My main issue is that I’m poking around in the dark here. My script ends with a
full_content.servable(). After that line I don’t have any control over the variables and that’s where they should be freed up. I don’t get any “hey, these variables could not be freed” error. I don’t have any information on what uses these variables. I don’t even know what the Bokeh server does with them. Yeah, I can use
tracemalloc to see the most heavily memory-intensive areas of the script, but wouldn’t tell me anything about the source of the leaks (I did not see any sings of those leaks I found later).
And to be honest, there are some alarms ringing in my head. While being pretty new with the Bokeh+Holoviz ecosystem, I have a bit of experience with developing web applications, and I’m definitely missing something here. I’m pretty sure this is not how professional development of a web application should look like. Like I can’t imagine that the best way of developing, the way experienced agencies are developing is putting together an app in weeks and then spending further weeks poking around in the dark.
Based on my research (I’ve read everything I could find) there is surprisingly few information on memory leaks in python, and also basically no tools fit for monitoring commercial level projects and providing detailed information on memory usage. That (and these articles stating it) makes me think that these issues are extremely rare and developers usually shouldn’t care about this.
On the other hand, my experience is that basically no matter what I do, whichever way I cut my project, it bleeds memory everywhere, originally creating the feeling that this is a pretty common issue. Which wouldn’t be a huge pain if memory management would be the developers’ responsibility with the necessary tools available (as it is in some languages like C++), but again, it doesn’t look like it is. So I’m stuck in this weird limbo, where I try to modify my script line by line, running it, then refreshing a few dozen times (every refresh of course takes ~10s) to see if anything changed, but even if it did, I don’t really have a clear idea what went wrong and how to solve it.
Like the first memory leak was related to an
opts(hooks=...) part coming from this comment of Phillip, that allowed for me to change the actual background-color of a Holoviews Overlay. It took literally a week to discover that in the N-th really dumbed-down version of the code. If that line was there, there was a heavy memory leak, as soon as I removed it, the dummy script worked like a charm. (I’m talking about either staying between 700-900MB even if constantly refreshed by
window.location.reload() for hours and consuming about (400+N*80) MB memory where N = number of refreshes and steadily climbing up to any gigabytes, so the difference is pretty striking.) I use hooks in multiple places and they usually don’t cause any problems, but this time the
combined = (tiles * layer1 * layer2).opts(hooks=...) wasn’t working out. So I did the same in my original script removing the same
opts(), and nothing changed. The reason for that being, I assume, that there are definitely more memory leaks.
The second example I found is a
@pn.depends() function that returns some widgets responsible for configuring the currently visible TIFF layers embedded in Cells and Rows. The function depends on the values of three widgets. If I remove the 3rd one (a DiscreteSlider) from the
@pn.depends(...) part, from the function parameters and remove every line that would use it, my dumbed down script runs without any leaks. If then I put it back into the
@pn.depends() part and into the function parameters but still not using the variable anywhere in the function, than the script will have a memory leak again. (About the same difference as described above.) The most fascinating experience. If I replace the DiscreteSlider with an IntSlider or a TextInput, nothing changes, exactly the same leak happens or not. Also, if I remove the widget from the same function in my original app, accepting the loss of functionality, it doesn’t even help. The most likely reason for that, I’m thinking, that my dumbed down script only referenced this widget in that one function, but the full script uses it in multiple places, so I remove it from everywhere, just to see that it doesn’t change anything. Maybe it helped, but there’s still another leak so I just don’t see the improvement? Most probably, but I’m just guessing.
So you get the picture. I can not search for memory leaks by commenting out a few bits as there are multiple leaks, and If I leave even one in, I will not realize I just commented out another one. So I have to create really dumbed down dummy scripts and building them up again line by line, spending multiple minutes with every refresh. And even that falls apart when I find a bit that without any question causes a memory leak, and I 1) have no idea why; 2) don’t know how what to do to eliminate the leak but keep the functionality; 3) even if I completely delete that part, it is possible that some other parts not present in the current dumbed down version will trigger the same issue.
Now my working theory as always is that I must be doing something wrong if this keeps happening, but anyways, sooner or later I will figure this out for this project. But what’s more interesting for me is if this is what I should expect every time I create something? I’m not surprised at me creating a few bugs using new tools, I’ve did that every time I switched to new languages or modules, but this is the first time I don’t see a clear path to learn to quickly resolve them. I assume, however, this is not how it is intended to work, and supposedly @philippjfr and @Marc and many others did create or regularly create professional commercially viable applications either for real or as an example. I’m really curious of your experience and want to learn what you do differently.