Multi-user app, memory leak (on EC2)

Hey All,
I have built a mitli-user app where the user gives access to a large zip file from s3 servers and some csv with metadata or from a database on MSSQL and then the data can be filtered and plotted in multiple ways.
All of this is served on a EC2 server.

Unfortunately I see a memory increase up to the maximum that i have on my server then the app crashes.
I have set the following parameters:

--mem-log-frequency 10000 \
--unused-session-lifetime 900000 \
--check-unused-sessions 60000 \
--log-file logs/panel-log.log \

the first few lines of the log are:
2022-03-28 09:01:09,752 [pid 4243] Memory usage: 129.53 MB (RSS), 306.61 MB (VMS)
but after few users use the app i get:
2022-04-01 08:15:26,940 [pid 315] Memory usage: 560.59 MB (RSS), 869.03 MB (VMS)

and then the session just crashes (either the ec2 stops responding or the server that is run through a screen command just closes the screen)

How do I deal with this? :frowning:
Can I set a callback on session timeout that erases some stuff?
Any other suggestions?
Thanks in advance

there was some related post, but I do not remember a solution being posted

I am trying to use “pn.state.kill_all_servers()” or make a callback that uses garbage collector to delete all instances of the app class, but it does not seem to solve the memory leak…

if anyone has any idea how to solve this, please help me

The solution was an updated numba if I remember correctly.

I am not using numba
And it is updated to the latest version on my env

Is there a way to access sessions manually and terminate them?

As I mentioned before I tried to make a button with a callback to remove all initiated class instances, but it does not remove anything from memory…

I also tried invoking pn.state.kill_all_servers() on my callback… No help

Unfortunately, it’s hard to debug (find the cause of the memory leak) without any example code.

As @ahung11 says, it is hard to know without any concrete code.

Something which stands out to me is the large zip file. Do you load the zip file into memory for each user? And if you do, are you caching it for faster access?

I will try to share some code, although its quite a big app already and might be hard to debug. Also, I might not be able to provide the input data since it is confidential.

I use smart_open to access the file in the following way

with smart_open(results) as f1:
with ZipFile(f1) as f2:
for path in path_list:
with as f3:

Is it possible that although i open the file in the following manner, it stays in memory?

with smart_open(results) as f1:
___with ZipFile(f1) as f2:
______for path in path_list:
_________with as f3: