Panel Performance larger Files

Hey there,
I started using panel for some fast data exploration website and it works fine, but has a huge amount of time for the first startup. I have some data files around 400-500MB I need to load into memory and I understand if on starting the whole application it takes once some time to load the data, but it takes this time after every page refresh. When the site is once loaded, the speed for calculation of plats and statistics is sufficient.
So for me it seems, that on page reload, the complete code gets executed and some heavy objects are not shared by default between different sessions. Is there some possibility to do so, so that I can somehow mark my data loaded into RAM to be available for all session?
Please correct me if my assumptions on how the underlying structure works is wrong.

1 Like

Hi @Richard

Welcome to the community :+1:

I believe pn.state.cache is what you are looking for. I have never used it (but I should!). From what I know its a simple dictionary where you can share key, values between sessions.

So maybe you can load the data into a dictionary, dataframe or similar and store that as a value in pn.state.cache before you serve the applications. Or maybe the first time the data is needed.

See https://panel.holoviz.org/user_guide/Overview.html

And if possible report back how it wen’t or with a small, reproducible code example if you cannot get it working.

2 Likes