Loading/rendering issues

Hello community,

In the course of my master’s thesis I use Panel. Despite its simplicity, I am unfortunately facing two problems.

  1. When loading a Tabulator with ~ 100 columns, the entire page freezes despite the limitation to page_size = 50 and pagination = ‘remote’. The number of rows seems to be irrelevant here. For data records with the same number of rows and fewer columns, loading takes significantly less time.
    The same (page freezing) happens when a widget is filled with ~ 10,000 options.
    Unfortunately, since everything is frozen, the loading spinner doesn’t work either, at least I don’t know how.
    Is there a trick how I can speed up the rendering?

  2. The app consists of several pages, each of it is called by a button in the sidebar of the template. Same problem as described in 1. Would it be possible to speed up the reloading when switching pages?
    Would be very grateful for your advice!

Thanks,
Torben

1 Like

Hi @torben335

Welcome to the community.

Could you provide a couple of minimal, reproducible code examples?

That will make it much easier to help solve the problem

Thanks.

1 Like

Hi Marc, thanks for your reply and offer of help. Here reproducible code to show the problem.

import panel as pn
import pandas as pd
import numpy as np


pn.extension(comms='vscode')
pn.config.sizing_mode = 'stretch_width'

df = pd.DataFrame()
  
ID_CreditSC_App = pn.template.BootstrapTemplate()

mainColumn = pn.Column()

df_tabulator = pn.widgets.Tabulator(df,name='Original Dataset', disabled =True, height=400, pagination='remote', page_size=50)
df_tabulator1 = pn.widgets.Tabulator(df,name='Original Dataset', disabled =True, height=400, pagination='remote', page_size=50)

tabs_dataset = pn.Tabs(
    (df_tabulator),
    (df_tabulator1)
)

multiSelect = pn.widgets.MultiSelect(name='Segmentation value', size=8)

btn_main    = pn.widgets.Button(name='Main')
btn_dataset = pn.widgets.Button(name='Dataset')
btn_loadSelector = pn.widgets.Button(name='Load')
btn_loadTable = pn.widgets.Button(name='Load')

def mainClick(event):
    mainColumn[:] = mainView

def datasetClick(event):
    mainColumn[:] = datasetView

def init_df():
    for i in range(100):
        df.insert(0, 'Column' + str(i), np.random.randint(0, 10, 10000))

def loadSelector(event):
    multiSelect.options = df['Column1'].values.tolist()

def loadTable(event):
    df_tabulator.value = df
    df_tabulator1.value = df



btn_main.on_click(mainClick)
btn_dataset.on_click(datasetClick)
btn_loadSelector.on_click(loadSelector)
btn_loadTable.on_click(loadTable)


mainView = [btn_loadSelector,multiSelect]
datasetView = [btn_loadTable, tabs_dataset]

init_df()

ID_CreditSC_App.sidebar.append(btn_main)
ID_CreditSC_App.sidebar.append(btn_dataset)
ID_CreditSC_App.main.append(mainColumn)
ID_CreditSC_App.servable()

To 1.) I am using a randomly generated dataframe (this serves as an example for our real data) with 10000 rows and 100 columns, which is loaded into a tabulator widget. I believe that the number of rows has little to no effect on speed, but the number of columns does. The original code I am working on uses two tabs (each with a tabulator widget, a copy of the data is loaded into the second tabulator and in the following workflow is modified step by step). This double loading can slow down the process a bit but is necessary for the app’s purpose to later have an overview and quickly be able to compare them.

The described process of loading the dataframe into the tabulator is initiated by clicking the “Load” button on the Dataset page (also for simulation purpose). The fact that it takes long the first time would not be a problem for my purposes. But it would be great if the page did not freeze during the loading process.

Similarly, on the main page the MultiSelect widget options can be updated with the corresponding “Load” button on that page. Here the contents of an entire column of the dataframe and its 10000 entries are used. This also takes a very long time. Is there any way to speed this up?

To 2.) Once the initial loading to the tabulator widget finished and I switch between the two pages (Main & Dataset), it seems like everything is reloaded again. I would really like to avoid this “reloading/rendering” to allow quick switching between the two pages.

Thanks!

Any idea @Marc ?

I’ve moved the discussion to Github as a bug. Please join the @torben335

Cannot load Tabulator with 100 columns: Access to fetch at ‘http://panel.pyviz.org/’ … blocked by CORS policy · Issue #2327 · holoviz/panel (github.com)

First of all, thanks @Marc for helping clarify the Tabulator issue. Seems like I still have to be patient about that.

Do you have any ideas about the other two problems?

Similarly, on the main page the MultiSelect widget options can be updated with the corresponding “Load” button on that page. Here the contents of an entire column of the dataframe and its 10000 entries are used. This also takes a very long time. Is there any way to speed this up?

To 2.) Once the initial loading to the tabulator widget finished and I switch between the two pages (Main & Dataset), it seems like everything is reloaded again. I would really like to avoid this “reloading/rendering” to allow quick switching between the two pages.

1 Like

Regarding question 2. I Think you could try using tabs instead. Or adding both pages and then hiding One or the other.

Hi Torben335,

Regarding the problem that you are mentioning with the columns for pn.Tabulator, I am having the same issue. I have try different Dataframe shapes but clearly for shapes with more than 100 columns, the component doesn’t work properly. Here some examples I have tested:

Shape= (81, 63)
Shape= (53, 12)
Shape= (206, 256)
Shape= (590, 21)
Shape= (53, 12)
Shape= (206, 256)
Shape= (240, 48)

It is also worth mentioning, that when loading sheets with at least 256 columns, something breaks and the application becomes unstable, the response time is really slow and sometimes I even get the image.
I have tried to switch to pn.DataFrame but the problem is even worse, so looks like there is not a simple way to walk away from pn.Tabulator. (version:Panel 0.14.3 py310haa95532_0)

Did you get a way around the number of columns problem for the Tabulator in panel ?
Thanks in advance for the assistance and any help you may provide.

Here a reproducible example, where it is easy to appreciate the performance and stability issues:

import panel as pn
import pandas as pd
import datetime as dt
import random as rnd

pn.extension('tabulator', css_files=[pn.io.resources.CSS_URLS['font-awesome']], template="fast", sizing_mode='stretch_both')

nrows = int(100)
ncolumns= int(256)

rows= [dt.datetime.now().strftime("%M%S")*rnd.randint(1,12) for i in range(nrows)]

data = {f'col{i}': rows for i in range(ncolumns)}


df = pd.DataFrame(data)

tabulator = pn.widgets.Tabulator( df,
                            name="test",
                            sizing_mode= 'stretch_width',
                            height= 600,
                            width= 800)

tabs = pn.Tabs(sizing_mode='stretch_both')
tabs.append(pn.Column(tabulator, name=f"tab 1"))

pn.Column(tabs).show()

Thanks