Memory Leak issue with "extra unexpexted referrers!" error

BranLWill · October 5, 2021, 2:42pm

ALL software version info

Windows OS
Google Chrome
bokeh - 2.3.2
panel 0.11.3
numpy 1.19.2
pandas 1.2.5
hvplot 0.7.1
holoviews 1.14.3
Pympler 0.9

Description of expected behavior and the observed behavior

We are trying to use classes that derive from panel components to create our own custom panels/components. We are running into a problem with a memory leak when using hv.tables and potentially with other components/classes. Let us know if there is an issue with the way we are implementing these libraries into our code or if there is actually a memory leak with holoviews holding on to references after a component is deleted. @philippjfr might have something to do with - Bokeh server getting killed by memory leak, and sometimes also getting Getting an "extra unexpected referrers!" error - #3 by Bryan - Community Support - Bokeh Discourse

Complete, minimal, self-contained example code that reproduces the issue

import gc
import holoviews as hv
import numpy as np
import panel as pn
import pandas as pd
import hvplot.pandas
from collections import OrderedDict as odict
from panel.template import DarkTheme, DefaultTheme
from pympler import tracker

hv.extension('bokeh')
pn.extension(sizing_mode='stretch_width')

class Application():
    hv.DynamicMap.cache_size = 0
    main_cmap = ['#265e3b','#326745','#3d7150','#497b5b','#548566','#608e71','#6b987c','#77a388','#83ad93','#8fb79f'] #Theme Light Blues/Greens

    def __init__(self, title, light_mode=True):
        if light_mode:
            self.main_page = pn.template.MaterialTemplate(title=title, theme=DefaultTheme)
        else:
            self.main_page = pn.template.MaterialTemplate(title=title, theme=DarkTheme)

        self.main_page.sidebar.append("\nBaby shark, doo doo doo doo doo doo\n"
                                      "\nBaby shark, doo doo doo doo doo doo\n"
                                      "\nBaby shark, doo doo doo doo doo doo\n"
                                      "\nBaby shark!\n\n"

                                      "\nMommy shark, doo doo doo doo doo doo\n"
                                      "\nMommy shark, doo doo doo doo doo doo\n"
                                      "\nMommy shark, doo doo doo doo doo doo\n"
                                      "\nMommy shark!\n\n"

                                      "\nDaddy shark, doo doo doo doo doo doo\n"
                                      "\nDaddy shark, doo doo doo doo doo doo\n"
                                      "\nDaddy shark, doo doo doo doo doo doo\n"
                                      "\nDaddy shark!\n\n"

                                      "\nGrandma shark, doo doo doo doo doo doo\n"
                                      "\nGrandma shark, doo doo doo doo doo doo\n"
                                      "\nGrandma shark, doo doo doo doo doo doo\n"
                                      "\nGrandma shark!\n\n"

                                      "\nGrandpa shark, doo doo doo doo doo doo\n"
                                      "\nGrandpa shark, doo doo doo doo doo doo\n"
                                      "\nGrandpa shark, doo doo doo doo doo doo\n"
                                      "\nGrandpa shark!\n\n"

                                      "\nLet’s go hunt, doo doo doo doo doo doo\n"
                                      "\nLet’s go hunt, doo doo doo doo doo doo\n"
                                      "\nLet’s go hunt, doo doo doo doo doo doo\n"
                                      "\nLet’s go hunt!\n\n")
                                
        self.delete_button = pn.widgets.Button(name='Delete', button_type='danger', disabled=True)
        self.delete_button.on_click(self.delete)
        self.view_button = pn.widgets.Button(name='View Data', button_type='success', disabled=False)
        self.view_button.on_click(self.view)
        self.info = InfoPanel(CustomTab(self.view_button, self.delete_button))
        self.main_page.main.append(self.info)

    def delete(self, event):
        """Delete button callback to delete a selected experiment."""
        self.info[-1] = CustomTab(self.view_button, self.delete_button)
        self.delete_button.disabled = True
        self.view_button.disabled = False
        self.show_memory_usage()

    def view(self, event):
        """Save selected experiment."""
        #Initializing the meta data
        self.info[0].add_tab(('Testing View', pn.Card(CustomPanel(self.select_data(), 'Test Plots', self.main_cmap), title='Data View')))
        self.delete_button.disabled = False
        self.view_button.disabled = True
        self.show_memory_usage()

    def select_data(self, num=1000000):
        np.random.seed(1)

        dists = {cat: pd.DataFrame(odict([('x',np.random.normal(x,s,num)), 
                                        ('y',np.random.normal(y,s,num)), 
                                        ('val',val), 
                                        ('cat',cat)]))      
                for x,  y,  s,  val, cat in 
                [(  2,  2, 0.03, 10, "d1"), 
                (  2, -2, 0.10, 20, "d2"), 
                ( -2, -2, 0.50, 30, "d3"), 
                ( -2,  2, 1.00, 40, "d4"), 
                (  0,  0, 3.00, 50, "d5")] }

        df = pd.concat(dists,ignore_index=True)
        df["cat"] = df["cat"].astype("category")

        return df

    def show_memory_usage(self):
        mem = tracker.SummaryTracker()
        memory = pd.DataFrame(mem.create_summary(), columns=['object', 'number_of_objects', 'memory'])
        memory['mem_per_object'] = memory['memory'] / memory['number_of_objects']
        memory['memory'] = round(memory['memory'] / 1000000, 2)
        print('\n####################################### Memory Allocation ##########################################')
        print(memory.sort_values('memory', ascending=False).head(10))
        print()
        print(f"Total Memory - {memory['memory'].sum()} MB")
        print('####################################################################################################\n')
        print(gc.collect()) # prints number of unreachable objects

class CustomPanel(pn.Column):
    def __init__(self, df, name, main_cmap, *objects, **params):
        run_meta_columns = ['val', 'x', 'y']
        plot = df.hvplot.scatter(x='x', y='y', title='Original',
                                cmap=main_cmap, grid=True, datashade=True, width=800, height=600).opts(bgcolor='lightgrey')
        
        super().__init__(plot, SelectionTablePanel(df, columns=['cat', *run_meta_columns]), name=name, *objects, **params) #induces memory leak
        # super().__init__(plot, name=name, *objects, **params) #no memory leak

class InfoPanel(pn.Column):
    def __init__(self, init_panel, *objects, **params):
        super().__init__(init_panel, *objects, **params)

class CustomTab(pn.Tabs):
    def __init__(self, view_button, delete_button, *objects, **params):
        init_tab = ('Overview', pn.Row(view_button, delete_button))
        super().__init__(init_tab, dynamic=False, *objects, **params)

    def refresh_tabs(self, tabs):
        self.clear()
        self = tabs

    def add_tab(self, panel):
        self.append(panel)

    def remove_tab(self, index):
        self.pop(index)

class SelectionTablePanel(pn.Row):
    def __init__(self, df, columns=None, sort_column='', *args, **kwargs):
        table_df = df[columns]

        if sort_column != '':
            table_df = table_df.sort_values(sort_column, ascending=False).reset_index(drop=True)

        super().__init__(hv.Table(table_df).opts(hooks=[self.hide_index]), *args, **kwargs)

    def hide_index(self, plot, element):
        plot.handles['table'].index_position = None

#app = Application(title='Memory Leak', light_mode=True).main_page
app = Application(title='Memory Leak', light_mode=False).main_page
app.servable()

Stack traceback and/or browser JavaScript console output

bokeh.document.document - ERROR - Module <module ‘bokeh_app_f972585d1ef142c1ab049361ea8dff71’ from ‘{app-file-path}’> has extra unexpected referrers! This could indicate a serious memory leak. Extra referrers: [<cell at 0x000001F13D878100: module object at 0x000001F13D8A7B80>]

Screenshots or screencasts of the bug in action

ahuang11 · October 5, 2021, 3:27pm

Should be fixed at:

github.com/numba/numba

CUDA: Fix potential leaks when initialization fails

numba:master ← gmarkall:fix-init-fail-leak

opened 09:42AM - 02 Sep 21 UTC

gmarkall

+161 -11

When CUDA driver initialization fails, the driver singleton object persists hold…ing on to an exception object that references calling frames and potentially other objects. This can create a leak in the case where modules that attempt to initialize the driver and then fail are created and destroyed. This change rectifies the issue by holding on to a string describing the error instead of the exception object. There are a couple of small related changes: - `initialize()` is changed to `ensure_initialized()`, and the caller no longer needs to check whether it should be called - it can always call it when it needs to ensure that the driver is initialized. - `cuda.cuda_error()` returns the error string instead of an exception object. Constructing an exception object here just to maintain the original behavior seems a bit convoluted; it's likely that any code using `cuda_error()` is checking whether its return value is `None` rather than looking for a specific instance of an exception class to see if an exception occurred. Some initialization tests are added - for the failing cases we need to run in a subprocess to avoid interfering with the initialization of the driver in the process in which we're actually running tests. The failure of `cuInit(0)` is accomplished by a slightly unorthodox patching of `driver.cuInit`, which is needed because driver functions are added to the `Driver` object on-demand, so there is nothing for `mock.patch.object()` to replace at the time we need to set up the mock.

github.com/holoviz/panel

panel serve logs stops working when importing datashader

opened 03:27PM - 09 May 21 UTC

Hoxbro

#### ALL software version info Windows 10 20H2 holoviews 1.14.3 panel 0.11.3… datashader 0.12.1 #### Description of expected behavior and the observed behavior I'm not sure if this a `panel`, `datashader`, or a `holoviews` issue. The problem is when importing `holoviews.operation.datashader` in my file, the logging in the command window stops sending logs. When running`panel serve tmp.py --autoreload` with `holoviews.operation.datashader` imported no logs show up at all and I have to guess when ready. #### Complete, minimal, self-contained example code that reproduces the issue ``` python import panel as pn import holoviews.operation.datashader pn.panel("# Test").servable() ``` And run the command `panel serve` or `panel serve --autoreload` #### Screenshots or screencasts of the bug in action | | No `datashade` import | `datashade` import | |--|--|--| | Without `--autoreload` | ![2021-05-09 17_02_50](https://user-images.githubusercontent.com/19758978/117577459-24225e00-b0ea-11eb-9e2b-e77b45932b3a.png) | ![2021-05-09 17_05_44](https://user-images.githubusercontent.com/19758978/117577513-5e8bfb00-b0ea-11eb-92da-f1d3a4c1882a.png) | | With `--autoreload` | ![2021-05-09 17_04_59](https://user-images.githubusercontent.com/19758978/117577490-3e5c3c00-b0ea-11eb-8f8b-6b1416677816.png) | ![2021-05-09 17_06_36](https://user-images.githubusercontent.com/19758978/117577521-677ccc80-b0ea-11eb-8a64-157f0c25e5b6.png) |

philippjfr · October 5, 2021, 6:24pm

Yes, please revisit this with latest versions.

BranLWill · October 5, 2021, 6:41pm

Using this method at the end of every class that derives from a panel component works at cleaning up the left over memory. This method gets called after a replacement/deletion of the given class.

    def _cleanup(self, root):
        for _ in range(len(self)):
            obj = self.pop(0)
            del obj
        return super()._cleanup(root)

Note: You still need to use gc.collect() somewhere after the replacement/deletion action is done.

Marc · October 6, 2021, 4:27pm

Is this needed @philippjfr ?

philippjfr · October 6, 2021, 4:32pm

I can see how it could help in a server context but it’s also not a safe thing to do since the object may be reused. So no this is not something we could add to Panel itself.

philippjfr · October 6, 2021, 4:57pm

There might be something like this we could do in server_destroy, if we can somehow determine that there are no other referrers to the object.