Large file does not load in file_input

Panel
,
1

Hello,

I am having issues importing files into my code with file_input. I have read Workaround for large files and tried but file was not load. I tried a little code to check if it enters sometimes but it seems it hangs and it is not a matter of time to load (Code is below)

class App(param.Parameterized):
    file_input = param.Parameter()
    
    def __init__(self, **params):
        self.param.file_input.default = pn.widgets.FileInput()
        super().__init__(**params)
        
    @pn.depends("file_input.value", watch=True)
    def file(self):
        value = self.file_input.value
        
        pl = pv.Plotter(notebook=True)
        
        print(len(value))
        
        if value:
            self.file_val = value
            print("file")
        else:
            print("error")

        print()
        
    def view(self):
        return pn.Column(
            self.file_input,
        )

    
    
app = App()

app_view = app.view()
app_view.show()

I used and 4.4 MB files can be read but over 50 MB I was not able to retrieve data from files.

Does anybody knows how to deal with this and read a large file?

Thanks for your help

2

Hi @Toni,

My understanding here is that file input utilises the browser, therefore your constrained to file sizes through the browser limits whatever they may be, you can increase the max message length but I don’t think this goes very far that I was able to tell before I gave up on this route. I was able to go from ~10 to 30mb I didn’t find the upper limit exactly but it was less than 80mb for myself. The below was done from command line rather than jupyter

panel serve --show SinglePTPlotter.ipynb --websocket-max-message-size 1000000000

For myself I use the multi - FileSelector or param, from this your able to obtain the path & filename selected in code and then process into panadas dataframe for example. I’m not sure how the multifile selector differs under the hood, my guess is it runs server side somewhat and therefore not restricted to browser file size limits and has access to path names that is otherwise restricted I think if the input mechanisim is using web browser features. I would like to see something like this but for single files, the param has single file but it’s more like a drop down list box and not a browser as such - if a drop down box would work for you then it’s an option. See my param multi file selector query it has a list box, you can see from the code - nb I don’t know if this is the right way but it’s a way and I’m interested in other ways too.

So I guess in short I make use of single file selctor param, multi file selector widget / param working with files 300MB and up.

Hope the above is of some help (& apologies for regurgitating everything and more you probably read in your link)

Thanks, Carl.

3

Hello @carl,
I will try first approach as I need to load files out from local server in my code. In case it does not work maybe I try to use FileSelector and work around to allow use from other computers.
Thanks for your help, I really appreciate. If I find some solution in future I will let you know.

1 Like
4

Hello,

Following advice from @carl, I tried out his advice with a combination of this post in forum and it worked. It loads all information of big files (I only tried in my personal case up to 56 MB) without problem.
My workaround based on previous forum post would look in my case like:

class App(param.Parameterized):
    file_input = param.Parameter()
    
    def __init__(self, **params):
        self.param.file_input.default = pn.widgets.FileInput()
        super().__init__(**params)
        
    @pn.depends("file_input.value", watch=True)
    def file(self):
        value = self.file_input.value
        
        pl = pv.Plotter(notebook=True)
        
        print(len(value))
        
        if value:
            self.file_val = value
            print("file")
        else:
            print("error")

        print()
        
    def view(self):
        return pn.Column(
            self.file_input,
        )

    
    
if __name__ == '__main__':
    app = App()
    app_view = app.view()
    
    server = pn.serve(
                      app_view, \
                      websocket_max_message_size=10485760000,\
                      max_buffer_size=10485760000,\
                      max_body_size=10485760000,\
                      max_header_size=10485760000,\
                      verbose=True,
                      threaded=False\
                     )

Thanks a lot for your help.

2 Likes
5

Happy you made it. Thanks for taking the time to share your solution.

1 Like