Max. upload size

As far as I could figure out, it should be possible to increase the maximum file size for uploads via something like this:

server = pn.serve(\
                  dashboard, 
                  websocket_max_message_size=10485760000,\
                  max_buffer_size=10485760000,\
                  max_body_size=10485760000
                 )

But neither for the built-in file upload widget nor for the DropzoneJs solution from this forum, this has any effect. Above 100 MB uploads do not work. (The built-in upload causes the tab in Firefox to creash, the dropzone reports a Code 0 error.)

Any ideas on why this has no effect? Do I have to start the server in a different way?

1 Like

Hi @rbps1

I don’t know the solution. But my idea for investigating would be

  1. Search for knowledge about this for bokeh via google, in the bokeh documentation and on the Bokeh Discourse - Bokeh.
  2. Try to debug what happens when you start the server. My hypothesis is that the max_buffer_size is not provided to the bokeh/ tornado server as it should. Maybe when you step through the code you can identify where it should be provided and fix it? Maybe even provide a PR to Panel.

Maybe see for a solution https://github.com/holoviz/panel/issues/1559

Hi @Marc,

Thanks for you reply. As for 1. I couldn’t solve the problem with the Bokeh documentation as I do no yet understand exactly what is wrong.

As for 2: server._tornado.settings gives me 'websocket_max_message_size': 10485760000, 'max_buffer_size': 10485760000, 'max_body_size': 10485760000, so I assume the settings have been passed to tornado.

I also played around with different browsers:

With the FileInput Widget:

  • Edge does nothing, doesn’t even upload small files
  • To my surprise Firefox can handle files slightly above 100 MB (120 / 130 MB), but if files are larger the tab crashes (not the server).
  • Chrome the same, 120 / 130 MB are still okay, above nothing happens. No crash, no error message, but also no file upload.

I’ll try a few more things

With the Dropzone:

  • Firefox, Edge, and Chrome have the limit at 100 MB (120 / 130 does not work). Dropzone show an error “Server responded with 0 code”

Hi @Hoxbro,
I increased buffer, body, and message size with the effects I discribed.

The first attempt was with a small file (13 MB) that upload sucessfully.
The second attempts was a bigger file that indeed shows no response code.
I am lost with understanding why the small file shows only 161 B (but was uploaded completly) and the aborted one 16.84 MB…

My UploadHandler is

class UploadHandler(web.RequestHandler):     
    def post(self, upload_n):
        for form_data_name in self.request.files:
            for file in self.request.files[form_data_name]:
                save = open(basefolder + '/' + upload_n + '/' + file['filename'], 'wb')
                save.write(file['body'])
         self.finish("Files are uploaded")
        print("File " + file['filename'] + " uploaded in " + upload_n)

and I register it via

server._tornado.add_handlers(r'.*', [
    (r"/upload/([^/]+)", UploadHandler)
])

Meanwhile, I tried to figure out what the problem is but could not make any progress.

With larger files, no matter what I do, the FileInput Widget keeps crashing Firefox Tabs and has not reaction in Chrome. Dropzone responds with 0 code. In both scenerios, the problem is somewhere before the tornado server even reaches the handler.

Hence, I consider these solutions a dead-end.

Is there any best-practice on how to upload large files to a server when using Panel?

Hi @rbps1

Could you post a minimum, reproducible code example that enables uploading as much as possible?

I think it would make it easier for the community to reproduce and start finding solutions instead of just providing ideas and hypothesis.

Make it concrete :+1:

Hi @rbps1

Have you seen this FileInput widget is limited by the size of the tornado `max_buffer_size1 · Issue #1559 · holoviz/panel · GitHub. If you are running in jupyter you need to change a setting in Jupyter as well.

Hi @Marc

I am not using Jupyter to start the server.

import panel as pn
pn.extension()

file_input = pn.widgets.FileInput(multiple=True)

def file_input_watch(event):
    print('File uploaded')

file_input.param.watch(file_input_watch,'value')

if __name__ == '__main__':
    server = pn.serve(\
                      file_input, \
                      start=False, show=False,\
                      websocket_max_message_size=10485760000,\
                      max_buffer_size=10485760000,\
                      max_body_size=10485760000,\
                      max_header_size=10485760000,\
                      port=8080,\
                      verbose=True,
                      threaded=False\
                     )

    server.run_until_shutdown()

Small files are fine. Bigger files show no reaction (i.e., the print command above is not reached) (Chrome) or the tabs crashes (FF). Multiples files exhibit the same behavior, even if they are small files.

3 Likes