Best practice for displaying high resolution camera images captured on server

Panel
,
1

Hi,

I’m trying to expose some camera functionallity on a panel dashboard.
At the moment, a PC has an industrial camera connected and serves panel. On that same PC I open up the dashboard in a browser, click a “Snapshot” button, which captures a frame and updates a param.Parameter. I have this code that depends on that param:


class Camera(param.Parameterized):
    latest = param.Parameter(default=(np.ones(shape=(50, 50), dtype=np.uint16) * 2000)) # Just some initial content


class InstrumentState(param.Parameterized):
    camera = Camera()

........

@pn.depends(image=instrumentState.camera.param.latest, watch=True)
def img0(image):
    return hv.Image(image, bounds=bounds).opts(*options)

layout = pn.Row(
    pn.Column(
        hv.DynamicMap(img0)
    )
)

I felt like it took ages to take a snapshot, so I made some very primitive profiling and found that it took me 1.3s to do:

instrumentState.camera.latest = nparray_reshaped # A 3208x2200 numpy 8bit array.

I can see in the developer console that there is a 7.1MB binary sent on the websocket when .latest is updated. It should take no time at all to send 7.1 from one application to a browser on the same PC, right?

Maybe I’m not doing this the right way at all. If anyone has previous experience of updating large images in panel i would be very interested to hear what kind of results you have achieved.

I will probably setup a minimal reproducable example tomorrow.

1 Like
2

Hi @Wiggan

One thing I notice is that you have watch=True on img0. This will make the Dynamic map update twice, so you can safely remove it.

3

Hi @Wiggan

I also tried building a minimum, reproducible example and I also experience that the it takes a very long time to transfer the high resolution data. The server also raises an exception for me before I reach the dimensions of your example.

I have raised it as a bug on Github here Transfer of high resolution image takes way too long and even raises WebSocketClosedError · Issue #3874 · holoviz/panel (github.com).

My hypothesis is that the issue is that the data is transfered as .json which is not efficient for this use case. But I don’t know.

import param
import panel as pn
import numpy as np
import holoviews as hv
import time

pn.extension()

bounds=(-1,-1,1,1)

X=2200
Y=3208

class Camera(param.Parameterized):
    
    xresolution = param.Integer(100, bounds=(100, 2200), step=100)
    yresolution = param.Integer(100, bounds=(100, 3200), step=100)
    
    take = param.Event()
    count = param.Integer()
    duration = param.Number()

    value = param.Parameter() # Just some initial content


    @pn.depends("take", watch=True, on_init=True)
    def _take_a_picture(self):

        xls = np.linspace(0, 10, self.xresolution)
        yls = np.linspace(0, 10, self.yresolution)
        xx, yy = np.meshgrid(yls, xls)
        
        start = time.time()
        self.value = np.sin(xx+self.count)*np.cos(yy)
        end = time.time()
        
        self.count += 1
        self.duration = end-start

camera = Camera()

@pn.depends(image=camera.param.value)
def img0(image):
    return hv.Image(image,bounds=bounds)
layout = pn.Column(pn.Param(camera, parameters=["xresolution", "yresolution", "take", "duration"]), hv.DynamicMap(img0), ).servable()

2022-09-22 15:51:15,374 Failed sending message as connection was closed
2022-09-22 15:51:15,374 Task exception was never retrieved
future: <Task finished name='Task-2604' coro=<WebSocketProtocol13.write_message.<locals>.wrapper() done, defined at /opt/conda/lib/python3.9/site-packages/tornado/websocket.py:1100> exception=WebSocketClosedError()>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/tornado/websocket.py", line 1102, in wrapper
    await fut
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/tornado/websocket.py", line 1104, in wrapper
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
2022-09-22 15:51:15,375 WebSocket connection closed: code=None, reason=None
4

Hi @Wiggan. We should have a new HoloViews out very soon which speeds this up considerably.

3 Likes
5

Hi, and thank you so much for fast response and even making a reprex! Wow!
I tried setting watch=False, but don’t see an impact on time spent in the assignment. It does however still work, so I will keep it that way.

1 Like
6

Great! I’ll keep an eye on the PR and upcoming release.
Meanwhile, is the method for live updating images above reasonable, or would you have solved the problem with some other method? I was thinking about WebRTC, but that would probably not be as well integrated with zoom tools and such. That would take me much more time to implement as well.

7

It works much better in recent version of holoviews! Thank you!

2 Likes
8

Happy that it worked @Wiggan

If you can please share some screenshots of your tool. I’m really curious to see how it works.

1 Like
9

image
image1444×640 26.1 KB

Sorry for late reply, Marc!
Here’s the camera card in all of its glory. :slight_smile:

I do still have problems with the live feed though. Sorry about digging up this old thread.

I think the problem is that my asyncio.task that acquires frames and updates a Param which causes a DynamicMap to update the image on the dashboard takes so much time that it blocks the eventloop. Sometimes the rest of the dashboard becomes unresponsive when livefeed is on. I don’t think the @depends can run in a nonblocking fashion?

I’m thinking of alternatives… Not sure if threading is of any help, as the bottleneck likely is CPU bound and not IO. Most of the async/threading mentioned in panel/param documentation is for when the server has to do something blocking, but here it is transferring the images from server to client that blocks. I haven’t found anything about that yet. Can Streamz be of any use? Probably not?

I realize I’m just rambling on here, any ideas are very much welcome!

1 Like
10

Hi @Wiggan

Thanks for reporting back. If you can provide a minimum, reproducible code example that replicates the problem and cause but can be run without a camera, then it would be much easier to try to help.

Could you somehow introduce an “artificial” camera that produces the images and then the rest replicates your code?

11

Yes, I’ll try to do that during the day.
Would be good to isolate the problem!

12

Hi, reviving this zombie-thread 40 years later.
I have so far not been able to reproduce the problem without using the camera, which is weird.

Here is a little too much code:

import asyncio
import PySpin
import numpy as np
import holoviews as hv
import panel as pn
import param

hv.extension('bokeh')


class Camera(param.Parameterized):
    camera = None
    system = None
    cam_list = None
    latest = param.Parameter(default=(np.ones(shape=(50, 50), dtype=np.uint16) * 2000))
    task: None = None
    livefeed = param.Boolean()

    async def get_frames(self):
        while self.livefeed:
            frame = await asyncio.to_thread(self.camera.GetNextImage, 2000)
            self.latest = frame.GetNDArray()  # This makes the program leak memory like crazy.
            # self.latest = (np.random.random((3200, 2200)) * 16000).astype(np.uint16) # If i use this line instead of previous, no leak.
            frame.Release()

    @pn.depends("livefeed", watch=True)
    async def main(self):
        print("In main")
        self.system = PySpin.System.GetInstance()
        self.cam_list = self.system.GetCameras()
        self.camera = self.cam_list.GetByIndex(0)
        self.camera.Init()
        self.camera.AcquisitionMode.SetValue(PySpin.AcquisitionMode_Continuous)
        self.camera.TLStream.StreamBufferHandlingMode.SetValue(PySpin.StreamBufferHandlingMode_NewestOnly)  # Make sure no old frames are sent
        self.camera.BeginAcquisition()

        self.task = asyncio.create_task(self.get_frames())
        print("Waiting for task...")
        await self.task

        if self.camera:
            if self.camera.IsStreaming():
                self.camera.EndAcquisition()
            self.camera.DeInit()
            self.camera = None
        self.cam_list.Clear()
        self.system.ReleaseInstance()


cam = Camera()


@pn.depends(image=cam.param.latest, watch=False)
def img0(image):
    plot = hv.Image(image)
    return plot


pn.serve(
    pn.Row(hv.DynamicMap(img0, cache_size=1), cam.param.livefeed)
)

It seems like the DynamicMap keeps a reference to the ndarray if it comes from the camera, but not if it is a random ndarray. If I create an hv.Image of the camera image but do not return it, there is no leak, but as soon as img0 returns it, there is like 13.5MB memory leak per frame.

I tried with an all black ndarray of same size too, but no leak.
If I do GetNDArray() * 0 it also does not leak.

The images that the camera produce at the moment are almost completely black with just some hot pixels. It has a lid on and is sitting on my desk at the moment.

I’d be very grateful if someone could shed some light here… :slight_smile:

Python 3.10.6
Panel: 0.14.2
Bokeh: 2.4.3
Param: 1.12.3
Holoviews: 1.15.4

1 Like