Behaviour of pointer for cross section of (DynamicMap vs RGB)

HoloViews
, ,
1

I use a camera, to capture a live video (this part is working), and I want to extract cross sections of some images of the video using for example PointerXY. This is working, but only when I use a dummy image ( i.e hv.RGB.load_image()). When I use the real camera feed (i.e. hv.DynamicMap(hv.RGB, streams=[pipe])`) i get problems

The commented code demonstrates some of my problems, please see the four alternative output lines at the bottom:

In my actual code, the camera video stream consists of frame, with type(frame) = numpy.ndarray and np.shape(frame) = (1024, 1280, 3) for example. In this MWE, I replaced that frame with img.data of a random jpg image. (giving the same type and shape once loaded). This works. If I feed that same frame to the pipe, it stops working in strange ways.

import numpy as np
import holoviews as hv
hv.extension('bokeh')
from holoviews.streams import Pipe


img_bounds = (0, 0, 1088, 821)

img = hv.RGB.load_image('./test.jpg', bounds=img_bounds).opts(width=img_bounds[2], height=img_bounds[3], xaxis=None, yaxis=None)
#np.shape(img.data)#(821, 1088, 3)

pipe = Pipe(data = [])
#pipe.send(img)#Send the complete image
#pipe.send(img.data)# Send just the data array
camera = hv.DynamicMap(hv.RGB, streams=[pipe]).opts(width=int(img_bounds[2]), height=int(img_bounds[3]),xaxis=None, yaxis=None)



data_red = img.data[:,:,0]
data_green = img.data[:,:,1]
data_blue = img.data[:,:,2]

img_red = hv.Image(data_red, bounds=img_bounds)
img_green = hv.Image(data_green, bounds=img_bounds)
img_blue = hv.Image(data_blue, bounds=img_bounds)

pointer = hv.streams.PointerXY(x=int(img_bounds[2]/2), y=int(img_bounds[3]/2), source=img)
#pointer = hv.streams.SingleTap(x=int(img_bounds[2]/2), y=int(img_bounds[3]/2), source=img)

def cross_hair_info(x, y):
    return hv.HLine(y) * hv.VLine(x)

crosshair = hv.DynamicMap(cross_hair_info, streams=[pointer])

crosssection_x_red = img_red.apply.sample(x=pointer.param.x).opts(width=100, yaxis=None, xaxis='bottom', color='red')
crosssection_y_red = img_red.apply.sample(y=pointer.param.y).opts(height=100, xaxis=None, yaxis='left', color='red')
crosssection_x_green = img_green.apply.sample(x=pointer.param.x).opts(color='green')
crosssection_y_green = img_green.apply.sample(y=pointer.param.y).opts(color='green')
crosssection_x_blue = img_blue.apply.sample(x=pointer.param.x).opts(color='blue')
crosssection_y_blue = img_blue.apply.sample(y=pointer.param.y).opts(color='blue')

# The (static) image is shown, and the pointer works as desired
((img * crosshair) << crosssection_x_red * crosssection_x_green * crosssection_x_blue << crosssection_y_red * crosssection_y_green * crosssection_y_blue)

# If I send.pipe(img), this shows the image, and correctly calculates the initial cross sections, but the pointer cannot be moved
#pipe.send(img)#Send the complete image
#((camera * crosshair) << crosssection_x_red * crosssection_x_green * crosssection_x_blue << crosssection_y_red * crosssection_y_green * crosssection_y_blue)

# In the other case, the pointer is also stuck at the initial position, but now the main canvas stays blank
#pipe.send(img.data)#Send just the data array
#((camera * crosshair) << crosssection_x_red * crosssection_x_green * crosssection_x_blue << crosssection_y_red * crosssection_y_green * crosssection_y_blue)

# This on the other hand always works, and shows the image correctly independently of how I call send.pipe()
#camera

I would very much like to be using pipe.send(img.data), since that is what my camera feed is using right now.
What is going wrong here? Am i somehow initializing the pipe or the DynamicMap wrong?

4

Hi @Feargus,

I’ve tried a couple of things, sorry made a hash of your notes here but this is my take on it. Firstly I changed the source for the pointer to the camera rather than image. I also added the bokeh hook for cross hair to remove the lag while I was trying it out.

I think the freezing was due to still looking at the img rather than camera as source. So I think that solves or is one step closer to resolution. However I think you may have encountered a bug for the img.data. As you noted if you just display the camera and crosshair you get the image, great. Once you add in the side graphs they function as you would expect but the image goes blank, this to me feels like a bug because it’s still functioning behind the scenes in that the side graphs update as you hover over where the image should be. Also I tried linking the cross hairs over the side graphs, whilst this worked zooming in on the image became problomatic I don’t think it functions well with the << operators was my feeling but as it wasn’t originally there I just removed. I like what you’ve put together here, I hope some of this helps.

import numpy as np
import holoviews as hv
hv.extension('bokeh')
from holoviews.streams import Pipe


img_bounds = (0, 0, 1088, 821)

img = hv.RGB.load_image('test.jpg', bounds=img_bounds).opts(width=img_bounds[2], height=img_bounds[3], xaxis=None, yaxis=None)
#np.shape(img.data)#(821, 1088, 3)

pipe = Pipe(data = [])
#pipe.send(img)#Send the complete image
#pipe.send(img.data)# Send just the data array
camera = hv.DynamicMap(hv.RGB, streams=[pipe]).opts(width=int(img_bounds[2]), height=int(img_bounds[3]),xaxis=None, yaxis=None)



data_red = img.data[:,:,0]
data_green = img.data[:,:,1]
data_blue = img.data[:,:,2]

img_red = hv.Image(data_red, bounds=img_bounds)
img_green = hv.Image(data_green, bounds=img_bounds)
img_blue = hv.Image(data_blue, bounds=img_bounds)

pointer = hv.streams.PointerXY(x=int(img_bounds[2]/2), y=int(img_bounds[3]/2), source=camera) #Changed the source to get one of the camera options functioning
#pointer = hv.streams.SingleTap(x=int(img_bounds[2]/2), y=int(img_bounds[3]/2), source=img)

def cross_hair_info(x, y):
    return hv.HLine(y) * hv.VLine(x)

crosshair = hv.DynamicMap(cross_hair_info, streams=[pointer])

crosssection_x_red = img_red.apply.sample(x=pointer.param.x).opts(width=100, yaxis=None, xaxis='bottom', color='red')
crosssection_y_red = img_red.apply.sample(y=pointer.param.y).opts(height=100, xaxis=None, yaxis='left', color='red')
crosssection_x_green = img_green.apply.sample(x=pointer.param.x).opts(color='green')
crosssection_y_green = img_green.apply.sample(y=pointer.param.y).opts(color='green')
crosssection_x_blue = img_blue.apply.sample(x=pointer.param.x).opts(color='blue')
crosssection_y_blue = img_blue.apply.sample(y=pointer.param.y).opts(color='blue')

# make crosshair object - less lagy than dynamic map
linked_crosshair = CrosshairTool(dimensions="both")

# make a hook to manipulate bokeh figure properties
def hook(plot, element):
    plot.state.add_tools(linked_crosshair)
    

# If I send.pipe(img), this shows the image, and correctly calculates the initial cross sections, but the pointer cannot be moved
pipe.send(img)#Send the complete image
((camera.opts(hooks=[hook])) << crosssection_x_red * crosssection_x_green * crosssection_x_blue << crosssection_y_red * crosssection_y_green * crosssection_y_blue)

# In the other case, the pointer is also stuck at the initial position, but now the main canvas stays blank
#pipe.send(img.data)#Send just the data array
#((camera * crosshair)) # NOTE seems fine, picture shows, cross hair functions
#NOTE picture vanishes but everything else seems to function possibly bug encountered here is what it feels like

#((camera * crosshair)) << crosssection_x_red )#* crosssection_x_green * crosssection_x_blue << crosssection_y_red * crosssection_y_green * crosssection_y_blue)
# This on the other hand always works, and shows the image correctly independently of how I call send.pipe()
#camera
5

Thank you very much @carl . I had a feeling that there are several things going on at once. The source=img instead of camera was definitely an oversight on my part. Thank you also for the bokeh native crosshair. I saw the other thread, and that would have been my next question :slight_smile: .

I spent the day trying to get this to work with my camera feed, and while it is still not working yet.

8

I played around some more, and it seems the bounds parameter is to blame. If I remove them from the pointer and from img_red/green/blue, then it starts working. If I remove them just from some, then I may get the opposite case, where the crosssections are blank, and the center camera image is showing.

However, now I realized that the crosssection plots are not updating for new items down the pipe.
And I cannot seem to address the subarray within the data as I did before. ( pipe.data[:,:,0] “are not stream instances”). I cannot pass them directly to a DynamicMap as I do when using streams=[pipe] ).