Access box_select attributes set in figure

[Wall.E]

I’m trying to get the image (x, y) coordinates hovered by hand-drawn rectangle over an image displayed with figure.image(). The app runs in Jupyter Lab, with synchronization.
To do that I’m trying to use the box_select tool set in the tools input of a figure object.

    TOOLTIPS = [
        ("x", "$x"),
        ("y", "$y"),
        ("value", "@image"),
    ]
    tools = ['box_select', 'hover', 'reset']
    # Create the figure axis
    fig = figure(
               tooltips=TOOLTIPS,
               tools=tools)

At first, I tried using data_source.selected.on_change('indices', callback) but that does not seem to trigger anything with an image. It only works when selecting glyphs in e.g. scatter plots.
The topic closest to this question does not seem to apply anymore, as fig.tool_events issues the error that tool_events is an unexpected attribute.

Any idea how I can get the image coordinates of the selection rectangle?

[gmerritt123]

Relevant: Correct way to use Selection (Lasso, Box) on an Image - #7 by Bryan

Additional question regarding your use case: 1) Do you just want the xy coords of an arbitrarily drawn box, or do you need the image “pixel” values contained in that selection?

Former would be pretty trivial with the draw tools, latter more complicated as you’d need to slice and dice a 2d array (as that’s how the image pixels are stored).

[Wall.E]

The former. xy coords of the drawn rectangle in the plotted image axes coordinates. Can be a top left vertex and width,height or two opposite vertices. I plot the image this way:

source = ColumnDataSource({'image': [samples[0]]})
fig = figure(x_axis_label='X [px]',
               y_axis_label='Y [px]',
               tooltips=TOOLTIPS,
               tools=tools)
im = fig.image(source=source, x=0, y=0, dw=600, dh=659)

where “samples” is just a list of 2D numpy arrays and my image arrays have dimensions of dw and dh pixels (here, 659 x 600 px).

At your linked topic, I see a SelectionGeometry mentionned but I don’t see how this can be used in combination with the BoxSelectTool.

As a simple workaround, I have used the Tap event which directly gives the clicked point in the x,y axes coordinates. Doing it twice can give me a rectangle but that’s not great. Then I round them up to integers and I can use them to work with my image array.

    def callback_tap(event):
        coords=(event.x,event.y)
        print(coords)
    fig.on_event(Tap, callback_tap)

[gmerritt123]

Right, I was linking to that just for reading up on the complexity of selecting image pixels (it can be done, but it’s more complicated than people seem to think).

Check out the BoxEditTool, this is what you’re after: Configuring plot tools — Bokeh 2.4.2 Documentation

Basically you create an empty CDS, drive a rect glyph with it (so you can control the alpha/formatting of it etc), and assign the box edit tool to be able to add to it. You can also specify the max number, so for you just limit it to one (so user draws one box at a time). Then with a callback retrieve the values that the user just added to the CDS and bam you have the coords of the box they drew.

[Wall.E]

Ok, I tried the BoxEditTool but the documentation would really need an example.

I tried this minimalistic example, but the rectangle won’t show dragging the mouse with the tool.

from bokeh.plotting import figure
from bokeh.layouts import column
from bokeh.models import ColumnDataSource, BoxEditTool
from bokeh.io import show
output_notebook()

def bkapp(doc):

    source0 = ColumnDataSource(data=dict(
    x0=[1, 2, 3, 4, 5],
    y0=[2, 5, 8, 2, 7],
    ))

    tools = ['hover', 'reset']

    plot = figure(x_axis_label='x',
                  y_axis_label='y)',
                  title="Test BoxEditTool",
                  tools=tools)
    
    source1 = ColumnDataSource({'x':[], 'y':[], 'width':[], 'height':[]})
    source2 = ColumnDataSource({'x':[], 'y':[], 'width':[], 'height':[]})

    r1 = plot.rect('x', 'y', 'width', 'height', source=source1)
    r2 = plot.rect('x', 'y', 'width', 'height', source=source2)
    tool = BoxEditTool(renderers=[r1, r2], num_objects=2)
    plot.add_tools(tool)
    
    # Add the data renderer (here a scatter plot)
    scatter = plot.scatter(x='x0', y='y0', source=source0)

    doc.add_root(column(plot))
    
show(bkapp)

[gmerritt123]

hold shift my friend

[gmerritt123]

I altered your example a tiny bit just to make it a quick and dirty standalone, but here’s what I mean by the callback part:

from bokeh.plotting import figure
from bokeh.layouts import column
from bokeh.models import ColumnDataSource, BoxEditTool, CustomJS
from bokeh.plotting import show

def bkapp():

    source0 = ColumnDataSource(data=dict(
    x0=[1, 2, 3, 4, 5],
    y0=[2, 5, 8, 2, 7],
    ))

    tools = ['hover', 'reset']

    plot = figure(x_axis_label='x',
                  y_axis_label='y)',
                  title="Test BoxEditTool",
                  tools=tools)
    
    source = ColumnDataSource({'x':[], 'y':[], 'width':[], 'height':[]})

    r1 = plot.rect('x', 'y', 'width', 'height', source=source)
    tool = BoxEditTool(renderers=[r1], num_objects=1)
    plot.add_tools(tool)
    
    cb = CustomJS(args=dict(source=source)
                  ,code='''
                  console.log(source.data)
                  const xl = source.data['x'][0]-source.data['width'][0]/2
                  const xr = source.data['x'][0]+source.data['width'][0]/2
                  const yl = source.data['y'][0]-source.data['height'][0]/2
                  const yu = source.data['y'][0]+source.data['height'][0]/2
                  console.log('xleft = '+xl.toString())
                  console.log('xright = '+xr.toString())
                  console.log('yupper = '+yu.toString())
                  console.log('ylower = '+yl.toString())
                  ''')
    source.js_on_change('data',cb)
    
    # Add the data renderer (here a scatter plot)
    scatter = plot.scatter(x='x0', y='y0', source=source0)

    return plot

p = bkapp()
show(p)

[Wall.E]

@gmerritt123

hold shift my friend

I did, doesn’t work in my Jupyter Lab. I saw it working in a browser when exporting an html file with output_file() but that is not our workflow… and that’s frustrating not to have it work in Jupyter Lab.

Regarding your way with customJS, we are JS-averse, but i’ll take a look.

[gmerritt123]

I did, doesn’t work in my Jupyter Lab.

That is a different problem then, way out of my domain.

Regarding your way with customJS, we are JS-averse, but i’ll take a look.

There 100% is a python callback style way to do this too, just retrieve source.data in the python callback function and do what you want with it then.

[Wall.E]

Ok, we’re nearly there then. I just need to fix that Jupyter Lab not showing the rectangles. I’ll create a new topic for this specific problem. Thanks.

[Wall.E]

We may have spoken too fast. For a python callback to be triggered with .on_change(), we still need an attribute to be changed. However, BoxEditTool seems to only have scatter glyph attritubes for handling that, due to the inheritence from EditTool:

renderers An explicit list of renderers corresponding to scatter glyphs that may be edited.

The customJS callback does show, but that JS side (browser side). Getting things synchronized with my underlying Python environment running the server remains unaddressed by BoxEditTool in combination with the image()

[gmerritt123]

I think you’ve read documentation that has been copy pasted from the point edit tool. Literally replace the word scatter in that sentence with rect… it handles rect glyphs (and probably quads but I dunno I’d have to test it) just fine. If it didn’t then my CustomJS approach wouldn’t work either. Anyway, here’s the python server side equivalent, adapted from your example. I replaced the console.logs with a div on the side that basically shows you how to get at the box dims in the same manner as my CustomJS effort above.

from bokeh.plotting import figure, curdoc
from bokeh.layouts import row
from bokeh.models import ColumnDataSource, BoxEditTool, Div
from bokeh.plotting import show



source0 = ColumnDataSource(data=dict(
x0=[1, 2, 3, 4, 5],
y0=[2, 5, 8, 2, 7],
))

tools = ['hover', 'reset']

plot = figure(x_axis_label='x',
              y_axis_label='y)',
              title="Test BoxEditTool",
              tools=tools)

source = ColumnDataSource({'x':[], 'y':[], 'width':[], 'height':[]})

r1 = plot.rect('x', 'y', 'width', 'height', source=source)
tool = BoxEditTool(renderers=[r1], num_objects=1)
plot.add_tools(tool)

div = Div(text='Box Info:')

def getBoxDims(attrname,old,new):
    xl = source.data['x'][0]-source.data['width'][0]/2
    xr = source.data['x'][0]+source.data['width'][0]/2
    yl = source.data['y'][0]-source.data['height'][0]/2
    yu = source.data['y'][0]+source.data['height'][0]/2
    div.text='Box Info: Xdims: '+str(xl)+','+str(xr)+', Ydims: '+str(yl)+','+str(yu) 
    
source.on_change('data',getBoxDims)    

# Add the data renderer (here a scatter plot)
scatter = plot.scatter(x='x0', y='y0', source=source0)

curdoc().add_root(row([plot,div]))

… I also don’t really see how the image glyph has anything to do with the implementation of simply drawing a box on a figure and getting the dimensions of said box, other than of course (for your specific use case) the box is gonna be drawn over top of an image.

[Bryan]

@gmerritt123

I think you’ve read documentation that has been copy pasted from the point edit tool.

Looks like. Bokeh’s docs comprise about 35000+ lines of text and code and have mostly been written by 2-3 people. For sure there’s copy-pasta here and there. Please create a GitHub Issue so it can be fixed before the next release.

[Wall.E]

@gmerritt123

Anyway, here’s the python server side equivalent, adapted from your example.

This is working pretty great if I dismiss my need for Jupyter Lab. I can live with that for now. That is very helpful, thanks a lot.
@Bryan suggested also to try the SelectionGeometry event with box_select. That would address the original question and is worth a look, as BoxSelectTool has no issue with Jupyter Lab.