Unexpected behavior: glyphs and renderer becoming dis-synchronized when updating glyphs in a button click

[xhongyi]

Hi, I will try to be succinct.

What I am trying to accomplish:
Change the color scheme with customized color maps after a button click.

My method:
According to the user guide, I try to style my circles by replacing the glyphs with clones.
Styling plot elements — Bokeh 3.8.0 Documentation

My setting:
I’m running with bokeh server.

Unexpected behavior:
If I update the glyphs outside of a button click callback, then the glyphs get updated fine.
However, if I update the glyphs within a button click callback, then the color changes will break.

My code:
Here is a minimum code snippet to reproduce my case:

from bokeh.models import ColumnDataSource, Div, Button
from bokeh.layouts import row as bokeh_row, column as bokeh_column
from bokeh.plotting import figure
from bokeh.transform import CategoricalColorMapper
from bokeh.io import curdoc


import pandas as pd
import numpy as np

N = 100  # or any number of rows you want
df = pd.DataFrame({
    'x': np.random.uniform(0, 20, N),
    'y': np.random.uniform(0, 20, N),
    'color1': np.random.choice(['a', 'b'], N),
    'color2': np.random.choice(['a', 'b'], N)
})

source = ColumnDataSource(df)

print(source)

cmap1 = CategoricalColorMapper(palette=[[32,14,23], [121,144,77]], factors=['a', 'b'])
cmap2 = CategoricalColorMapper(palette=["red", "blue"], factors=['a', 'b'])

TOOLS="hover,crosshair,pan,wheel_zoom,zoom_in,zoom_out,box_zoom,undo,redo,reset,tap,save,box_select,poly_select,lasso_select,examine,fullscreen,help"

p = figure(tools=TOOLS)

color_mapper1 = {"field": "color1", "transform": cmap1}
color_mapper2 = {"field": "color2", "transform": cmap2}

renderer = p.circle(
        source=source,
        x="x", y="y",
        radius=0.25,
        fill_color="yellow"
        #fill_color=color_mapper1
        )

def color_callback1(): 
    print("current field:", renderer.glyph.fill_color["field"])

    renderer.glyph = renderer.glyph.clone(fill_alpha=1, fill_color=color_mapper1, line_color=None)

    renderer.selection_glyph = renderer.glyph.clone(fill_alpha=1, fill_color=color_mapper1, line_color=None)
    
    renderer.nonselection_glyph = renderer.glyph.clone(fill_alpha=0.2, fill_color=color_mapper1, line_color=None)


def color_callback2(): 
    renderer.glyph.fill_color = color_mapper1
    
def color_callback3(): 
    renderer.glyph.fill_color = color_mapper2

def print_field():
    #renderer.glyph.fill_color = color_mapper1
    print(renderer.glyph.fill_color["field"])
    div.text = renderer.glyph.fill_color["field"]


color_button1 = Button(label="update color1", button_type="primary")
color_button1.on_click(color_callback1)

print_button = Button(label="print color", button_type="primary")
print_button.on_click(print_field)

color_button2 = Button(label="color to 1", button_type="primary")
color_button2.on_click(color_callback2)

color_button3 = Button(label="color to 2", button_type="primary")
color_button3.on_click(color_callback3)

div = Div(text="Hello")

curdoc().add_root(bokeh_column([p, bokeh_row(color_button1, print_button, color_button2, color_button3), div]))

renderer.glyph = renderer.glyph.clone(fill_alpha=1, fill_color=color_mapper2, line_color=None)

print(renderer.glyph.fill_color["field"]

Behavior:
a. initially, the glyphs are colored just fine by replacing the glyphs with clones outside of button click call back.

b. color changes are responsive if I just replace the color map (click “color to 1” and “color to 2” back and forth).

c. the color update is broken if I try to update the glyphs through the cloning method (not replacing the color map but replace the glyphs with clones) in a button callback (click “update color1”). Even though I do see that the color map is updated (click “print color”). Reassigning color_map would break too.

Root cause:
The glyphs are indeed replaced with new glyphs in the renderer. However this is not reflected in the plot. The plot is stuck with the old glyph.

This behavior is quite weird to be honest. I would expect that if a glyph styling through replacing with clones to behave consistently regardless of whether the update is called within a button callback or not.

Is there any way to work around this? Basically, I want to update my color map through bespoke color mapper.

[xhongyi]

I also want to report a related issue.

The reason that I had to resort to replacing the glyphs with the cloning method, rather than just reassigning a new color map, is because of another unexpected behavior:

If I simply replace the color map, and I use lasso select, then the color map of the selected item will revert back to the old color:
A, the color maps are reassigned.

B, but in lasso select, the color scheme remains to be the old one:

It seems that the core reason for this inconsistency is that the selected color is maintained by the renderer in “selection_fill_color”.
However, I cannot find a way to manipulate the selection_fill_color separately other than replacing the glyphs through cloning as recommended in user guide.
Updating the selection_fill_color to a new color map will result in error stating that selection_fill_color is a string and it cannot be assigned with a dictionary.

If there is a workaround where I can separately manipulate selection_fill_color such that the selected color and the data color is consistent, please let me know.

[xhongyi]

Lastly, while we are on this issue. There is another unexpected behavior:

If I swap the color map of the glyph outside of a button click callback, such as:

renderer.glyph.fill_color = color_mapper1

Then it seems that both the static color map, as well as the selection_color_map will be updated.

However, if I update the color map inside a button click callback, then only the static color map is updated but not the selection_color_map.

def color_callback2():
    renderer.glyph.fill_color = color_mapper1

I can understand that this probably has something to do with how button callback is handled differently than changing the glyphs in general. But this inconsistent behavior is very confusing.

So the bottom line is: is there any way to update selection_color_map in a callback function or this behavior is just not supported completely.

[xhongyi]

While the confusion persists, I think I have found a workaround of this problem.
I think the core issue is that the fill_color behavior is actually quite complicated.
In my code, when I initialize all the circle glyphs, I did not explicitly provide a dictionary color map to the selection_fill_color and nonselection_fill_color.

renderer = p.circle(
        source=source,
        x="x", y="y",
        radius=0.25,
        fill_color="yellow"
        #fill_color=color_mapper1
        )

Internally, this will make bokeh creating a default color map configuration which is a string “auto”.
Therefore, later on any effort of reassigning selection_fill_color with a dictionary will run into a mis-matched data type error.
Actually, a simple fix would be initializing the circle with both selection_fill_color and non_selection_fill_color to anything but None. This will force bokeh to create an instance for the color property rather than a NullObject.

renderer = p.circle(
        source=source,
        x="x", y="y",
        radius=0.25,
        fill_color="yellow",
        selection_fill_color="yellow",
        nonselection_fill_color="yellow"
        )

Then later on I can update the color map via:

renderer.selection_glyph.fill_color=my_color_map

I know this is quite nuanced and I only understood it when looking into the GlyphRenderer initialization. I think manipulating color is quite a common practice and it would be great if in the future there could be a section in the tutorial explaining the nuance of it.

This does not resolve my confusion about the disconnection between the renderer and the glyphs. But for now it serves my design requirement and I can move on.

[Bryan]

@xhongyi swapping in and out lots of actual objects in the Document is usually not advised. There is lot a internal signal handling plumbing between objects, and it is always reliable to swap them out entirely. We always advise the best practice is to update the smallest thing possible, i.e. to update properties of existing objects, rather than adding / deleting entire objects.

In this case, I am just not entirely clear on your situation. It looks like you actually have two different sets of data? In that case I would make two separate glyphs up front, each configured exactly how you want to look for each data. Then just have callbacks toggle the visibility, as needed. This is as simple as setting renderer.visible = True etc.

Here is a complete example along those lines. If it doesn’t exactly match your situation hopefully at least it might provide some ideas.

from bokeh.models import ColumnDataSource, Button
from bokeh.layouts import column
from bokeh.plotting import figure
from bokeh.transform import factor_cmap
from bokeh.io import curdoc

import pandas as pd
import numpy as np

N = 100

df = pd.DataFrame({
    'x': np.random.uniform(0, 20, N),
    'y': np.random.uniform(0, 20, N),
    'data1': np.random.choice(['a', 'b'], N),
    'data2': np.random.choice(['a', 'b'], N)
})
source = ColumnDataSource(df)

palette1 = ["green", "orange"]
palette2 = ["red", "blue"]

p = figure(tools="lasso_select")

r1 = p.circle(
    source=source,
    x="x", y="y",
    radius=0.25,
    fill_color=factor_cmap('data1', palette1, ["a", "b"]),
    nonselection_fill_color="yellow",
)

r2 = p.circle(
    source=source,
    x="x", y="y",
    radius=0.25,
    fill_color=factor_cmap('data2', palette2, ["a", "b"]),
    nonselection_fill_color="yellow",
)
r2.visible = False

def callback():
    r1.visible = not r1.visible
    r2.visible = not r2.visible

button = Button(label="Toggle data")
button.on_click(callback)

curdoc().add_root(column(p, button))

[xhongyi]

Hi Bryan,

I am trying to make a plotting tool UI. In this case I want to create an interactive interface where user can dynamically decide using which color to paint each type of categories in the data.
In addition, I want to let users to dynamically update the categories of the data. That is why I ended up with dynamically swapping the color map.

There is always a single set of data.

[Bryan]

I want to let users to dynamically update the categories of the data. That is why I ended up with dynamically swapping the color map.

I think we are having a terminology misunderstanding, since I find the above statement to contradict this one:

There is always a single set of data.

Another option, if there really is only one set of data, is to only create one color mapper object up front, and then update it’s palette property. That is a much “smaller” change that avoids changing the Document object graph.

Any any case, my advice still stands: look for ways to achieve the UX you want that don’t involve adding/deleting actual objects. There is almost always a way to achieve anything you’d want to do by creating the right set up objects all up front, then merely updating their various properties in some way. Hopefully my example above can at least inspire some possible ideas along those lines.

[xhongyi]

Yeah. I think the problem I ran into was that I have to update all three color maps:

renderer.glyph.fill_color
renderer.selection_glyph.fill_color
renderer.nonselection_glyph.fill_color

Also in order to be able to update all these dynamically, all of them have to be initialized to be anything but None when I create the glyphs. I didn’t know that they have to be not None at the time of creating the glyphs.
Here is my current solution, for anyone who is interested in the future:

from bokeh.models import ColumnDataSource, Div, Button
from bokeh.layouts import row as bokeh_row, column as bokeh_column
from bokeh.plotting import figure
from bokeh.transform import CategoricalColorMapper
from bokeh.io import curdoc


import pandas as pd
import numpy as np

N = 100  # or any number of rows you want
df = pd.DataFrame({
    'x': np.random.uniform(0, 20, N),
    'y': np.random.uniform(0, 20, N),
    'color1': np.random.choice(['a', 'b'], N),
    'color2': np.random.choice(['a', 'b'], N)
})

source = ColumnDataSource(df)

print(source)

cmap1 = CategoricalColorMapper(palette=[[32,14,23], [121,144,77]], factors=['a', 'b'])
cmap2 = CategoricalColorMapper(palette=["red", "teal"], factors=['a', 'b'])

TOOLS="hover,crosshair,pan,wheel_zoom,zoom_in,zoom_out,box_zoom,undo,redo,reset,tap,save,box_select,poly_select,lasso_select,examine,fullscreen,help"

p = figure(tools=TOOLS)

color_mapper1 = {"field": "color1", "transform": cmap1}
color_mapper2 = {"field": "color2", "transform": cmap2}

renderer = p.circle(
        source=source,
        x="x", y="y",
        radius=0.25,
        fill_color=color_mapper1,
        selection_fill_color="yellow",
        nonselection_fill_color="yellow",
        )

def color_callback1():
    renderer.glyph.fill_color = color_mapper1
    renderer.selection_glyph.fill_color = color_mapper1
    renderer.nonselection_glyph.fill_color = color_mapper1
    
def color_callback2(): 
    renderer.glyph.fill_color = color_mapper2
    renderer.selection_glyph.fill_color = color_mapper2
    renderer.nonselection_glyph.fill_color = color_mapper2

def print_field():
    #renderer.glyph.fill_color = color_mapper1
    print(renderer.glyph.fill_color["field"])
    div.text = renderer.glyph.fill_color["field"]


print_button = Button(label="print color", button_type="primary")
print_button.on_click(print_field)

color_button1 = Button(label="color to 1", button_type="primary")
color_button1.on_click(color_callback1)

color_button2 = Button(label="color to 2", button_type="primary")
color_button2.on_click(color_callback2)

div = Div(text="Hello")

curdoc().add_root(bokeh_column([p, bokeh_row(color_button1,  color_button2, print_button), div]))

[Bryan]

My question above about data sets is because color_mapper1 is configured to refer to the color1 column (one set of data) but color_mapper2 is configured to refer to the color2 column (another set of data).

If that’s really representative of your situation, then I re-iterate my suggestion to create a dedicated glyph renderer for each case (separate calls to p.circle) and then handle the UX by toggling visibilities as appropriate. I’ve seen that approach used successfully many times.

[Bryan]

Specifically, note that you can do this, e.g.

cmap1 = factor_cmap('data1', palette1, ["a", "b"])
r1 = p.circle(
    source=source,
    x="x", y="y",
    radius=0.25,
    fill_color=cmap1,
    nonselection_fill_color=cmap1,
)

cmap2 = factor_cmap('data2', palette2, ["a", "b"])
r2 = p.circle(
    source=source,
    x="x", y="y",
    radius=0.25,
    fill_color=cmap2,
    nonselection_fill_color=cmap2,
)
r2.visible = False

In my example above I thought you wanted the yellow non-selection color, so that’s the only reason I put it. But this code does what your version does without adding/removing models.

[xhongyi]

Thanks Bryan. Appreciate it. I do have a concern: would invisible glyph still be stored in the browser? Or are they cleared and retransmitted from the backend when it becomes visible again?
I was a bit worried that they would still be stored in the browser and after a while it would flood the memory.

[Bryan]

@xhongyi

would invisible glyph still be stored in the browser?

Yes, but glyphs are very lightweight compared to data sources. You’ve indicated there is only one data source, so the glyphs can all share that same data source. I’m assuming your situation might require ~10s of glyphs… I suppose if you would actually need thousands of glyphs, then that might add unacceptable overhead.

[xhongyi]

Haha I actually have around tens of thousands of glyphs… When I finish, I will send you my github link!

[xhongyi]

Relying on bokeh for some really heavy-duty visualization tasks.