Bug with DeserializationError when changing content layout in Bokeh 3.1.1

Community Support
1

This is a copy of my post on SO: python - Bokeh 3.1.1 DeserializationError when changing layout content - Stack Overflow

I have a project that worked fine under Bokeh 2.4.3 but after me upgrading to 3.1.1 produces bokeh.core.serialization.DeserializationErrors.

The culprit is me having frames which first display a “loading screen” and afterwards an actual plot.
I have a class that manages a frame and looks somewhat like this, simplified:

class MyFrame:

    def __init__(self):
        self.layout = column()
        self.loading = loading_pic()
        self.layout.children.append(self.loading)

    def display_loading(self) -> None:
        self.layout.children[0] = self.loading

    def plot(self, data) -> None:
        self.my_plot = MyPlot(data)
        self.layout.children[0] = self.my_plot

In this, the function loading_pic() simply generates a (Bokeh) plot that contains the label “Loading”, while the MyPlot class does some more sophisticated rendering.

This produces the error

2023-06-19 13:29:44,116 error handling message
 message: Message 'PATCH-DOC' content: {'events': [{'kind': 'ModelChanged', 'model': {'id': 'p1004'}, 'attr': 'inner_width', 'new': 865}, {'kind': 'ModelChanged', 'model': {'id': 'p1004'}, 'attr': 'inner_height', 'new': 540}, {'kind': 'ModelChanged', 'model': {'id': 'p1004'}, 'attr': 'outer_width', 'new': 900}, {'kind': 'ModelChanged', 'model': {'id': 'p1004'}, 'attr': 'outer_height', 'new': 550}]}
 error: DeserializationError("can't resolve reference 'p1004'")

Reference p1004 is the loading screen, and if I turn the code of display_loading into a comment, the error goes away (but of course so does the loading screen functionality).
Interestingly by my code as is, the loading screen plot object exists during the error, and also the loading screen is still actually used in the __init__ method. And I actually see the loading screen during the initial loadup.

Now the error does not crash the program and it actually works despite it, but leaving it as is, suppressing it, or not using loading screens all feel like code smell to me.

Does anybody have an idea what causes this? Is Bokeh trying to resize an item that is not visible?

2

@Aziuth it is impossible to speculate without a complete Minimal Reproducible Example to actually run and investigate.

3

Took me a while to assemble one, given that my actual code is a big bulk, but here it is.
Supposed to be executed as a Bokeh server.

To produce the bug, change the value of the range slider twice.

from bokeh.plotting import figure, curdoc
from bokeh.models import ColumnDataSource, Label, RangeSlider
from bokeh.layouts import column
from random import *

def loading_pic(width: int = 500, height: int = 500) -> figure:
    plot = figure(x_range=(0,width), y_range=(0,height), width=width, height=height, tools='')
    pos_x = width/2 
    pos_y = height/2    
    
    label_size = min(height/4, width/4)
    label_size_str = str(label_size)+'px'
    
    loading_label = Label(x=pos_x, y=pos_y, text='Loading', text_font_size = label_size_str,
                          text_align='center', text_baseline='middle')
    plot.add_layout(loading_label)
    
    plot.xaxis.visible = False
    plot.yaxis.visible = False
    plot.xgrid.grid_line_color = None
    plot.ygrid.grid_line_color = None
    
    return plot
    
def random_coords(maximum: float = 500, amount: int = 1000) -> ColumnDataSource:
    data = ColumnDataSource()
    data.data['x'] = [random()*maximum for i in range(amount)]
    data.data['y'] = [random()*maximum for i in range(amount)]
    return data
    
class MyPlot:
    
    def __init__(self, data: ColumnDataSource):
        self.plot = figure(x_range=(0,500), y_range=(0,500), width=500, height=500)
        self.plot.circle(source=data, x='x', y='y')
    
class MyFrame:

    def __init__(self):
        self.layout = column()
        self.loading = loading_pic()
        self.layout.children.append(self.loading)

    def display_loading(self) -> None:
        self.layout.children[0] = self.loading

    def plot(self, data) -> None:
        self.my_plot = MyPlot(data)
        self.layout.children[0] = self.my_plot.plot
        
    def plot_delayed(self, attr, old, new) -> None:
        self.display_loading()
        data = random_coords()
        curdoc().add_next_tick_callback(lambda: self.plot(data)) 
        
my_frame = MyFrame()
        
range_slider = RangeSlider(start = 0, end = 10, value = (0,10), step = 1, title = 'Range')
range_slider.on_change('value_throttled', my_frame.plot_delayed)
       
layout = column(my_frame.layout, range_slider)

curdoc().add_root(layout)
4

@Aziuth FWIW this message was changed recently. Here is how it will appear in 3.2

2023-06-19 20:26:49,756 Dropping a patch because it contains a previously known reference (id=‘p1002’). Most of the time this is harmless and usually a result of updating a model on one side of a communications channel while it was being removed on the other end.

We display this message for “completeness” sake, but it’s actually not usually anything to be concerned with. So to be honest, I think you are worrying more than you need to be, and can just forget about it.

If you really want to get rid of it, though, your best bet is to add both plots to the layout up front an then toggle their visibility rather than removing/replacing them in the layout.

5

Yeah, I kinda thought so, but I am always hesitant to ignore warnings.

I guess I will suppress the error type since I need to be able to read the rest of the output and this one kinda spams the log.
Probably won’t go with the invisibility variant, as I think that would make my code less readable - I’d have to add an explanation why I do it that way.

Thanks for the answer.

6

Now, I said that I’m going to suppress the error, but how to do so? I know how to silence warnings in python, but I’m not sure how to do that with errors. Putting a huge try-catch around ther whole program doesn’t sound like the right way to go.

7

This is certainly a matter of opinion and style. Personally I find setting a visible property to be a self-documenting action that states something should be shown or hidden from the user.

I would also be remiss if I don’t remind that the express purpose of the Bokeh server is to automatically synchronize a document state over the network. By making a “big” change, e.g. adding and removing entire plots from the layout, you will cause the plots, all their sub-components (including data sources and all their data) to be serialized, trasnsmitted over the network, deserialized, and initialized— on every single slider movement. This is the Bokeh server performing exactly as designed and intended By contrast, setting the visible property only results in a very small update to toggle the value of an existing property.

FWIW the consistent best practice we have always demonstrated and stated is “make the smallest change possible”.

Now, I said that I’m going to suppress the error, but how to do so?

This isn’t an exception that is surfaced to you. It’s an exception that is caught internally and logged informationally. I don’t actually have any suggestions for you here, except to lower the log level.

8

I took your advice and used an invisibility approach.

However, now I am facing the next wave of the same error, but this time for the actual plot. The error is now triggered when the actual plot is exchanged for an entirely new one.

I could try to change the source of the old plot, but that feels strange. And also error-prone - a completely new plot contains only the elements it should, a modified old plot might contain artifacts from the old one, old settings or old elements.

Below code to trigger the bug, again by changing the slider twice. I added a debug output that shows the ID of the created plot, having us clearly see that the last actual plot is the culprit reference.

Also note that despite me now using hold and unhold like you suggested to me in another thread, the invisibility approach causes a hickup, showing both the loading screen and the plot at once for a split second.
If I reverse the visibility assignments, one sees none for a split second, trading a “down” hickup for an “up” hickup. That’s a bit suboptimal - am I doing something wrong here? Or is hold only working for actual plots, but not layouts? In a different place where I use the arrows from the other thread, hold works fine.

from bokeh.plotting import figure, curdoc
from bokeh.models import ColumnDataSource, Label, RangeSlider
from bokeh.layouts import column
from random import *

def loading_pic(width: int = 500, height: int = 500) -> figure:
    plot = figure(x_range=(0,width), y_range=(0,height), width=width, height=height, tools='')
    pos_x = width/2 
    pos_y = height/2    
    
    label_size = min(height/4, width/4)
    label_size_str = str(label_size)+'px'
    
    loading_label = Label(x=pos_x, y=pos_y, text='Loading', text_font_size = label_size_str,
                          text_align='center', text_baseline='middle')
    plot.add_layout(loading_label)
    
    plot.xaxis.visible = False
    plot.yaxis.visible = False
    plot.xgrid.grid_line_color = None
    plot.ygrid.grid_line_color = None
    
    return plot
    
def random_coords(maximum: float = 500, amount: int = 10000) -> ColumnDataSource:
    data = ColumnDataSource()
    data.data['x'] = [random()*maximum for i in range(amount)]
    data.data['y'] = [random()*maximum for i in range(amount)]
    return data
    
class MyPlot:
    
    def __init__(self, data: ColumnDataSource):
        self.plot = figure(x_range=(0,500), y_range=(0,500), width=500, height=500)
        self.plot.circle(source=data, x='x', y='y')
    
class MyFrame:

    def __init__(self):

        self._loading = loading_pic()

        self._loading_container = column(self._loading)
        self._render_container = column(column())
        self._render_container.visible = False
        
        self.layout = column(self._loading_container,  self._render_container)

    def display_loading(self) -> None:
        curdoc().hold("combine")
        self._loading_container.visible = True
        self._render_container.visible = False
        curdoc().unhold()

    def plot(self) -> None:
        curdoc().hold("combine")
        data = random_coords()
        self._my_plot = MyPlot(data)
        print("plot:", self._my_plot.plot)
        self._render_container.children[0] = self._my_plot.plot
        self._loading_container.visible = False
        self._render_container.visible = True
        curdoc().unhold()
        
    def plot_delayed(self, attr, old, new) -> None:
        self.display_loading()
        curdoc().add_next_tick_callback(self.plot) 
        
my_frame = MyFrame()
        
range_slider = RangeSlider(start = 0, end = 10, value = (0,10), step = 1, title = 'Range')
range_slider.on_change('value_throttled', my_frame.plot_delayed)
       
layout = column(my_frame.layout, range_slider)

curdoc().add_root(layout)
9

I could try to change the source of the old plot, but that feels strange

Bokeh is expressly geared towards updating data sources. That is the usage that has been heavily optimized and thoroughly tested and understood. Pretty much all of the interactive examples in docs and this forum achieve their results by updating data sources. I can’t make you take my advice, all I can say is: this is how the library is intended to be used, and by not doing so, you are fighting the tool.

That’s a bit suboptimal - am I doing something wrong here?

No but update events are ultimately processed individually some order. There’s probably improvements or new features that could be made here, resources allowing. I’d suggest you make a focused and detailed GitHub Issue.

I can tell you how I would solve this using pure Bokeh (put everything in one plot, including the “loading” text, and just update that one plot), but I don’t have the impression you would find it acceptable.

At this point I might actually suggest you take a look at Panel, which is built on top of Bokeh, intended for creating more dashboard-y output from a higher level. They also have some corporate resources backing their development, so they can afford to put more effort into some areas. E.g. they have a loading spinner built in: LoadingSpinner — Panel v1.3.4

10

Thanks for the advice, I followed it and just created an issue:

11

@Aziuth My suggestion was directly in response to your statements about “hickups”:

If I reverse the visibility assignments, one sees none for a split second, trading a “down” hickup for an “up” hickup. That’s a bit suboptimal - am I doing something wrong here?

That (improvements to reduce “hickups” when doing multiple updates) is what I was suggesting an issue for.

The “deserialization error” is not really an error and, as stated earlier, has in fact been changed to reflect this in the 3.2 latest release by no longer presenting as an error message at all. There is no work to do regarding the “deserialization error”.

1 Like
12

Ah kay. Did so:

13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.