Updating values watched by a CommsHandle (in a notebook)

I am trying to animate a Bokeh plot inside a Layout which is itself contained within an object, all within a Jupyter notebook, using the stream interface and push_notebook. I would like to be able to dynamically update the dataset being plotted using a method, call it set_data. When I instantiate the object I start with an empty Layout, and when I set the dataset to be plotted, I generate a figure and set the Layout's children property to be a list containing just the figure. Below the Layout I have a couple of ipywidgets, a play button and a slider, along with some text. Pushing the play button or moving the slider should cause the frame value to advance, i.e., the ColumnDataSource powering the plot should update, and the plot display itself should update via push_notebook.

When I set the children property inside the set_data method, I can plot the initial (from the first frame) data points. The Layout changes (by adding children) and this propagates. Changes to the children of the layout don’t propagate: even though the ColumnDataSource is updated via stream (which can be verified by displaying some statistic like the median of the changed values) and the slider, which is jslinked to the play widget, advances, the plot itself doesn’t change when calling push_notebook.

On the other hand, if I set the Layout's children before I create the notebook handle, using show, everything works as expected.

I think that I have this narrowed down to the fact that the CommsHandledoesn’t “see” the Layout's children. Here’s the handle.doc.to_json_string() for the object that doesn’t update:

{"roots":{"references":[{"attributes":{"children":[{"id":"1735","type":"Row"}]},"id":"1660","type":"Column"}],"root_ids":["1660"]},"title":"Bokeh Application","version":"1.3.4"}

And here’s the handle.doc.to_json_string() for the object that does:

{"roots":{"references":[{"attributes":{"children":[{"id":"1838","type":"Row"}]},"id":"1839","type":"Column"},{"attributes":{"callback":null},"id":"1804","type":"DataRange1d"},{"attributes":{},"id":"1821","type":"WheelZoomTool"},{"attributes":{"source":{"id":"1800","type":"ColumnDataSource"}},"id":"1837","type":"CDSView"},{"attributes":{},"id":"1887","type":"BasicTickFormatter"},{"attributes":{"dimension":1,"ticker":{"id":"1816","type":"BasicTicker"}},"id":"1819","type":"Grid"},{"attributes":{},"id":"1824","type":"ResetTool"},{"attributes":{},"id":"1825","type":"HelpTool"},{"attributes":{"fill_color":{"value":"#1f77b4"},"line_color":{"value":"#1f77b4"},"size":{"field":"size","units":"screen"},"x":{"field":"x"},"y":{"field":"y"}},"id":"1834","type":"Scatter"},{"attributes":{"callback":null,"data":{"size":{"__ndarray__":"8RsvooaSI0AkYEyUmgIdQCISACrGKBlAorePxFCgBUAckW8aMXwjQKvIvWVDvxVA4j57wMQmDECRDPEwdgoeQGQ0uTjOgRNA0YLMGfJKGkA=","dtype":"float64","shape":[10]},"x":{"__ndarray__":"qBJfVuGP4T8zh2DP1OLmP0uJbmvWSeM/9ZVj265v4T/2Dv0EKR3bP1ubiSEqq+Q/0llhym0B3D/3EjqIZ4nsPzYzF0lT1u4/FkLSS06K2D8=","dtype":"float64","shape":[10]},"y":{"__ndarray__":"KKk/v89V6T8if+sJtezgPy02kslrLeI/CCVY13ye7T8I88tKay+yP/DSvRQbTrY/4IkiryG0lD97INRf0qTqPx24j/yo5ug/J/h6tyPX6z8=","dtype":"float64","shape":[10]}},"selected":{"id":"1891","type":"Selection"},"selection_policy":{"id":"1892","type":"UnionRenderers"}},"id":"1800","type":"ColumnDataSource"},{"attributes":{"active_drag":"auto","active_inspect":"auto","active_multi":null,"active_scroll":"auto","active_tap":"auto","tools":[{"id":"1820","type":"PanTool"},{"id":"1821","type":"WheelZoomTool"},{"id":"1822","type":"BoxZoomTool"},{"id":"1823","type":"SaveTool"},{"id":"1824","type":"ResetTool"},{"id":"1825","type":"HelpTool"}]},"id":"1826","type":"Toolbar"},{"attributes":{"below":[{"id":"1810","type":"LinearAxis"}],"center":[{"id":"1814","type":"Grid"},{"id":"1819","type":"Grid"}],"left":[{"id":"1815","type":"LinearAxis"}],"plot_height":400,"plot_width":800,"renderers":[{"id":"1836","type":"GlyphRenderer"}],"title":{"id":"1885","type":"Title"},"toolbar":{"id":"1826","type":"Toolbar"},"x_range":{"id":"1802","type":"DataRange1d"},"x_scale":{"id":"1806","type":"LinearScale"},"y_range":{"id":"1804","type":"DataRange1d"},"y_scale":{"id":"1808","type":"LinearScale"}},"id":"1801","subtype":"Figure","type":"Plot"},{"attributes":{},"id":"1808","type":"LinearScale"},{"attributes":{},"id":"1816","type":"BasicTicker"},{"attributes":{"fill_alpha":{"value":0.1},"fill_color":{"value":"#1f77b4"},"line_alpha":{"value":0.1},"line_color":{"value":"#1f77b4"},"size":{"field":"size","units":"screen"},"x":{"field":"x"},"y":{"field":"y"}},"id":"1835","type":"Scatter"},{"attributes":{},"id":"1823","type":"SaveTool"},{"attributes":{"formatter":{"id":"1889","type":"BasicTickFormatter"},"ticker":{"id":"1816","type":"BasicTicker"}},"id":"1815","type":"LinearAxis"},{"attributes":{},"id":"1811","type":"BasicTicker"},{"attributes":{},"id":"1892","type":"UnionRenderers"},{"attributes":{},"id":"1891","type":"Selection"},{"attributes":{},"id":"1806","type":"LinearScale"},{"attributes":{"ticker":{"id":"1811","type":"BasicTicker"}},"id":"1814","type":"Grid"},{"attributes":{},"id":"1889","type":"BasicTickFormatter"},{"attributes":{"overlay":{"id":"1893","type":"BoxAnnotation"}},"id":"1822","type":"BoxZoomTool"},{"attributes":{"formatter":{"id":"1887","type":"BasicTickFormatter"},"ticker":{"id":"1811","type":"BasicTicker"}},"id":"1810","type":"LinearAxis"},{"attributes":{"callback":null},"id":"1802","type":"DataRange1d"},{"attributes":{"text":""},"id":"1885","type":"Title"},{"attributes":{},"id":"1820","type":"PanTool"},{"attributes":{"bottom_units":"screen","fill_alpha":{"value":0.5},"fill_color":{"value":"lightgrey"},"left_units":"screen","level":"overlay","line_alpha":{"value":1.0},"line_color":{"value":"black"},"line_dash":[4,4],"line_width":{"value":2},"render_mode":"css","right_units":"screen","top_units":"screen"},"id":"1893","type":"BoxAnnotation"},{"attributes":{"data_source":{"id":"1800","type":"ColumnDataSource"},"glyph":{"id":"1834","type":"Scatter"},"hover_glyph":null,"muted_glyph":null,"nonselection_glyph":{"id":"1835","type":"Scatter"},"selection_glyph":null,"view":{"id":"1837","type":"CDSView"}},"id":"1836","type":"GlyphRenderer"},{"attributes":{"children":[{"id":"1801","subtype":"Figure","type":"Plot"}]},"id":"1838","type":"Row"}],"root_ids":["1839"]},"title":"Bokeh Application","version":"1.3.4"}

Can someone help me with the last 10% and let me know how I can update what that CommsHandle is watching? Also let me know if anything’s unclear. Here’s some code:

First cell, imports and context:

from bokeh.models.sources import ColumnDataSource
from bokeh.plotting import figure
from bokeh.layouts import layout, row
from bokeh.io import output_notebook, show, push_notebook
from ipywidgets import Play, IntSlider, HBox, jslink, Label
import numpy as np

class DummyChange:
    """Replicates the relevant part of the ipywidget's change event"""
    def __init__(self, val):
        self.new = val

output_notebook()

Next cell, test data:

np.random.seed(0)
data = {"x": np.random.rand(10),
        "y": np.random.rand(10),
        "size": 10*np.random.rand(10, 100)}

Next cell, dynamically updating the children property when setting data (this will not animate on pressing play):

class Streamer:
    def __init__(self):
        self._data = None
        self._n_points = 0
        self._cds = ColumnDataSource(data=dict(x=[], y=[], size=[]))
        self._layout = layout([])  # Bokeh complains about an empty layout
        self._pw = Play(value=0,
                        min=0,
                        max=1,
                        step=1,
                        description="",
                        disabled=True)
        self._is = IntSlider(value=0,
                             min=0,
                             max=1,
                             step=1,
                             disabled=True)
        self._t = Label(value="0")
        jslink((self._is, "value"), (self._pw, "value"))
        self._pw.observe(self._animate, names="value")        

        self._handle = show(self._layout, notebook_handle=True)
        display(HBox([self._pw, self._is, self._t]))

    def _animate(self, change):
        frame = change.new
        self._cds.stream(new_data=dict(x=self._data["x"], # no change
                                       y=self._data["y"], # no change
                                       size=self._data["size"][:, frame]),
                         rollover=self._n_points)
        push_notebook(handle=self._handle)
        self._t.value = f"median size: {np.median(self._cds.data['size']):0.3f}"

    def set_data(self, data):
        self._data = data
        self._n_points = data["x"].size

        fig = figure(plot_width=800, plot_height=400)
        fig.scatter("x", "y", size="size", source=self._cds)
        self._layout.children = [row([fig])]

        # update player/slider widget values
        self._is.disabled = self._pw.disabled = False
        self._is.max = self._pw.max = self._data["size"].shape[1]

        self._animate(DummyChange(0))

s = Streamer()
s.set_data(data)

Next cell, creating the Layout with its children already in place (this will animate:

class Streamer2:
    def __init__(self, data):
        self._data = data
        self._n_points = data["x"].size
        self._cds = ColumnDataSource(data=dict(x=[], y=[], size=[]))

        fig = figure(plot_width=800, plot_height=400)
        fig.scatter("x", "y", size="size", source=self._cds)

        self._layout = layout(row([fig]))
        self._pw = Play(value=0,
                        min=0,
                        max=self._data["size"].shape[1],
                        step=1,
                        description="",
                        disabled=False)
        self._is = IntSlider(value=0,
                             min=0,
                             max=self._data["size"].shape[1],
                             step=1,
                             disabled=False)
        self._t = Label(value="0")
        jslink((self._is, "value"), (self._pw, "value"))
        self._pw.observe(self._animate, names="value")        

        self._handle = show(self._layout, notebook_handle=True)
        display(HBox([self._pw, self._is, self._t]))
        
        self._animate(DummyChange(0))

    def _animate(self, change):
        frame = change.new
        self._cds.stream(new_data=dict(x=self._data["x"], # no change
                                       y=self._data["y"], # no change
                                       size=self._data["size"][:, frame]),
                         rollover=self._n_points)
        push_notebook(handle=self._handle)
        self._t.value = f"median size: {np.median(self._cds.data['size']):0.3f}"

s2 = Streamer2(data)

Probably just a bug of some sort. Being frank, the use cases we had in mind for push_notebook were simpler than this, essentially setting something up once, up front, then updating data and simple properties, not entire layouts. Do you have a link to a complete notebook that can be downloaded for investigation?

I don’t have it on this machine, but those four code blocks are the entirety of the notebook I constructed to make this post. You can paste each of them into a separate cell to see the effect I’m talking about.

I’m happy to try a complete ready-to-run notebook when it is available.

You’re the boss: https://github.com/aliddell/misc-notebooks/blob/master/commshandle-demo.ipynb

(not trying to be bossy, but there are many users and one of me, even a small amount of “prep” work adds up very quickly on a single person)

I can’t find any immediate or obvious problems running this on master or 1.3.4, so I would say a GitHub issue with full details, so it can be tracked and investigated more thoroughly at in the future, would be appropriate. I’ll reiterate that I think the second (working) version is more in line with the typical supported usage pattern: show something structurally “complete” up front, and modify mostly/only the data afterward. Another option to try is embedding a Bokeh server app instead of using push_notebook.

(I mean it’s your forum and your software, so your rules; not as a gripe. I get the support burden.)

I’ll open the issue. A server app is not really an option for my application, but the second pattern can work for me. Thanks.

1 Like