Filtering/CDSViewing + Streaming

What are you trying to do?

I have two CDSs (s1 and s2), related by an id field in both of them. The idea is to use CustomJS coupled with an IndexFilter to allow the user to tap-select a single index in s1 and filter s2 for only the selected id in s2. This facilitates essentially a one-to-many filter action. I’m not using CustomJSFilter intentionally because in my actual use case there’s more going on than just this.

So this one to many IndexFilter approach works fine:

works1

The next thing I’m trying to do is append (i.e. stream) data to s2 via CustomJS. For the MRE below I’ve made a button and a CustomJS instance that will retrieve the current maximum x-y value in s2 and add one to both for the next data point (so the first button click will add an ‘apple’ at 101,101, next button click will add another apple at 102,102 etc. With the CDSView removed from the s2 renderer, this also works as intended:

works2

What have you tried that did NOT work as expected?

Now the combination of these two features is what’s not behaving the way I expected. When the user selects an apple from s1 (i.e. filters down to just apples on s2), then attempts to add an apple, I want the filter indices to update accordingly. So in my MRE, I show that you can accomplish this two different ways → execute the filter callback in the stream callback (probably pretty inefficient and unnecessary) or check if the currently selected thing is an apple, and if so, add the new index to the filter. In either case, based on the console.log, s2.data’s value arrays grow in length by one, and the current filter indices are reporting that the new value’s index is added:

works3

BUT when the button is pressed, the new point does not render!

works4

^^should be displaying a new red point at 101,101 but doesn’t!^^

Now where it gets even stranger is if I click the button twice (or more than twice):

works5

What’s happening as far as I can tell, is that the renderer is always one update behind the callback → click once, nothing shows up, click twice, what should have shown up on the first click now shows up but what should show up on the second click doesn’t etc. etc.

Finally, If I click the button once (and the new datapoint doesn’t render), then change the selected indices of s1), then go back to that selection, the new datapoint actually shows up:

works6

Which leads me to my current workaround → in the stream callback, store the currently selected s1 indices in a dummy variable, set s1.selected.indices to an empty array, then set it back to the dummy variable. And that works:

works7

Is this a bug, or is there some sort of intentional sequential execution of filter updating/renderer drawing going on here? Would definitely appreciate an explanation…I think this is related to Changing CDSView filters' attributes does not trigger rendering · Discussion #11560 · bokeh/bokeh · GitHub but I don’t know how to proceed (i.e. do I go with my workaround or there a better way?)

MRE:

import numpy as np
import pandas as pd
from bokeh.plotting import figure, save
from bokeh.models import MultiLine, IndexFilter, CDSView, ColumnDataSource,Scatter,CustomJS, Button
from bokeh.layouts import layout

fdict = {'banana':'yellow','apple':'red','orange':'orange'}
d1 = {'x':np.arange(0,3,1),'y':np.arange(0,3,1),'id':list(fdict.keys()),'c':list(fdict.values())}
n = 101
d2 = {'x':np.arange(1,n,1),'y':np.arange(1,n,1),'id':np.random.choice(list(fdict.keys()),n-1)}
d2['c'] = [fdict[f] for f in d2['id']]

s1 = ColumnDataSource(d1)
s2 = ColumnDataSource(d2) 

#first figure, user to make a selection to update the filter
f1 = figure(tools=['wheel_zoom','tap'],width=400,height=400)
f1_glyph = Scatter(x='x',y='y',size=25,fill_color='c')
f1_rend = f1.add_glyph(s1,f1_glyph)
#second figure, user to view the corresponding fruits for the one selected in the other source
f2 = figure(width=400,height=400,x_range=(0,105),y_range=(0,105))
f2_glyph = Scatter(x='x',y='y',size=10,fill_color='c')

#make the IndexFilter and the callback to update the filter's indices 
filt = IndexFilter(indices=[])
filt_cb = CustomJS(args=dict(s1=s1,s2=s2,filt=filt)
              ,code='''
              var sel_ind = s1.selected.indices[0]
              var sel_fruit = s1.data['id'][sel_ind]
              var filt_inds = []
              for (var i = 0; i<s2.data['id'].length; i++){
                      if (s2.data['id'][i]==sel_fruit){
                              filt_inds.push(i)}
                      }
              filt.indices = filt_inds
              s2.change.emit()
              ''')
view = CDSView(source=s2,filters=[filt])

f2_rend = f2.add_glyph(s2,f2_glyph
                        ,view=view
                       )
s1.selected.js_on_change('indices',filt_cb) 


btn = Button(label='Add_Apple')

add_cb = CustomJS(args=dict(s2=s2,filt=filt,s1=s1,filt_cb=filt_cb,view=view)
                  ,code='''
                  //get the new data to stream
                  var nx = Math.max(...s2.data['x'])+1
                  var ny = Math.max(...s2.data['y'])+1
                  var new_data = {'x':[nx],'y':[ny], 'id':['apple'],'c':['red']}
                  s2.stream(new_data)
                  console.log(s2.data)
                  //two ways to update the filter indices
                  //1 simple but probably slow, run the filt_cb
                  //filt_cb.execute()
                  //2 check the selected index is an apple and if so just push the new index to filt.indices
                  var sel_id = s1.data['id'][s1.selected.indices[0]]
                  console.log(sel_id)
                  if ( sel_id == 'apple'){
                          filt.indices.push(s2.data['id'].length-1)
                          }
                  view.properties.filters.change.emit() //kinda grasping at straws emitting changes etc.
                  s2.change.emit() //kinda grasping at straws emitting changes etc.

                  console.log(view._indices) // take a look at what the view's indices are reporting after all this'
                  
                  // my workaround
                  //var si = s1.selected.indices
                  //s1.selected.indices=[]
                  //s1.selected.indices=si
                  '''
                  )


btn.js_on_click(add_cb)


lo = layout([[f1,f2],btn])
save(lo,'filt_bug_testing.html')

I’m not sure these two features have ever been explicitly considered together. I’ll try to take a look this weekend.

1 Like

Hi @gmerritt123 here is another version that works. I am not sure it is better, per se:

add_cb = CustomJS(args=dict(s1=s1, s2=s2, filt=filt), code="""
    const nx = Math.max(...s2.data.x) + 1
    const ny = Math.max(...s2.data.y) + 1
    const new_data = {x: [nx], y: [ny], id: ['apple'], c: ['red']}
    const id = s1.data.id[s1.selected.indices[0]]
    if (id == 'apple') {
        filt.indices.push(s2.data.id.length)  # remove the -1
    }
    s2.stream(new_data)
""")

I believe the issue comes down to making a change that is “forceful” enough to really trigger a proper re-render, including some data preparation steps. Above, I have done that by moving the s2.stream to be last. But also notice, I had to change the index that was pushed on to the filter indices in a slightly annoying way, by pushing an index that won’t exist until the subsequent stream is done.

None of this is great, of course. Maybe we can find some ways to make this less fragile. I’d encourage you to make a GH issue about it.

1 Like

Thanks so much for looking at this. Post as a GH issue here: [BUG] IndexFilter + Stream, renderer update not behaving as expected · Issue #12227 · bokeh/bokeh · GitHub