Updating multiple plots simultaneously

jphalip · June 7, 2017, 8:15pm

Hi,

I’m building a dashboard that contains multiple plots inside the same document. Each plot has a separate ColumnSource object attached to it. The data for each plot is retrieved with different SQL queries executed on a remote database server. The dashboard also has some controls (e.g. Select widgets) that allow the user to update those plots with different parameters.

Everything is working well, however all the plots are currently rendered synchronously in sequence (i.e. one after the other). This is especially problematic since each my DB queries are a bit slow and can take several seconds to execute. So when the user selects a different value in the controls, plots are updated in sequence at a few seconds interval.

Here’s some pseudo code showing my use case:

def update_plots(attrname, old, new):

    update_plot_1(attrname, old, new)  # Causes a new DB request being executed (takes 2-3 seconds) followed by plot #1 being updated.

    update_plot_2(attrname, old, new)  # This causes plot #2 to be updated but is only executed once upload_plot_1() is completed.

my_select = Select(value='a', options=['a', 'b', 'c'])

my_select.on_change('value', update_plots)

In order to speed up the rendering process, I’d like to have all plots be updated simultaneously. Would you recommend some strategies for letting the on_change() control callbacks instruct multiple plots to be updated at the same time? Perhaps using Python multiprocessing or threading? Or should each plot perhaps live in a separate document and in separate Tornado connections?

Thanks a lot,

Julien

jphalip · June 9, 2017, 2:12am

Hello again,

I’ve tried a different approach:

@gen.coroutine

@without_document_lock

def update_plot_1(attrname, old, new):

    ...

@gen.coroutine

@without_document_lock

def update_plot_2(attrname, old, new):

    ...

def update_plots(attrname, old, new):

    global doc

    doc.add_next_tick_callback(partial(update_plot_1, attrname, old, new))

    doc.add_next_tick_callback(partial(update_plot_2, attrname, old, new))

doc = currdoc()

my_select = Select(value='a', options=['a', 'b', 'c'])

my_select.on_change('value', update_plots)

Unfortunately I’m still getting the same undesired effect, i.e. the plots get updated in sequence. Is there no way to simultaneously/asynchronously update multiple plots in a same document? If so, is there a way to serve multiple sessions/documents (presumably one for each plot) from the same server, and if so what would the code look like based on my example above? Otherwise is the only way to serve each plot separately from a different server?

See also these similar threads:

“Streaming Data to Multiple Documents Simultaneously”: Redirecting to Google Groups
“Plot on different documents at the same time”: https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/snm3qxAIomI

···

On Wednesday, June 7, 2017 at 1:17:39 PM UTC-7, Julien Phalip wrote:

Hi,

I’m building a dashboard that contains multiple plots inside the same document. Each plot has a separate ColumnSource object attached to it. The data for each plot is retrieved with different SQL queries executed on a remote database server. The dashboard also has some controls (e.g. Select widgets) that allow the user to update those plots with different parameters.

Everything is working well, however all the plots are currently rendered synchronously in sequence (i.e. one after the other). This is especially problematic since each my DB queries are a bit slow and can take several seconds to execute. So when the user selects a different value in the controls, plots are updated in sequence at a few seconds interval.

Here’s some pseudo code showing my use case:

def update_plots(attrname, old, new):

    update_plot_1(attrname, old, new)  # Causes a new DB request being executed (takes 2-3 seconds) followed by plot #1 being updated.

    update_plot_2(attrname, old, new)  # This causes plot #2 to be updated but is only executed once upload_plot_1() is completed.

my_select = Select(value='a', options=['a', 'b', 'c'])

my_select.on_change('value', update_plots)

In order to speed up the rendering process, I’d like to have all plots be updated simultaneously. Would you recommend some strategies for letting the on_change() control callbacks instruct multiple plots to be updated at the same time? Perhaps using Python multiprocessing or threading? Or should each plot perhaps live in a separate document and in separate Tornado connections?

Thanks a lot,

Julien

Bryan · June 9, 2017, 1:10pm

Hi,

Batched updates are still an open issue:

feature: add support for batched model updates in Bokeh server applications · Issue #5438 · bokeh/bokeh · GitHub

The low level machinery at the protocol level should actually already be in place, but there is work to do (including choosing an API) at the python level.

In the mean time, the best suggestion I can think of would be to re-order the work. Split the functions update_plot_1 and update_plot_1 into two parts, one that fetches the data and one that updates the Bokeh models. Then do both the expensive fetch data calls first, before both Bokeh update calls.

Thanks,

Bryan

···

On Jun 8, 2017, at 21:12, [email protected] wrote:

Hello again,

I've tried a different approach:

    @gen.coroutine
    @without_document_lock
    def update_plot_1(attrname, old, new):
        ...

    @gen.coroutine
    @without_document_lock
    def update_plot_2(attrname, old, new):
        ...

    def update_plots(attrname, old, new):
        global doc
        doc.add_next_tick_callback(partial(update_plot_1, attrname, old, new))
        doc.add_next_tick_callback(partial(update_plot_2, attrname, old, new))

    doc = currdoc()
    my_select = Select(value='a', options=['a', 'b', 'c'])
    my_select.on_change('value', update_plots)

Unfortunately I'm still getting the same undesired effect, i.e. the plots get updated in sequence. Is there no way to simultaneously/asynchronously update multiple plots in a same document? If so, is there a way to serve multiple sessions/documents (presumably one for each plot) from the same server, and if so what would the code look like based on my example above? Otherwise is the only way to serve each plot separately from a different server?

See also these similar threads:
  • "Streaming Data to Multiple Documents Simultaneously": Redirecting to Google Groups
  • "Plot on different documents at the same time": https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/snm3qxAIomI

On Wednesday, June 7, 2017 at 1:17:39 PM UTC-7, Julien Phalip wrote:
Hi,

I'm building a dashboard that contains multiple plots inside the same document. Each plot has a separate ColumnSource object attached to it. The data for each plot is retrieved with different SQL queries executed on a remote database server. The dashboard also has some controls (e.g. Select widgets) that allow the user to update those plots with different parameters.

Everything is working well, however all the plots are currently rendered synchronously in sequence (i.e. one after the other). This is especially problematic since each my DB queries are a bit slow and can take several seconds to execute. So when the user selects a different value in the controls, plots are updated in sequence at a few seconds interval.

Here's some pseudo code showing my use case:

    def update_plots(attrname, old, new):
        update_plot_1(attrname, old, new) # Causes a new DB request being executed (takes 2-3 seconds) followed by plot #1 being updated.
        update_plot_2(attrname, old, new) # This causes plot #2 to be updated but is only executed once upload_plot_1() is completed.

    my_select = Select(value='a', options=['a', 'b', 'c'])
    my_select.on_change('value', update_plots)

In order to speed up the rendering process, I'd like to have all plots be updated simultaneously. Would you recommend some strategies for letting the on_change() control callbacks instruct multiple plots to be updated at the same time? Perhaps using Python multiprocessing or threading? Or should each plot perhaps live in a separate document and in separate Tornado connections?

Thanks a lot,

Julien

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4631c0f9-2674-4fe2-8047-a1bb9d615c2d%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

jphalip · June 9, 2017, 10:03pm

Thanks for the info. Could you provide more pointers about the low-level machinery you’re referring to? I might be interested in taking a look, see if I can help push this forward.

···

On Friday, June 9, 2017 at 6:10:36 AM UTC-7, Bryan Van de ven wrote:

Hi,

Batched updates are still an open issue:
    [https://github.com/bokeh/bokeh/issues/5438](https://github.com/bokeh/bokeh/issues/5438)
The low level machinery at the protocol level should actually already be in place, but there is work to do (including choosing an API) at the python level.

In the mean time, the best suggestion I can think of would be to re-order the work. Split the functions update_plot_1 and update_plot_1 into two parts, one that fetches the data and one that updates the Bokeh models. Then do both the expensive fetch data calls first, before both Bokeh update calls.

Thanks,

Bryan

On Jun 8, 2017, at 21:12, [email protected] wrote:

Hello again,

I’ve tried a different approach:
@gen.coroutine
@without_document_lock
def update_plot_1(attrname, old, new):
    ...
@gen.coroutine
@without_document_lock
def update_plot_2(attrname, old, new):
    ...
def update_plots(attrname, old, new):
    global doc
    doc.add_next_tick_callback(partial(update_plot_1, attrname, old, new))
    doc.add_next_tick_callback(partial(update_plot_2, attrname, old, new))
doc = currdoc()
my_select = Select(value='a', options=['a', 'b', 'c'])
my_select.on_change('value', update_plots)
Unfortunately I’m still getting the same undesired effect, i.e. the plots get updated in sequence. Is there no way to simultaneously/asynchronously update multiple plots in a same document? If so, is there a way to serve multiple sessions/documents (presumably one for each plot) from the same server, and if so what would the code look like based on my example above? Otherwise is the only way to serve each plot separately from a different server?

See also these similar threads:
    • "Streaming Data to Multiple Documents Simultaneously": [https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/9vZJdhIpP0c](https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/9vZJdhIpP0c)
    • "Plot on different documents at the same time": [https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/snm3qxAIomI](https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/snm3qxAIomI)
On Wednesday, June 7, 2017 at 1:17:39 PM UTC-7, Julien Phalip wrote:

Hi,

I’m building a dashboard that contains multiple plots inside the same document. Each plot has a separate ColumnSource object attached to it. The data for each plot is retrieved with different SQL queries executed on a remote database server. The dashboard also has some controls (e.g. Select widgets) that allow the user to update those plots with different parameters.

Everything is working well, however all the plots are currently rendered synchronously in sequence (i.e. one after the other). This is especially problematic since each my DB queries are a bit slow and can take several seconds to execute. So when the user selects a different value in the controls, plots are updated in sequence at a few seconds interval.

Here’s some pseudo code showing my use case:
def update_plots(attrname, old, new):
    update_plot_1(attrname, old, new)  # Causes a new DB request being executed (takes 2-3 seconds) followed by plot #1 being updated.
    update_plot_2(attrname, old, new)  # This causes plot #2 to be updated but is only executed once upload_plot_1() is completed.
my_select = Select(value='a', options=['a', 'b', 'c'])
my_select.on_change('value', update_plots)
In order to speed up the rendering process, I’d like to have all plots be updated simultaneously. Would you recommend some strategies for letting the on_change() control callbacks instruct multiple plots to be updated at the same time? Perhaps using Python multiprocessing or threading? Or should each plot perhaps live in a separate document and in separate Tornado connections?

Thanks a lot,

Julien

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4631c0f9-2674-4fe2-8047-a1bb9d615c2d%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Bryan · June 11, 2017, 9:07pm

Hi,

The issue is here:

feature: add support for batched model updates in Bokeh server applications · Issue #5438 · bokeh/bokeh · GitHub

What I mean specifically in reference to "low-level plumbing" is that the PATCH-DOC protocol message can already contain multiple events. Multiple events in one message are all applied sequentially in one operation on the client:

https://github.com/bokeh/bokeh/blob/master/bokehjs/src/coffee/document.coffee#L574

So far, so good. The issue then is that currently all property changes, etc. immediately trigger and fire off a PATCH-DOC with a single event. So the task is to add some mechanism to postpone messages and collect events when properties change, and then generate a single PATCH-DOC with all the collected events.

Thanks,

Bryan

···

On Jun 9, 2017, at 17:03, [email protected] wrote:

Thanks for the info. Could you provide more pointers about the low-level machinery you're referring to? I might be interested in taking a look, see if I can help push this forward.

On Friday, June 9, 2017 at 6:10:36 AM UTC-7, Bryan Van de ven wrote:
Hi,

Batched updates are still an open issue:

feature: add support for batched model updates in Bokeh server applications · Issue #5438 · bokeh/bokeh · GitHub

The low level machinery at the protocol level should actually already be in place, but there is work to do (including choosing an API) at the python level.

In the mean time, the best suggestion I can think of would be to re-order the work. Split the functions update_plot_1 and update_plot_1 into two parts, one that fetches the data and one that updates the Bokeh models. Then do both the expensive fetch data calls first, before both Bokeh update calls.

Thanks,

Bryan

> On Jun 8, 2017, at 21:12, jph...@gmail.com wrote:
>
> Hello again,
>
> I've tried a different approach:
>
> @gen.coroutine
> @without_document_lock
> def update_plot_1(attrname, old, new):
> ...
>
> @gen.coroutine
> @without_document_lock
> def update_plot_2(attrname, old, new):
> ...
>
> def update_plots(attrname, old, new):
> global doc
> doc.add_next_tick_callback(partial(update_plot_1, attrname, old, new))
> doc.add_next_tick_callback(partial(update_plot_2, attrname, old, new))
>
> doc = currdoc()
> my_select = Select(value='a', options=['a', 'b', 'c'])
> my_select.on_change('value', update_plots)
>
> Unfortunately I'm still getting the same undesired effect, i.e. the plots get updated in sequence. Is there no way to simultaneously/asynchronously update multiple plots in a same document? If so, is there a way to serve multiple sessions/documents (presumably one for each plot) from the same server, and if so what would the code look like based on my example above? Otherwise is the only way to serve each plot separately from a different server?
>
> See also these similar threads:
> • "Streaming Data to Multiple Documents Simultaneously": Redirecting to Google Groups
> • "Plot on different documents at the same time": https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/snm3qxAIomI
>
> On Wednesday, June 7, 2017 at 1:17:39 PM UTC-7, Julien Phalip wrote:
> Hi,
>
> I'm building a dashboard that contains multiple plots inside the same document. Each plot has a separate ColumnSource object attached to it. The data for each plot is retrieved with different SQL queries executed on a remote database server. The dashboard also has some controls (e.g. Select widgets) that allow the user to update those plots with different parameters.
>
> Everything is working well, however all the plots are currently rendered synchronously in sequence (i.e. one after the other). This is especially problematic since each my DB queries are a bit slow and can take several seconds to execute. So when the user selects a different value in the controls, plots are updated in sequence at a few seconds interval.
>
> Here's some pseudo code showing my use case:
>
> def update_plots(attrname, old, new):
> update_plot_1(attrname, old, new) # Causes a new DB request being executed (takes 2-3 seconds) followed by plot #1 being updated.
> update_plot_2(attrname, old, new) # This causes plot #2 to be updated but is only executed once upload_plot_1() is completed.
>
> my_select = Select(value='a', options=['a', 'b', 'c'])
> my_select.on_change('value', update_plots)
>
> In order to speed up the rendering process, I'd like to have all plots be updated simultaneously. Would you recommend some strategies for letting the on_change() control callbacks instruct multiple plots to be updated at the same time? Perhaps using Python multiprocessing or threading? Or should each plot perhaps live in a separate document and in separate Tornado connections?
>
> Thanks a lot,
>
> Julien
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
> To post to this group, send email to bo...@continuum.io.
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4631c0f9-2674-4fe2-8047-a1bb9d615c2d%40continuum.io\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/a7a7527a-7ba5-4d2b-96a4-80363789e1a6%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.