Share data between sources of different documents

Hi,
I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream
only the information that they haven’t retrieved yet to the documents’ ColumnDataSources.
I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it’s not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.
My setup is the following, I run “bokeh serve” and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:

def update_server():

state.backlog.source1[“time”].append(someNewData)
state.backlog.source1[“y”].append(someNewData)

def on_server_loaded(server_context):
state.backlog.source1 = {“time”:,“y”:,“color”:}
server_context.add_periodic_callback(update_server, 10)

``

When a new session is created, I pass the backlog on to it:

setattr(session_context._document,‘backlog’, state.backlog)

``

I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:

@gen.coroutine
def locked_update(dataDic):
source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update
source1.data.update() #also this doesn’t help to update the plot on the webpage

@gen.coroutine
@without_document_lock
def update_session():
print “Session update…”
#…run an CPU-intensive thread…
doc.add_next_tick_callback(partial(locked_update,dict(time=, y=)))
print “…done (len source:{})”.format(len(source1.data.values()[0]))

doc.add_periodic_callback(update_session, 200)

``

Do I have to call a function to manually make sure @notify_owner or similar is done after setting `source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?

Thanks for all the replies and merry Chrismas.
`

Hi,

It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:

  https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram

However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:

Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models -- plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run "just once", it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user's session. But here is the crucial part: any given Bokeh Model can belong to one and *only one* Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session's document.

So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.

Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data "atomically" or all in one go instead of updating individual columns one at at time. That is, do this:

  source.data = new_data

NOT this:

  source.data['foo'] = stuff
  source.data['bar'] = other_stuff

The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are *always* of the same length. The next release will start to warn on this condition, and later releases will throw an error.

Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:

  http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream

A good example of it's use is the OHLC demo:

  https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc

Thanks,

Bryan

···

On Dec 26, 2016, at 3:11 AM, Colin <[email protected]> wrote:

Hi,

I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven't retrieved yet to the documents' ColumnDataSources.

I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it's not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.

My setup is the following, I run "bokeh serve" and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:

def update_server():
...
    state.backlog.source1["time"].append(someNewData)
    state.backlog.source1["y"].append(someNewData)

def on_server_loaded(server_context):
    state.backlog.source1 = {"time":,"y":,"color":}
    server_context.add_periodic_callback(update_server, 10)

When a new session is created, I pass the backlog on to it:
...
    setattr(session_context._document,'backlog', state.backlog)
...

I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:

@gen.coroutine
def locked_update(dataDic):
    source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update
    source1.data.update() #also this doesn't help to update the plot on the webpage

@gen.coroutine
@without_document_lock
def update_session():
    print "Session update..."
    #...run an CPU-intensive thread...
    doc.add_next_tick_callback(partial(locked_update,dict(time=, y=)))
    print "...done (len source:{})".format(len(source1.data.values()[0]))

doc.add_periodic_callback(update_session, 200)

Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?

Thanks for all the replies and merry Chrismas.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update

``

It’s helpful to confirm my understanding of documents, sessions and so on by reading your little summary. It’s also good to know that using bokeh model won’t help me in this case (because I also tried that).
In the code I posted I was using a normal python dict and list (for the backlog). In that respect, I did what you suggested and furthermore replace all of the data atomically with this line:
`
However, why doesn’t the plot update then? I assume line should trigger exactly one update event!?

Lastly, why is streaming in this case more efficient. On the python side, I have to keep the data twice and cope the data every time whereas by setting source.data = x would refer to the same object. For plotting I could understand because not all points have to be redrawn. Is that so?

`

···

On Tuesday, 27 December 2016 01:20:57 UTC+9, Bryan Van de ven wrote:

Hi,

It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram](https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram)

However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:

Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models – plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run “just once”, it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user’s session. But here is the crucial part: any given Bokeh Model can belong to one and only one Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session’s document.

So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.

Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data “atomically” or all in one go instead of updating individual columns one at at time. That is, do this:

    source.data = new_data

NOT this:

    source.data['foo'] = stuff

    source.data['bar'] = other_stuff

The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are always of the same length. The next release will start to warn on this condition, and later releases will throw an error.

Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:

    [http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream](http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream)

A good example of it’s use is the OHLC demo:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc](https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc)

Thanks,

Bryan

On Dec 26, 2016, at 3:11 AM, Colin [email protected] wrote:

Hi,

I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven’t retrieved yet to the documents’ ColumnDataSources.

I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it’s not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.

My setup is the following, I run “bokeh serve” and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:

def update_server():

state.backlog.source1["time"].append(someNewData)
state.backlog.source1["y"].append(someNewData)

def on_server_loaded(server_context):

state.backlog.source1 = {"time":[],"y":[],"color":[]}
server_context.add_periodic_callback(update_server, 10)

When a new session is created, I pass the backlog on to it:

setattr(session_context._document,'backlog', state.backlog)

I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:

@gen.coroutine

def locked_update(dataDic):

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update
source1.data.update() #also this doesn't help to update the plot on the webpage

@gen.coroutine

@without_document_lock

def update_session():

print "Session update..."
#...run an CPU-intensive thread...
doc.add_next_tick_callback(partial(locked_update,dict(time=[], y=[])))
print "...done (len source:{})".format(len(source1.data.values()[0]))

doc.add_periodic_callback(update_session, 200)

Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?

Thanks for all the replies and merry Chrismas.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

I had some success notifying the owner after the replacement of the data myself. But i think this is not the way it should be :wink:

p = next(iter(source1.data._owners))
p[1]._notify_mutated(p[0],None)
···

On Tuesday, 27 December 2016 10:36:43 UTC+9, Colin wrote:

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update

``

It’s helpful to confirm my understanding of documents, sessions and so on by reading your little summary. It’s also good to know that using bokeh model won’t help me in this case (because I also tried that).
In the code I posted I was using a normal python dict and list (for the backlog). In that respect, I did what you suggested and furthermore replace all of the data atomically with this line:
`
However, why doesn’t the plot update then? I assume line should trigger exactly one update event!?

Lastly, why is streaming in this case more efficient. On the python side, I have to keep the data twice and cope the data every time whereas by setting source.data = x would refer to the same object. For plotting I could understand because not all points have to be redrawn. Is that so?

`
On Tuesday, 27 December 2016 01:20:57 UTC+9, Bryan Van de ven wrote:

Hi,

It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram](https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram)

However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:

Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models – plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run “just once”, it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user’s session. But here is the crucial part: any given Bokeh Model can belong to one and only one Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session’s document.

So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.

Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data “atomically” or all in one go instead of updating individual columns one at at time. That is, do this:

    source.data = new_data

NOT this:

    source.data['foo'] = stuff

    source.data['bar'] = other_stuff

The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are always of the same length. The next release will start to warn on this condition, and later releases will throw an error.

Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:

    [http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream](http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream)

A good example of it’s use is the OHLC demo:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc](https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc)

Thanks,

Bryan

On Dec 26, 2016, at 3:11 AM, Colin [email protected] wrote:

Hi,

I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven’t retrieved yet to the documents’ ColumnDataSources.

I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it’s not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.

My setup is the following, I run “bokeh serve” and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:

def update_server():

state.backlog.source1["time"].append(someNewData)
state.backlog.source1["y"].append(someNewData)

def on_server_loaded(server_context):

state.backlog.source1 = {"time":[],"y":[],"color":[]}
server_context.add_periodic_callback(update_server, 10)

When a new session is created, I pass the backlog on to it:

setattr(session_context._document,'backlog', state.backlog)

I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:

@gen.coroutine

def locked_update(dataDic):

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update
source1.data.update() #also this doesn't help to update the plot on the webpage

@gen.coroutine

@without_document_lock

def update_session():

print "Session update..."
#...run an CPU-intensive thread...
doc.add_next_tick_callback(partial(locked_update,dict(time=[], y=[])))
print "...done (len source:{})".format(len(source1.data.values()[0]))

doc.add_periodic_callback(update_session, 200)

Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?

Thanks for all the replies and merry Chrismas.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi Colin,

It's hard to say more without seeing or being able to run code. For instance in

  source1.data = doc.backlog.source1

I have no idea what the type of source1 is for certain, or where exactly you are trying to make this update. In general, an assignment like that is all that is needed. You should almost certainly not be looking to _notify_mutated, etc. Can you possibly make a minimal complete example available for inspection?

Regarding streaming, I don't know for certain that it is more efficient in your case (again, because I haven't really seen a complete example code). But you mentioned "streaming" in your original post, so I thought I would mention the streaming API in case it was relevant. As I said, if by "streaming" you mean "periodic append-only updates" then that is what the streaming API makes much more efficient (by sending only the marginal updates, and not unnecessarily re-sending all the older data).

Thanks,

Bryan

···

On Dec 26, 2016, at 7:36 PM, Colin <[email protected]> wrote:

It's helpful to confirm my understanding of documents, sessions and so on by reading your little summary. It's also good to know that using bokeh model won't help me in this case (because I also tried that).
In the code I posted I was using a normal python dict and list (for the backlog). In that respect, I did what you suggested and furthermore replace all of the data atomically with this line:

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update

However, why doesn't the plot update then? I assume line should trigger exactly one update event!?

Lastly, why is streaming in this case more efficient. On the python side, I have to keep the data twice and cope the data every time whereas by setting source.data = x would refer to the same object. For plotting I could understand because not all points have to be redrawn. Is that so?

On Tuesday, 27 December 2016 01:20:57 UTC+9, Bryan Van de ven wrote:
Hi,

It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:

        https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram

However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:

Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models -- plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run "just once", it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user's session. But here is the crucial part: any given Bokeh Model can belong to one and *only one* Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session's document.

So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.

Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data "atomically" or all in one go instead of updating individual columns one at at time. That is, do this:

        source.data = new_data

NOT this:

        source.data['foo'] = stuff
        source.data['bar'] = other_stuff

The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are *always* of the same length. The next release will start to warn on this condition, and later releases will throw an error.

Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:

        http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream

A good example of it's use is the OHLC demo:

        https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc

Thanks,

Bryan

> On Dec 26, 2016, at 3:11 AM, Colin <[email protected]> wrote:
>
> Hi,
>
> I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven't retrieved yet to the documents' ColumnDataSources.
>
> I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it's not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.
>
> My setup is the following, I run "bokeh serve" and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:
>
> def update_server():
> ...
> state.backlog.source1["time"].append(someNewData)
> state.backlog.source1["y"].append(someNewData)
>
> def on_server_loaded(server_context):
> state.backlog.source1 = {"time":,"y":,"color":}
> server_context.add_periodic_callback(update_server, 10)
>
>
>
> When a new session is created, I pass the backlog on to it:
> ...
> setattr(session_context._document,'backlog', state.backlog)
> ...
>
> I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:
>
> @gen.coroutine
> def locked_update(dataDic):
> source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update
> source1.data.update() #also this doesn't help to update the plot on the webpage
>
> @gen.coroutine
> @without_document_lock
> def update_session():
> print "Session update..."
> #...run an CPU-intensive thread...
> doc.add_next_tick_callback(partial(locked_update,dict(time=, y=)))
> print "...done (len source:{})".format(len(source1.data.values()[0]))
>
> doc.add_periodic_callback(update_session, 200)
>
>
> Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?
>
> Thanks for all the replies and merry Chrismas.
>
> This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/b0bbb7ae-948c-4c8c-8254-ed8656959774%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi Bryan,

the assignment alone definitely does not help in my case. I attach a minimal working example. In the main.py there are 2 lines commented. Toggle this comment to see what I mean.
I wanted in no way to attack the streaming method. Actually I like it but I just fear that I would have to have to kee pcopies of data for each document. As long as I don’t want to modify (just display) the data in the sessions, why would I want to do that?

To start I run “bokeh serve folderName --show”

Thank you for your persistent and amazingly responsive support,
Colin

archive.zip (1.54 KB)

···

On Tuesday, 27 December 2016 14:45:19 UTC+9, Bryan Van de ven wrote:

Hi Colin,

It’s hard to say more without seeing or being able to run code. For instance in

    source1.data = doc.backlog.source1

I have no idea what the type of source1 is for certain, or where exactly you are trying to make this update. In general, an assignment like that is all that is needed. You should almost certainly not be looking to _notify_mutated, etc. Can you possibly make a minimal complete example available for inspection?

Regarding streaming, I don’t know for certain that it is more efficient in your case (again, because I haven’t really seen a complete example code). But you mentioned “streaming” in your original post, so I thought I would mention the streaming API in case it was relevant. As I said, if by “streaming” you mean “periodic append-only updates” then that is what the streaming API makes much more efficient (by sending only the marginal updates, and not unnecessarily re-sending all the older data).

Thanks,

Bryan

On Dec 26, 2016, at 7:36 PM, Colin [email protected] wrote:

It’s helpful to confirm my understanding of documents, sessions and so on by reading your little summary. It’s also good to know that using bokeh model won’t help me in this case (because I also tried that).

In the code I posted I was using a normal python dict and list (for the backlog). In that respect, I did what you suggested and furthermore replace all of the data atomically with this line:

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update

However, why doesn’t the plot update then? I assume line should trigger exactly one update event!?

Lastly, why is streaming in this case more efficient. On the python side, I have to keep the data twice and cope the data every time whereas by setting source.data = x would refer to the same object. For plotting I could understand because not all points have to be redrawn. Is that so?

On Tuesday, 27 December 2016 01:20:57 UTC+9, Bryan Van de ven wrote:

Hi,

It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram](https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram)

However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:

Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models – plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run “just once”, it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user’s session. But here is the crucial part: any given Bokeh Model can belong to one and only one Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session’s document.

So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.

Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data “atomically” or all in one go instead of updating individual columns one at at time. That is, do this:

    source.data = new_data

NOT this:

    source.data['foo'] = stuff
    source.data['bar'] = other_stuff

The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are always of the same length. The next release will start to warn on this condition, and later releases will throw an error.

Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:

    [http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream](http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream)

A good example of it’s use is the OHLC demo:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc](https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc)

Thanks,

Bryan

On Dec 26, 2016, at 3:11 AM, Colin [email protected] wrote:

Hi,

I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven’t retrieved yet to the documents’ ColumnDataSources.

I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it’s not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.

My setup is the following, I run “bokeh serve” and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:

def update_server():

state.backlog.source1[“time”].append(someNewData)
state.backlog.source1[“y”].append(someNewData)

def on_server_loaded(server_context):
state.backlog.source1 = {“time”:,“y”:,“color”:}
server_context.add_periodic_callback(update_server, 10)

When a new session is created, I pass the backlog on to it:

setattr(session_context._document,‘backlog’, state.backlog)

I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:

@gen.coroutine
def locked_update(dataDic):
source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update
source1.data.update() #also this doesn’t help to update the plot on the webpage

@gen.coroutine
@without_document_lock
def update_session():
print “Session update…”
#…run an CPU-intensive thread…
doc.add_next_tick_callback(partial(locked_update,dict(time=, y=)))
print “…done (len source:{})”.format(len(source1.data.values()[0]))

doc.add_periodic_callback(update_session, 200)

Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?

Thanks for all the replies and merry Chrismas.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/b0bbb7ae-948c-4c8c-8254-ed8656959774%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Let me just add version details here:
bokeh 0.12.3
python 2.7.10
firefox 50.1
linux

···

On Wednesday, 28 December 2016 15:42:55 UTC+9, Colin wrote:

Hi Bryan,

the assignment alone definitely does not help in my case. I attach a minimal working example. In the main.py there are 2 lines commented. Toggle this comment to see what I mean.
I wanted in no way to attack the streaming method. Actually I like it but I just fear that I would have to have to kee pcopies of data for each document. As long as I don’t want to modify (just display) the data in the sessions, why would I want to do that?

To start I run “bokeh serve folderName --show”

Thank you for your persistent and amazingly responsive support,
Colin

On Tuesday, 27 December 2016 14:45:19 UTC+9, Bryan Van de ven wrote:

Hi Colin,

It’s hard to say more without seeing or being able to run code. For instance in

    source1.data = doc.backlog.source1

I have no idea what the type of source1 is for certain, or where exactly you are trying to make this update. In general, an assignment like that is all that is needed. You should almost certainly not be looking to _notify_mutated, etc. Can you possibly make a minimal complete example available for inspection?

Regarding streaming, I don’t know for certain that it is more efficient in your case (again, because I haven’t really seen a complete example code). But you mentioned “streaming” in your original post, so I thought I would mention the streaming API in case it was relevant. As I said, if by “streaming” you mean “periodic append-only updates” then that is what the streaming API makes much more efficient (by sending only the marginal updates, and not unnecessarily re-sending all the older data).

Thanks,

Bryan

On Dec 26, 2016, at 7:36 PM, Colin [email protected] wrote:

It’s helpful to confirm my understanding of documents, sessions and so on by reading your little summary. It’s also good to know that using bokeh model won’t help me in this case (because I also tried that).

In the code I posted I was using a normal python dict and list (for the backlog). In that respect, I did what you suggested and furthermore replace all of the data atomically with this line:

source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update

However, why doesn’t the plot update then? I assume line should trigger exactly one update event!?

Lastly, why is streaming in this case more efficient. On the python side, I have to keep the data twice and cope the data every time whereas by setting source.data = x would refer to the same object. For plotting I could understand because not all points have to be redrawn. Is that so?

On Tuesday, 27 December 2016 01:20:57 UTC+9, Bryan Van de ven wrote:

Hi,

It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram](https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram)

However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:

Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models – plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run “just once”, it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user’s session. But here is the crucial part: any given Bokeh Model can belong to one and only one Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session’s document.

So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.

Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data “atomically” or all in one go instead of updating individual columns one at at time. That is, do this:

    source.data = new_data

NOT this:

    source.data['foo'] = stuff
    source.data['bar'] = other_stuff

The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are always of the same length. The next release will start to warn on this condition, and later releases will throw an error.

Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:

    [http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream](http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream)

A good example of it’s use is the OHLC demo:

    [https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc](https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc)

Thanks,

Bryan

On Dec 26, 2016, at 3:11 AM, Colin [email protected] wrote:

Hi,

I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven’t retrieved yet to the documents’ ColumnDataSources.

I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it’s not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.

My setup is the following, I run “bokeh serve” and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:

def update_server():

state.backlog.source1[“time”].append(someNewData)
state.backlog.source1[“y”].append(someNewData)

def on_server_loaded(server_context):
state.backlog.source1 = {“time”:,“y”:,“color”:}
server_context.add_periodic_callback(update_server, 10)

When a new session is created, I pass the backlog on to it:

setattr(session_context._document,‘backlog’, state.backlog)

I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:

@gen.coroutine
def locked_update(dataDic):
source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn’t update
source1.data.update() #also this doesn’t help to update the plot on the webpage

@gen.coroutine
@without_document_lock
def update_session():
print “Session update…”
#…run an CPU-intensive thread…
doc.add_next_tick_callback(partial(locked_update,dict(time=, y=)))
print “…done (len source:{})”.format(len(source1.data.values()[0]))

doc.add_periodic_callback(update_session, 200)

Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?

Thanks for all the replies and merry Chrismas.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/b0bbb7ae-948c-4c8c-8254-ed8656959774%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Colin,

I have a workaround that I can't explain offhand, but perhaps that's better than nothing, until it can be investigated further. If I replace these two lines:

  state.backlog.source1["time"].append(i)
  state.backlog.source1["y"].append(y)

with this line below:

  state.backlog.source1 = dict(time=state.backlog.source1["time"] + [i],
                               y=state.backlog.source1["y"] + [y])

then things seem to work as expected.

I'm not sure why that would be. In the absence of threads I would not expect any race condition that could result in state.backlog.source being only partially updated before it is assigned to source1.data in main.py. And indeed adding a print of all the column lengths there seems to confirm that they are always the same, even when things are not working.

As I said this will require some additional investigation, can you submit at GH issue with this example? If you are able making the code py3 runnable with print_function would be helpful.

Thanks,

Bryan

···

On Dec 28, 2016, at 8:20 PM, Colin <[email protected]> wrote:

Let me just add version details here:
bokeh 0.12.3
python 2.7.10
firefox 50.1
linux

On Wednesday, 28 December 2016 15:42:55 UTC+9, Colin wrote:
Hi Bryan,

the assignment alone definitely does not help in my case. I attach a minimal working example. In the main.py there are 2 lines commented. Toggle this comment to see what I mean.
I wanted in no way to attack the streaming method. Actually I like it but I just fear that I would have to have to kee pcopies of data for each document. As long as I don't want to modify (just display) the data in the sessions, why would I want to do that?

To start I run "bokeh serve folderName --show"

Thank you for your persistent and amazingly responsive support,
Colin

On Tuesday, 27 December 2016 14:45:19 UTC+9, Bryan Van de ven wrote:
Hi Colin,

It's hard to say more without seeing or being able to run code. For instance in

        source1.data = doc.backlog.source1

I have no idea what the type of source1 is for certain, or where exactly you are trying to make this update. In general, an assignment like that is all that is needed. You should almost certainly not be looking to _notify_mutated, etc. Can you possibly make a minimal complete example available for inspection?

Regarding streaming, I don't know for certain that it is more efficient in your case (again, because I haven't really seen a complete example code). But you mentioned "streaming" in your original post, so I thought I would mention the streaming API in case it was relevant. As I said, if by "streaming" you mean "periodic append-only updates" then that is what the streaming API makes much more efficient (by sending only the marginal updates, and not unnecessarily re-sending all the older data).

Thanks,

Bryan

> On Dec 26, 2016, at 7:36 PM, Colin <[email protected]> wrote:
>
> It's helpful to confirm my understanding of documents, sessions and so on by reading your little summary. It's also good to know that using bokeh model won't help me in this case (because I also tried that).
> In the code I posted I was using a normal python dict and list (for the backlog). In that respect, I did what you suggested and furthermore replace all of the data atomically with this line:
>
> source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update
>
> However, why doesn't the plot update then? I assume line should trigger exactly one update event!?
>
> Lastly, why is streaming in this case more efficient. On the python side, I have to keep the data twice and cope the data every time whereas by setting source.data = x would refer to the same object. For plotting I could understand because not all points have to be redrawn. Is that so?
>
>
>
> On Tuesday, 27 December 2016 01:20:57 UTC+9, Bryan Van de ven wrote:
> Hi,
>
> It is certainly possible to have a single thread that updates a single data object of some sort, and to use that data to inform multiple sessions. The best example of that I know if is probably the spectrogram example:
>
> https://github.com/bokeh/bokeh/tree/master/examples/app/spectrogram
>
> However, note that this app does not try to share actual Bokeh models, such as ColumnDataSource, between sessions. And in fact, this is not possible. Some description of how things work might help:
>
> Every user connection creates a session. Each session creates a Document (which is a collection of Bokeh Models -- plots, ranges, column data sources, etc.) that represents what the browser should render for the user. The Document is created by executing the app main.py code when the session is initiated. This is perhaps not always obvious, but is worth stressing: the app code is not run "just once", it is run every single time a user connects to the app URL. It is what creates the objects that serve a particular user's session. But here is the crucial part: any given Bokeh Model can belong to one and *only one* Document. When a session is created, it is expected to make a new Document, with new Models, that are not part of any other session's document.
>
> So, you can share data between sessions, as the spectrogram example does. But you will have to do it with non-Bokeh data objects, such as NumPy arrays or Pandas dataframes, or python data structures, as the spectrogram example does. Then your app code should copy whatever is needed or appropriate into a new CDS that the app creates.
>
> Based on the code you shared, a few other comments might be tangentially helpful. You should almost always update .data "atomically" or all in one go instead of updating individual columns one at at time. That is, do this:
>
> source.data = new_data
>
> NOT this:
>
> source.data['foo'] = stuff
> source.data['bar'] = other_stuff
>
> The latter method generates tow update events, instead of one, and can also break the fundamental assumption of CDS that columns are *always* of the same length. The next release will start to warn on this condition, and later releases will throw an error.
>
> Second, if you are actually streaming data (i.e. only adding new data points the the end of things) there is a .stream method that can be much more efficient:
>
> http://bokeh.pydata.org/en/latest/docs/reference/models/sources.html#bokeh.models.sources.ColumnDataSource.stream
>
> A good example of it's use is the OHLC demo:
>
> https://github.com/bokeh/bokeh/tree/master/examples/app/ohlc
>
> Thanks,
>
> Bryan
>
>
> > On Dec 26, 2016, at 3:11 AM, Colin <[email protected]> wrote:
> >
> > Hi,
> >
> > I would like to run a server that updates a global dictionary. Sessions, which each have their own document, should make use of that data. Alternatively, stream only the information that they haven't retrieved yet to the documents' ColumnDataSources.
> >
> > I have been trying to find an answer to my problem but did not see any good example so far (on this lilst and in bokeh examples). I hope it's not due to the conceptual understanding of server/session/document. I am just beginning with bokeh.
> >
> > My setup is the following, I run "bokeh serve" and have a server_lifecycle.py defined, which adds a periodic callback from (later zmq but now just) dummy data and append to a dict with various columns called backlog.source1:
> >
> > def update_server():
> > ...
> > state.backlog.source1["time"].append(someNewData)
> > state.backlog.source1["y"].append(someNewData)
> >
> > def on_server_loaded(server_context):
> > state.backlog.source1 = {"time":,"y":,"color":}
> > server_context.add_periodic_callback(update_server, 10)
> >
> >
> >
> > When a new session is created, I pass the backlog on to it:
> > ...
> > setattr(session_context._document,'backlog', state.backlog)
> > ...
> >
> > I the main for the sessions I define a ColumnDataSource (source1) and plot that uses this data. I want that each session shares the same data that is updated constantly. Therefore I run a slightly slower periodic callback attached to the current document:
> >
> > @gen.coroutine
> > def locked_update(dataDic):
> > source1.data = doc.backlog.source1 #I tried not to set this, the data is there but the plot doesn't update
> > source1.data.update() #also this doesn't help to update the plot on the webpage
> >
> > @gen.coroutine
> > @without_document_lock
> > def update_session():
> > print "Session update..."
> > #...run an CPU-intensive thread...
> > doc.add_next_tick_callback(partial(locked_update,dict(time=, y=)))
> > print "...done (len source:{})".format(len(source1.data.values()[0]))
> >
> > doc.add_periodic_callback(update_session, 200)
> >
> >
> > Do I have to call a function to manually make sure @notify_owner or similar is done after setting source1.data = doc.backlog.source1? Or is the proper way tokeep an ID of the columns that have been streamed so far and stream the new ones to the ColumnDataSource of each document?
> >
> > Thanks for all the replies and merry Chrismas.
> >
> > This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> > To post to this group, send email to [email protected].
> > To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0d68ed1b-7fff-429b-a7fa-d357abc2ae03%40continuum.io.
> > For more options, visit https://groups.google.com/a/continuum.io/d/optout.
>
>
> This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/b0bbb7ae-948c-4c8c-8254-ed8656959774%40continuum.io.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/74057adc-49e8-4e9b-bf4a-af0598e63e4b%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

OK, I will do this once I am back from my holiday (sorry, not before 10th Jan)
but yes, it’s strange because when one appends to a list and sets it with

@property
def foo:…
@foo.setter
def foo(bar): …

python registers a change. Anyway, this has been helpful so far.