From C++ to Bokeh Server

Mauricio_Hernandes · November 8, 2016, 12:42am

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+":5555")
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?
What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

Matthew_Rocklin · November 8, 2016, 3:31pm

We do something similar with the dask.distributed web interface: http://distributed.readthedocs.io/en/latest/web.html

Some extra things you might want to consider:

You could use the PyZMQ event loop to do non-blocking communication. This would play slightly more nicely with Bokeh, which also uses an event loop, although isn’t at all necessary if you’re ok with a very small amount of contention: https://pyzmq.readthedocs.io/en/latest/eventloop.html
As you’ve written it now every session will connect to your c++ server instead of having just one connection. For Dask we do a single connection in the server_lifecycle.py script that populates a globally accessible dictionary of current state and then every session (main.py) periodically checks this global state. This helps reduce communication overhead if you have multiple sessions.
If you’re looking for examples then Dask’s solution may be of value (although plenty of it is Dask specific):

https://github.com/dask/distributed/blob/master/distributed/bokeh/background/server_lifecycle.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/components.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/status/main.py

For communicating string-like or json-like data I personally use msgpack, which is fairly fast and broadly accessible.

-matt

···

On Tuesday, November 8, 2016 at 10:09:31 AM UTC-5, [email protected] wrote:

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+“:5555”)
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?

What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

Mauricio_Hernandes · November 8, 2016, 11:40pm

Thank you for the reply.

I was looking at more details of PYZMQ yesterday and I am planning to use to prevent the server to get locked waiting for messages. I may go for that.

Thank you for the example with Dask. Your suggestion is to have a main.py that connects to the c++ code and each bokeh client asks that main.py if there is new data? In this way there would be only one connection to the c++ code.

Did I understand it correctly?

Thank you

Mauricio

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

···

On Wed, Nov 9, 2016 at 12:31 AM, [email protected] wrote:

We do something similar with the dask.distributed web interface: http://distributed.readthedocs.io/en/latest/web.html

Some extra things you might want to consider:

You could use the PyZMQ event loop to do non-blocking communication. This would play slightly more nicely with Bokeh, which also uses an event loop, although isn’t at all necessary if you’re ok with a very small amount of contention: https://pyzmq.readthedocs.io/en/latest/eventloop.html

As you’ve written it now every session will connect to your c++ server instead of having just one connection. For Dask we do a single connection in the server_lifecycle.py script that populates a globally accessible dictionary of current state and then every session (main.py) periodically checks this global state. This helps reduce communication overhead if you have multiple sessions.
If you’re looking for examples then Dask’s solution may be of value (although plenty of it is Dask specific):

https://github.com/dask/distributed/blob/master/distributed/bokeh/background/server_lifecycle.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/components.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/status/main.py

For communicating string-like or json-like data I personally use msgpack, which is fairly fast and broadly accessible.

-matt

On Tuesday, November 8, 2016 at 10:09:31 AM UTC-5, [email protected] wrote:

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+“:5555”)
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?

What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

–

Mauricio S.C. Hernandes | Strategic Software Development | SBI BITS
Roppongi T-Cube 20F, 3-1-1 Roppongi, Minato-ku, Tokyo 106-0032 Japan
T +81-3-4510-7000 | E [email protected]
www.sbibits.com

Matthew_Rocklin · November 8, 2016, 11:43pm

Yes, although main.py is used for each web client (of which there are many). The server_lifecycle.py file is where you place code that should run exactly once.

http://bokeh.pydata.org/en/latest/docs/user_guide/server.html#directory-format

···

On Tue, Nov 8, 2016 at 6:40 PM, Mauricio Hernandes [email protected] wrote:

Thank you for the reply.

I was looking at more details of PYZMQ yesterday and I am planning to use to prevent the server to get locked waiting for messages. I may go for that.

Thank you for the example with Dask. Your suggestion is to have a main.py that connects to the c++ code and each bokeh client asks that main.py if there is new data? In this way there would be only one connection to the c++ code.

Did I understand it correctly?

Thank you

Mauricio

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

On Wed, Nov 9, 2016 at 12:31 AM, [email protected] wrote:

We do something similar with the dask.distributed web interface: http://distributed.readthedocs.io/en/latest/web.html

Some extra things you might want to consider:

You could use the PyZMQ event loop to do non-blocking communication. This would play slightly more nicely with Bokeh, which also uses an event loop, although isn’t at all necessary if you’re ok with a very small amount of contention: https://pyzmq.readthedocs.io/en/latest/eventloop.html

As you’ve written it now every session will connect to your c++ server instead of having just one connection. For Dask we do a single connection in the server_lifecycle.py script that populates a globally accessible dictionary of current state and then every session (main.py) periodically checks this global state. This helps reduce communication overhead if you have multiple sessions.
If you’re looking for examples then Dask’s solution may be of value (although plenty of it is Dask specific):

https://github.com/dask/distributed/blob/master/distributed/bokeh/background/server_lifecycle.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/components.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/status/main.py

For communicating string-like or json-like data I personally use msgpack, which is fairly fast and broadly accessible.

-matt

On Tuesday, November 8, 2016 at 10:09:31 AM UTC-5, [email protected] wrote:

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+“:5555”)
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?

What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

–

Mauricio S.C. Hernandes | Strategic Software Development | SBI BITS
Roppongi T-Cube 20F, 3-1-1 Roppongi, Minato-ku, Tokyo 106-0032 Japan
T +81-3-4510-7000 | E [email protected]
www.sbibits.com

Mauricio_Hernandes · November 10, 2016, 2:22pm

I went back to have a closer look at Dask but I still did not understand how the main.py gets the messages updated by the lifecycle.

Could you walk me through some details?

I see that the periodic_call_back stores the msgs in the distributed.bokeh.messages

But how exactly main.py see those messages? Where are them instantiated?

Thank you for the links again.

Mau

···

On Wednesday, November 9, 2016 at 12:37:26 AM UTC+9, [email protected] wrote:

We do something similar with the dask.distributed web interface: http://distributed.readthedocs.io/en/latest/web.html

Some extra things you might want to consider:

You could use the PyZMQ event loop to do non-blocking communication. This would play slightly more nicely with Bokeh, which also uses an event loop, although isn’t at all necessary if you’re ok with a very small amount of contention: https://pyzmq.readthedocs.io/en/latest/eventloop.html

As you’ve written it now every session will connect to your c++ server instead of having just one connection. For Dask we do a single connection in the server_lifecycle.py script that populates a globally accessible dictionary of current state and then every session (main.py) periodically checks this global state. This helps reduce communication overhead if you have multiple sessions.
If you’re looking for examples then Dask’s solution may be of value (although plenty of it is Dask specific):

https://github.com/dask/distributed/blob/master/distributed/bokeh/background/server_lifecycle.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/components.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/status/main.py

For communicating string-like or json-like data I personally use msgpack, which is fairly fast and broadly accessible.

-matt

On Tuesday, November 8, 2016 at 10:09:31 AM UTC-5, [email protected] wrote:

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+“:5555”)
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?

What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

Matthew_Rocklin · November 10, 2016, 2:25pm

For example in distributed/bokeh/status/main.py

messages = distributed.bokeh.messages # global message store

task_stream = TaskStream(1000, sizing_mode=SIZING_MODE, width=WIDTH, height=300)

doc.add_periodic_callback(lambda: task_stream.update(messages), messages[‘task-events’][‘interval’])

···

On Thu, Nov 10, 2016 at 9:22 AM, [email protected] wrote:

I went back to have a closer look at Dask but I still did not understand how the main.py gets the messages updated by the lifecycle.

Could you walk me through some details?

I see that the periodic_call_back stores the msgs in the distributed.bokeh.messages

But how exactly main.py see those messages? Where are them instantiated?

Thank you for the links again.

Mau

On Wednesday, November 9, 2016 at 12:37:26 AM UTC+9, [email protected] wrote:

We do something similar with the dask.distributed web interface: http://distributed.readthedocs.io/en/latest/web.html

Some extra things you might want to consider:

You could use the PyZMQ event loop to do non-blocking communication. This would play slightly more nicely with Bokeh, which also uses an event loop, although isn’t at all necessary if you’re ok with a very small amount of contention: https://pyzmq.readthedocs.io/en/latest/eventloop.html

As you’ve written it now every session will connect to your c++ server instead of having just one connection. For Dask we do a single connection in the server_lifecycle.py script that populates a globally accessible dictionary of current state and then every session (main.py) periodically checks this global state. This helps reduce communication overhead if you have multiple sessions.
If you’re looking for examples then Dask’s solution may be of value (although plenty of it is Dask specific):

https://github.com/dask/distributed/blob/master/distributed/bokeh/background/server_lifecycle.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/components.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/status/main.py

For communicating string-like or json-like data I personally use msgpack, which is fairly fast and broadly accessible.

-matt

On Tuesday, November 8, 2016 at 10:09:31 AM UTC-5, [email protected] wrote:

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+“:5555”)
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?

What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

Mauricio_Hernandes · November 10, 2016, 2:35pm

Oh, See.

In a quick private change of emails you told me that you set the global variable in the init file.

I did not notice that.

Anyway. I think now I understand.

Thank you

···

On Thursday, November 10, 2016 at 11:25:41 PM UTC+9, Matthew Rocklin wrote:

For example in distributed/bokeh/status/main.py

messages = distributed.bokeh.messages # global message store

task_stream = TaskStream(1000, sizing_mode=SIZING_MODE, width=WIDTH, height=300)

doc.add_periodic_callback(lambda: task_stream.update(messages), messages[‘task-events’][‘interval’])

On Thu, Nov 10, 2016 at 9:22 AM, [email protected] wrote:

I went back to have a closer look at Dask but I still did not understand how the main.py gets the messages updated by the lifecycle.

Could you walk me through some details?

I see that the periodic_call_back stores the msgs in the distributed.bokeh.messages

But how exactly main.py see those messages? Where are them instantiated?

Thank you for the links again.

Mau

On Wednesday, November 9, 2016 at 12:37:26 AM UTC+9, [email protected] wrote:

We do something similar with the dask.distributed web interface: http://distributed.readthedocs.io/en/latest/web.html

Some extra things you might want to consider:

You could use the PyZMQ event loop to do non-blocking communication. This would play slightly more nicely with Bokeh, which also uses an event loop, although isn’t at all necessary if you’re ok with a very small amount of contention: https://pyzmq.readthedocs.io/en/latest/eventloop.html

As you’ve written it now every session will connect to your c++ server instead of having just one connection. For Dask we do a single connection in the server_lifecycle.py script that populates a globally accessible dictionary of current state and then every session (main.py) periodically checks this global state. This helps reduce communication overhead if you have multiple sessions.
If you’re looking for examples then Dask’s solution may be of value (although plenty of it is Dask specific):

https://github.com/dask/distributed/blob/master/distributed/bokeh/background/server_lifecycle.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/components.py

https://github.com/dask/distributed/blob/master/distributed/bokeh/status/main.py

For communicating string-like or json-like data I personally use msgpack, which is fairly fast and broadly accessible.

-matt

On Tuesday, November 8, 2016 at 10:09:31 AM UTC-5, [email protected] wrote:

Hi,
I am running a C++ process and I want to display its output with a bokeh server application.
Ideally, the C++ program would run during the day while sending the updates to the bokeh server. Then, the (bokeh) server sends the data to the client.
My options so far are as follows:
1 [Bad Idea] - After a number of events the c++ program fetch a buffer to a csv file. On the next periodic_callback the bokeh server sees the changes and updates the client.
2 [Ok Idea] - Using ZeroMQ, every second the Bokeh serves asks the c++ program if there is any update. If there is, bokeh server sends to the client.
The idea 2 looks like this:
Bokeh server:
source = some_data
def update_client(new_data):
source.stream(new_data, 100)
#Open ZeroMQ connection:
context = zmq.Context()
socket = context.socket(zmq.REQ)
ip = ‘localhost’
socket.connect(“tcp://”+ip+“:5555”)
def ask_mzq():
message = socket.recv()
if message is not None:
update_client(message)
#Add a figure to the app
plot = figure()
plot.circle(source=source, x=‘x’, y=‘y’)
curdoc().add_root(plot)
curdoc().add_periodic_callback(ask_mzq, 1000)
QUESTIONS:

Does bokeh has something pre-built for this sort of task?

What would be the most appropriate data structure to send string-like data from a buffer to Python.

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。

This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。