I am using the following code to push a plot (actually, multiple plots in a single doc) to the user’s browser:
class SingleShotServer:
def __init__(self, log, plot, template=None, title=None, template_vars=None):
self.__log = log
self.__plot = plot
self.__template = template
self.__title = title
self.__template_vars = {} if template_vars is None else template_vars
self.__server = Server(self.__modify_doc)
self.__server.start()
self.__log.info('Opening Bokeh application on http://localhost:5006/')
self.__server.io_loop.add_callback(self.__server.show, "/")
self.__server.io_loop.call_later(5, self.__stop)
self.__server.io_loop.start()
def __modify_doc(self, doc):
doc.add_root(self.__plot)
doc.title = self.__title
# this extends bokeh.core.templates.FILE - see code in bokeh.embed.elements
doc.template = self.__template
doc.template_variables.update(self.__template_vars)
def __stop(self):
self.__log.info('Stopping server')
self.__server.io_loop.stop()
self.__server.stop()
self.__server.unlisten()
``
and it works nicely. Everything is good in the world - templating, variables, display. Both log messages are displayed.
BUT:
1 - It reconfigures Python’s logging so that I start to see a lot of output on stdout (or stderr). This is a disaster as my calling code is running inside an curses-like environment; the logging destroys my TUI.
2 - If I call the code a second time it fails with an error “Models must be owned by a single document XXX is already in a doc” where XXX seems to vary, but is what seems to be some kind of tool like zoom, etc. Sorry for the lack of detail here - I can’t cut + paste because the error message is appearing only in a terminal expecting curses (see above) - there is nothing in the logs I actually configured for the app (this is all related to point 1 above). The plots are “freshly” generated each time - no explicit state is re-used, that I can see.
So I guess I’d like to know how to stop Bokeh / Tornado taking over Python’s logs, and what I need to do in addition to the above to release resources on server shutdown?
Hopefully examining that will suggest what you might need to do to avoid issues with your own codebase. If there is something that would make life easier for certain use cases, we would be happy to consider a PR. We just want to ensure that we continue to force deprecation warnings to be emitted by default (it still boggles my mind that the Python itself hobbles deprecation warnings by silencing them).
For the other issue: The likely explanation is that you are creating a "global" Bokeh model in some module you are importing. Due to the way Python itself caches module imports, this means only one instance of the Bokeh model is ever created, and then if multiple sessions are created on the server, they all try to use this one instance. That is not permissible, as the error states -- every session/document must have its own unique set of Bokeh models (for a variety of very good reasons). So you need to avoid creating any Bokeh model objects at module scope outside the "main" app module.
and it works nicely. Everything is good in the world - templating, variables, display. Both log messages are displayed.
BUT:
1 - It reconfigures Python's logging so that I start to see a lot of output on stdout (or stderr). This is a disaster as my calling code is running inside an curses-like environment; the logging destroys my TUI.
2 - If I call the code a second time it fails with an error "Models must be owned by a single document XXX is already in a doc" where XXX seems to vary, but is what seems to be some kind of tool like zoom, etc. Sorry for the lack of detail here - I can't cut + paste because the error message is appearing only in a terminal expecting curses (see above) - there is nothing in the logs I actually configured for the app (this is all related to point 1 above). The plots are "freshly" generated each time - no explicit state is re-used, that I can see.
So I guess I'd like to know how to stop Bokeh / Tornado taking over Python's logs, and what I need to do in addition to the above to release resources on server shutdown?
Some debugging shows that the problem is (at least sometimes) the
WMTSTileSource associated with the STAMEN_TONER map tiles, which seem
to be cached in bokeh.tile_providers.py
That WMTSTileSource's self._document is (I guess) still set to the
previous document. This is consistent with what you say, I think,
implying that I should not reuse this tile source.
But the tile source is cached by Bokeh itself. How do I avoid this?
Do I need to execute
myself (what Bokeh has cached)? I guess that will solve this,
although I have not tried it. But it seems very odd to me that this
is necessary. Is this really as expected?
I suspect I am confused, sorry...
(Haven't yet looked at logs, but thanks for the pointer there)
Andrew
···
On Thu, Dec 27, 2018 at 10:10:19AM -0800, Bryan Van de ven wrote:
For the other issue: The likely explanation is that you are creating
a "global" Bokeh model in some module you are importing. Due to the
way Python itself caches module imports, this means only one
instance of the Bokeh model is ever created, and then if multiple
sessions are created on the server, they all try to use this one
instance. That is not permissible, as the error states -- every
session/document must have its own unique set of Bokeh models (for a
variety of very good reasons). So you need to avoid creating any
Bokeh model objects at module scope outside the "main" app module.
You must be using a somewhat old version of Bokeh? This was addressed in Bokeh 0.12.10 so that accessing any attribute from tile_providers.py now always returns a brand new instance (precisely to avoid the situation you describe). You can see how things are implemented today:
On Thu, Dec 27, 2018 at 10:10:19AM -0800, Bryan Van de ven wrote:
For the other issue: The likely explanation is that you are creating
a "global" Bokeh model in some module you are importing. Due to the
way Python itself caches module imports, this means only one
instance of the Bokeh model is ever created, and then if multiple
sessions are created on the server, they all try to use this one
instance. That is not permissible, as the error states -- every
session/document must have its own unique set of Bokeh models (for a
variety of very good reasons). So you need to avoid creating any
Bokeh model objects at module scope outside the "main" app module.
I don't fully understand this, sorry.
Some debugging shows that the problem is (at least sometimes) the
WMTSTileSource associated with the STAMEN_TONER map tiles, which seem
to be cached in bokeh.tile_providers.py
That WMTSTileSource's self._document is (I guess) still set to the
previous document. This is consistent with what you say, I think,
implying that I should not reuse this tile source.
But the tile source is cached by Bokeh itself. How do I avoid this?
Do I need to execute
myself (what Bokeh has cached)? I guess that will solve this,
although I have not tried it. But it seems very odd to me that this
is necessary. Is this really as expected?
I suspect I am confused, sorry...
(Haven't yet looked at logs, but thanks for the pointer there)
Ahhh. No, I am stupid. I get how that should work now. Thanks, Andrew
···
On Thu, Dec 27, 2018 at 12:52:20PM -0800, Bryan Van de ven wrote:
Hi,
You must be using a somewhat old version of Bokeh? This was addressed in Bokeh 0.12.10 so that accessing any attribute from tile_providers.py now always returns a brand new instance (precisely to avoid the situation you describe). You can see how things are implemented today:
You can either update to a newer version of Bokeh, or if that is not an option, create your own tile source instances manually, as you suggest.
Thanks,
Bryan
> On Dec 27, 2018, at 12:44, andrew cooke <[email protected]> wrote:
>
> On Thu, Dec 27, 2018 at 10:10:19AM -0800, Bryan Van de ven wrote:
>> For the other issue: The likely explanation is that you are creating
>> a "global" Bokeh model in some module you are importing. Due to the
>> way Python itself caches module imports, this means only one
>> instance of the Bokeh model is ever created, and then if multiple
>> sessions are created on the server, they all try to use this one
>> instance. That is not permissible, as the error states -- every
>> session/document must have its own unique set of Bokeh models (for a
>> variety of very good reasons). So you need to avoid creating any
>> Bokeh model objects at module scope outside the "main" app module.
>
> I don't fully understand this, sorry.
>
> Some debugging shows that the problem is (at least sometimes) the
> WMTSTileSource associated with the STAMEN_TONER map tiles, which seem
> to be cached in bokeh.tile_providers.py
>
> That WMTSTileSource's self._document is (I guess) still set to the
> previous document. This is consistent with what you say, I think,
> implying that I should not reuse this tile source.
>
> But the tile source is cached by Bokeh itself. How do I avoid this?
> Do I need to execute
>
> return WMTSTileSource(
> url='http://tile.stamen.com/terrain/{Z}/{X}/{Y}@2x.png',
> attribution=self._STAMEN_ATTRIBUTION % '<a href="https://creativecommons.org/licenses/by-sa/3.0">CC BY SA</a>'
> )
>
> myself (what Bokeh has cached)? I guess that will solve this,
> although I have not tried it. But it seems very odd to me that this
> is necessary. Is this really as expected?
>
> I suspect I am confused, sorry...
>
> (Haven't yet looked at logs, but thanks for the pointer there)
>
> Andrew
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/20181227204454.imsdxxrfj3menvaq%40acooke.org\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.
Not at all. It is a complicated cross-runtime library, and the cross-runtime part means that the library is occasionally opinionated (if not even "un-pythonic") in at least a few ways. Glad you have things working,
Ahhh. No, I am stupid. I get how that should work now. Thanks, Andrew
On Thu, Dec 27, 2018 at 12:52:20PM -0800, Bryan Van de ven wrote:
Hi,
You must be using a somewhat old version of Bokeh? This was addressed in Bokeh 0.12.10 so that accessing any attribute from tile_providers.py now always returns a brand new instance (precisely to avoid the situation you describe). You can see how things are implemented today:
On Thu, Dec 27, 2018 at 10:10:19AM -0800, Bryan Van de ven wrote:
For the other issue: The likely explanation is that you are creating
a "global" Bokeh model in some module you are importing. Due to the
way Python itself caches module imports, this means only one
instance of the Bokeh model is ever created, and then if multiple
sessions are created on the server, they all try to use this one
instance. That is not permissible, as the error states -- every
session/document must have its own unique set of Bokeh models (for a
variety of very good reasons). So you need to avoid creating any
Bokeh model objects at module scope outside the "main" app module.
I don't fully understand this, sorry.
Some debugging shows that the problem is (at least sometimes) the
WMTSTileSource associated with the STAMEN_TONER map tiles, which seem
to be cached in bokeh.tile_providers.py
That WMTSTileSource's self._document is (I guess) still set to the
previous document. This is consistent with what you say, I think,
implying that I should not reuse this tile source.
But the tile source is cached by Bokeh itself. How do I avoid this?
Do I need to execute
myself (what Bokeh has cached)? I guess that will solve this,
although I have not tried it. But it seems very odd to me that this
is necessary. Is this really as expected?
I suspect I am confused, sorry...
(Haven't yet looked at logs, but thanks for the pointer there)
for the record, in case this is useful to anyone else, doing the
following as soon as possible (eg at the top of the __init__ of the
module that python is going to run):
from logging import getLogger, NullHandler
getLogger('bokeh').addHandler(NullHandler())
getLogger('tornado').addHandler(NullHandler())
signals to bokeh/tornado that you will be taking care of logging.
then you can configure logging in the normal way within your app.
cheers,
andrew
···
On Thu, Dec 27, 2018 at 10:10:19AM -0800, Bryan Van de ven wrote:
Hi,
I'll be honest and say that Python logging configuration (beyond trivial usage) is not an area of expertise for me. you can see what Bokeh does here:
Hopefully examining that will suggest what you might need to do to
avoid issues with your own codebase. If there is something that
would make life easier for certain use cases, we would be happy to
consider a PR. We just want to ensure that we continue to force
deprecation warnings to be emitted by default (it still boggles my
mind that the Python itself hobbles deprecation warnings by
silencing them).
So that makes sense. Would an environment variable to control things would be sufficient? It would be fairly simple to condition that import on the value of a variable defined in bokeh.settings. Alternatively (or perhaps in addition) we could consider and API that "undoes" the Bokeh logging, basically as you have done below. I am happy to help provide guidance and information if you are interested in working up a contribution.
Hopefully examining that will suggest what you might need to do to
avoid issues with your own codebase. If there is something that
would make life easier for certain use cases, we would be happy to
consider a PR. We just want to ensure that we continue to force
deprecation warnings to be emitted by default (it still boggles my
mind that the Python itself hobbles deprecation warnings by
silencing them).
for the record, in case this is useful to anyone else, doing the
following as soon as possible (eg at the top of the __init__ of the
module that python is going to run):
from logging import getLogger, NullHandler
getLogger('bokeh').addHandler(NullHandler())
getLogger('tornado').addHandler(NullHandler())
signals to bokeh/tornado that you will be taking care of logging.
then you can configure logging in the normal way within your app.
i didn't think anything was necessary tbh. what i described is what
is tested for in line 40 of logconfig.py and line 495 of tornado's
ioloop.py
from those two i assumed (and i think it's reasonable) that libraries
in general check if handlers have already been configured and, if so,
do nothing. what i posted simply sets null handlers so that those
tests trigger correctly (this needs to be done early if libraries
configure logging during import).
so i thought this was probably a standard approach that i just
discovered. it makes sense, is easy to implement, and already works.
i just sent the email to educate others...
cheers,
andrew
···
On Thu, Dec 27, 2018 at 03:06:14PM -0800, Bryan Van de ven wrote:
Hi Andrew,
Thanks for sharing the solution. Bokeh does import util.logconfig at its own top level:
So that makes sense. Would an environment variable to control things would be sufficient? It would be fairly simple to condition that import on the value of a variable defined in bokeh.settings. Alternatively (or perhaps in addition) we could consider and API that "undoes" the Bokeh logging, basically as you have done below. I am happy to help provide guidance and information if you are interested in working up a contribution.
Thanks,
Bryan
> On Dec 27, 2018, at 14:17, andrew cooke <[email protected]> wrote:
>
> On Thu, Dec 27, 2018 at 10:10:19AM -0800, Bryan Van de ven wrote:
>> Hi,
>>
>> I'll be honest and say that Python logging configuration (beyond trivial usage) is not an area of expertise for me. you can see what Bokeh does here:
>>
>> https://github.com/bokeh/bokeh/blob/master/bokeh/util/logconfig.py
>>
>> Hopefully examining that will suggest what you might need to do to
>> avoid issues with your own codebase. If there is something that
>> would make life easier for certain use cases, we would be happy to
>> consider a PR. We just want to ensure that we continue to force
>> deprecation warnings to be emitted by default (it still boggles my
>> mind that the Python itself hobbles deprecation warnings by
>> silencing them).
>
> for the record, in case this is useful to anyone else, doing the
> following as soon as possible (eg at the top of the __init__ of the
> module that python is going to run):
>
> from logging import getLogger, NullHandler
> getLogger('bokeh').addHandler(NullHandler())
> getLogger('tornado').addHandler(NullHandler())
>
> signals to bokeh/tornado that you will be taking care of logging.
>
> then you can configure logging in the normal way within your app.
>
> cheers,
> andrew
>
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/20181227221729.t2cf7c3trahjjda4%40acooke.org\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.
