Running Bokeh behind Traefik

I’m trying to run several python panel apps, each in individual docker containers behind a Traefik proxy. I see the Nginx and Apache config examples in the docs here, but I was wondering if anyone had successfully done the same/similar using Traefik?

I’m no expert in any of Apache/Nginx nor Traefik.

The panel+bokeh (trace) logs look normal except for:

Module <module 'bokeh_app_2704603430aa4ca9905b7c746f61a016' from '/usr/src/app/xxx.py'> has extra unexpected referrers! This could indicate a serious memory leak. Extra referrers: [<cell at 0x7f839c363070: module object at 0x7f839c248c70>]

1 Like

@bill.sim It’s not clear from your question, are you saying that:

  • things are working, but you see that error message and are concerned about it? Or
  • things are not working at all?

The error message is due to some interactions between the way Panel uses Bokeh. I expect updating everything to latest versions might make it go away, given some recent work on both projects. But if not, this would be an issue to take up with the Panel project (it is maintained by a different group of people).

AS for Traefik I have never heard of it or have any experience with it so I can’t offer any concrete guidance. All I can say is that the proxy needs to be able to be configured to handle websockets. You can looks at the Bokeh server logs to see how far along the process gets (i.e. there should be an HTTP request and repsonse, and then subsequently, a websocket connection established) which might offer clues in addition to the proxy logs.

Hi @Bryan, thanks for getting back to me. Yes you’re right, my description was a bit poor.

The UI renders a couple of bits of static content (images) but fails to do anything else. Thinking further, I believe you are right, I need to take things up with the Panel project next.

(I hadn’t heard of Traefik until a year or so ago either, but now, having used it in front of our docker infrastructure, I’m super impressed, and their development progress seems good.)

Your notes about following the proxy needing to be configured to handle websockets may be helpful, I’ll follow that line next. I’ve changed the logging level in Panel from “error” to “trace”. There are bokeh log messages coming out there. But is there a more specific way of getting Bokeh logs? Or again is that more of a qustion for the Panel guys? These are the kind of logs I’m getting:

2021-10-31 21:17:19,113 Starting Bokeh server version 2.3.3 (running on Tornado 6.1)
2021-10-31 21:17:19,115 Host wildcard '*:9002' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
2021-10-31 21:17:19,118 User authentication hooks NOT provided (default user enabled)
2021-10-31 21:17:19,118 These host origins can connect to the websocket: ['*:9002']
2021-10-31 21:17:19,119 Patterns are:
2021-10-31 21:17:19,120 [('/oscillator_simulator_app/?',
2021-10-31 21:17:19,120 <class 'panel.io.server.DocHandler'>,
2021-10-31 21:17:19,120 {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7ff909c22790>,
2021-10-31 21:17:19,121 'bokeh_websocket_path': '/oscillator_simulator_app/ws'}),
2021-10-31 21:17:19,121 ('/oscillator_simulator_app/ws',
2021-10-31 21:17:19,121 <class 'bokeh.server.views.ws.WSHandler'>,
2021-10-31 21:17:19,121 {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7ff909c22790>,
2021-10-31 21:17:19,121 'bokeh_websocket_path': '/oscillator_simulator_app/ws',
2021-10-31 21:17:19,121 'compression_level': None,
2021-10-31 21:17:19,121 'mem_level': None}),
2021-10-31 21:17:19,121 ('/oscillator_simulator_app/metadata',
2021-10-31 21:17:19,121 <class 'bokeh.server.views.metadata_handler.MetadataHandler'>,
2021-10-31 21:17:19,121 {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7ff909c22790>,
2021-10-31 21:17:19,121 'bokeh_websocket_path': '/oscillator_simulator_app/ws'}),
2021-10-31 21:17:19,122 ('/oscillator_simulator_app/autoload.js',
2021-10-31 21:17:19,122 <class 'panel.io.server.AutoloadJsHandler'>,
2021-10-31 21:17:19,122 {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7ff909c22790>,
2021-10-31 21:17:19,122 'bokeh_websocket_path': '/oscillator_simulator_app/ws'}),
2021-10-31 21:17:19,122 ('/resources/(.*)',
2021-10-31 21:17:19,122 <class 'tornado.web.StaticFileHandler'>,
2021-10-31 21:17:19,122 {'path': '/usr/src/app/resources'}),
2021-10-31 21:17:19,122 ('/?',
2021-10-31 21:17:19,122 <class 'panel.io.server.RootHandler'>,
2021-10-31 21:17:19,122 {'applications': {'/oscillator_simulator_app': <bokeh.server.contexts.ApplicationContext object at 0x7ff909c22790>},
2021-10-31 21:17:19,122 'index': '/usr/local/lib/python3.8/site-packages/panel/io/../_templates/index.html',
2021-10-31 21:17:19,122 'prefix': '',
2021-10-31 21:17:19,122 'use_redirect': True}),
2021-10-31 21:17:19,122 ('/static/extensions/(.*)',
2021-10-31 21:17:19,122 <class 'bokeh.server.views.multi_root_static_handler.MultiRootStaticHandler'>,
2021-10-31 21:17:19,122 {'root': {'panel': '/usr/local/lib/python3.8/site-packages/panel/dist'}}),
2021-10-31 21:17:19,122 ('/static/(.*)',
2021-10-31 21:17:19,122 <class 'bokeh.server.views.static_handler.StaticHandler'>)]
2021-10-31 21:17:19,127 Bokeh app running at: http://localhost:9002/oscillator_simulator_app
2021-10-31 21:17:19,127 Starting Bokeh server with process id: 8
2021-10-31 21:17:34,143 Running stats log job
2021-10-31 21:17:34,143 [pid 8] 0 clients connected
2021-10-31 21:17:34,143 [pid 8] /oscillator_simulator_app has 0 sessions with 0 unused
2021-10-31 21:17:36,130 Running session cleanup job
2021-10-31 21:17:49,140 Running stats log job

snip, and then after hitting the site:

2021-10-31 21:28:19,138 [pid 8] /oscillator_simulator_app has 1 sessions with 1 unused
2021-10-31 21:28:22,132 Running session cleanup job
2021-10-31 21:28:25,131 Running keep alive job
2021-10-31 21:28:34,137 Running stats log job
2021-10-31 21:28:34,137 [pid 8] 0 clients connected
2021-10-31 21:28:34,137 [pid 8] /oscillator_simulator_app has 1 sessions with 1 unused
2021-10-31 21:28:39,133 Running session cleanup job
2021-10-31 21:28:39,133 Scheduling 1 sessions to discard
2021-10-31 21:28:39,133 Discarding session 'cPS1EOW1reHChbmF4sBsY4saYX1Z58f3Vvhrc0GHkTRk' last in use 30132.78970000008 milliseconds ago
2021-10-31 21:28:39,134 Deleting 1 modules for <bokeh.document.document.Document object at 0x7ff909c33370>
2021-10-31 21:28:39,157 Module <module 'bokeh_app_77dcdeae870d4677a6fa64127bc155db' from '/usr/src/app/oscillator_simulator_app.py'> has extra unexpected referrers! This could indicate a serious memory leak. Extra referrers: [<cell at 0x7ff909c33070: module object at 0x7ff90eb5e450>]
2021-10-31 21:28:39,233 Session 'cPS1EOW1reHChbmF4sBsY4saYX1Z58f3Vvhrc0GHkTRk' was successfully discarded
2021-10-31 21:28:49,138 Running stats log job

Ok, and I can also make sure I’m running the latest and greatest of Panel and Bokeh… so I’ll do that next.

If you don’t see log messages like

2021-10-31 18:09:13,792 WebSocket connection opened
2021-10-31 18:09:13,793 ServerConnection created

Then that means the incoming websocket upgrade is never making it to the Bokeh server (not making past the proxy, most likely)

@Bryan that got me on the right track. I managed to get traefik settings to work under some conditions (not with a path, not with https, but ok on a basic 80). So anyway, it’s a lack of my understanding of how traefik works. I’m going to leave it for now, and come back to it when I have more time. Thanks for your help.

1 Like