Bokeh server embedded in gunicorn/Flask hosted to cloud

This topic is cross-referenced to Holoviz Panel discourse topic 1124 because the panel server is the bokeh server, and the issue is really one of low-level server setup and communication.

I don’t have any experience with Heroku (maybe others do and can chime in) so all I can offer is speculation based on reported information.

What is the exact error? Is the timeout on the initial HTTP request, or on the webscocket upgrade? Is there any more information if you raise BOKEH_LOG_LEVEL to debug? What about the Flask/Bokeh server console logs, any messages there?

Hey @Bryan

Thanks for the quick reply.

The following info is all from a simple example very much like the bokeh Github flask_gunicorn_embed.py example.

The error that shows up in the JavaScript console approximately 5 min after the boilerplate bokeh example div/banner as in all the server embedding examples is …

And the info under that link is (with my actual application name removed/masked out) …

http://<masked-app-name>.herokuapp.com:43096/bkapp/autoload.js?bokeh-autoload-element=1001&bokeh-app-path=/bkapp&bokeh-absolute-url=http://<masked-app-name>.herokuapp.com:43096/bkapp

I need to figure out how to get the bokeh server logs on Heroku. The environment variable is set in my app workspace but I don’t see anything in the Heroku logs from bokeh. (This did work on when I tried on Google App Engine, but there the bokeh server just kept periodically reporting that it had zero clients connected.)

@Bryan

There are no JavaScript bokeh logs generated at the DEBUG log level. There are Python bokeh logs generated at the DEBUG log level.

The following are the logs from the Heroku app with my application name masked out from the URL. gunicorn has 3 worker processes in my configuration file.

2020-08-24T16:12:05.749354+00:00 heroku[web.1]: Restarting
2020-08-24T16:12:05.751681+00:00 heroku[web.1]: State changed from up to starting
2020-08-24T16:12:06.485783+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2020-08-24T16:12:06.509684+00:00 app[web.1]: [2020-08-24 16:12:06 +0000] [10] [INFO] Worker exiting (pid: 10)
2020-08-24T16:12:06.509724+00:00 app[web.1]: [2020-08-24 16:12:06 +0000] [11] [INFO] Worker exiting (pid: 11)
2020-08-24T16:12:06.509731+00:00 app[web.1]: [2020-08-24 16:12:06 +0000] [4] [INFO] Handling signal: term
2020-08-24T16:12:06.511342+00:00 app[web.1]: [2020-08-24 16:12:06 +0000] [9] [INFO] Worker exiting (pid: 9)
2020-08-24T16:12:06.810778+00:00 app[web.1]: [2020-08-24 16:12:06 +0000] [4] [INFO] Shutting down: Master
2020-08-24T16:12:06.958100+00:00 heroku[web.1]: Process exited with status 0
2020-08-24T16:12:18.519919+00:00 heroku[web.1]: Starting process with command `gunicorn --config gunicorn_config.py main:app`
2020-08-24T16:12:21.520369+00:00 app[web.1]: [2020-08-24 16:12:21 +0000] [4] [INFO] Starting gunicorn 20.0.4
2020-08-24T16:12:21.520922+00:00 app[web.1]: [2020-08-24 16:12:21 +0000] [4] [INFO] Listening at: http://0.0.0.0:19015 (4)
2020-08-24T16:12:21.521041+00:00 app[web.1]: [2020-08-24 16:12:21 +0000] [4] [INFO] Using worker: sync
2020-08-24T16:12:21.525772+00:00 app[web.1]: [2020-08-24 16:12:21 +0000] [9] [INFO] Booting worker with pid: 9
2020-08-24T16:12:21.545715+00:00 app[web.1]: [2020-08-24 16:12:21 +0000] [10] [INFO] Booting worker with pid: 10
2020-08-24T16:12:21.623643+00:00 app[web.1]: [2020-08-24 16:12:21 +0000] [11] [INFO] Booting worker with pid: 11
2020-08-24T16:12:22.098875+00:00 heroku[web.1]: State changed from starting to up
2020-08-24T16:12:26.420319+00:00 app[web.1]: INFO:bokeh.server.tornado:User authentication hooks NOT provided (default user enabled)
2020-08-24T16:12:26.420488+00:00 app[web.1]: INFO:bokeh.server.tornado:User authentication hooks NOT provided (default user enabled)
2020-08-24T16:12:26.420490+00:00 app[web.1]: DEBUG:bokeh.server.tornado:These host origins can connect to the websocket: ['<masked-app-name>.herokuapp.com', '0.0.0.0']
2020-08-24T16:12:26.420613+00:00 app[web.1]: DEBUG:bokeh.server.tornado:These host origins can connect to the websocket: ['<masked-app-name>.herokuapp.com', '0.0.0.0']
2020-08-24T16:12:26.420700+00:00 app[web.1]: DEBUG:bokeh.server.tornado:Patterns are:
2020-08-24T16:12:26.420782+00:00 app[web.1]: DEBUG:bokeh.server.tornado:Patterns are:
2020-08-24T16:12:26.422155+00:00 app[web.1]: DEBUG:bokeh.server.tornado:  [('/bkapp/?',
2020-08-24T16:12:26.422251+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.doc_handler.DocHandler'>,
2020-08-24T16:12:26.422341+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e5b0e5b0>,
2020-08-24T16:12:26.422436+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.422546+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/ws',
2020-08-24T16:12:26.422645+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.ws.WSHandler'>,
2020-08-24T16:12:26.422645+00:00 app[web.1]: DEBUG:bokeh.server.tornado:  [('/bkapp/?',
2020-08-24T16:12:26.422732+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e5b0e5b0>,
2020-08-24T16:12:26.422736+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.doc_handler.DocHandler'>,
2020-08-24T16:12:26.422821+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.422910+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/metadata',
2020-08-24T16:12:26.422959+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3a42280>,
2020-08-24T16:12:26.423024+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.metadata_handler.MetadataHandler'>,
2020-08-24T16:12:26.423079+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e5b0e5b0>,
2020-08-24T16:12:26.423152+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.423226+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/autoload.js',
2020-08-24T16:12:26.423301+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>,
2020-08-24T16:12:26.423329+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.423370+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e5b0e5b0>,
2020-08-24T16:12:26.423444+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.423501+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/ws',
2020-08-24T16:12:26.423502+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/?',
2020-08-24T16:12:26.423564+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.root_handler.RootHandler'>,
2020-08-24T16:12:26.423633+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.ws.WSHandler'>,
2020-08-24T16:12:26.423679+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'applications': {'/bkapp': <bokeh.server.contexts.ApplicationContext object at 0x7f30e5b0e5b0>},
2020-08-24T16:12:26.423735+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3a42280>,
2020-08-24T16:12:26.423793+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'index': None,
2020-08-24T16:12:26.423832+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.423904+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'prefix': '',
2020-08-24T16:12:26.423931+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/metadata',
2020-08-24T16:12:26.424013+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'use_redirect': True}),
2020-08-24T16:12:26.424017+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.metadata_handler.MetadataHandler'>,
2020-08-24T16:12:26.424102+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/static/extensions/(.*)',
2020-08-24T16:12:26.424105+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3a42280>,
2020-08-24T16:12:26.424192+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.multi_root_static_handler.MultiRootStaticHandler'>,
2020-08-24T16:12:26.424196+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.424300+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/autoload.js',
2020-08-24T16:12:26.424316+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'root': {}}),
2020-08-24T16:12:26.424401+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>,
2020-08-24T16:12:26.424451+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/static/(.*)',
2020-08-24T16:12:26.424501+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3a42280>,
2020-08-24T16:12:26.424584+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.static_handler.StaticHandler'>)]
2020-08-24T16:12:26.424639+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.424742+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/?',
2020-08-24T16:12:26.424840+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.root_handler.RootHandler'>,
2020-08-24T16:12:26.424941+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'applications': {'/bkapp': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3a42280>},
2020-08-24T16:12:26.425046+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'index': None,
2020-08-24T16:12:26.425139+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'prefix': '',
2020-08-24T16:12:26.425272+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'use_redirect': True}),
2020-08-24T16:12:26.425375+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/static/extensions/(.*)',
2020-08-24T16:12:26.425474+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.multi_root_static_handler.MultiRootStaticHandler'>,
2020-08-24T16:12:26.425571+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'root': {}}),
2020-08-24T16:12:26.425672+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/static/(.*)',
2020-08-24T16:12:26.425775+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.static_handler.StaticHandler'>)]
2020-08-24T16:12:26.428273+00:00 app[web.1]: INFO:bokeh.server.tornado:User authentication hooks NOT provided (default user enabled)
2020-08-24T16:12:26.428388+00:00 app[web.1]: DEBUG:bokeh.server.tornado:These host origins can connect to the websocket: ['<masked-app-name>.herokuapp.com', '0.0.0.0']
2020-08-24T16:12:26.428520+00:00 app[web.1]: DEBUG:bokeh.server.tornado:Patterns are:
2020-08-24T16:12:26.429827+00:00 app[web.1]: DEBUG:bokeh.server.tornado:  [('/bkapp/?',
2020-08-24T16:12:26.429923+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.doc_handler.DocHandler'>,
2020-08-24T16:12:26.430018+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3bfac10>,
2020-08-24T16:12:26.430100+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.430204+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/ws',
2020-08-24T16:12:26.430306+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.ws.WSHandler'>,
2020-08-24T16:12:26.430407+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3bfac10>,
2020-08-24T16:12:26.430508+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.430615+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/metadata',
2020-08-24T16:12:26.430718+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.metadata_handler.MetadataHandler'>,
2020-08-24T16:12:26.430817+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3bfac10>,
2020-08-24T16:12:26.430916+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.431014+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/bkapp/autoload.js',
2020-08-24T16:12:26.431114+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>,
2020-08-24T16:12:26.431209+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3bfac10>,
2020-08-24T16:12:26.431304+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'bokeh_websocket_path': '/bkapp/ws'}),
2020-08-24T16:12:26.431402+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/?',
2020-08-24T16:12:26.431497+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.root_handler.RootHandler'>,
2020-08-24T16:12:26.431591+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'applications': {'/bkapp': <bokeh.server.contexts.ApplicationContext object at 0x7f30e3bfac10>},
2020-08-24T16:12:26.431685+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'index': None,
2020-08-24T16:12:26.431778+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'prefix': '',
2020-08-24T16:12:26.431872+00:00 app[web.1]: DEBUG:bokeh.server.tornado:     'use_redirect': True}),
2020-08-24T16:12:26.431964+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/static/extensions/(.*)',
2020-08-24T16:12:26.432070+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.multi_root_static_handler.MultiRootStaticHandler'>,
2020-08-24T16:12:26.432164+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    {'root': {}}),
2020-08-24T16:12:26.432260+00:00 app[web.1]: DEBUG:bokeh.server.tornado:   ('/static/(.*)',
2020-08-24T16:12:26.432355+00:00 app[web.1]: DEBUG:bokeh.server.tornado:    <class 'bokeh.server.views.static_handler.StaticHandler'>)]
2020-08-24T16:12:41.445517+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10] 0 clients connected
2020-08-24T16:12:41.445585+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:12:41.449455+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9] 0 clients connected
2020-08-24T16:12:41.449456+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11] 0 clients connected
2020-08-24T16:12:41.449542+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:12:41.449546+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:12:55.085552+00:00 heroku[router]: at=info method=GET path="/" host=<masked-app-name>.herokuapp.com request_id=1ba438cf-e580-4a93-a175-1dff1a81d79f fwd="71.206.168.180" dyno=web.1 connect=1ms service=13ms status=200 bytes=1185 protocol=http
2020-08-24T16:12:55.087722+00:00 app[web.1]: 10.30.61.229 - - [24/Aug/2020:16:12:55 +0000] "GET / HTTP/1.1" 200 1023 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
2020-08-24T16:12:56.445503+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10] 0 clients connected
2020-08-24T16:12:56.445514+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9] 0 clients connected
2020-08-24T16:12:56.445566+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:12:56.445593+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:12:56.449434+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11] 0 clients connected
2020-08-24T16:12:56.449506+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:13:11.441119+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10] 0 clients connected
2020-08-24T16:13:11.441206+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:13:11.441568+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9] 0 clients connected
2020-08-24T16:13:11.441661+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:13:11.443266+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11] 0 clients connected
2020-08-24T16:13:11.443399+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:13:26.441512+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11] 0 clients connected
2020-08-24T16:13:26.441529+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10] 0 clients connected
2020-08-24T16:13:26.441576+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 11]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:13:26.441620+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 10]   /bkapp has 0 sessions with 0 unused
2020-08-24T16:13:26.442057+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9] 0 clients connected
2020-08-24T16:13:26.442185+00:00 app[web.1]: DEBUG:bokeh.server.tornado:[pid 9]   /bkapp has 0 sessions with 0 unused

Looking at that there does not ever seem to be any Websocket connection hitting the server. Perhaps there is some configuration necessary on Heroku to allow WS connections to pass? (There definitely is on AWS Elastic Beanstalk, for instance.)

@Bryan

Thanks.

I know that Heroku supports websocket connections and in some circumstances there don’t seem to be any special configuration steps, although I’ve only seen there documentation referencing use of the Flask-sockets extension.

The panel server documentation also shows specific steps to deploy a server on Heroku and by extension this should transfer directly to deploying the bokeh server. The challenge is when this is combined with a Flask web infrastructure b/c Flask is being used to serve additional content, manage user authenticated roles for specific routes, etc.

I have a support ticket open with Heroku to understand what is and what is not possible.

Perhaps it is just my recent attention to this topic, but it seems to me like a number of users are at the stage of being able to deploy to self-managed servers running on Linux or OSX but hosting to the cloud platforms as a service like Heroku, Google App Engine, or Azure is the next frontier.

@Bryan

My understanding is that current Heroku versions enable websockets by default although they dictate which port you’re allowed to use.

I have been able to extend my debugging app based on Bokeh’s Flask gunicorn embed example to use nginx via the heroku-community/nginx buildpack. And I can hit the index route just fine, but I get errors when it goes to retrieve the server document. Specifically, the JavaScript console shows a net:ERR_CONNECTION_REFUSED error.

As before, there are Python messages with BOKEH_PY_LOG_LEVEL equals debug, basically say there are no clients connected. And there are no JavaScript messages with BOKEH_LOG_LEVEL equals debug, which makes sense b/c the connection is being refused preemptively.

The following is the NGINX config file, which is simply the default for the Heroku-community buildpack. It uses a Unix socket, and my gunicorn configuration file properly binds to that, so I get to the boilerplate banner familiar in the bokeh GitHub examples.

Any ideas on what needs to be done with the nginx configuration and/or setup of the Bokeh server to get at the websocket? (In the screenshot above, I am using the default localhost address 127.0.0.1, but I’ve tried other things here even briefly looking at Tornado’s bind_unix_socket methods, but to no avail.)

NGINX configuration

daemon off;
#Heroku dynos have at least 4 cores.
worker_processes <%= ENV['NGINX_WORKERS'] || 4 %>;

events {
	use epoll;
	accept_mutex on;
	worker_connections <%= ENV['NGINX_WORKER_CONNECTIONS'] || 1024 %>;
}

http {
        gzip on;
        gzip_comp_level 2;
        gzip_min_length 512;

	server_tokens off;

	log_format l2met 'measure#nginx.service=$request_time request_id=$http_x_request_id';
	access_log <%= ENV['NGINX_ACCESS_LOG_PATH'] || 'logs/nginx/access.log' %> l2met;
	error_log <%= ENV['NGINX_ERROR_LOG_PATH'] || 'logs/nginx/error.log' %>;

	include mime.types;
	default_type application/octet-stream;
	sendfile on;

	#Must read the body in 5 seconds.
	client_body_timeout 5;

	upstream app_server {
		server unix:/tmp/nginx.socket fail_timeout=0;
 	}

	server {
		listen <%= ENV["PORT"] %>;
		server_name _;
		keepalive_timeout 5;

		location / {
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header Host $http_host;
			proxy_redirect off;
			proxy_pass http://app_server;
		}
	}
}

It’s been a long time since I’ve written this config for NGINX + Tornado, but it works.

http {
    upstream app_server {
        server 127.0.0.1:55775;
    }

    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    server {
        location / {
            proxy_pass http://app_server;
            proxy_pass_header Server;
            proxy_http_version 1.1;
            proxy_read_timeout 60h;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;

            proxy_redirect off;
            proxy_set_header Host $host:$server_port;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
        }
    }
}

I’ve removed everything that doesn’t seem to be related to WebSockets. Hopefully I didn’t remove something vital.

As far as I recall, the main points of interest are that map directive and those proxy_set_header directives for the Upgrade and Connection headers. I think proxy_read_timeout is also required because WebSocket connections are usually long-living.

1 Like

@p-himik

Thanks for the help. I sincerely appreciate it.

Everything works well with nginx+gunicorn+Flask+bokeh in my scenario on a self-managed computer but not a platform as a service like Heroku. I think fundamentally it comes down to my not fully appreciating the differences / restrictions with what they refer to as dynos for hosting an app and something more concrete to me like a computer running Linux.

Perhaps the difference is that I am trying to run bokeh server as a separate process on localhost and this simply is not compatible with the concept of a dyno.

I’ve seen a suggestion you made helping others in the past on similar topics to wrap the Flask app in Tornado. Coupled with your response above, I think this will allow me to embed Bokeh server as a library and attach to the existing Tornado IOLoop?

Let me know if I understand correctly, and I will explore that route in the meantime.

Thanks again so much for the help.

Hmm, I think you’re right. At some point, the config above was used to run a Django app on Heroku. But both the Django and the Tornado apps were using a WebSocket connection via the same port as the main app. If you embed Bokeh as a library, you will be able to use the same port as well. After all, Heroku doesn’t allow you to use any port but the one it provides.

@p-himik @Bryan

I am trying a different architecture currently using separate Heroku dynos for the Flask app and the bokeh server.

In the following description of the problem currently faced, <masked-app-name> is a placeholder for my actual application’s name.

I can navigate to the server just fine via a web browser, e.g.

https://<masked-app-name>.herokuapp.com/

However, when I try to embed it from Flask via pull_session() within a client context manager, it fails.

Here’s the Flask app code.

from flask import Flask, render_template

from bokeh.client import pull_session
from bokeh.embed import server_session

PANEL_URL = "https://<masked-app-name>.herokuapp.com"

app = Flask(__name__)

@app.route('/', methods=['GET'])
def bkapp_page():
    with pull_session(url=PANEL_URL) as session:
        script = server_session(session_id=session.id, url=PANEL_URL)
        return render_template("embed.html", script=script, template="Flask")

The web browser for the Flask app shows 500 Server Internal Error, which is also reflected in the JavaScript console.

And the Heroku log excerpt shows the following error stack trace from Bokeh client session logic …

2020-08-28T20:07:30.436217+00:00 app[web.1]: File "/app/main.py", line 12, in bkapp_page

2020-08-28T20:07:30.436217+00:00 app[web.1]: with pull_session(url=PANEL_URL) as session:

2020-08-28T20:07:30.436217+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.8/site-packages/bokeh/client/session.py", line 120, in pull_session

2020-08-28T20:07:30.436218+00:00 app[web.1]: session.pull()

2020-08-28T20:07:30.436218+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.8/site-packages/bokeh/client/session.py", line 381, in pull

2020-08-28T20:07:30.436218+00:00 app[web.1]: self.check_connection_errors()

2020-08-28T20:07:30.436220+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.8/site-packages/bokeh/client/session.py", line 367, in check_connection_errors

2020-08-28T20:07:30.436220+00:00 app[web.1]: raise OSError(f"Check your application path! The given Path is not valid: {self.url}")

2020-08-28T20:07:30.436220+00:00 app[web.1]: OSError: Check your application path! The given Path is not valid: ws://<masked-app-name>.herokuapp.com/ws

Is there a way to work around the ws:// protocol substitution in the URL to make it behave as if a user is navigating directly to the server via https?

It’s not about the protocol, it’s about the URL as a whole:

if self.error_code == 404:
    raise OSError(f"Check your application path! The given Path is not valid: {self.url}")

Note that 404.

Alas, I cannot add anything else without an MRE.

I see. It appeared to be related to the protocol and address substitution, b/c when I enter that as the URL manually in a browser, Google Chrome shows the following.

For others that have application architectures similar that embed a bokeh server in a gunicorn/Flask framework and want to deploy it to a platform as a service / cloud service, this can be done in Heroku using two dynos. After exhaustively exploring different mechanisms and helpful exchanges with the Heroku support team, getting everything to work in one dyno is not possible.

DETAILS

Flask-app which embeds the bokeh server on one dyno:

Create a python Flask app, e.g. a simple one as follows, and use the Heroku Procfile to run it via gunicorn, which is well documented on Heroku’s site.

from flask import Flask, render_template

from bokeh.client import pull_session
from bokeh.embed import server_session

BOKEH_URL = "https://<bokeh-app-name>.herokuapp.com/<bokeh-server-name>"

app = Flask(__name__)

@app.route('/', methods=['GET'])
def bkapp_page():
    with pull_session(url=PANEL_URL) as session:
        script = server_session(session_id=session.id, url=BOKEH_URL)
        return render_template("embed.html", script=script, template="Flask")

Bokeh server app which runs the bokeh server on second dyno:

Implement this as usual and create a Heroku Procfile that runs the server, e.g.

web: bokeh serve --address="0.0.0.0" --port=$PORT <bokeh-server-name> --allow-websocket-origin=<flask-app-name>.herokuapp.com

NB: The port argument is important b/c Heroku assigns the port that is used; you cannot choose it yourself.

NB: The quantities in <…> above should reflect the app names for your Heroku apps and the name of your bokeh server following the bokeh conventions if it is in single-file or directory format. Caveat, that only the single module bokeh-server-name.py case has been tested in determining the viability of the approach.

Thanks to @p-himik and @Bryan for the valuable help getting this to work!