Websocket connection failed when depoly Bokeh server on AWS EC2

I got some troubles on deploying Bokeh server on AWS EC2.
(same Question also posted in here)
I am using Apache 2.4 and Windows server 2019 on my EC2 instance, and I used Bokeh build-in Tornado http-server to run bokeh successfully in localhost:5006 on ec2.
The application source code is shown as following:

 # -*- coding: utf-8 -*-
    from bokeh.io import curdoc
    from bokeh.layouts import column
    from bokeh.models import Select,  Div
    
    ticker1 = Select(title="test_ID:",value = "choose test_ID:", options = ["test001"], width=250)
    welcome_div = Div(text='<b>Hello World</b>',width=1000, height=20, style={'font-size': '150%'})
    
    layout = column(ticker1,welcome_div)
    curdoc().add_root(layout)
    curdoc().title = "test"

And I ran it with command:

` bokeh serve --show .\test1.py --allow-websocket-origin="*"`

Then I use Apache as to proxy the localhost:5006 to port80, httpd-vhosts.conf is as following:

<VirtualHost *:80>
        ServerName localhost

        CustomLog "C:/DeployTest/access_log" combined2
        ErrorLog "C:/DeployTest/error_log"
        SSLProxyEngine on
        ProxyPreserveHost On
        ProxyPass /test1/ws ws://localhost:5006/test1
        ProxyPassReverse /test1/ws ws://localhost:5006/test1

        ProxyPass /test1 http://localhost:5006/test1/
        ProxyPassReverse /test1 http://localhost:5006/test1/

        <Directory />
            Require all granted
            Options -Indexes
        </Directory>

        Alias "/static" "C:/Users/Administrator/AppData/Local/Programs/Python/Python39/lib/sitepackages/bokeh/server/static"
        <Directory "C:/Users/Administrator/AppData/Local/Programs/Python/Python39/lib/sitepackages/bokeh/server/static">
            Require all granted
            Options +Indexes
        </Directory>
    </VirtualHost>

And in httpd.conf, I also have loaded all necessary module which are as following

LoadModule proxy_module modules/mod_proxy.so
    LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
    LoadModule proxy_http_module modules/mod_proxy_http.so
    LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

However, when I tried to access from another computer by chrome, I get blank page except the title, which indicated that http connection may worked well, but websocket did not. I also checked chrome console and I get error which may prove my thinking:

Finally, I also checked the access log of Apache:

 x.xxx.xxx.xxx - - [23/Jun/2021:10:24:28 +0900] "GET /test1 HTTP/1.1" 200 3779 "-" "Amazon CloudFront" "TdW2sVmIwd3dfjU"
    xx.xxx.xxx.xxx - - [23/Jun/2021:10:24:29 +0900] "GET /test1/ws HTTP/1.1" 200 3803 "-" "Amazon CloudFront" "TdW2sVmIwd3dfjU"

Any comment and help would be appreciated.

Can you access the console logs of the Bokeh server. It seems like the ws upgrade is making it to Apache, so the next question is whether it is making it to the Bokeh server or not (and if it is, was it rejected for some reason? that would also be in those logs)

Also @X_L, if you must cross-post questions in different forums (e.g. StackOverflow), we appreciate if you add cross-links everywhere, so that future users can always get to an answer, regardless of where they find find the question.

Thanks,Bryan.
I ran the bokeh server with --log–level debug, and I got the logs as following:

PS C:\DeployTest> bokeh serve .\test1.py --allow-websocket-origin="*" --log-level debug
2021-06-24 08:16:58,304 Starting Bokeh server version 2.3.2 (running on Tornado 6.1)
2021-06-24 08:16:58,306 Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
2021-06-24 08:16:58,309 User authentication hooks NOT provided (default user enabled)
2021-06-24 08:16:58,309 These host origins can connect to the websocket: ['*']
2021-06-24 08:16:58,310 Patterns are:
2021-06-24 08:16:58,311   [('/test1/?',
2021-06-24 08:16:58,311     <class 'bokeh.server.views.doc_handler.DocHandler'>,
2021-06-24 08:16:58,311     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000017350ACAD00>,
2021-06-24 08:16:58,312      'bokeh_websocket_path': '/test1/ws'}),
2021-06-24 08:16:58,313    ('/test1/ws',
2021-06-24 08:16:58,313     <class 'bokeh.server.views.ws.WSHandler'>,
2021-06-24 08:16:58,313     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000017350ACAD00>,
2021-06-24 08:16:58,314      'bokeh_websocket_path': '/test1/ws',
2021-06-24 08:16:58,315      'compression_level': None,
2021-06-24 08:16:58,315      'mem_level': None}),
2021-06-24 08:16:58,315    ('/test1/metadata',
2021-06-24 08:16:58,316     <class 'bokeh.server.views.metadata_handler.MetadataHandler'>,
2021-06-24 08:16:58,316     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000017350ACAD00>,
2021-06-24 08:16:58,321      'bokeh_websocket_path': '/test1/ws'}),
2021-06-24 08:16:58,321    ('/test1/autoload.js',
2021-06-24 08:16:58,322     <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>,
2021-06-24 08:16:58,322     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000017350ACAD00>,
2021-06-24 08:16:58,323      'bokeh_websocket_path': '/test1/ws'}),
2021-06-24 08:16:58,323    ('/?',
2021-06-24 08:16:58,324     <class 'bokeh.server.views.root_handler.RootHandler'>,
2021-06-24 08:16:58,324     {'applications': {'/test1': <bokeh.server.contexts.ApplicationContext object at 0x0000017350ACAD00>},
2021-06-24 08:16:58,325      'index': None,
2021-06-24 08:16:58,325      'prefix': '',
2021-06-24 08:16:58,325      'use_redirect': True}),
2021-06-24 08:16:58,325    ('/static/extensions/(.*)',
2021-06-24 08:16:58,325     <class 'bokeh.server.views.multi_root_static_handler.MultiRootStaticHandler'>,
2021-06-24 08:16:58,326     {'root': {}}),
2021-06-24 08:16:58,326    ('/static/(.*)',
2021-06-24 08:16:58,327     <class 'bokeh.server.views.static_handler.StaticHandler'>)]
2021-06-24 08:16:58,332 Bokeh app running at: http://localhost:5006/test1
2021-06-24 08:16:58,332 Starting Bokeh server with process id: 3612
2021-06-24 08:17:13,347 [pid 3612] 0 clients connected
2021-06-24 08:17:13,347 [pid 3612]   /test1 has 14 sessions with 14 unused
2021-06-24 08:17:28,343 [pid 3612] 0 clients connected
2021-06-24 08:18:05,588 [pid 3612]   /test1 has 14 sessions with 14 unused
2021-06-24 08:18:05,588 Scheduling 14 sessions to discard
2021-06-24 08:18:05,588 Discarding session 'VgYdmbP9gC2DC1F6om1vduyTjbfDJamD5WVeQDIK4bKb' last in use 63157.0 milliseconds ago
2021-06-24 08:18:05,588 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350AFD100>
2021-06-24 08:18:05,620 Discarding session 'P8gmEsjwjkeLi0RMbiBNjPaJK5tAqbyrilwddqAOR6tn' last in use 62969.0 milliseconds ago
2021-06-24 08:18:05,620 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B3B940>
2021-06-24 08:18:05,651 Discarding session '8y5utrQu1p5WwEvUJhpUhYo3iBcUu5yJ8VqxyArpXjGp' last in use 57922.0 milliseconds ago
2021-06-24 08:18:05,651 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B3B250>
2021-06-24 08:18:05,667 Discarding session 'kFfyDxgBSjVJO2zr0VCk5E3IE94eVtrhlRbsNa0daqIx' last in use 57703.0 milliseconds ago
2021-06-24 08:18:05,667 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B22F40>
2021-06-24 08:18:05,682 Discarding session 'TTEWARyOfTiArGkz7RHyX2OEvt4MNohtfEsMCybPWIM1' last in use 56640.0 milliseconds ago
2021-06-24 08:18:05,682 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B22820>
2021-06-24 08:18:05,698 Discarding session 'sc2FkTn5C1GEuRrFni8Ehukharyvh7G4OEMoRSN9opSQ' last in use 56484.0 milliseconds ago
2021-06-24 08:18:05,713 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B58610>
2021-06-24 08:18:05,729 Discarding session 'dCJLAiassrjmfeqWrt9xkwkWayWS5k3oIEfJJecSw82m' last in use 56172.0 milliseconds ago
2021-06-24 08:18:05,729 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B58CD0>
2021-06-24 08:18:05,760 Discarding session 'HmRO3fa2FMCEGzbXtjljSDuu5IGwdBel2qVER0cDaIVA' last in use 56016.0 milliseconds ago
2021-06-24 08:18:05,760 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B68400>
2021-06-24 08:18:05,792 Discarding session 'lcNbiysPjn1DQG9wmR09oe3ssO4thv1kAd1Vv8PvFZFX' last in use 55985.0 milliseconds ago
2021-06-24 08:18:05,792 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B68AC0>
2021-06-24 08:18:05,823 Discarding session 'VjvAeDsCec5771xvwfwtIaerE6Y1BlP6nq8KmwvF4BHn' last in use 55844.0 milliseconds ago
2021-06-24 08:18:05,823 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B761C0>
2021-06-24 08:18:05,854 Discarding session '4XqJa8hPcuLLRo5aijdBG2DS4VP9d8xxvllP85qnEIhY' last in use 55828.0 milliseconds ago
2021-06-24 08:18:05,854 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B76880>
2021-06-24 08:18:05,885 Discarding session '9qM5M5Imvq2kQuvWRJAOznjo8kQSz6cLRHxt2tHuUyva' last in use 55704.0 milliseconds ago
2021-06-24 08:18:05,885 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B76F40>
2021-06-24 08:18:05,901 Discarding session 'Txsb6qbqtqEfmFcMAvnJAgLvLOEVRvT77CbIEivzw1L3' last in use 55640.0 milliseconds ago
2021-06-24 08:18:05,901 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B84640>
2021-06-24 08:18:05,932 Discarding session 'qOxVcAmBsfdDWnU79YTc2CrhefUGCeY7vojzR91cosKU' last in use 55500.0 milliseconds ago
2021-06-24 08:18:05,932 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B84D00>
2021-06-24 08:18:13,340 [pid 3612] 0 clients connected
2021-06-24 08:18:13,340 [pid 3612]   /test1 has 4 sessions with 4 unused
2021-06-24 08:18:23,354 Scheduling 4 sessions to discard
2021-06-24 08:18:57,121 Discarding session 'T9sr9Ltkb8V3pKsFxbwryoavM3DLgWPxskwaxgL2fazJ' last in use 51172.0 milliseconds ago
2021-06-24 08:18:57,121 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B84F40>
2021-06-24 08:18:57,152 Discarding session 'XUdgujq5V8HjI62gGsopRZIKK0tnhGOU8SST9DVW7OtE' last in use 51203.0 milliseconds ago
2021-06-24 08:18:57,152 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B84B80>
2021-06-24 08:18:57,183 Discarding session 'MImmkvT0yCgGHtfoZtIwUHdPUb2h1ppbRQXZXxQmI6Ma' last in use 51219.0 milliseconds ago
2021-06-24 08:18:57,183 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B58040>
2021-06-24 08:18:57,214 Discarding session 'rTNkOAd5SB16a3YH2M9EzjeP0nxjiGNtEmOn0oUWtnzh' last in use 50968.0 milliseconds ago
2021-06-24 08:18:57,214 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B58160>
2021-06-24 08:18:57,246 [pid 3612] 0 clients connected
2021-06-24 08:18:57,246 [pid 3612]   /test1 has 2 sessions with 2 unused
2021-06-24 08:18:57,261 Scheduling 2 sessions to discard
2021-06-24 08:18:57,261 Discarding session 'wdMbbfjhqtrIuFME0WFqe9turPPq1E2LfkVjxDDMMwAG' last in use 40594.0 milliseconds ago
2021-06-24 08:18:57,261 Deleting 1 modules for <bokeh.document.document.Document object at 0x0000017350B3B070>
2021-06-24 08:18:57,277 Discarding session 'rTTuHctPmwC80Vt0vIrjvRBXHN6CyikNjv5J0ZDXQUuu' last in use 40360.0 milliseconds ago
2021-06-24 08:18:57,292 Deleting 1 modules for <bokeh.document.document.Document object at 0x00000173500E1820>
2021-06-24 08:18:58,339 [pid 3612] 0 clients connected
2021-06-24 08:18:58,339 [pid 3612]   /test1 has 0 sessions with 0 unused

It seems like the client get connected to the apache, but not Bokeh(Tornado) server sit behind, just as you mentioned.

OK, I will put stack overflow link in my poster.

1 Like

I think these lines are wrong:

 ProxyPass /test1/ws ws://localhost:5006/test1
 ProxyPassReverse /test1/ws ws://localhost:5006/test1

and should be instead

 ProxyPass /test1/ws ws://localhost:5006/test1/ws
 ProxyPassReverse /test1/ws ws://localhost:5006/test1/ws

Thanks,Bryan.
I tried but it did not work.

PS C:\DeployTest> bokeh serve .\test1.py --allow-websocket-origin="*" --log-level debug
2021-06-24 08:38:54,162 Starting Bokeh server version 2.3.2 (running on Tornado 6.1)
...(same as previous) ...
2021-06-24 08:38:54,194 Bokeh app running at: http://localhost:5006/test1
2021-06-24 08:38:54,194 Starting Bokeh server with process id: 2104
2021-06-24 08:38:58,584 426 GET /test1/ws (::1) 0.00ms
2021-06-24 08:39:09,194 [pid 2104] 0 clients connected
2021-06-24 08:39:09,194 [pid 2104]   /test1 has 1 sessions with 1 unused

It seems like Bokeh server started to get request(GET).

@X_L @X_L I’ve never actually encountered a 426, which apparently is

The HTTP 426 Upgrade Required client error response code indicates that the server refuses to perform the request using the current protocol but might be willing to do so after the client upgrades to a different protocol.

My best guess is that the websocket upgrade is not finishing, or a plain HTTP request is landing when the websocket upgrade is expected. I’m not sure. That definitely seems like it would point to a configuration issue on the proxy but I am afraid I am at the limit of my expertise with Apache.

Edit: well I guess the log is explicit:

2021-06-24 08:38:58,584 426 GET /test1/ws 

an HTTP GET request is landing on the /ws endpoint which definitely seems wrong. The JS in the page that is initially rendered to embed the app uses JS new Websocket(...) to open the ws connection back to the server. This must be getting muddled by the proxy somehow.

Thanks for keeping following, Bryan.
Is there anyway to check contents of the request back to Bokeh server? I want to know how does the request was muddled by proxy.

As I mentioned above, BokehJS calls the browser-standard new Websocket(...) API, with the Bokeh server’s ws:// URL, in order to open the websocket connection. I am afraid I don’t have any idea how to ask Apache about what might have happened to that ws request between when it received and when it forwarded it on.

In case it is useful, this error code is coming from Tornado. You can see the function that can fail here:

Specifically, it looks like it can fail (only) if the Sec-WebSocket-Version is not what Tornado expects. I can think of two reasons for this:

  • Your browser itself is somehow weird and does not support the right kinds of websockets
  • Apache is somehow munging the Sec-WS version header before sending on the request

To rule out the first bullet you can just make sure that you try things with a recent Chrome or FF, etc. Assuming things still fail, I have to assume the ws request is getting messed up by Apache, but I don’t know anything beyond that speculation.

@Bryan
Thanks for sharing your ideas, Bryan.
With respect to the Sec-WebSocket-Version, I confirmed with my chrome(lastest) console and it shows my ws request are using WebSocket-Version 13.
Maybe it is possible that ws request might be messed up by CloudFront of AWS rather than Apache, because CloudFront now is configurated to automatically add a custom header to all external request.(refer this))
Anyway, thanks, I will try to figure it out and keep updating here when I made any progress.

1 Like

@Bryan My problem was fixed by configurating CloudFront of AWS to enable websocket request, it was done by technical staff of company and I am sorry that I don`t know many details about that.
Anyway, Thanks.

2 Likes