Bryan - I can ssh onto the server but just want to be sure I get the right logs - are you laking about /var/logs/eb-docker/container/eb-something/–latestfil–.log
@andrew.waites I don’t know the log file name, it’s whichever has the bokeh server output that starts like
2019-11-29 08:31:42,921 Starting Bokeh server version 2.0.0dev4-71-g752e1471a (running on Tornado 6.0.3)
2019-11-29 08:31:42,925 User authentication hooks NOT provided (default user enabled)
2019-11-29 08:31:42,927 Bokeh app running at: http://localhost:5006/selection_histogram
2019-11-29 08:31:42,927 Starting Bokeh server with process id: 16698
FYI if you add --log-level=debug to the server start it will output some additional information.
2019-12-01 18:50:44,004 Starting Bokeh server version 1.4.0 (running on Tornado 6.0.3)
2019-12-01 18:50:44,007 Host wildcard ‘’ will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
2019-12-01 18:50:44,007 User authentication hooks NOT provided (default user enabled)
2019-12-01 18:50:44,007 These host origins can connect to the websocket: [’’]
2019-12-01 18:50:44,007 Patterns are:
2019-12-01 18:50:44,009 [(’/ceoperformance/?’,
2019-12-01 18:50:44,009 <class ‘bokeh.server.views.doc_handler.DocHandler’>,
2019-12-01 18:50:44,009 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf950>,
2019-12-01 18:50:44,009 ‘bokeh_websocket_path’: ‘/ceoperformance/ws’}),
2019-12-01 18:50:44,009 (’/ceoperformance/ws’,
2019-12-01 18:50:44,010 <class ‘bokeh.server.views.ws.WSHandler’>,
2019-12-01 18:50:44,010 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf950>,
2019-12-01 18:50:44,010 ‘bokeh_websocket_path’: ‘/ceoperformance/ws’}),
2019-12-01 18:50:44,010 (’/ceoperformance/metadata’,
2019-12-01 18:50:44,010 <class ‘bokeh.server.views.metadata_handler.MetadataHandler’>,
2019-12-01 18:50:44,010 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf950>,
2019-12-01 18:50:44,011 ‘bokeh_websocket_path’: ‘/ceoperformance/ws’}),
2019-12-01 18:50:44,011 (’/ceoperformance/autoload.js’,
2019-12-01 18:50:44,011 <class ‘bokeh.server.views.autoload_js_handler.AutoloadJsHandler’>,
2019-12-01 18:50:44,011 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf950>,
2019-12-01 18:50:44,011 ‘bokeh_websocket_path’: ‘/ceoperformance/ws’}),
2019-12-01 18:50:44,012 (’/electoralmaths/?’,
2019-12-01 18:50:44,012 <class ‘bokeh.server.views.doc_handler.DocHandler’>,
2019-12-01 18:50:44,012 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf910>,
2019-12-01 18:50:44,012 ‘bokeh_websocket_path’: ‘/electoralmaths/ws’}),
2019-12-01 18:50:44,012 (’/electoralmaths/ws’,
2019-12-01 18:50:44,012 <class ‘bokeh.server.views.ws.WSHandler’>,
2019-12-01 18:50:44,013 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf910>,
2019-12-01 18:50:44,013 ‘bokeh_websocket_path’: ‘/electoralmaths/ws’}),
2019-12-01 18:50:44,013 (’/electoralmaths/metadata’,
2019-12-01 18:50:44,013 <class ‘bokeh.server.views.metadata_handler.MetadataHandler’>,
2019-12-01 18:50:44,013 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf910>,
2019-12-01 18:50:44,013 ‘bokeh_websocket_path’: ‘/electoralmaths/ws’}),
2019-12-01 18:50:44,014 (’/electoralmaths/autoload.js’,
2019-12-01 18:50:44,014 <class ‘bokeh.server.views.autoload_js_handler.AutoloadJsHandler’>,
2019-12-01 18:50:44,014 {‘application_context’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf910>,
2019-12-01 18:50:44,014 ‘bokeh_websocket_path’: ‘/electoralmaths/ws’}),
2019-12-01 18:50:44,014 (’/?’,
2019-12-01 18:50:44,014 <class ‘bokeh.server.views.root_handler.RootHandler’>,
2019-12-01 18:50:44,015 {‘applications’: {’/ceoperformance’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf950>,
2019-12-01 18:50:44,015 ‘/electoralmaths’: <bokeh.server.contexts.ApplicationContext object at 0x7ff2cadbf910>},
2019-12-01 18:50:44,015 ‘index’: ‘/index.html’,
2019-12-01 18:50:44,015 ‘prefix’: ‘’,
2019-12-01 18:50:44,015 ‘use_redirect’: True}),
2019-12-01 18:50:44,015 (’/static/(.*)’,
2019-12-01 18:50:44,016 <class ‘bokeh.server.views.static_handler.StaticHandler’>)]
2019-12-01 18:50:44,026 Bokeh app running at: http://localhost:5006/ceoperformance
2019-12-01 18:50:44,028 Bokeh app running at: http://localhost:5006/ceoperformance
2019-12-01 18:50:44,028 Bokeh app running at: http://localhost:5006/electoralmaths
2019-12-01 18:50:44,028 Bokeh app running at: http://localhost:5006/electoralmaths
2019-12-01 18:50:44,028 Starting Bokeh server with process id: 8
2019-12-01 18:50:44,028 Starting Bokeh server with process id: 7
2019-12-01 18:50:59,045 [pid 8] 0 clients connected
2019-12-01 18:50:59,045 [pid 8] /ceoperformance has 0 sessions with 0 unused
2019-12-01 18:50:59,045 [pid 8] /electoralmaths has 0 sessions with 0 unused
2019-12-01 18:50:59,045 [pid 7] 0 clients connected
2019-12-01 18:50:59,045 [pid 7] /ceoperformance has 0 sessions with 0 unused
2019-12-01 18:50:59,045 [pid 7] /electoralmaths has 0 sessions with 0 unused
.
.
.
2019-12-01 18:52:14,034 [pid 7] 0 clients connected
2019-12-01 18:52:14,035 [pid 7] /ceoperformance has 0 sessions with 0 unused
2019-12-01 18:52:14,035 [pid 7] /electoralmaths has 0 sessions with 0 unused
2019-12-01 18:52:23,771 400 GET /ceoperformance/ws?bokeh-protocol-version=1.0&bokeh-session-id=NbCHWHwCBqx9ASz0yGqtOYJfoI9L3gH7eHZCDfhhYGZF (172.17.0.1) 0.66ms
2019-12-01 18:52:29,032 [pid 8] 0 clients connected
2019-12-01 18:52:29,032 [pid 8] /ceoperformance has 0 sessions with 0 unused
2019-12-01 18:52:29,033 [pid 8] /electoralmaths has 0 sessions with 0 unused
2019-12-01 18:52:29,033 [pid 7] 0 clients connected
2019-12-01 18:52:29,034 [pid 7] /ceoperformance has 1 sessions with 1 unused
2019-12-01 18:52:29,034 [pid 7] /electoralmaths has 0 sessions with 0 unused
2019-12-01 18:52:43,037 Scheduling 1 sessions to discard
2019-12-01 18:52:43,037 Discarding session ‘NbCHWHwCBqx9ASz0yGqtOYJfoI9L3gH7eHZCDfhhYGZF’ last in use 19427.084345042706 milliseconds ago
2019-12-01 18:52:43,037 Deleting 1 modules for <bokeh.document.document.Document object at 0x7ff2caef8c10>
And the relevent lines in nginx/access.log
172.31.2.113 - - [01/Dec/2019:18:52:23 +0000] “GET /ceoperformance HTTP/1.1” 200 2699 “http://apps.amalfiblue.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36”
172.31.2.113 - - [01/Dec/2019:18:52:23 +0000] “GET /ceoperformance/ws?bokeh-protocol-version=1.0&bokeh-session-id=NbCHWHwCBqx9ASz0yGqtOYJfoI9L3gH7eHZCDfhhYGZF HTTP/1.1” 400 34 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36”
@andrew.waites Do you not have outbound TCP traffic opened up, by chance? It looks from the log like the websocket connection request is making it in to the server and then a reply is never making it out. For the demo site we have the same config for outbound as inbound.
Bryan - outbound is wide open - everything on every security group I can find - am looking for AWS traffic logs/denials now
Out of interest, do you have a step-by-step on how you set up the EB - perhaps because I started with the demo docker environment then just deployed the new dockerfile ++ perhaps there is a difference there
I notice that you don’t have
proxy_set_header X-Forwarded-Proto $scheme;
in your nginx config, which is included in the sample nginx conf in the docs. Does this help?
I don’t have that yet, I was hoping to be able to work on this this weekend but it did not work out.
Bryan - just tried adding that to the config and no immediate joy with a service nginx restart
I might try a full restart
Bryan CloudWatch logs picked up this reject which is strange as am operating on 5006
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2 630057245058 eni-064f582960f584470 51.158.25.170 172.31.5.72 4040 5005 17 1 476 1575270531 1575270587 REJECT OKThat is strange. It occurs to me that the “400 GET” log message must be coming from Tornado, since Bokeh does not itself raise any 400 errors anywhere. That might be a place to search. I have not studied it yet, but this link did pop up immediately:
Bryan -trying a few things but if you could post your nginx.conf and /sites-available/alastic…conf files that might help me find the difference.
thanks
@andrew.waites The demo site does not use Nginx it’s just a Bokeh server with an index.html specified. I’ve never personally run a Bokeh server on AWS behind Nginx. I’m not familiar with the other config file either, or if there even is one in our case, but if you tell me where to look I will see if I can find it.
Bryan
I didn’t set nginx up - EB/Docker demo (I guess) uses it but if you have an index.html something must be serving it. If you do have nginx you will have /etc/nginx/nginx.conf and /etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy.conf
Strange because the Dockerfile doesnt seem to ask for nginx yet it is there - makes me think maybe it is an AWS/EB thing.
I am not sure how to get on the instance. I did not have a key pair set up initially, and these were two reserved instances that EB grabbed when Application started. I am not averse to shutting down/restarting the EB for the demo site, but I would need guidance on how to make sure the instances are ssh-able afterwards.
Edit: Looking at “download logs” there are definitely nginx logs present. So, this seems to be something AWS sets up by default. We have not modified whatever the default nginx config AWS provides is. As an example, the nginx logs show 200 entries:
172.31.90.24 - - [02/Dec/2019:21:47:12 +0000] “GET /movies/ws?bokeh-protocol-version=1.0&bokeh-session-id=TDnvXkeKrDwY4QKRUV6Mrrqua6nd36beekkiytDHiRkp HTTP/1.1” 101 1866229 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36”
Bryan - thanks - that 101 is what I see when I run it all locally (protocol upgrade).
My thinking is that perhaps the difference is that I set the application up using the AWS docker demo container then replaced it with my (and later I tried your) dockerfile. Perhaps it is something in the AWS demo setup that is lingering. Certainly seem to be many related problems scattered throughout SO. I might try setting up the bokeh demo from a straight upload and see if that makes a difference - if I can get it going and ssh in then I will see what the difference is.
From memory, to set up ssh if you have the eb command line tool and try eb ssh it should automatically work on setting one up but yes the process would involve a restart of the demo site so I will ponder. Setting up for ssh for the first time in AWS will also ask you to set up a key pair (once you have one you can reuse it on other instances).
Also - do you know who wrote the README.md on the bokeh demo deployment as it references websocket setup but is a bit vague on detail.
@andrew.waites That would be me, and it’s vague because (as is probably evident!) I don’t really have great expertise in this. But AFAIK for the ELB websocket comment I meant this:
Bryan - that would be it, now to validate with HTTPS but I basically rolled back to the standard nginx config (yes EB does use it), left the 5006 inbound port mapping on (not sure if that is required but working) and went to the load balancer (i.e. nowhere near EB) and just had that line and TCP 443 to TCP 80 with a certificate.
Actually just checked and HTTPS works too:
I will add a final reply with the steps but bottom line is:
- don’t play with nginx config
- just set up as classic and key is to change the load balancer from HTTP to TCP
Many many thanks
1 Like
@andrew.waites 
Thank you for your perseverance! I definitely want to distill want to distill this in to some resources for users, e.g maybe (one of a set of?) sample projects that users can refer to. If you have time to give any thoughts on how best to make this simpler (or at least better demonstrated) please do let me know!