We made a new section on our website regarding plot data and display and we decided to go with bokeh. The architecture we choose is the flask embed option and the good news is the app works perfectly when running on our computers. Once we moved to AWS the apps refuses to work the error we get is
We suspect that the error happens because of nginx, AWS or letsEncrypt certificate however we made the correct changes however we are still not able to open the connection our flask code
Our nginx configuration
Finally our AWS security group inbound traffic is as follow
as you can see we open the custom TCP rule for port 5006 and also we opened port 8000 any help is appreciated
Connection refused typically means allow_websocket_origin is not set correctly. In this case I might suspect that the ultimate end viewers are not actually navigating to port 8000. The allowed origin list needs to include what the &actual viewer’s browser will report as the origin, and typically this is “what is in the URL bar”.
Perhaps it’s Nginx refusing the connection then? Hard to say much without logs. The first question to answer is: who/where is rejecting the connection.
BTW, if you are just using Nginx in order to terminal SSL, know that as of Bokeh 1.4 the Bokeh server can terminate HTTPS connections directly.
@Bryan I think it is nginx. This is all behind an AWS classic load balancer but I have tracked this down to the websocket reverse proxy and have tried adding this block to nginx configuration:
The result is a shift from a 400 error to a 502 error which, being a change, is as good as a holiday I suppose. I amthinking perhaps I should try something other than 127.0.0.1 (maybe localhost although not sure what difference that would make).
I’m afraid I don’t know what else to suggest at this point. Are you able to retrieve the Bokeh server logs? If the server is intentionally rejecting the connection that will show up. Another possibility is if the app code is raising an exception, that would probably also result in a 502 in fact that seem more likely. But if the app code is blowing up that would also show up in the logs.
Bryan - appreciate the help - driving me a bit mad and having to learn way to many things for this old dog.
The code runs fine in a local standalone docker container, also fine on EB is there is no load balancer
Can you provide some clarity as to what happens with that webserver call. The JS on client is initiating a call to /ceoperformance/ws?.. I am guessing the bokeh server should be interpreting the /ws as a web service call (as opposed to some for of page load) so the client is sending a request (TCP port 5006?) which is somehow seen as malformed (400) - the load balancer is mapping 5006 to 5006 (is that right) - and is it coming in on 5006 from the client?
I am pretty confused (clearly) because I can’t directly address 5006 from my client unless it is running locally.
On your demo setup did you have to do anything with nginx (default docker EB web server) config? What did you map 5006 to on the EB load balancer?
There seem to have been a few questions along these lines in the past.
Ok this is a problem on the ws: call which (when successful) returns a 101 (protocol change). I gather that this is some form of client side swap from http to tcp (guessing) and it may (somewhere) swap to using the bokeh port (here 5006).
I have 5006 mapped to 5006 on the ELB as HTTP on both sides but I wonder if one or both sides of that should be TCP.
I would REALLY LOVE to know how you have you AWS LB configured.
@andrew.waites If you can tell me what to look for, I am happy to report it back. Failing that, could potentially do a screen share to look over the config together. FYI ours is a classic ELB, I am not sure the “new” ELB supports websockets, just in case that is the issue.
which is actually an AWS config. Note that we currently use this config and pull images from docker hub, but we didn’t always. It’s also possible to just upload docker files directly to EB. But you might start just duplicating everything, including the dockerhub image we publish.
EDIT: Note, when we did just upload the Docker file directly, we had this in it:
EXPOSE 5006
EXPOSE 80
But I’m not actually sure that’s correct… I don’t recall where the container/host port mapping was configured when the docker file was uploaded.
Bryan I just tried using the Dockerfile on GitHub - bokeh/demo.bokeh.org: Hosted Bokeh App Demos loading it into a standard docker AWS EB instance. Works fine without load balancing. As soon as I set up load balancing I get exactly the error I have been seeing all week. I didn’t touch the ELB defaults so they are the same as you indicated.
Also, based on your previous response I tried adding a listener 5006 → 80 on the AWS ELB (for my instance) still no joy. Am thinking that config though might be telling docker something rather than AWS which “might” be reflected in your nginx config (/etc/nginx/sites-available/something)
Okay I found that I had that dockerfile.aws.json in the demo folder so I checked the EB setup and that seems to have added an an upstream conf file to nginx (in /etc/conf.d) which implies that docker should (perhaps) be handling all this hence no need for AWS ELB configuration changes.
Bryan - http://bokehdemo.usspnmr7nm.ap-southeast-2.elasticbeanstalk.com/
is the copy of the demo system - as I say, it works when there is no load balancer but does not work as soon as it is behind a load balancer and it is clearly related to the server call to upgrade the protocol (websocket).
@andrew.waites I’m not sure I will have to try to reproduce things step by step from scratch this weekend after the holiday. We definitely have two instances running behind a classic ELB. I still think if you can obtain the Bokeh server console logs that will be very helpful. (I know AWS does not make it simple)