Bokeh Server Error with Flask on AWS EC2 instance

Waleed · April 26, 2019, 6:05pm

We made a new section on our website regarding plot data and display and we decided to go with bokeh. The architecture we choose is the flask embed option and the good news is the app works perfectly when running on our computers. Once we moved to AWS the apps refuses to work the error we get is

Screen Shot 2019-04-26 at 8.03.26 PM.png

We suspect that the error happens because of nginx, AWS or letsEncrypt certificate however we made the correct changes however we are still not able to open the connection our flask code

Screen Shot 2019-04-26 at 8.19.06 PM.png

Our nginx configuration

Screen Shot 2019-04-26 at 8.38.09 PM.png

Finally our AWS security group inbound traffic is as follow

Screen Shot 2019-04-26 at 8.54.16 PM.png

as you can see we open the custom TCP rule for port 5006 and also we opened port 8000 any help is appreciated

andrew.waites · November 26, 2019, 8:59pm

Waleed - did you get any joy on this?

Bryan · November 26, 2019, 10:52pm

Connection refused typically means allow_websocket_origin is not set correctly. In this case I might suspect that the ultimate end viewers are not actually navigating to port 8000. The allowed origin list needs to include what the &actual viewer’s browser will report as the origin, and typically this is “what is in the URL bar”.

andrew.waites · November 26, 2019, 11:44pm

@Bryan sadly not that as I am running with

CMD bokeh serve
–allow-websocket-origin="*"
–index=/index.html
–num-procs=${NUM_PROCS}
ceoperformance.py
electoralmaths.py

Bryan · November 27, 2019, 12:14am

Perhaps it’s Nginx refusing the connection then? Hard to say much without logs. The first question to answer is: who/where is rejecting the connection.

BTW, if you are just using Nginx in order to terminal SSL, know that as of Bokeh 1.4 the Bokeh server can terminate HTTPS connections directly.

andrew.waites · November 27, 2019, 12:27am

@Bryan I think it is nginx. This is all behind an AWS classic load balancer but I have tracked this down to the websocket reverse proxy and have tried adding this block to nginx configuration:

        location /ceoperformance {
            proxy_pass            http://127.0.0.1:5006;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host:$server_port;
            proxy_buffering off;
        }

The result is a shift from a 400 error to a 502 error which, being a change, is as good as a holiday I suppose. I amthinking perhaps I should try something other than 127.0.0.1 (maybe localhost although not sure what difference that would make).

2019/11/26 21:54:39 [error] 15735#0: *2 connect() failed (111: Connection refused) 
while connecting to upstream, client: 172.31.30.233, server: , 
request: "GET /ceoperformance HTTP/1.1", 
upstream: "http://127.0.0.1:5006/ceoperformance", host: "apps.amalfiblue.com", 
referrer: "http://apps.amalfiblue.com/"

I’m not sure what the existing nginx proxy_pass / → http://docker is all about so am a bit gunshy about playing with that.

Bryan · November 27, 2019, 2:49am

I guess it’s worth verifying: Are nginx and the Bokeh server running on the same instance?

Also does your security group allow TCP (websocket) connections in and out of the instance?

andrew.waites · November 27, 2019, 4:09am

Bryan - both on same instance. Pretty sure I have managed to allow the TCP on the security group as when I changed the nginx config adding:

        location /ceoperformance {
            proxy_pass            http://localhost:5006;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host:$server_port;
            proxy_buffering off;
        }

The nginx error log gave a 502 error which was not there before - I suspect that means the packet got to the server.

Bryan · November 27, 2019, 6:16am

I’m afraid I don’t know what else to suggest at this point. Are you able to retrieve the Bokeh server logs? If the server is intentionally rejecting the connection that will show up. Another possibility is if the app code is raising an exception, that would probably also result in a 502 in fact that seem more likely. But if the app code is blowing up that would also show up in the logs.

andrew.waites · November 27, 2019, 7:17am

Bryan - appreciate the help - driving me a bit mad and having to learn way to many things for this old dog.

The code runs fine in a local standalone docker container, also fine on EB is there is no load balancer

Can you provide some clarity as to what happens with that webserver call. The JS on client is initiating a call to /ceoperformance/ws?.. I am guessing the bokeh server should be interpreting the /ws as a web service call (as opposed to some for of page load) so the client is sending a request (TCP port 5006?) which is somehow seen as malformed (400) - the load balancer is mapping 5006 to 5006 (is that right) - and is it coming in on 5006 from the client?

I am pretty confused (clearly) because I can’t directly address 5006 from my client unless it is running locally.

On your demo setup did you have to do anything with nginx (default docker EB web server) config? What did you map 5006 to on the EB load balancer?

There seem to have been a few questions along these lines in the past.

andrew.waites · November 27, 2019, 7:23pm

Ok this is a problem on the ws: call which (when successful) returns a 101 (protocol change). I gather that this is some form of client side swap from http to tcp (guessing) and it may (somewhere) swap to using the bokeh port (here 5006).

I have 5006 mapped to 5006 on the ELB as HTTP on both sides but I wonder if one or both sides of that should be TCP.

I would REALLY LOVE to know how you have you AWS LB configured.

Bryan · November 27, 2019, 8:25pm

@andrew.waites If you can tell me what to look for, I am happy to report it back. Failing that, could potentially do a screen share to look over the config together. FYI ours is a classic ELB, I am not sure the “new” ELB supports websockets, just in case that is the issue.

andrew.waites · November 27, 2019, 8:32pm

@Bryan if you look at your EB application → configuration → load balancer

There should be a table of listener mappings at the top:

Port 80 HTTP → 80 HTTP etc - I’d like to know what it says for other ports (5006) or whatever.

I have an application LB in front of this but I am currently testing the classic load balancer directly.

Bryan · November 27, 2019, 8:45pm

It’s very minimal, AFIAK the mapping to the external port (at least in our case) is handled at the Docker level:

andrew.waites · November 27, 2019, 8:51pm

Bryan - does handled at the docker level mean nginx config?

I know the Dockerfile has EXPOSE 5006 and the default nginx config seems to map everything to http://docker - is that what you mean?

Not really sure what that actually means in practice

HAve removed my 5006 mapping and no difference

andrew.waites · November 27, 2019, 10:20pm

Bryan - I will duplicate your gallery and see how that goes then work back from there

Bryan · November 28, 2019, 12:19am

@andrew.waites I have have mis-spoke, I mean here:

github.com

bokeh/demo.bokeh.org/blob/main/Dockerrun.aws.json

{
  "AWSEBDockerrunVersion": "1",
  "Image": {
    "Name": "562605262404.dkr.ecr.us-east-1.amazonaws.com/bokeh",
    "Update": "true"
  },
  "Ports": [
    {
      "ContainerPort": "80",
      "HostPort": "80"
    }
  ]
}

which is actually an AWS config. Note that we currently use this config and pull images from docker hub, but we didn’t always. It’s also possible to just upload docker files directly to EB. But you might start just duplicating everything, including the dockerhub image we publish.

EDIT: Note, when we did just upload the Docker file directly, we had this in it:

EXPOSE 5006
EXPOSE 80

But I’m not actually sure that’s correct… I don’t recall where the container/host port mapping was configured when the docker file was uploaded.

andrew.waites · November 28, 2019, 1:49am

Bryan I just tried using the Dockerfile on GitHub - bokeh/demo.bokeh.org: Hosted Bokeh App Demos loading it into a standard docker AWS EB instance. Works fine without load balancing. As soon as I set up load balancing I get exactly the error I have been seeing all week. I didn’t touch the ELB defaults so they are the same as you indicated.

Also, based on your previous response I tried adding a listener 5006 → 80 on the AWS ELB (for my instance) still no joy. Am thinking that config though might be telling docker something rather than AWS which “might” be reflected in your nginx config (/etc/nginx/sites-available/something)

Okay I found that I had that dockerfile.aws.json in the demo folder so I checked the EB setup and that seems to have added an an upstream conf file to nginx (in /etc/conf.d) which implies that docker should (perhaps) be handling all this hence no need for AWS ELB configuration changes.

Still doesn’t work though.

andrew.waites · November 28, 2019, 2:45am

Bryan - http://bokehdemo.usspnmr7nm.ap-southeast-2.elasticbeanstalk.com/
is the copy of the demo system - as I say, it works when there is no load balancer but does not work as soon as it is behind a load balancer and it is clearly related to the server call to upgrade the protocol (websocket).

Bryan · November 28, 2019, 7:05am

@andrew.waites I’m not sure I will have to try to reproduce things step by step from scratch this weekend after the holiday. We definitely have two instances running behind a classic ELB. I still think if you can obtain the Bokeh server console logs that will be very helpful. (I know AWS does not make it simple)