HTTP/2, varnish, nginx, haproxy, and mixing TCP/HTTP mode

@lee_ars wrote:

I've been using HAProxy for SSL termination as part of a stack that looks like this:

          https           http           http
Internet <-----> haproxy <----> varnish <----> nginx

Everything works great, but adding HTTP/2 support has slammed me hard into a wall and I can't figure a way out of it. I don't want to jettison HAProxy in favor of Hitch, but I think I'm about to unless I can figure out some magical voodoo configuration options to get things working

Problem, the short version: The only way I can figure out how to make HTTP/2 work correctly and in conjunction with HSTS (i.e., using haproxy to redirect all client http attempts to https) is to use mode http for the frontend (necessary because I have about 10 sites each using various http-header settings for various things) and mode tcp for the back end )because I have to communicate to Varnish using the proxy protocol, because varnish in turn needs to communicate to nginx with the proxy protocol).

Problem, the short version, continued: If I switch the frontend to mode tcp, I get beautiful HTTP/2-served web sites with no problem. Everything works great. However, http-request redirect scheme https if http obviously no longer works and I don't know if there are substitutes for that and all the other http-request commands I'm using for domain redirects and the like. I need that functionality and I need it at the termination layer, not deeper in the stack. Conversely, switching the backend to mode http breaks everything and nothing works—no pages get served and all I get is a ERR_SPDY_PROTOCOL_ERROR when I try to connect to anything.

Problem, the long version: ugh, I don't know if I have the energy to type all this out, but here are some config snippets at each piece in the stack to show what I'm doing:

1) haproxy:

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  forwardfor

...

frontend my_front
        bind :::80 v4v6
        bind :::443 v4v6 ssl crt mycert.pem ecdhe secp384r1 alpn h2,http/1.1
        acl http ssl_fc,not
        acl letsencryptrequest path_beg -i /.well-known/acme-challenge/
        acl mastodon hdr(host) beg -i mastodon.bigdinosaur.org
        use_backend letsencrypt if letsencryptrequest
        use_backend mastodonwtf if mastodon

        http-request redirect prefix https://www.bigdinosaur.org code 301 if { hdr(host) -i bigdinosaur.org }
  
        ...(imagine lots more 301s here)...

        http-request redirect scheme https if http
        http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload;"
        http-response set-header Referrer-Policy "strict-origin-when-cross-origin"
        http-response set-header X-Content-Type-Options "nosniff"
        http-response set-header X-XSS-Protection "1; mode=block"
        use_backend mastodon if mastodon

        rsprep ^Set-Cookie:\ (.*) Set-Cookie:\ \1;\ Secure if { ssl_fc }
        default_backend tovarnish

...

backend tovarnish
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        server local 127.0.0.1:6081 send-proxy-v2

backend mastodon
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        http-response set-header Content-Security-Policy redacted for length
        http-response set-header Public-Key-Pins redacted for length
        server local 127.0.0.1:6081 send-proxy-v2

backend letsencrypt
        server letsencrypt 127.0.0.1:54321

2) Varnish:

# Backend definition. Set this to point to your content server. Nginx listens on 2 ports,
# one for non-upgraded and non-http2 requests (default), and the other for http2 requests.

backend default {
        .host = "127.0.0.1";
        .port = "8086";
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 800;
        .proxy_header = 1;
}

backend h2 {
        .host = "127.0.0.1";
        .port = "8088";
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 800;
        .proxy_header = 1;
}

...

sub vcl_recv {
        # Happens before we check if we have this in cache already.
        #
        # Typically you clean up the request here, removing cookies you don't need,
        # rewriting the request, etc.

        if (req.http.protocol ~ "HTTP/2") {
                set req.backend_hint = h2;
        }
        else {
                set req.backend_hint = default;
        }

}

3) Nginx:

server {
	server_name www.bigdinosaur.org;
	listen 8088 http2 proxy_protocol default_server;
	listen 8086 proxy_protocol;

...

(all the rest of the vhost file)

I know based on some quick testing with a virtual machine that I can rip out HAProxy and drop in Hitch and everything Just Works , but I like the additional flexibility HAProxy gives me with being able to do redirects and backend voodoo (for example, if I rip out haproxy, I have to rethink my entire LetsEncrypt setup, ugh).

Does anyone have any insight on potential ways forward here that will let me keep HAProxy? I've been banging on this for a couple of days on and off and I just can't seem to reach a solution that works.

Posts: 3

Participants: 2

Read full topic