HAProxy bufferbloat/not buffering? when using in front of any caching proxy

I am trying to use HAProxy for TLS termination and some extra http rules in front of varnish cache. For some weird reason, whenever I have used HAProxy in front of any caching server, it does not buffer anything ( and I think that’s what the issue is here because nginx works just fine, but I cannot use it ), and there is no official resource that assures whether or not HAProxy can actually buffer response bodies or not in the first place.

Both servers are using bbr congestion algorithm and have high ulimit settings. I have also tested the servers with a high 100 segment initial tcp window all the way down to the default 10. Of course, I have also played around with the wmem rmem buffers on both servers. None of these settings changed anything.

The HAProxy and Varnish server are both running Fedora 39 and talk to each other via a local interface.

haproxy -vv (click for more details)

I have tested this on multiple operating systems and different haproxy versions etc. with the same results, so I do not think this has to do with any specific haproxy version. I have tried messing with nodelay, request buffering, changing the maxrewrite and other buf values, but I was still unable to fix the issue. The only way I was able to make it act normal was to use maxconn 1 on the backend, but obviously that would never work in production.

Here is my problem. Note: all of those objects are warm in varnish cache.

Ideally, it would stream every object at once like this:

Or buffer everything like this:

(In all of the above photos, I am loading a page with 250 50kb images, from a 100mbps connection and 12ms latency)

My basic config. Of course, I have messed around with option http-no-delay, option http-buffer-request, option http-keep-alive, http-reuse always/safe.

global
    daemon
    maxconn 12000
defaults
    timeout server 30s
    timeout client 30s
    timeout connect 10s
    mode http
listen http
    mode http
    bind :443 ssl crt /root/mycert.pem alpn h2,http/1.1
    server s1 ip.ip.ip.ip:80

Can anyone guide me on how I can get HAProxy to send objects more consistently, or whether or not this is even possible to fix?

Thanks in advance!

1 post - 1 participant

Read full topic