I have a somewhat complicated setup with 2 instances of HAProxy running on a single server, fronted by Nginx on another server.
Even though there are two instances of haproxy running on the same server, only one of them is ever being used (via iptables rule).
nginx
| nginx server (10.0.0.1)
---------------------------------------------
80
| haproxy server (10.0.0.2)
[ IPTABLES]
/
/
HAproxy_A HAproxy_B
This setup has worked for me for a long time without any issues. When I want to make a config change for haproxy, I will update the instance which is not being used, restart it, and update iptables, for example:
nginx
| nginx server (10.0.0.1)
---------------------------------------------
80
| haproxy server (10.0.0.2)
[ IPTABLES]
\
\
HAproxy_A HAproxy_B
I recently discovered that I'm able to get haproxy to hang and use 100% CPU when using the keepalive option in nginx, and the http-reuse option in haproxy. This only happens when REMOVING backend servers in haproxy and using ApacheBench to simulate load on the servers.
For example:
Nginx config routing to haproxy:
upstream myapp {
server 10.0.0.2;
keepalive 64;
}
server {
listen 80;
server_name mynginx;
location / {
proxy_pass http://myapp$request_uri;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
Haproxy config routing to a single backend app on 10.134.8.221:
backend backend_myapp
balance roundrobin
http-reuse safe
server 10.134.8.221_31403 10.134.8.221:31403 maxconn 128 weight 100 check
-
haproxy_a is accepting requests
-
Add another backend servers into haproxy_b (backend app is running on same server, just different port), for example:
backend backend_myapp
balance roundrobin
http-reuse safe
server 10.134.8.221_31403 10.134.8.221:31403 maxconn 128 weight 100 check
server 10.134.8.221_31404 10.134.8.221:31404 maxconn 128 weight 100 check
Restart haproxy_b with
/usr/sbin/haproxy_b -p /tmp/haproxy_b.pid -f /etc/haproxy/haproxy_b.cfg -sf <old haproxy_b pid>
- Update iptables to
haproxy_b port
- Run
ab test via nginx. Everything works as expected. Haproxy_b is being used and requests are getting routed to backend servers appropriately.
However, when removing a backend server. For example:
-
haproxy_b is accepting requests
- Restart
haproxy_a (even though the config hasn't changed and only contains the one backend server) with
/usr/sbin/haproxy_a -p /tmp/haproxy_a.pid -f /etc/haproxy/haproxy_a.cfg -sf <old haproxy_a pid>
- Update iptables to point to
haproxy_a port
- Run
ab test against nginx. Haproxy_b starts using 100% CPU, even though requests should only be coming to haproxy_a. Note that the CPU doesn't spike to 100% until the requests starting coming through nginx to haproxy.
This only happens when using the keepalive option in nginx AND the http-reuse option in haproxy. If I remove either one of these options, I am unable to reproduce the issue.
Is it possible that nginx is keeping the connections alive to the inactive instance of haproxy, and then the next set of requests is coming to the old (non running) instance?
Any ideas on why this would be happening? I realize it's a complicated setup, so would be happy to provide any details I may have forgotten.
Thanks for the help in advance.