Strange timeout behavior during load testing

@cmendelson wrote:

I’ve been doing some load testing in anticipation of our busy season. I’m able to push through 800 connections per second to a single backend without much problem. When I add additional
servers to the backend pool, and increase the load on the haproxy machine to 3000-4000 connections per second, haproxy starts marking my backends as down, even though I’ve increased my backend capacity 10x, and only 4x-5x’ed my load.

Running: watch -n1 'curl --silent "http://localhost:9000/haproxy_stats;csv" | cut -d "," -f 2,5,37'

Shows backends going from L7OK -> *L7OK (what does the * mean?) -> L4TOUT -> Healthy. The downstream server is always healthy, and tcp connections from another machine succeed without any issue.

CPU on the haproxy machine I’m using is fine (an r4.large EC2 instance), as is memory usage. The machine is maxing out at 50MBps bandwidth usage, which is a 1/10th of what we push in production, so I’m not bandwidth limited.

Any ideas why haproxy isn’t able to successfully complete a healthcheck? Any suggestions on debugging this?

Here is my haproxy.cfg:

global
  maxconn 60000
  user haproxy
  group haproxy
  daemon
  tune.ssl.default-dh-param 2048

defaults
  mode http
  retries 3
  option redispatch
  timeout connect 5000
  timeout client 50000
  timeout server 50000

resolvers dns
    nameserver dns 169.254.169.253:53
backend server_pool
  balance hdr(x-mi-cbe)
  option httpchk GET /?health=true

  server a 10.2.2.183:9292 check inter 5000
  server b 10.2.1.184:9292 check inter 5000
  server c 10.2.2.174:9292 check inter 5000
  server d 10.2.0.41:9292 check inter 5000
  server e 10.2.0.16:9292 check inter 5000
  server f 10.2.2.216:9292 check inter 5000
  server g 10.2.1.135:9292 check inter 5000
  server h 10.2.0.162:9292 check inter 5000
  server i 10.2.1.232:9292 check inter 5000
  server j 10.2.1.253:9292 check inter 5000
  server k 10.2.2.141:9292 check inter 5000

backend stats
  balance roundrobin
  stats enable
frontend cors_proxy_http
  bind 0.0.0.0:80
  monitor-uri /haproxy?health
  default_backend server_pool
  option forwardfor
  option http-server-close
  maxconn 60000
  log /var/lib/haproxy/dev/log local2 debug
  option httplog
  option dontlognull

frontend stats
  bind 0.0.0.0:9000
  monitor-uri /haproxy?health
  default_backend stats
  stats uri /haproxy_stats

Posts: 4

Participants: 2

Read full topic