@captainark wrote:
Hi !
I’ve been trying to load 100.000 distinct SSL certificates with HAProxy v1.8.13, to no avail.
My test server is a virtual machine with 4 vCPU and 8GB of RAM.
Here’s my configuration so far :
global daemon user haproxy group haproxy log 127.0.0.1 local0 log 127.0.0.1 local1 notice stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s stats maxconn 10 ca-base /etc/ssl/certs crt-base /etc/ssl/haproxy tune.ssl.default-dh-param 2048 ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl-default-bind-options no-sslv3 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl-default-server-options no-sslv3 no-tls-tickets maxconn 100000 spread-checks 4 nbthread 4 defaults log global option httplog option dontlognull maxconn 100000 timeout client 60s timeout server 60s timeout queue 60s timeout connect 4s timeout http-request 5s option httpclose option abortonclose option forwardfor option httpchk GET / HTTP/1.0\r\nUser-agent:\ haproxy-httpchk retries 2 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http frontend ft_frontend mode http bind 127.0.0.1:80 bind ::1:80 acl servers_down nbsrv(bk_backend) lt 1 monitor-uri /ping monitor fail if servers_down option http-server-close default_backend bk_backend frontend ft_frontends mode http bind 127.0.0.1:443 ssl crt /etc/ssl/haproxy bind ::1:443 ssl crt /etc/ssl/haproxy acl servers_down nbsrv(bk_backend) lt 1 monitor-uri /ping monitor fail if servers_down option http-server-close default_backend bk_backend backend bk_backend mode http balance roundrobin server www1 10.0.0.1:8080 check inter 5s maxconn 25000 server www2 10.0.0.1:8081 check inter 5s maxconn 25000 server www3 10.0.0.2:8080 check inter 5s maxconn 25000 server www4 10.0.0.2:8081 check inter 5s maxconn 25000I’ve generated 100.000 self-signed SSL certificates using EasyRSA (2048 bits RSA), all of them stored in the
/etc/ssl/haproxyfolder.–
Here are the result I am seeing when I reload HAProxy :
- with 10.000 certificates
time service haproxy reload [ ok ] Reloading haproxy: haproxy. service haproxy reload 49,56s user 6,02s system 97% cpu 56,987 total
- with 20.000 certificates
time service haproxy reload [ ok ] Reloading haproxy: haproxy. service haproxy reload 94,24s user 9,59s system 99% cpu 1:44,11 total
- with 30.000 certificates
time service haproxy reload [ ok ] Reloading haproxy: haproxy. service haproxy reload 143,05s user 14,41s system 99% cpu 2:37,94 total
- with 40.000 certificates
time service haproxy reload [ ok ] Reloading haproxy: haproxy. service haproxy reload 190,40s user 23,97s system 96% cpu 3:41,17 total
- with 50.000 certificates
The process gets killed with an out of memory error.
–
The reload process seems to be pretty CPU heavy (100% usage during the whole reload process), but to only use one of the available vCPUs.
It would also seem HAProxy takes 5 more seconds to reload per 1.000 certificates I add. Is there any way to speed up the reload process ? I have not found anything regarding this in the documentation.
I’m also suprised to get an out of memory issue while loading ~400MB certificates. Am I missing something obvious ?
I also had a benchmark running during the reloads (1000 request per seconds towards the HAProxy internal monitoring URL I configured on /ping). No requests were lost during any of my tests, which is quite impressive !
Any help or tips would be appreciated !
Thanks !
Posts: 3
Participants: 2