"SSL handshake failures" on big amount of requests

@SorokinPA wrote:

Hi guys,
I'd appreciate if anyone can give me couple of suggestions for the issue I have with SSL.
I know that sounds like certificate issue, but it happens only when I have big spike of new connections.

I am running haproxy 1.5.14 on Azure and using SSL termination.
Haproxy works perfectly well when load rises gradually, but everything goes bad if I have instant load.

In normal situation qmax goes up to 3000 and per thread and cpu core is loaded not higher than 75%.

So if I restart haproxy during daily load, haproxy might fill CPU usage up to 100% and be unable to handle more than 700-800 requests per thread.
When it comes to that limit, I see rate of new requests lowers down to 2-5
Haproxy log become mostly filled with tls/1: SSL handshake failure errors.

If I add more haproxy instances into balance, it becomes normal.

I don't have issues with entropy:

cat /proc/sys/kernel/random/entropy_avail
885

I tried to add conneciton rate limits:

maxsessrate 100
maxsslrate 100
maxconnrate 100
that had no effect. Everything stops at about 800 connections and then whole log filled with SSL handshake failures.

I tried to play around with timeouts
changed timeout connect as:
- 500
- 50000
- 30s

No effect

Can anyone suggest anything here? I have no idea how to debug that.

Here is the config file I use:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        stats socket /var/run/haproxy.p1.sock mode 660 group nagios level admin process 1
        stats socket /var/run/haproxy.p2.sock mode 600 level admin process 2
        stats socket /var/run/haproxy.p3.sock mode 600 level admin process 3
        stats socket /var/run/haproxy.p4.sock mode 600 level admin process 4
        stats timeout 2m  #Wait up to 2 minutes for input
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        nbproc 4
        cpu-map 1 0  # first arg is process number (1-based); second arg is cpu number (0-based)
        cpu-map 2 1
        cpu-map 3 2
        cpu-map 4 3

        # SSL/TLS settings
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        tune.ssl.default-dh-param 2048
        tune.ssl.cachesize 10000000
        tune.ssl.lifetime 86400
        #tune.ssl.maxrecord 2859
        tune.ssl.maxrecord 1400  # TCP window size
        ssl-default-bind-options no-sslv3 no-tls-tickets
        ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

        maxconn 60000
        maxsslconn 60000
        # maxsessrate 100
        # maxsslrate 100
        # maxconnrate 100

defaults
        log     global
        option  dontlognull
        option  dontlog-normal
        timeout connect 5000
        timeout client 50000
        timeout server 50000
        bind-process all  # not needed, but worthwhile being explicit

listen stats
        bind :2100 process 1
        bind :2101 process 2
        bind :2102 process 3
        bind :2103 process 4
        mode http
        log global
        stats enable
        stats realm stats_process
        stats uri /
        stats refresh 15s
        stats show-legends
        stats show-node
        stats auth xxxxxxxxxxxxx

frontend tls
        mode tcp
        maxconn 60000
        option tcplog
        bind *:443 ssl crt-list /etc/ssl/private/certificates.txt npn http/1.1
        default_backend frontend_service

backend frontend_service
        mode tcp
        option tcplog
        option httpchk GET /status
        fullconn 60000

        # 2 second 'inter'val between health checks. 2 failures to remove a server. 2 successes to add it back
        default-server inter 8s fall 2 rise 2
        timeout check 8s

        balance leastconn

        server SRV1 SRV1:80 maxconn 2000 check port 3000
        ....
        server SRV60 SRV1:80 maxconn 2000 check port 3000

Thank you!
Pavel

Posts: 4

Participants: 2

Read full topic