SSL handshake failure hangs HAProxy

@kustodian wrote:

We are using HAProxy 1.8.9, but the same thing happens on 1.7.x versions. From time to time we get the following messages in HAProxy log (source IP is hidden):

Jul 12 15:43:36 hap-01 haproxy[26141]: x.y.z.w:47996 [12/Jul/2018:15:43:36.514] www-https/1: SSL handshake failure
Jul 12 15:43:37 hap-01 haproxy[26141]: x.y.z.w:48986 [12/Jul/2018:15:43:37.071] www-https/1: SSL handshake failure
Jul 12 15:43:37 hap-01 haproxy[26141]: x.y.z.w:49222 [12/Jul/2018:15:43:37.255] www-https/1: SSL handshake failure
Jul 12 15:43:37 hap-01 haproxy[26141]: x.y.z.w:49922 [12/Jul/2018:15:43:37.833] www-https/1: SSL handshake failure
Jul 12 15:43:38 hap-01 haproxy[26141]: x.y.z.w:50294 [12/Jul/2018:15:43:38.188] www-https/1: SSL handshake failure
Jul 12 15:43:53 hap-01 haproxy[26141]: x.y.z.w:55843 [12/Jul/2018:15:43:48.203] www-https/1: SSL handshake failure
Jul 12 15:44:03 hap-01 haproxy[26141]: x.y.z.w:60427 [12/Jul/2018:15:43:53.214] www-https/1: SSL handshake failure
Jul 12 15:44:08 hap-01 haproxy[26141]: x.y.z.w:60423 [12/Jul/2018:15:43:53.214] www-https/1: SSL handshake failure
Jul 12 15:44:08 hap-01 haproxy[26141]: x.y.z.w:50304 [12/Jul/2018:15:43:43.195] www-https/1: SSL handshake failure
Jul 12 15:44:13 hap-01 haproxy[26141]: x.y.z.w:60381 [12/Jul/2018:15:43:53.214] www-https/1: SSL handshake failure
Jul 12 15:44:28 hap-01 haproxy[26141]: x.y.z.w:50308 [12/Jul/2018:15:43:43.195] www-https/1: SSL handshake failure
Jul 12 15:44:38 hap-01 haproxy[26141]: x.y.z.w:52772 [12/Jul/2018:15:44:13.234] www-https/1: SSL handshake failure
Jul 12 15:44:43 hap-01 haproxy[26141]: x.y.z.w:50255 [12/Jul/2018:15:44:38.256] www-https/1: SSL handshake failure
Jul 12 15:44:48 hap-01 haproxy[26141]: x.y.z.w:44924 [12/Jul/2018:15:44:38.256] www-https/1: SSL handshake failure
Jul 12 15:44:48 hap-01 haproxy[26141]: x.y.z.w:34625 [12/Jul/2018:15:44:28.246] www-https/1: SSL handshake failure

During this time HAProxy becomes unresponsive. We are monitoring HAProxy with Prometheus, and the haproxy_exporter reports the following error in it’s log:

time="2018-07-12T15:44:04+02:00" level=error msg="Can't scrape HAProxy: Get http://localhost:9000/;csv: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="haproxy_exporter.go:304" 
time="2018-07-12T15:44:09+02:00" level=error msg="Can't scrape HAProxy: Get http://localhost:9000/;csv: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="haproxy_exporter.go:304" 
time="2018-07-12T15:44:34+02:00" level=error msg="Can't scrape HAProxy: Get http://localhost:9000/;csv: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="haproxy_exporter.go:304" 
time="2018-07-12T15:44:39+02:00" level=error msg="Can't scrape HAProxy: Get http://localhost:9000/;csv: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="haproxy_exporter.go:304" 

At first we thought that it is a problem with the exporter, but one of our devs managed to login into the host during this period and his curl requests also timed out. This HAProxy isn’t that active, but this happens on more active ones from time to time. You can view a snapshot of the Grafana dashboard for this host with all metrics collected by node_exporter. Looking at the metrics I would say the there was some connection spam, but doesn’t look anything special to be able to make HAProxy unresponsive.

Do you have an idea what we can do to resolve this issue? Maybe we can enable additional logging or something to get more information.

Posts: 1

Participants: 1

Read full topic