Hello,
I faced a segfault in HAProxy when using tcp-check connect configuration setup. Google showed a lot of people having the same issue and saying the bug has been fixed in version 1.9.6 or later versions (https://www.mail-archive.com/haproxy@formilux.org/msg33141.html).
However I check with the last available version at https://haproxy.debian.net/ and still have the issue.
I have made several tests and it turns out that the more frontends / backends you have the more frequently the coredumps occurs. With 4 frontends I had this about 4 times a day, with over 100 frontend it happens every 2 or 3 minutes event if only one of them is using the tcp-check connect.
In dmsg I got logs like that:
[Mon Jul 1 10:55:38 2019] haproxy[29988]: segfault at 10 ip 000055d2a1731e85 sp 00007f01f37e2390 error 4 in haproxy[55d2a1660000+1e6000]
[Mon Jul 1 10:55:58 2019] haproxy[30027]: segfault at 10 ip 0000560f4a9e7e85 sp 00007f94fd508390 error 4 in haproxy[560f4a916000+1e6000]
[Mon Jul 1 10:56:11 2019] haproxy[30114]: segfault at 10 ip 000056534d544e85 sp 00007fa56ac1a390 error 4 in haproxy[56534d473000+1e6000]
[Mon Jul 1 10:56:16 2019] haproxy[30150]: segfault at 6 ip 00005565aa6b4203 sp 00007f5b8ee442f0 error 4
[Mon Jul 1 10:56:16 2019] haproxy[30151]: segfault at 6 ip 00005565aa6b4203 sp 00007f5b8e6432f0 error 4
[Mon Jul 1 10:56:16 2019] in haproxy[5565aa5e4000+1e6000]
[Mon Jul 1 10:56:16 2019] in haproxy[5565aa5e4000+1e6000]
[Mon Jul 1 10:57:12 2019] traps: haproxy[30168] general protection ip:55f8194a1ec8 sp:7f9fb7606390 error:0
[Mon Jul 1 10:57:12 2019] in haproxy[55f8193d0000+1e6000]
[Mon Jul 1 10:57:17 2019] haproxy[30353]: segfault at 6 ip 000055a43df1b203 sp 00007f4fa68832f0 error 4 in haproxy[55a43de4b000+1e6000]
[Mon Jul 1 10:57:29 2019] haproxy[30378]: segfault at 6 ip 000056028e238203 sp 00007f88d36882f0 error 4 in haproxy[56028e168000+1e6000]
How to reproduce on Debian Stretch:
- Install HAproxy 1.9.x
- Add
EXTRAOPTS="-f /etc/haproxy/sites-enabled/" in /etc/default/haproxy
- Use the default
haproxy.cfg provided by the package and add nbthread 4 in the global section (I didn’t manage to trigger the bug without adding threads).
- generate extra configuration using the following script:
#!/bin/bash
cd /etc/haproxy
mkdir -p sites-enabled
rm sites-enabled/*
cat <<EOF > sites-enabled/00-stats.cfg
listen stats
bind *:80
stats uri /
stats admin if TRUE
EOF
for i in $(seq 50); do
cat <<EOF > sites-enabled/front-$i.cfg
listen frontend$i
mode tcp
option tcplog
bind 127.0.0.$i:12345
option tcp-check
tcp-check connect port 25
tcp-check expect rstring "^220 "
tcp-check send "quit\r\n"
server 127.0.0.1 127.0.0.1:25 check inter 500ms
EOF
done
- Make sure you have a smtp server installed on localhost (exim, postfix, opensmtpd would do the job).
- restart HAProxy and wait.
I can provide further details if required.
My HAProxy installed version on a Debian Stretch.
HA-Proxy version 1.9.8-1~bpo9+1 2019/05/18 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-1.9.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.22 2016-07-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
