I am trying to use haproxy 2.8.5-stable or 2.9.0 with WolfSSL 5.6.4-stable but I am not getting any replies from HAProxy. If I compile haproxy with OpenSSL 3.0.2 or 1.1.1w it works fine.
I also tried this patch:
which is for HAProxy 2.4-dev18 with same results.
I am using ubuntu 22.04 server.
In both cases (openssl, wolfssl) I am hitting HAProxy with:
$ wrk -t12 -c400 -d100s -H"Connection: Close" "https://my-public-reachable-hostname?code=hi"
The only things that differs in the following two scenarios is the SSL shared library the haproxy binary is linked with:
1. With openssl
compiled openssl 1.1.1w, and also used apt’s 3.0.2 same behaviour, haproxy builds and works fine:
$ make clean && make -j $(nproc) TARGET=linux-glibc USE_OPENSSL=1 USE_SYSTEMD=1
$ ldd haproxy
linux-vdso.so.1 (0x00007ffdacd4f000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x000015213dbf3000)
libssl.so.1.1 => /usr/local/lib/libssl.so.1.1 (0x000015213db5a000)
libcrypto.so.1.1 => /usr/local/lib/libcrypto.so.1.1 (0x000015213d86b000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x000015213d7a4000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000015213d57c000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x000015213d54f000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x000015213d480000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x000015213d460000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x000015213d455000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x000015213d317000)
/lib64/ld-linux-x86-64.so.2 (0x000015213e1b4000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x000015213d2ef000)
$ sudo ./haproxy -V -d -f /etc/haproxy/haproxy.cfg
[NOTICE] (137738) : haproxy version is 2.8.5-aaba8d0
[NOTICE] (137738) : path to executable is ./haproxy
[WARNING] (137738) : config : Proxy 'pub-https': no-sslv3/no-tlsv1x are ignored for bind 'HAPROXY_PUBLIC_REACHABLE_IP:443' at [/etc/haproxy/haproxy.cfg:77]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
[WARNING] (137738) : config : Proxy 'int-https': no-sslv3/no-tlsv1x are ignored for bind 'HAPROXY_PRIVATE_REACHABLE_IP:443' at [/etc/haproxy/haproxy.cfg:119]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Note: setting global.maxconn to 799929.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Using epoll() as the polling mechanism.
00000000:int-https.accept(000b)=00b2 from [IP_OF_WRK:50664] ALPN=<none>
00000000:int-https.clireq[00b2:ffffffff]: GET / HTTP/1.1
00000000:int-https.clihdr[00b2:ffffffff]: host: my_reachable_hostname
00000000:abc_c39_session_id.srvrep[00b2:0115]: HTTP/1.1 200 OK
00000000:abc_c39_session_id.srvhdr[00b2:0115]: content-type: text/plain
00000000:abc_c39_session_id.srvhdr[00b2:0115]: content-length: 0
00000000:abc_c39_session_id.srvhdr[00b2:0115]: date: Wed, 13 Dec 2023 17:27:14 GMT
00000000:abc_c39_session_id.srvcls[00b2:0115]
00000000:abc_c39_session_id.clicls[00b2:0115]
00000000:abc_c39_session_id.closed[00b2:0115]
...
getting responses to client (wrk)
$ make clean && sudo make uninstall
2. with WolfSSL
Downloaded wolfssl-5.6.4-stable from:
https://github.com/wolfSSL/wolfssl/archive/refs/tags/v5.6.4-stable.tar.gz
$ ./autogen.sh
$ ./configure --enable-haproxy
also tried:
$ ./configure --enable-opensslextra --enable-haproxy
$ make -j $(nproc)
no errors during building, but the following check reports one:
$ make check
...
SKIP: scripts/openssl.test
SKIP: scripts/external.test
PASS: scripts/trusted_peer.test
PASS: scripts/resume.test
PASS: scripts/google.test
PASS: scripts/tls13.test
PASS: scripts/crl-revoked.test
PASS: scripts/ocsp.test
PASS: scripts/pem.test
PASS: scripts/ocsp-stapling.test
PASS: scripts/ocsp-stapling-with-ca-as-responder.test
PASS: scripts/ocsp-stapling2.test
PASS: testsuite/testsuite.test
FAIL: scripts/unit.test
============================================================================
Testsuite summary for wolfssl 5.6.4
============================================================================
# TOTAL: 14
# PASS: 11
# SKIP: 2
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
...
$ sudo make install
*** Back in haproxy 2.8.5 (clean dir) ***
$ make clean && make -j $(nproc) TARGET=linux-glibc USE_OPENSSL_WOLFSSL=1 USE_SYSTEMD=1 SSL_INC=/usr/local/include/wolfssl SSL_LIB=/usr/local/lib
$ sudo make install
$ ldd haproxy
linux-vdso.so.1 (0x00007fffa8dd0000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x000014ddf962a000)
libwolfssl.so.41 => /usr/local/lib/libwolfssl.so.41 (0x000014ddf931a000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x000014ddf9253000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ddf902b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ddf8f44000)
/lib64/ld-linux-x86-64.so.2 (0x000014ddf9bea000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x000014ddf8f17000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x000014ddf8e48000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x000014ddf8e28000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x000014ddf8e1d000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x000014ddf8cdf000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x000014ddf8cb7000)
Now I get no HTTP_responses/output:
$ sudo ./haproxy -V -d -f /etc/haproxy/haproxy.cfg
[NOTICE] (155142) : haproxy version is 2.8.5-aaba8d0
[NOTICE] (155142) : path to executable is ./haproxy
[WARNING] (155142) : config : Proxy 'pub-https': no-sslv3/no-tlsv1x are ignored for bind 'haproxy_public_ipv6_here:443' at [/etc/haproxy/haproxy.cfg:77]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
[WARNING] (155142) : config : Proxy 'int-https': no-sslv3/no-tlsv1x are ignored for bind 'haproxy_private_ip_here:443' at [/etc/haproxy/haproxy.cfg:119]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Note: setting global.maxconn to 799929.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Using epoll() as the polling mechanism.
… no more output here … stats page also empty
system info
uname -a
Linux myhostname 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
/etc/haproxy/haproxy.cfg
global
daemon
quiet
nbthread 48
cpu-map auto:1/1-48 24-47,48-71
# cpu-map auto:1/1-48 0-47
user haproxy
group haproxy
chroot /var/lib/haproxy
# these were commented out so that it starts with wolfssl too:
# tune.ssl.cachesize 100000000
# tune.ssl.cachesize 20000000
# tune.ssl.ssl-ctx-cache-size 100000
# tune.ssl.lifetime 3000s
log /dev/log local0
log /dev/log local1 notice
no log
# these were commented out so that it starts with wolfssl too:
# ssl-default-bind-options ssl-min-ver TLSv1.0 no-sslv3 no-tls-tickets
# ssl-default-bind-ciphers 6:kEDH+AESGCM:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:@SECLEVEL=0
# ssl-default-server-options ssl-min-ver TLSv1.0 no-sslv3 no-tls-tickets
# ssl-default-server-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES256-GCM-SHA384:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
defaults
no log
mode http
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
option redispatch
retries 3
maxconn 300000
# bind-process 1
listen stats
bind 0.0.0.0:8888
stats enable
stats uri /haproxy?stats
stats auth usr:pass
stats refresh 5s
# bind-process 1-64
# acl network_allowed src 10.0.0.0/16
# tcp-request connection reject if !network_allowed
frontend pub-http
# bind-process 1-64
bind haproxy_public_ip_here:80
bind haproxy_public_ipv6_here:80 v6only
mode http
option forwardfor
http-request add-header X-Forwarded-Proto https
maxconn 6000000
timeout http-keep-alive 25000ms
timeout client 25000ms
timeout http-request 16000ms
option http-ignore-probes
acl abc urlp(sessionRoot) -m found
acl def urlp(code) -m found
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def
frontend pub-https
# bind-process 1-64
bind haproxy_public_ip_here:443
bind haproxy_public_ipv6_here:443 v6only ssl no-sslv3 crt /etc/haproxy/certs/ ssl-min-ver TLSv1.0 alpn h2,http/1.1
mode http
option forwardfor
http-request add-header X-Forwarded-Proto https
maxconn 6000000
timeout http-keep-alive 25000ms
timeout client 25000ms
timeout http-request 16000ms
option http-ignore-probes
acl abc urlp(sessionRoot) -m found
acl def urlp(code) -m found
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def
frontend int-http
# bind-process 1-64
bind haproxy_private_ip_here:80
mode http
option forwardfor
http-request add-header X-Forwarded-Proto https
maxconn 6000000
#timeout http-keep-alive 25000ms
#timeout client 25000ms
timeout client 8000ms
#timeout http-request 16000ms
#option http-ignore-probes
acl abc urlp(sessionRoot) -m found
acl def urlp(code) -m found
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def
frontend int-https
# bind-process 1-64
bind haproxy_private_ip_here:443 ssl no-sslv3 crt /etc/haproxy/certs/ ssl-min-ver TLSv1.0 alpn h2,http/1.1
mode http
option forwardfor
http-request add-header X-Forwarded-Proto https
maxconn 6000000
#timeout http-keep-alive 25000ms
#timeout client 25000ms
timeout client 8000ms
#timeout http-request 16000ms
#option http-ignore-probes
acl hname urlp(sessionRoot) -m found
acl def urlp(code) -m found
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def
...
backends configuration following ...
...
Any ideas how to troubleshoot this?
Thank you all
3 posts - 2 participants