Hi,

we are running haproxy 2.0.14 on our dev environment and it is crashing after sometime with code 134.

[ALERT] 101/035225 (1) : Current worker #1 (51) exited with code 134 (Aborted)
[ALERT] 101/035225 (1) : exit-on-failure: killing every processes with SIGTERM
[WARNING] 101/035225 (1) : All workers exited. Exiting… (134)

Can you please give some clue about the reason and what to do to get around of this problem ?
Thank you very much

Following are info of

1. haproxy -vv
2. uname -a
3. haproxy logs
4. haproxy.cfg

haproxy -vv
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
: mode=HTX side=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS

Available services : none

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

uname -a

Linux mnc-haproxy 4.18.0-240.el8.x86_64 #1 SMP Wed Sep 23 05:13:10 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

logs of haproxy :-:

Thread 3 is about to kill the process.

Thread 1 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
stuck=1 fdcache=1 prof=0 harmless=0 wantrdv=0
cpu_ns: poll=2119763113 now=4043048166 diff=1923285053
curr_task=0
Thread 2 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
stuck=1 fdcache=1 prof=0 harmless=0 wantrdv=0
cpu_ns: poll=2889080862 now=4914504896 diff=2025424034
curr_task=0
*>Thread 3 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
stuck=1 fdcache=1 prof=0 harmless=0 wantrdv=0
cpu_ns: poll=2280737678 now=5008029696 diff=2727292018
curr_task=0
…
…
…

Thread 15: act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
stuck=0 fdcache=1 prof=0 harmless=0 wantrdv=0
cpu_ns: poll=2215901996 now=2224364418 diff=8462422
curr_task=0x7fad6803c050 (tasklet) calls=2
fct=0x42c670=main+47584 (?) ctx=0x7fad80049220
Thread 16: act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
stuck=0 fdcache=0 prof=1 harmless=1 wantrdv=0
cpu_ns: poll=4711777691 now=6464605747 diff=1752828056
curr_task=0
[ALERT] 101/035225 (1) : Current worker #1 (51) exited with code 134 (Aborted)
[ALERT] 101/035225 (1) : exit-on-failure: killing every processes with SIGTERM
[WARNING] 101/035225 (1) : All workers exited. Exiting… (134)

haproxy.cfg

global
log stdout format raw local0 debug
tune.ssl.default-dh-param 2048
user otn
group otn
tune.http.maxhdr 32767

tune.maxrewrite 1024

    tune.bufsize 163840  
master-worker        

defaults
mode http
log global
option http-server-close
option httplog
option dontlognull
option redispatch
option forwardfor
retries 3
backlog 4096
timeout client 200s
timeout client-fin 20s
timeout connect 5s
timeout server 100s
timeout tunnel 15m
timeout http-keep-alive 30s
timeout http-request 10s
timeout queue 5s
timeout tarpit 60s
maxconn 10000
option accept-invalid-http-response
default-server inter 3s rise 2 fall 3 init-addr last,libc,none

frontend default_ports
bind *:80 user otn group otn
option httplog
http-request add-header X-Forwarded-Proto https
http-request redirect location https://"$APP_IP":"$port"/cas/login

frontend default_ports_443
bind *:443 user otn group otn ssl crt /usr/local/etc/cert.crt
option tcplog
acl fault_path path_beg -i /FaultManagement
use_backend nspos_serv1 if fault_path
default_backend nspos_serv

frontend web_gateway
bind *:"$port" ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
http-request set-query %[query,regsub(:"$port",g)]
http-request set-uri %[url,regsub(otntomcat,"$APP_IP",g)]
redirect scheme https if !{ ssl_fc }
acl mnc_spcms_path path_beg -i /spcms
use_backend mnc_spcms_serv if mnc_spcms_path
acl otntomcat_path path_beg -i /oms1350
acl dojoroot_path path_beg -i /dojoroot
acl p1 path_beg -i /jointroot
acl p2 path_beg -i /jquery
acl p3 path_beg -i /lodash
acl p4 path_beg -i /openlayersroot
acl p6 path_beg -i /svgutilsroot
acl p7 path_beg -i /d3root
acl p8 path_beg -i /backbone
acl p9 path_beg -i /Cpb
use_backend otntomcat if otntomcat_path || dojoroot_path || p1 || p2 || p3 || p4 || p6 || p7 || p8
acl cpb_path path_beg -i /Cpb
use_backend cpb_serv if cpb_path
acl cas_path path_beg -i /cas
http-request set-var(txn.path) path
acl session_path path_beg -i /session-manager
use_backend nspos_serv if cas_path || session_path
acl fault_path path_beg -i /FaultManagement
acl mncfm_help_path path_beg /HelpCenter
acl mncfm_search_path path_beg /SearchApp
use_backend nspos_serv1 if fault_path || mncfm_help_path || mncfm_search_path
acl int_path path_beg -i /internal
use_backend nspos_serv if int_path
use_backend cpb_serv if p9
acl ntsm_path path_beg -i /ntsm
use_backend ntsm_serv if ntsm_path
acl systemmonitor_path path_beg -i /systemmonitor
use_backend systemmonitor_serv if systemmonitor_path
acl wso_path path_beg -i /wso
use_backend wso_serv if wso_path
acl motn_path path_beg -i /onc
use_backend motn_serv if motn_path
acl nrct_tapi_path path_beg -i /tapi
use_backend nrct_tapi_serv if nrct_tapi_path
acl mnc-esm_path path_beg -i /esm
use_backend mnc-esm_serv if mnc-esm_path
acl mnc-map_path path_beg -i /mnc-map
use_backend mnc-map_serv if mnc-map_path
acl mncpm_path path_beg -i /mncpm
use_backend pm-components_serv if mncpm_path
acl search_path path_beg /search
use_backend otntomcat_search if search_path
default_backend nspos_serv

frontend nrct_tapi_frontend
bind *:8543 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl nrct_tapi_path path_beg -i /tapi
use_backend nrct_tapi_serv if nrct_tapi_path

frontend motn_frontend
bind *:8444 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl motn_path path_beg -i /onc
use_backend motn_serv if motn_path

frontend event_web_gateway
bind *:8443 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
http-request set-query %[query,regsub(:"$port",g)]
http-request set-uri %[url,regsub(otntomcat,"$APP_IP",g)]
redirect scheme https if !{ ssl_fc }
acl oms1350_path path_beg -i /oms1350
use_backend otntomcat if oms1350_path
acl search_path path_beg /search
use_backend otntomcat_search if search_path

frontend ntsm_frontend
bind *:8546 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl ntsm_path path_beg -i /ntsm
use_backend ntsm_serv if ntsm_path

frontend mnc_admin_frontend2
bind *:8548 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl systemmonitor_path path_beg -i /systemmonitor
use_backend systemmonitor_serv if systemmonitor_path

frontend wso_frontend
bind *:8549 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl wso_path path_beg -i /wso
use_backend wso_serv if wso_path

frontend mnc-esm_frontend
bind *:8547 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl mnc-esm_path path_beg -i /esm
use_backend mnc-esm_serv if mnc-esm_path

frontend cpb_frontend
bind *:8001 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl cpb_path path_beg -i /Cpb
use_backend cpb_serv if cpb_path

frontend pm_components_frontend
bind *:8751 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl mncpm_path1 path_beg -i /mncpm
acl mncpm_path2 path_beg -i /pmservice
use_backend pm-components_serv if mncpm_path1 || mncpm_path2

frontend mnc_spcms_frontend
bind *:8550 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl mnc_spcms_path path_beg -i /spcms
use_backend mnc_spcms_serv if mnc_spcms_path

frontend mnc_fm_frontend
bind *:8544 ssl crt /usr/local/etc/cert.crt
http-request add-header X-Forwarded-Proto https
redirect scheme https if !{ ssl_fc }
acl mncfm_help_path path_beg -i /HelpCenter
acl mnc_fm_path path_beg -i /FaultManagement
acl mncfm_search_path path_beg /SearchApp
use_backend nspos_serv1 if mnc_fm_path || mncfm_help_path || mncfm_search_path
default_backend nspos_serv

resolvers docker_nfmt-net
nameserver dns1 127.0.0.11:53

backend otntomcat
mode http
balance roundrobin
log stdout local0 debug
server serv01 otntomcat:8443 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\botntomcat\b)(.)$ \1"APP_IP"\3 http-response replace-header Location (.*)(\b:80\/\b)(.*) \1:"port"\/\3 http-response replace-header Location (.*)(\b:443\/\b)(.*) \1:"port"\/\3 http-response replace-header Location (.*)(\b:8544\/\b)(.*) \1:"port"\/\3 http-response replace-header Location (.*)(\b:8545\/\b)(.*) \1:"$port"/\3

http-response replace-header Location (.)(\b:8443/\b)(.)$ \1:"$port"/\3

http-response replace-header Location  (.*)(\b\/oms1350\/pages\/otn\b)(.*) \1:"$port"\/oms1350\/pages\/otn\3
http-response replace-header Location  (.*)(\b\/oms1350\/login\/cas\b)(.*) \1:"$port"\/oms1350\/login\/cas\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend otntomcat_search
mode http
balance roundrobin
log stdout local0 debug
server serv01 otntomcat:8443 resolvers docker_nfmt-net check ssl verify none

http-response replace-header Location (.)(\b:8443/\b)(.)(?=?) \1:"$port"/\3

http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend nspos_serv
mode http
balance roundrobin
no option checkcache
log stdout local0 debug
acl path_cas var(txn.path) -m beg -i /cas
acl c1 hdr(Location) urlp(service)
server srv01 nspos:443 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\botntomcat\b)(.)$ \1"$APP_IP"\3
http-response replace-header Location (.)(\b"$APP_IP"/cas/login\b)(.) \1"$APP_IP":"$port"/cas/login\3
http-response replace-header Location (.)(\b"$APP_IP"/oms1350/login\b)(.) \1"$APP_IP":"$port"/oms1350/login\3
http-response replace-header Location (.)(\b"$APP_IP"/session-manager/login\b)(.) \1"$APP_IP":"$port"/session-manager/login\3
http-response replace-header Location (.)(\b:"$port":"$port"/session-manager/login\b)(.) \1:"$port"/session-manager/login\3

http-response replace-header Location (.)(\b:8443/\b)(.)$ \1:"$port"/\3

http-response replace-header Location (.)(\b:8443/\b)(.)(?=?) \1:"$port"/\3

http-response replace-header Location   ^(https://[^:]+):(\d+)(.*) \1:"$port"\3
http-response replace-header Location  (.*)(\b\/cas\/logout\b)(.*) \1:"$port"\/cas\/logout\3
http-response replace-header Location   (.*)(\bhttps:\/\/(.*):)\b(.*) \1https:\/\/"$APP_IP":\4  if path_cas 
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend nspos_serv1
mode http
balance roundrobin
no option checkcache
server srv01 nspos:8544 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8544/\b)(.)$ \1:"$port"/\3

http-response replace-header Location (.)(\b:443/\b)(.)$ \1:"$port"/\3

http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend ntsm_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 ntsm:8546 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8546/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend systemmonitor_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 mnc-admin:8548 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8548/\b)(.)$ \1:"port"\/\3 http-response replace-header Location (.*)(\b:443\/\b)(.*) \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend wso_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 wso:8549 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8549/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend motn_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 otncore:8444 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8444/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend nrct_tapi_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 nrct-tapi:8543 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8543/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend mnc-map_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 mnc-map:8443 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8880/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend mnc-esm_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 mnc-esm:8547 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8547/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend cpb_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 mnc-cpb:8443 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8001/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend pm-components_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 pm-components:8751 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8751/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend mnc_spcms_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 mnc-syspref:8550 resolvers docker_nfmt-net check ssl verify none
http-response replace-header Location (.)(\b:8550/\b)(.)$ \1:"$port"/\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

listen stats
bind *:1974
stats enable
stats uri /

1 post - 1 participant

Read full topic