Backend server not receiving traffic after failover with log-forward and mode log

Hi,
I’m using HAProxy to load balance TCP syslog streams with log-forward and mode log in an active passive setup with keepalived. It seems to work great at first, but I noticed that after a failover, backend servers do not receive any connections. The floating IP does bind correctly and HTTP frontend and backends work fine, but the syslog backend servers do not receive any traffic whatsoever. Restarting HAProxy on the backup node after it has entered master state resolves the issue. This is especially weird since the Prometheus metric haproxy_process_recv_logs_total does happily report messages flowing in, but they never leave on the other end, tcpdump does suggest the same.

I did also notice a similar behavior if a backend server goes down and is reported up again. It does not receive traffic until HAProxy is restarted.

Here’s the full configuration:

# https://www.haproxy.com/documentation/

global
  log stdout format raw daemon
  log stderr format raw daemon notice

  log-tag alloy-haproxy
  log backend@alloy_syslog_tcp_rfc5424_a format rfc5424 local0

  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /path/to/cert
  crt-base /path/to/cert/private

  # https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options ssl-min-ver TLSv1.2

  tune.ssl.default-dh-param 2048

  # Performance and protection tweaks
  nbthread 5
  maxconn 60000

defaults
  log     global
  option  dontlognull
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

  # Ensure outgoing connections do not use the floating IP
  default-server source 10.0.0.2


# https://www.haproxy.com/documentation/haproxy-configuration-tutorials/authentication/basic-authentication/
userlist alloy_credentials
  user a password xxxx

userlist alloy_admin_credentials
  user b password xxxx

backend static_404
  mode http
  http-request deny deny_status 404

# https://www.haproxy.com/documentation/haproxy-configuration-tutorials/load-balancing/syslog/#load-balance-syslog-over-tcp
log-forward forward_syslog_tcp_rfc5424_a
  bind 10.0.0.2:10000 ssl crt /path/to/full.chain
  #Virtual IP
  bind 10.0.0.4:10000 ssl crt /path/to/full.chain

  timeout client 30s
  log backend@alloy_syslog_tcp_rfc5424_a format rfc5424 local4

backend alloy_syslog_tcp_rfc5424_a
  mode log
  balance roundrobin

  # We need to send a properly formatted message to avoid generating errors
  option tcp-check
  tcp-check send <14>1\ -\ lb01.my.tld\ haproxy-health-check\ -\ -\ -\ HAProxy\ health\ check

  # Server  names  must  be  unique, otherwise
  # logs will be sent to all  servers  sharing
  # the same name due to the ring buffer being
  # shared. This might be a bug.
  server server01_10000 10.0.0.10:10000 log-proto octet-count log-bufsize 65536 check on-marked-down shutdown-sessions
  server server02_10000 10.0.0.11:10000 log-proto octet-count log-bufsize 65536 check on-marked-down shutdown-sessions
  server server03_10000 10.0.0.12:10000 log-proto octet-count log-bufsize 65536 check on-marked-down shutdown-sessions

log-forward forward_syslog_tcp_rfc5424_b
  bind 10.0.0.2:10001 ssl crt /path/to/full.chain
  #Virtual IP
  bind 10.0.0.4:10001 ssl crt /path/to/full.chain

  timeout client 30s
  log backend@alloy_syslog_tcp_rfc5424_b format rfc5424 local4

backend alloy_syslog_tcp_rfc5424_b
  mode log
  balance roundrobin

  # We need to send a properly formatted message to avoid generating errors
  option tcp-check
  tcp-check send <14>1\ -\ lb01.my.tld\ haproxy-health-check\ -\ -\ -\ HAProxy\ health\ check

  # Server  names  must  be  unique, otherwise
  # logs will be sent to all  servers  sharing
  # the same name due to the ring buffer being
  # shared. This might be a bug.
  server server01_10001 10.0.0.10:10001 log-proto octet-count log-bufsize 65536 check on-marked-down shutdown-sessions
  server server02_10001 10.0.0.11:10001 log-proto octet-count log-bufsize 65536 check on-marked-down shutdown-sessions
  server server03_10001 10.0.0.12:10001 log-proto octet-count log-bufsize 65536 check on-marked-down shutdown-sessions
  timeout client 30s
  log backend@alloy_syslog_tcp_rfc5424_b format rfc5424 local4

backend alloy_http_prometheus
  mode http
  balance roundrobin
  option forwardfor
  http-request replace-path /prom(/)?(.*) /\2
  http-request auth unless { http_auth(alloy_credentials) }
  http-response set-header X-Server %s


  server server01_9000 10.0.0.10:9000 check
  server server02_9000 10.0.0.11:9000 check
  server server03_9000 10.0.0.12:9000 check

backend alloy_http_loki
  mode http
  balance roundrobin
  option forwardfor
  http-request replace-path /loki(/)?(.*) /\2
  http-request auth unless { http_auth(alloy_credentials) }
  http-response set-header X-Server %s

  option httpchk GET /ready
  http-check expect status 200

  server server01_9001 10.0.0.10:9001 check
  server server02_9001 10.0.0.11:9001 check
  server server03_9001 10.0.0.12:9001 check

backend alloy_http_otlp
  mode http
  balance roundrobin
  option forwardfor
  http-request replace-path /otlp(/)?(.*) /\2
  http-request auth unless { http_auth(alloy_credentials) }
  http-response set-header X-Server %s


  server server01_9002 10.0.0.10:9002 ssl verify none check
  server server02_9002 10.0.0.11:9002 ssl verify none check
  server server03_9002 10.0.0.12:9002 ssl verify none check

backend alloy_http_admin_server01
  mode http
  option forwardfor
  http-request auth unless { http_auth(alloy_admin_credentials) }
  http-response set-header X-Server %s

  # https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#cookie
  cookie alloy_host insert nocache
  server server01 10.0.0.10:443 ssl verify none check cookie server01
backend alloy_http_admin_server02
  mode http
  option forwardfor
  http-request auth unless { http_auth(alloy_admin_credentials) }
  http-response set-header X-Server %s

  # https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#cookie
  cookie alloy_host insert nocache

  server server02 10.0.0.11:443 ssl verify none check cookie server02
backend alloy_http_admin_server03
  mode http
  option forwardfor
  http-request auth unless { http_auth(alloy_admin_credentials) }
  http-response set-header X-Server %s

  # https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#cookie
  cookie alloy_host insert nocache

  server server03 10.0.0.12:443 ssl verify none check cookie server03

frontend metrics
  mode    http
  option  httplog

  bind 10.0.0.2:8080
  bind 10.0.0.2:8443 ssl crt /path/to/full.chain
  default_backend static_404

  http-request set-log-level silent if { path /metrics }
  http-request use-service prometheus-exporter if { path /metrics }

frontend www
  mode    http
  option  httplog

  bind 10.0.0.2:80
  bind 10.0.0.2:443 ssl crt /path/to/full.chain

  # Virtual IP
  bind 10.0.0.4:80
  bind 10.0.0.4:443 ssl crt /path/to/full.chain

  monitor-uri /health
  default_backend static_404

  stats enable
  stats uri /stats
  stats refresh 10s
  stats hide-version

  monitor fail if { nbsrv(alloy_http_prometheus) eq 0 }
  monitor fail if { nbsrv(alloy_http_loki) eq 0 }
  monitor fail if { nbsrv(alloy_http_otlp) eq 0 }

  monitor fail if { nbsrv(alloy_http_admin_server01) eq 0 }
  monitor fail if { nbsrv(alloy_http_admin_server02) eq 0 }
  monitor fail if { nbsrv(alloy_http_admin_server03) eq 0 }

  http-request set-log-level silent if { path /health } || { path /stats }

  stick-table type ip size 100k expire 30s store http_req_rate(10s)
  http-request track-sc0 src
  http-request deny deny_status 429 if { sc_http_req_rate(0) gt 30 }

  use_backend alloy_http_prometheus if { path /prom } || { path_beg /prom/ }
  use_backend alloy_http_loki if { path /loki } || { path_beg /loki/ }
  use_backend alloy_http_otlp if { path /otlp } || { path_beg /otlp/ }

  use_backend alloy_http_admin_server01 if { urlp(alloy_host) server01 } || !{ urlp(alloy_host) -m found } { cook(alloy_host) server01 }
  use_backend alloy_http_admin_server02 if { urlp(alloy_host) server02 } || !{ urlp(alloy_host) -m found } { cook(alloy_host) server02 }
  use_backend alloy_http_admin_server03 if { urlp(alloy_host) server03 } || !{ urlp(alloy_host) -m found } { cook(alloy_host) server03 }

  http-response set-log-level silent if { status lt 400 }

haproxy -vv:

HAProxy version 3.0.8-1ppa1~jammy 2025/01/29 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.8.html
Running on: Linux 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64
Build options :
  TARGET  = linux-glibc
  CC      = x86_64-linux-gnu-gcc
  CFLAGS  = -O2 -g -fwrapv -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   =

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=6).
Built with OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
Running on OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [  OT] opentracing
        [SPOE] spoe
        [TRACE] trace

1 post - 1 participant

Read full topic