Server Disconnects / TCP Window Full

Hello together,

we are using HAproxy widely, but we’re actually facing a strange problem, I can’t get behind and hope, someone can help

Our constellation: Client => HAproxy => Dockerhost with Camunda-Container

To classify the IPs in the attached config, here are some Infos:

Client: 10.8.11.150
HAproxy: 10.1.2.180
Backend (Docker with Camunda-Container): 10.1.3.103

If the Client sends small POST-Requests, all works fine. When the request are getting bigger, you see the following in the HAproxy-Log:

Apr 28 11:59:26 kag-lx-pxy-t1 haproxy[6979]: 10.8.11.150:58270 [28/Apr/2023:11:59:26.804] haproxy-intern-test-frontend-http backend_camunda-dev/kag-docker-03 0/0/0/2/101 401 149 - - SD-- 12/1/0/0/0 0/0 "POST /engine-rest/message HTTP/1.1"

First, I thought, that https in the frontend is the problem, so I enabled http for the frontend. Unfortunately, this doesn’t help. We made further tests with the client communicating directly to the Docker-Host and the problem disappeared. So, I think, the problem is HAproxy related.

In the next step, I made tcpdumps on frontend- and backend-side. Here the TCP window size messages in the backend communication were strange, while the client continued to talk diligently with the frontend. I think this is where the problem lies, but I don’t know, how to resolve this issue.

Any help on this topic? What else can I do to narrow down the problem?

Attached, some usefull informations (config, tcpdump-screenshots)

thanks and best regards,
markus

root@kag-lx-pxy-t1:~# haproxy -vv
HAProxy version 2.6.12-1ppa1~bionic 2023/04/01 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.12.html
Running on: Linux 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-rnXPxH/haproxy-2.6.12=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -QUIC +RT +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

global
        log /dev/log    local0 info
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats socket /var/lib/haproxy/stats
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 60000

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

        tune.ssl.default-dh-param 2048

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
        bind *:8000
        mode http
        stats enable
        stats realm Haproxy\ Statistics
        stats uri /
        stats refresh 5s

peers haproxy-intern-test-peers
        peer kag-lx-pxy-t1 10.1.2.180:50000

######################
#                    #
#  Frontend - HTTP   #
#                    #
######################

frontend haproxy-intern-test-frontend-http
        bind *:80

        acl valid_http_method method GET HEAD POST PUT DELETE OPTIONS
        http-request deny if ! valid_http_method

        # Umleitung von http zu https (für bestimmte Webs unterdrücken, da sie auch per http erreichbar sein müssen)
        redirect scheme https code 301 if !{ ssl_fc } !{ hdr_beg(host) -i camunda-test } !{ hdr_beg(host) -i camunda-dev }

        option forwardfor
        default_backend haproxy-intern-test-backend_no_match

######################
#                    #
# Backend-Zuordnung  #
#        HTTP        #
#                    #
######################

        use_backend backend_camunda-test                if { hdr_beg(host) -i camunda-test }
        use_backend backend_camunda-dev                 if { hdr_beg(host) -i camunda-dev }

######################
#                    #
#  Frontend - HTTPS  #
#                    #
######################

frontend haproxy-intern-test-frontend-https
        bind *:443 ssl crt /etc/haproxy/ssl/ alpn h2,http/1.1

        acl valid_http_method method GET HEAD POST PUT DELETE OPTIONS
        http-request deny if ! valid_http_method

        http-request set-header X-Forwarded-Host %[req.hdr(Host)]
        http-request set-header X-Forwarded-Proto https

        http-response set-header Strict-Transport-Security max-age=15552000;\ includeSubdomains

        option forwardfor

        default_backend haproxy-intern-test-backend_no_match

######################
#                    #
# Backend-Zuordnung  #
#       HTTPS        #
#                    #
######################

        use_backend backend_camunda-dev                 if { hdr_beg(host) -i camunda-dev }
        use_backend backend_camunda-test                if { hdr_beg(host) -i camunda-test }

######################
#                    #
#      Backends      #
#                    #
######################

#Default Backend -> 403 ausgeben!
backend haproxy-intern-test-backend_no_match
        http-request deny deny_status 403

#########################################################

backend backend_camunda-dev
        stick-table type ip size 200k expire 30m peers haproxy-intern-test-peers
        stick on src
        server docker-03 10.1.3.103:9002 check inter 10000
        server docker-04 10.1.3.104:9002 check inter 10000 backup

        option httpchk GET /
               http-check expect status 200
               http-check send-state

        option redispatch
        option forwardfor

        http-response del-header Server
        http-response del-header X-Powered-By

#########################################################

backend backend_camunda-test
        stick-table type ip size 200k expire 30m peers haproxy-intern-test-peers
        stick on src
        server docker-03 10.1.3.103:9102 check inter 10000
        server docker-04 10.1.3.104:9102 check inter 10000 backup

        option httpchk GET /
               http-check expect status 200
               http-check send-state

        option redispatch
        option forwardfor

        http-response del-header Server
        http-response del-header X-Powered-By

1 post - 1 participant

Read full topic