Hi folks, need some help troubleshooting a 503 response being returned by haproxy.

Specifically, I am not sure how to interpret what haproxy is logging, we have a custom log format and haproxy is logging that the server IP it tried to connect to is the exact same IP as the frontend IP. This does not seem correct as the backend server has a completely different IP.

Initially, I had believed maybe we messed up the DNS record for the server and mistakenly added the frontend IP but at this point this does not seem to be the case.

"%fi": "10.255.1.2"  # frontend IP
"%fp": "443"         # frontend port
"%si": "10.255.1.2"  # server IP - why is this the same?
"%sp": "0"           # server port
"%bq": "0".          # backend queue 0
"%s": "srvr1"        # server name
"%rc" :"3"           # connection retries

timing shows high Tw but we don’t have “timeout queue” configured and everything else is set to 60s which if I’m understanding the docs correctly should result in using timeout connect of 60s so not sure how it’s even getting to almost 200s.

global
    maxconn 1000000 # we never get close to this connection limit

defaults
    timeout connect 60s
    timeout client 60s
    timeout server 60s
    mode http
    balance roundrobin
    retries 3
    option redispatch

# ... snip ...

backend artifactory:9582
    server-template srvr 1-25 artifactory.company.com:9582 ssl verify none crt /var/certs/identity.pem alpn h2,http/1.1 resolvers systemdns init-addr none maxconn 2000

"Ta":"+264005"
"Tc":"-1"
"Td":"-1"
"Th":"0"
"Ti":"0"
"Tq":"0"
"TR":"0"
"Tr":"-1"
"Tt":"+264005"
"Tu":"+264005"
"Tw":"198004" # This looks suspect to me but not sure what queue it's waiting on

2 posts - 1 participant

Read full topic

I’m using HAProxy to load balancing gRPC backends. Sometimes I need to temporarily remove a backend from backends list in the config file to upgrade it, then I re-add it to the list later. I’m curious what will happen to ongoing requests of the removed backend? Does HAProxy immediately close these requests when I reload the config?

2 posts - 2 participants

Read full topic

When a server’s IP changes during runtime, HAProxy does not resolve the hostname again when using external Python health check scripts. It will hold on to the old IP forever.

Here’s a timeline of events that triggers an issue for us:

1. HAProxy starts up and resolves the name of server.com to the IP 123.123.123.123. Backend is marked as UP.
2. The backend runs a python external health check script for the server every 10 seconds. Python resolves server.com by itself, does it’s tests against the server and keeps it UP.
3. The server.com IP changes to 234.234.234.234.
4. Python health checks resolve to the new IP, run their tests and everything works fine. Server kept UP.
5. A request we want to route to server.com comes from a client to HAProxy. HAProxy still has the old 123.123.123.123 IP configured. The request is routed there and we get a 404 as our expected service is no longer there (but it’s still a valid IP with a response). The 404 is returned to the client.
6. Python health check runs again, resolves to the new IP again and passes the checks. Server kept UP.

We haven’t found a way to force HAProxy to resolve names again at set intervals. Instead it will hold on to valid IPs until either restarted or reloaded.

Do you know any approach we could utilize here to make HAProxy to re-resolve a hostname and take the new IP into use even if the old IP is still functional?

Running our own custom health check scripts is a hard requirement.

Here are some of our related HAProxy configuration snippets. We are using HAProxy 2.9.

resolvers default
  parse-resolv-conf
  hold other           15s
  hold refused         15s
  hold nx              15s
  hold timeout         15s
  hold valid           10s
  hold obsolete        15s

backend server.com
  option external-check
  external-check command /health-check.py
  server server.com server.com:443 init-addr libc,none

Looking forward to any recommendations!

4 posts - 2 participants

Read full topic

Hi All,

I need some input as to whether I’m going about this the wrong way or not.
I am currently building a HaProxy Docker container. The goal is that the container specifies the global and default settings and these shouldn’t be changed (overwriting the defaults via frontend/backend is of course possible).

I remember that includes doesn’t work the same way as in nginx The question is to what extent my plan is technically possible.

The goal would be to have only two files in the pvc, one for the frontend config and the other for the backend config, (Or one file if that makes things easier).

But the way I see it, I can’t give HaProxy two folders where it pulls configs from, can I?
Thanks,

Alex

1 post - 1 participant

Read full topic

Hi,

I need to set up a proxy server that will be used by multiple clients in my network. The clients will send their internet requests through this proxy. The requirement for the proxy is that it should not be able to terminate the SSL connection or decrypt the message content. It simply needs to forward the encrypted traffic without needing to inspect the message.

I’m using the Server Name Indication (SNI) to determine which backend server to route the requests to. However, I’m facing an issue with some of my clients that use the Java Apache Client. When these clients send a request through a proxy, they first send a CONNECT request. In my case, this CONNECT request is reaching the backend servers, which should not be happening.

How can I configure HAProxy to only forward the SSL connection to the backend servers, and not the CONNECT request? I need HAProxy to handle the CONNECT request without passing it along to the backend.

2 posts - 2 participants

Read full topic

Hello,

I have setup an HAProxy service (version 2.8) with OPNSense and I’m struggling to make it work consistently.
I tried to follow a guide that was available on OPNsense forum as base for my setup.

Here is my current configuration for HAProxy

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 60s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): Frontend_HTTP ()

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80 
    bind 0.0.0.0:443 name 0.0.0.0:443 
    mode tcp
    default_backend SSL_backend

    # logging options
    option log-separate-errors
    option tcplog

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.10.10.2:80 name 127.10.10.2:80 accept-proxy 
    mode http
    option http-keep-alive

    # logging options
    option log-separate-errors
    option httplog
    # ACL: no-ssl
    acl acl_606b47270c87a3.09968459 ssl_fc

    # ACTION: HTTPRedirect
    http-request redirect scheme https code 301 if !acl_606b47270c87a3.09968459

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000"
    bind 127.10.10.2:443 name 127.10.10.2:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets strict-sni ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/66a957c77230c2.86454741.certlist 
    mode http
    option http-keep-alive

    # logging options
    option log-separate-errors
    option httplog
    # ACL: nc-host
    acl acl_606b4708173456.71367641 hdr_sub(host) -i nc.roxblic.xyz

    # ACTION: NC_rule
    use_backend NC_backend if acl_606b4708173456.71367641

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    server SSL_server 127.10.10.2 send-proxy-v2 check-send-proxy

# Backend: NC_backend ()
backend NC_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    http-reuse safe
    option forwardfor
    server NC_server 10.1.1.2:443 ssl verify none


# statistics are DISABLED

First of all why frontend needs to be prefixed with 0_ / 1_ ? Without these prefixes, I consistently encountered the error below when accessing the HTTP port. After adding the 0_ prefix to SNI_frontend, the error now only occurs approximately once in every four attempts.

$ curl -vv http://domain.tld/
* Host domain.tld:80 was resolved.
* IPv6: (none)
* IPv4: 10.1.1.2
*   Trying 10.1.1.2:80...
* Connected to domain.tld (10.1.1.2) port 80
> GET / HTTP/1.1
> Host: domain.tld
> User-Agent: curl/8.8.0
> Accept: */*
>
* Request completely sent off
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

Debug log corresponding in HAProxy :

2024-08-02T13:23:47	Informational	10.2.2.1:41028 [02/Aug/2024:13:23:47.399] SNI_frontend SSL_backend/SSL_server 1/0/22 0 -- 1/1/0/0/0 0/0	
2024-08-02T13:23:47	Error	127.10.10.2:19006 [02/Aug/2024:13:23:47.399] 1_HTTP_frontend/127.10.10.2:80: Received something which does not look like a PROXY protocol header

Also, I am getting SSL_ERROR_SYSCALL error (or PR_END_OF_FILE_ERROR with Chrome/Firefox) for HTTPS requests that occurs approximately one time out of every 2/3 attempts.

$ curl -vv https://domain.tld/
* Host domain.tld:443 was resolved.
* IPv6: (none)
* IPv4: 10.1.1.2
*   Trying 10.1.1.2:443...
* Connected to domain.tld (10.1.1.2) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to domain.tld:443
* Closing connection
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to domain.tld:443

Debug log corresponding in HAProxy :

2024-08-02T13:21:24	Informational	10.2.2.1:60118 [02/Aug/2024:13:21:24.339] SNI_frontend SSL_backend/SSL_server 1/0/19 0 -- 1/1/0/0/0 0/0	
2024-08-02T13:21:24	Error	127.10.10.2:15595 [02/Aug/2024:13:21:24.339] 2_HTTPS_frontend/127.10.10.2:443: Received something which does not look like a PROXY protocol header

I have run a TCP dump and it’s seems that sometimes I don’t receive the proxy header the first packet being “Client Hello”.

1 post - 1 participant

Read full topic

Hi everybody,

I’ve been trying to get my listener to attach to my Gateway in kubernetes, and I just can’t get it to work! Here’s what I’ve written till now:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
  namespace: bind9
  name: bind9-gatewayclass
spec:
  controllerName: haproxy.org/gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: bind9-gateway
  namespace: bind9
spec:
  gatewayClassName: bind9-gatewayclass
  listeners:
    - allowedRoutes:
        kinds:
          - group: gateway.networking.k8s.io
            kind: TCPRoute
        namespaces:
          from: All
      name: bind9-dns-listener
      port: 53
      protocol: TCP
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: bind9-dns-route
  namespace: bind9
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: bind9-gateway
      namespace: bind9
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: bind9-dns-service
          port: 53
---
apiVersion: v1
kind: Service
metadata:
  namespace: bind9
  name: bind9-dns-service
spec:
  selector:
    app: bind9
  ports:
    - name: bind9-dns
      protocol: TCP
      port: 53
      targetPort: 53
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bind9-deployment
  namespace: bind9
  labels:
    app: bind9
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bind9
  template:
    metadata:
      labels:
        app: bind9
    spec:
      containers:
      - name: bind9
        image: ubuntu/bind9:edge  # Use a pre-built BIND9 image
        ports:
        - containerPort: 53
          protocol: TCP
        args:
        - "/usr/sbin/named"
        - "-g"  # Start BIND9 in the foreground
        - "-c"
        - "/etc/bind/named.conf"

Sorry to make you go through such a long file, but any help would be greatly appreciated! The error message is:

$ kubectl describe gateway bind9-gateway -n bind9

 Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2024-08-06T03:42:19Z
      Message:
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2024-08-06T03:42:19Z
      Message:
      Observed Generation:   1
      Reason:                Attached
      Status:                False
      Type:                  Detached
      Last Transition Time:  2024-08-06T03:42:19Z
      Message:
      Observed Generation:   1
      Reason:                Ready
      Status:                True
      Type:                  Ready
    Name:                    bind9-dns-listener

Thanks!

1 post - 1 participant

Read full topic

Hello I am using the above together with Apache2 and Ubuntu on 2 seperate VPS servers with the same hosting companies.
Where do I get the virtual IP address? I was told to ask the hosting company but they have no clue what i am talking about. Is there a way to figure it out? Could I use the nameserver address as its the same for them both? There has to be a way? Thanks

1 post - 1 participant

Read full topic

Where i can find example of this ?

1 post - 1 participant

Read full topic

Hello everyone,

I am trying to update the expiry of stick table entries in HAProxy based on the value of a specific HTTP header. Specifically, I need to check if the header test contains the value 1, and if it does, set a new expiry time for the corresponding stick table entry.

1 post - 1 participant

Read full topic

Hello,
we have HAproxy set up with ~200 backend servers, and I noticed that the haproxy_server_response_time_average_seconds is acceptable (around 400ms) but the metrics for haproxy_server_total_time_average_seconds is almost around 2 minutes for each server, and would like to debug the issue. I would like to ask, what is the fundamental difference between response_time and total_time? Since I don’t know their components I don’t know why the difference is this big. Also would appreciate any ideas for debugging the issue. Thanks

1 post - 1 participant

Read full topic

Hi Team,

I would like to know if HA Proxy version 1.5 compatible on Oracle linux 8.9 as I have acknowledged that the HA Proxy was end of life but seems i could not found any official statement or documentation stated about compatibality issue. Could i know if it compatible or not?

If it not compatible, shall we proceed to directly upgrade from version 1.5 to the latest or we have to upgrade from 1.5 then proceed to 1.8 then to the latest?
Please let me know.

1 post - 1 participant

Read full topic

Hello everyone,

I currently have a KEMP server used for an Exchange 2016 Reverse Proxy that I want to replace with HAProxy. I configured everything to make it work and only one point does not work: Smartphones with ActiveSync.

If I use the Microsoft Outlook application on Android, it doesn’t work.
But with Gmail application, the same settings work.

This is the log with Gmail App :

2024-08-09T09:22:08.603356+02:00 S35 haproxy[58500]: 87.88.147.213:8224 [09/Aug/2024:09:22:08.563] ft_ex2019~ bk_ex2019_eas/exchange1 0/0/6/33/39 200 714 - - ---- 14/14/0/0/0 0/0 “POST /Microsoft-Server-ActiveSync?Cmd=Sync&User=domain.intra%5Cuser
&DeviceId=androidc672123383&DeviceType=Android HTTP/1.1”
2024-08-09T09:22:08.735679+02:00 S35 haproxy[58500]: 87.88.147.213:8224 [09/Aug/2024:09:22:08.702] ft_ex2019~ bk_ex2019_eas/exchange1 0/0/0/32/32 200 714 - - ---- 14/14/0/0/0 0/0 “POST /Microsoft-Server-ActiveSync?Cmd=Sync&User=domain.intra%5Cuser&DeviceId=androidc672123383&DeviceType=Android HTTP/1.1”
2024-08-09T09:22:21.102618+02:00 S35 haproxy[58500]: 87.88.147.213:8224 [09/Aug/2024:09:22:21.086] ft_ex2019~ bk_ex2019_eas/exchange1 0/0/0/16/16 200 989 - - ---- 14/14/0/0/0 0/0 “POST /Microsoft-Server-ActiveSync?Cmd=Settings&User=domain.intra%5Cuser&DeviceId=androidc672123383&DeviceType=Android HTTP/1.1”

The log with Outlook App

2024-08-09T10:09:19.907924+02:00 S35 haproxy[58500]: 52.97.187.109:54379 [09/Aug/2024:10:09:19.903] ft_ex2019~ bk_ex2019_eas/exchange1 0/0/1/3/4 401 259 - - ---- 13/13/0/0/0 0/0 “OPTIONS /Microsoft-Server-ActiveSync?Cmd=Options&User=domain.intra%5Cuser&DeviceId=OPCC286B2E6E33A3AAC152A4DE796B59&DeviceType=Outlook HTTP/1.1”
2024-08-09T10:09:19.928580+02:00 S35 haproxy[58500]: 52.97.187.109:54379 [09/Aug/2024:10:09:19.918] ft_ex2019~ bk_ex2019_eas/exchange1 0/0/0/9/9 200 1042 - - ---- 13/13/0/0/0 0/0 “OPTIONS /Microsoft-Server-ActiveSync?Cmd=Options&User=domain.intra%5Cuser&DeviceId=OPCC286B2E6E33A3AAC152A4DE796B59&DeviceType=Outlook HTTP/1.1”

Does anyone have a solution because I’m completely stuck there and it’s not possible for users to use anything other than the Outlook application on Mobile for their professional address.

HAProxy version : 2.6.12 on Debian 12

global

log         127.0.0.1:514 local0 debug
chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
user        haproxy
group       haproxy
daemon

h1-case-adjust www-authenticate WWW-Authenticate
h1-case-adjust persistent-auth Persistent-Auth

# turn on stats unix socket
stats socket /var/run/haproxy.stat

#--------------------------

SSL tuning / hardening

#--------------------------
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:DH+AES256:!aNULL:!MD5:!DSS
ssl-default-server-options no-sslv3
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:DH+AES256:!aNULL:!MD5:!DSS
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
option forwardfor except 127.0.0.0/8
option redispatch

option contstats

option h1-case-adjust-bogus-client
no option httpclose
no option http-server-close
retries                 3
timeout http-request    15s
timeout queue           1m
timeout connect         15s
timeout client          15m # this value should be rather high with Exchange
timeout server          15m # this value should be rather high with Exchange
timeout http-keep-alive 15s
timeout check           15s
maxconn                 100

#---------------------------------------------------------------------

Main front-end which proxies to the back-ends

#---------------------------------------------------------------------

frontend ft_ex2019
http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload
http-response set-header X-Frame-Options SAMEORIGIN
http-response set-header X-Content-Type-Options nosniff
option http-use-proxy-header
mode http
log /dev/log local0 info
bind *:443 ssl crt /etc/haproxy/ssl alpn h2,http/1.1
bind 10.10.10.35:1936 ssl crt /etc/haproxy/ssl
redirect scheme https code 301 if !{ ssl_fc } # redirect 80 → 443 (for owa)
acl autodiscover url_beg /Autodiscover
acl autodiscover url_beg /autodiscover
acl mapi url_beg /mapi
acl rpc url_beg /rpc
acl owa url_beg /owa
acl owa url_beg /OWA
acl eas url_beg /Microsoft-Server-ActiveSync
acl ecp url_beg /ecp
acl ews url_beg /EWS
acl ews url_beg /ews
acl oab url_beg /OAB

http-request deny if ecp

use_backend bk_ex2019_autodiscover if autodiscover
use_backend bk_ex2019_mapi if mapi
use_backend bk_ex2019_rpc if rpc
use_backend bk_ex2019_owa if owa
use_backend bk_ex2019_eas if eas

use_backend bk_ex2019_ecp if ecp

use_backend bk_ex2019_ews if ews
use_backend bk_ex2019_oab if oab
default_backend bk_ex2019

#------------------------------

Back-end section

#------------------------------

backend bk_ex2019_autodiscover
mode http
balance roundrobin
option httpchk GET /autodiscover/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

backend bk_ex2019_mapi
mode http
balance roundrobin
option httpchk GET /mapi/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

backend bk_ex2019_rpc
mode http
balance roundrobin
option httpchk GET /rpc/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

backend bk_ex2019_owa
mode http
balance roundrobin
option httpchk GET /owa/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

backend bk_ex2019_eas
mode http
log 127.0.0.1 local0 info
log 127.0.0.1 local0 warning
log 127.0.0.1 local0 err
option httpchk GET /microsoft-server-activesync/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

#backend bk_ex2019_ecp

mode http

balance roundrobin

option httpchk GET /ecp/healthcheck.htm

option log-health-checks

http-check expect status 200

server exchange1 SRV_Intern:443 check ssl inter 15s verify none

backend bk_ex2019_ews
mode http
log 127.0.0.1 local0 info
log 127.0.0.1 local0 warning
log 127.0.0.1 local0 err
option httpchk GET /ews/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 30s verify none

backend bk_ex2019_oab
mode http
option httpchk GET /oab/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

backend bk_ex2019
mode http
server exchange1 SRV_Intern:443 check ssl inter 15s verify none

Thank you.

1 post - 1 participant

Read full topic

Hi All,
we are trying that domain base routing in haproxy,in my backend server we are running 3 application. but when hit my url it navigate to only onethe domain.
Please find the below file haproxy configuration file.
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

    log-format "%ci %f %b %ST %HM %HU"

frontend http_front
bind *:80

#whitelist hosts

acl is_host1 hdr(host) -i portal.xxxx.com

acl is_host2 hdr(host) -i api.xxxxx.com
acl is_host3 hdr(host) -i skooner.xxxxxx.com

http-request redirect scheme https if is_host1
http-request redirect scheme https if is_host2
http-request redirect scheme https if is_host3

#letsencrypt
acl letsencrypt_acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt_back if letsencrypt_acl

#all unmatched hosts
default_backend deny_back

frontend https_front
bind *:443 ssl crt /etc/haproxy/certs/

#whitelist hosts
acl is_host1 hdr(host) -i portal.xxxx.com
acl is_host2 hdr(host) -i api.xxxx.com
acl is_host3 hdr(host) -i skooner.xxxx.com
#apps
use_backend https_back if is_host1
use_backend https_back if is_host2
use_backend https_back if is_host3

#websocket
acl is_websocket hdr(Connection) -i upgrade
#acl is_websocket hdr(Upgrade) -i websocket
use_backend ws_back if is_websocket

#all unmatched hosts
default_backend deny_back

backend deny_back
#errorfile 403 /var/www/403.http
#http-request deny deny_status 403
http-request silent-drop if TRUE

backend letsencrypt_back
server letsencrypt 127.0.0.1:54321

backend ws_back
#todo: add sticky session
#option http-server-close
#timeout tunnel 1h
server mdm-dev-backend-01 128.199.20.206:6000 check

backend https_back
balance roundrobin
server mdm-dev-apps-01 10.122.64.3:80 check
server mdm-dev-apps-02 10.122.64.4:80 check

1 post - 1 participant

Read full topic

Summary

Upgrading a HTTP connection to TCP no long work for versions greater than v2.3.
This is NOT about upgrading a connection in a TCP frontend using switch-mode http, nor is it about WebSocket. It’s about the Connection: Upgrade and Upgrade: tcp HTTP headers.

With HAProxy 2.2: client and server could upgrade the connection and converse bidirectionnaly using a proprietary binary format on the connection.

With HAProxy 3.0: proxy returns HTTP 200 (instead of expected 101) and connection upgrade fails.

Looking at traces, the problem seems to originate from the parsing of the HTTP request. The Upgrade header seems ignored and is not passed to the backend, so no upgrade takes place.

Looking at the code, the h1_parse_upgrade_header seems to ignore the Upgrade header unless it’s set to websocket.

Do you think this is the intended behavior? Or is it a shortcoming?

Context

The docker-socket-proxy project uses HAproxy to filter requests to the Docker daemon using ACLs (like here). It worked well with HAproxy v2.2 and broke when I updated HAproxy to v3.0.

The proxied protocol is the Docker Engine API. It’s mostly HTTP REST but for some endpoints, the connection is “hijacked”: upgraded to TCP, then used for streaming a proprietary binary format in both directions. Hijacking is detailed here in Docker doc.

An example of problematic call is the POST to /v1.46/exec/{id}/start.

Observations

Note: output of various commands were redacted for brevity

When working (HAproxy 2.2)

Proxy logs

dockerhttp dockerhttp/dockersocket 0/0/0/0/29 101 316 - - ---- 2/2/0/0/0 0/0 "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1"

Wireshark capture, between client and proxy

Hypertext Transfer Protocol
    POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1\r\n
    Connection: Upgrade\r\n
    Content-Type: application/json\r\n
    Upgrade: tcp\r\n

Hypertext Transfer Protocol
    HTTP/1.1 101 UPGRADED\r\n
    content-type: application/vnd.docker.multiplexed-stream\r\n
    connection: Upgrade\r\n
    upgrade: tcp\r\n

Transmission Control Protocol
    # A paquet with binary data that respect the docker.multiplexed-stream content-type

Traces

[04|h1|0|mux_h1.c:1534] H1 request fully rcvd : [F] [MSG_DONE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf090(0x00000000) h1s=0x7551d2ecf7b0(0x00000410) ibuf=280@0x7551d2a02f00+48/16384 obuf=0@0+0/0
	htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=118    - addr=0     	POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=118   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=136   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=174   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=190   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=207   	content-type: application/json
		[6] type=HTX_BLK_HDR       - size=10     - addr=235   	upgrade: tcp
		[7] type=HTX_BLK_EOH       - size=1      - addr=245   	<empty>
		[8] type=HTX_BLK_DATA      - size=29     - addr=246
		[9] type=HTX_BLK_EOM       - size=1      - addr=275   	<empty>

# snip

[04|h1|0|mux_h1.c:1739] sending request headers : [B] [MSG_RQBEFORE, MSG_RPBEFORE] - "POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00000010) ibuf=0@0+0/0 obuf=0@0x7551d2a10a50+0/16384
	htx=0x7551d2a07810(size=16336,data=276,used=10,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=9,tail_addr=276,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=118    - addr=0     	POST /v1.46/exec/a84c8fa7af83d2d6fb8438e107f82fc569284db962ba8f57f0794686c5cb7068/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=118   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=136   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=174   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=190   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=207   	content-type: application/json
		[6] type=HTX_BLK_HDR       - size=10     - addr=235   	upgrade: tcp
		[7] type=HTX_BLK_EOH       - size=1      - addr=245   	<empty>
		[8] type=HTX_BLK_DATA      - size=29     - addr=246
		[9] type=HTX_BLK_EOM       - size=1      - addr=275   	<empty>

# snip

[04|h1|0|mux_h1.c:1501] rcvd H1 response headers : [B] [MSG_TUNNEL, MSG_TUNNEL] - VAL=210 - "HTTP/1.1 101 UPGRADED" - h1c=0x7551d2ecf680(0x00000000) h1s=0x7551d2ecf9a0(0x00004020) ibuf=210@0x7551d2a02f00+0/16384 obuf=0@0+0/0
	htx=0x7551d2a07810(size=16336,data=201,used=9,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=8,tail_addr=201,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_RES_SL    - size=43     - addr=0     	HTTP/1.1 101 UPGRADED
		[1] type=HTX_BLK_HDR       - size=53     - addr=43    	content-type: application/vnd.docker.multiplexed-stream
		[2] type=HTX_BLK_HDR       - size=17     - addr=96    	connection: Upgrade
		[3] type=HTX_BLK_HDR       - size=10     - addr=113   	upgrade: tcp
		[4] type=HTX_BLK_HDR       - size=15     - addr=123   	api-version: 1.46
		[5] type=HTX_BLK_HDR       - size=24     - addr=138   	docker-experimental: false
		[6] type=HTX_BLK_HDR       - size=11     - addr=162   	ostype: linux
		[7] type=HTX_BLK_HDR       - size=27     - addr=173   	server: Docker/27.1.1 (linux)
		[8] type=HTX_BLK_EOH       - size=1      - addr=200   	<empty>

When broken (HAproxy 3.0)

Proxy logs

dockerhttp dockerhttp/dockersocket 0/0/0/0/20 200 156 - - ---- 1/1/0/0/0 0/0 "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1"

Wireshark capture, between client and proxy

Hypertext Transfer Protocol
    POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1\r\n
    Connection: Upgrade\r\n
    Content-Type: application/json\r\n
    Upgrade: tcp\r\n

Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
    content-type: application/vnd.docker.raw-stream\r\n
    connection: close\r\n

# No TCP packet with binary data

Traces

[02|h1|1|mux_h1.c:2087] H1 request fully rcvd : [F,EMB] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29)  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00000010) sd=0x7341fd80ec90(0x00500021) ibuf=280@0x7341fce8a530+48/16384 obuf=0@0+0/0
	htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=114    - addr=0     	POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=114   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=132   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=170   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=186   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=203   	content-type: application/json
		[6] type=HTX_BLK_EOH       - size=1      - addr=231   	<empty>
		[7] type=HTX_BLK_DATA      - size=29     - addr=232


# snip

[02|h1|1|mux_h1.c:2291] sending request headers : [B,RUN] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1" - h1c=0x7341fce5e4c0(0x80000000) conn=0x7341fce5e3d0(0x00000300) h1s=0x7341fce5e5b0(0x00100010) sd=0x7341fd80ec10(0x40400001) sc=0x7341fd515f00(0x00001c31) ibuf=0@0+0/0 obuf=0@0x7341fce85c20+0/16384
	htx=0x7341fce7ca00(size=16336,data=261,used=8,wrap=NO,flags=0x00000010,extra=0,first=0,head=0,tail=7,tail_addr=261,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_REQ_SL    - size=114    - addr=0     	POST /v1.46/exec/239ed0d2f465f6fca17c9d8a68e72b8b41e396d34516e910d810bbc8fbc2b160/start HTTP/1.1
		[1] type=HTX_BLK_HDR       - size=18     - addr=114   	host: 127.0.0.1:2375
		[2] type=HTX_BLK_HDR       - size=38     - addr=132   	user-agent: Docker-Client/27.1.1 (linux)
		[3] type=HTX_BLK_HDR       - size=16     - addr=170   	content-length: 29
		[4] type=HTX_BLK_HDR       - size=17     - addr=186   	connection: Upgrade
		[5] type=HTX_BLK_HDR       - size=28     - addr=203   	content-type: application/json
		[6] type=HTX_BLK_EOH       - size=1      - addr=231   	<empty>
		[7] type=HTX_BLK_DATA      - size=29     - addr=232

# snip

[02|h1|1|mux_h1.c:2376] sending response headers : [F,RUN] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=29)  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "HTTP/1.1 200 OK" - h1c=0x7341fce5e1f0(0x00000000) conn=0x7341fce5e100(0x80000300) h1s=0x7341fce5e2e0(0x00100010) sd=0x7341fd80ec90(0x50404001) sc=0x7341fd515ea0(0x00001422) ibuf=0@0+0/0 obuf=0@0x7341fce8a530+0/16384
	htx=0x7341fce7ca00(size=16336,data=156,used=7,wrap=NO,flags=0x00000000,extra=18446744073709551615,first=0,head=0,tail=6,tail_addr=156,head_addr=0,end_addr=0)
		[0] type=HTX_BLK_RES_SL    - size=33     - addr=0     	HTTP/1.1 200 OK
		[1] type=HTX_BLK_HDR       - size=45     - addr=33    	content-type: application/vnd.docker.raw-stream
		[2] type=HTX_BLK_HDR       - size=15     - addr=78    	api-version: 1.46
		[3] type=HTX_BLK_HDR       - size=24     - addr=93    	docker-experimental: false
		[4] type=HTX_BLK_HDR       - size=11     - addr=117   	ostype: linux
		[5] type=HTX_BLK_HDR       - size=27     - addr=128   	server: Docker/27.1.1 (linux)
		[6] type=HTX_BLK_EOH       - size=1      - addr=155   	<empty>

How to reproduce

haproxy.cfg:

global
    log stdout len 4096 format raw daemon debug
    stats socket ipv4@:10000 level admin
    maxconn 100

defaults
    log global
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

frontend dockerhttp
    bind :2375
    mode http
    option httplog
    default_backend dockerhttp

backend dockerhttp
    mode http
    server dockersocket /var/run/docker.sock

As an user that has access to /var/run/docker.sock (upgrade version as needed and make sure Docker daemon is up)

 docker run --rm --name haproxy \
    --user root --privileged \
    -v ./haproxy.cfg:/haproxy.cfg \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 127.0.0.1:2375:2375 \
    -p 127.0.0.1:10000:10000 \
    haproxy:2.2-alpine haproxy -f /haproxy.cfg

In another terminal:

export DOCKER_HOST=tcp://127.0.0.1:2375
docker exec haproxy ls

2 posts - 2 participants

Read full topic

Hi

Is it possible to prioritize a tcp frontend over an other with same or overlapping backend?

Our use-case: I have a priority and a normal tcp frontend with same config and backend servers. Now I have to make sure that the priority frontend will always be prioritized in the queue. Is there a nice solution without separation of the backends?

2 posts - 2 participants

Read full topic

I have a haproxy infront of Percona MySQL cluster. HAproxy is TCP proxying connection towards Percona with send-proxy-v2.

In the log lines, I’m seeing around 40% of SD connection termination states which haproxy logs as an error. If I disable SSL login, then the connections terminates OK and there is no problem. I have search the web for similar issues but couldn’t find a solution. The closest was Intermittent "SD" termination state - #3 by uliromahn, but we are not using nolinger option.

Error:

Aug 9 16:13:49 mysql-lb1 haproxy[346659]: 2001:yyyy:xxxx:44::4:51724 [09/Aug/2024:16:13:49.260] mysql-db1X mysql-db1X/mysql-db1a 1/0/149 39165 SD 44/42/41/41/0 0/0

HAproxy, client and DB servers are all in the same L2 network and I tried disabling firewall on the servers just to be sure firewall is not messing things up. No win:)

tcpdump of a request that returns SD state (cja host is the client):

When I disable SSL, the last RST package does not happen and seems like the only difference in the tcpdump.

Haproxy config:

global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  maxconn  2048
  pidfile  /var/run/haproxy.pid
  user  haproxy

defaults
  log  global
  maxconn  8000
  option  redispatch
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen mysql-pdc2-db1X
  bind xxxx:3306
  bind yyy:3306
  mode tcp
  balance leastconn
  option httpchk
  option clitcpka
  option srvtcpka
  option tcplog
  timeout queue 1m
  timeout connect 10s
  timeout client 8h
  timeout server 8h
  server mysql-pdc2-db1a x.y.v.z:3306 check send-proxy-v2 port 9200 inter 2s rise 3 fall 2
  server mysql-pdc2-db1b x.y.v.z1:3306 check send-proxy-v2 backup port 9200 inter 2s rise 3 fall 2
  server mysql-pdc2-db1c x.y.v.z:3306 check send-proxy-v2 backup port 9200 inter 2s rise 3 fall 2

The application (client) works OK as I always get the whole response from the MySQL server back, but haproxy is incrementing error counters which is not ideal:)

We are using haproxy 2.4.22 (bundled with RHEL 9).

Any idea / help would be appreciated, because I have no idea where to search for solution.

Thanks.

1 post - 1 participant

Read full topic

Help needed.
I have 1 Exchange 2019 behind Haproxy. Everything works fine before today. Today Outllok became slow and it’s a problem of Haproxy i beleive.
In Haproxy logs everytihng looks fine:

# tail -f /var/log/haproxy.log | grep 192.168.21.90
Aug 14 10:49:37 localhost haproxy[717]: 192.168.21.90:2870 [14/Aug/2024:10:49:37.775] fe_ex2019~ be_ex2019_mapi/exch01 0/0/1/13/14 401 514 - - ---- 41/41/25/25/0 0/0 "POST /mapi/nspi/?MailboxId=5faabed4-11b1-4793-9aec-84578f08cdb7@mydomain.com HTTP/1.1"
Aug 14 10:49:37 localhost haproxy[717]: 192.168.21.90:2870 [14/Aug/2024:10:49:37.790] fe_ex2019~ be_ex2019_mapi/exch01 0/0/0/39/39 200 1235 - - ---- 41/41/25/25/0 0/0 "POST /mapi/nspi/?MailboxId=5faabed4-11b1-4793-9aec-84578f08cdb7@mydomain.com HTTP/1.1"
Aug 14 10:49:37 localhost haproxy[717]: 192.168.21.90:2870 [14/Aug/2024:10:49:37.837] fe_ex2019~ be_ex2019_mapi/exch01 0/0/0/33/34 200 1604 - - ---- 41/41/25/25/0 0/0 "POST /mapi/nspi/?MailboxId=5faabed4-11b1-4793-9aec-84578f08cdb7@mydomain.com HTTP/1.1"
Aug 14 10:49:37 localhost haproxy[717]: 192.168.21.90:2870 [14/Aug/2024:10:49:37.891] fe_ex2019~ be_ex2019_mapi/exch01 0/0/0/30/31 200 1216 - - ---- 41/41/25/25/0 0/0 "POST /mapi/nspi/?MailboxId=5faabed4-11b1-4793-9aec-84578f08cdb7@mydomain.com HTTP/1.1"
Aug 14 10:49:37 localhost haproxy[717]: 192.168.21.90:2870 [14/Aug/2024:10:49:37.928] fe_ex2019~ be_ex2019_mapi/exch01 0/0/0/32/33 200 1604 - - ---- 41/41/25/25/0 0/0 "POST /mapi/nspi/?MailboxId=5faabed4-11b1-4793-9aec-84578f08cdb7@mydomain.com HTTP/1.1"
Aug 14 10:49:38 localhost haproxy[717]: 192.168.21.90:2873 [14/Aug/2024:10:49:38.382] fe_ex2019~ be_ex2019_mapi/exch01 0/0/2/3/5 401 514 - - ---- 41/41/25/25/0 0/0 "POST /mapi/emsmdb/?MailboxId=5faabed4-11b1-4793-9aec-84578f08cdb7@mydomain.com HTTP/1.1"

But in tcpdump i see that Haproxy responds to 443 port with 5sec delay:

# tcpdump host 192.168.21.90 and port 443 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:38:15.253295 IP 192.168.21.90.2737 > 192.168.20.37.443: Flags [S], seq 1842537036, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
10:38:15.253331 IP 192.168.20.37.443 > 192.168.21.90.2737: Flags [S.], seq 175281901, ack 1842537037, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
10:38:16.258620 IP 192.168.20.37.443 > 192.168.21.90.2737: Flags [S.], seq 175281901, ack 1842537037, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
10:38:18.274606 IP 192.168.20.37.443 > 192.168.21.90.2737: Flags [S.], seq 175281901, ack 1842537037, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
10:38:19.769188 IP 192.168.20.37.443 > 192.168.21.90.2544: Flags [P.], seq 1584716687:1584716730, ack 195633256, win 501, length 43
10:38:19.810825 IP 192.168.21.90.2544 > 192.168.20.37.443: Flags [.], ack 43, win 1024, length 0
10:38:22.402580 IP 192.168.20.37.443 > 192.168.21.90.2737: Flags [S.], seq 175281901, ack 1842537037, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
10:38:22.402892 IP 192.168.21.90.2737 > 192.168.20.37.443: Flags [.], ack 1, win 1026, options [nop,nop,sack 1 {0:1}], length 0
10:38:23.780456 IP 192.168.20.37.443 > 192.168.21.90.2640: Flags [P.], seq 131916949:131916992, ack 3368509531, win 501, length 43
10:38:23.826849 IP 192.168.21.90.2640 > 192.168.20.37.443: Flags [.], ack 43, win 8209, length 0
10:38:24.592484 IP 192.168.21.90.2737 > 192.168.20.37.443: Flags [P.], seq 1:215, ack 1, win 1026, length 214
10:38:24.592825 IP 192.168.20.37.443 > 192.168.21.90.2737: Flags [P.], seq 1:142, ack 215, win 501, length 141
10:38:24.593895 IP 192.168.21.90.2737 > 192.168.20.37.443: Flags [P.], seq 215:266, ack 142, win 1026, length 51
10:38:24.634614 IP 192.168.20.37.443 > 192.168.21.90.2737: Flags [.], ack 266, win 501, length 0

icmp from pc 192.168.21.90 to mailserver 192.168.20.37 works fine also without delays. So i think the problem is in Haproxy.
How can i resolve it?
Appreciate any help.

1 post - 1 participant

Read full topic

i believe what im trying to do is pretty simple but i cant for the life of me find the right documentation, guides or forum articles about this. im not sure whether squid or haproxy are the right pick for me but ive tried both and i couldnt get it to work

basically im trying to make my own proxies (something like webshare for example) but with ipv6, since i have a subnet of /48 ipv6 proxies on my virtual private server. my server is running arch linux and according to ip addr i have the following under the eth0 interface: inet6 xxxx:xxxx:xxx:xxxx::1/48 scope global. so if im not mistaken that means that i have access to more than a quintillion ipv6 addresses and i would like to be able to make requests through them, whether theyre randomly picked or specified by me through the proxy authentication process (for example when specifying the ip with curl’s -x parameter)

my end goal is basically to be able to either:
run curl -x http://[ipv6ip]:port -6 http://ifconfig.co and get a unique ipv6 address in each response
or be able to specify different ipv6 ips to curl and receive the specified ipv6 address in the response, for example
run curl -x http://[ipv6:ip1:23ex:ampl:eeee]:port -6 http://ifconfig.co and receive ipv6:ip1:23ex:ampl:eeee as the response
and
run curl -x http://[ipv6:ip1:23ex:ampl:ee22]:port -6 http://ifconfig.co and receive ipv6:ip1:23ex:ampl:ee22 as the response
i know these arent proper ipv6 ips but im just trying to get the idea across
i appreciate any help as ive been trying to pull this off for hours but have had no luck

1 post - 1 participant

Read full topic

Hi all, new to haproxy and in the process of translating my config to it from nginx (specifically swag if that matters), have the majority working but unsure how to translate the following into haproxy for hassio auth with Authentik:

proxy_set_header X-ak-hass-user $X_ak_hass_user;
auth_request_set $X_ak_hass_user $upstream_http_x_ak_hass_user;

Docs for what i’m trying to do is here: Home Assistant | authentik

Below is my config:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 40000
        ulimit-n 81000

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
       ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

       # Crowdsec bouncer
       lua-prepend-path /usr/lib/crowdsec/lua/haproxy/?.lua
       lua-load /usr/lib/crowdsec/lua/haproxy/crowdsec.lua
       setenv CROWDSEC_CONFIG /etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf

       # Authentik Auth Lua files
       lua-prepend-path /usr/local/share/lua/5.3/?.lua
       lua-load /usr/local/share/lua/5.3/auth-request.lua

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option forwardfor
        timeout connect 30s
        timeout client  30s
        timeout server  5s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
    bind *:8404
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /haproxy_stats
    stats auth HAProxy:Password

# Frontend to redirect HTTP to HTTPS with code 301
frontend http-redirect
    bind *:80
    http-request redirect scheme https code 301
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl

# LE Backend
backend letsencrypt-backend
    server letsencrypt 127.0.0.1:8888

# Frontend for redirecting traffic to the required frontend
frontend https-redirect
    bind *:443 ssl crt /etc/ssl/home.MYDOMAIN.com.pem
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    acl internal src 0.0.0.0/24 #IP RANGE OF MY NETWORK

    use_backend internal if internal

# Frontend for users
frontend internal
    bind abns@internal accept-proxy
    # Crowdsec bouncer
    stick-table type ip size 10k expire 30m # declare a stick table to cache captcha verifications
    http-request lua.crowdsec_allow # action to identify crowdsec remediation
    http-request track-sc0 src if { var(req.remediation) -m str "captcha-allow" } # cache captcha allow decision
    http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial url
    http-request use-service lua.reply_captcha if { var(req.remediation) -m str "captcha" } # serve captcha template if remediation is captcha
    http-request use-service lua.reply_ban if { var(req.remediation) -m str "ban" } # serve ban template if remediation is ban

    #Authentik config
    acl protected-frontends hdr(host) -m reg -i ^(?i)(arrs|downloads)\.(home|media)\.home\.MYDOMAIN\.com
    acl is_authentikoutpost path -m reg ^/outpost.goauthentik.io/

    http-request set-var(req.scheme) str(http) if !{ ssl_fc }
    http-request set-var(req.questionmark) str(?) if { query -m found }

    http-request set-header X-Real-IP %[src]

    http-request set-header X-Forwarded-Method %[method]
    #http-request set-header X-Forwarded-Proto  %[var(req.scheme)] #had to remove this as hassio breaks if its enabled, everything works fine without it
    http-request set-header X-Forwarded-Host   %[req.hdr(Host)]
    http-request set-header X-Original-URL     %[url]
    # websockets break if all headers are passed to be_auth_requests as the Upgrade header screws the process up - so we pass manual list
    http-request lua.auth-intercept be_auth_request /outpost.goauthentik.io/auth/nginx HEAD x-original-url,x-real-ip,x-forwarded-host,x-forwarded-proto,user-agent,cookie,accept,x-forwarded-method x-authentik-username,x-authentik-uid,x-authentik-email,x-authentik-name,x-authentik-groups - if protected-frontends !is_authentikoutpost

    http-request redirect code 302 location /outpost.goauthentik.io/start?rd=%[hdr(X-Original-URL)] if protected-frontends !{ var(txn.auth_response_successful) -m bool } { var(txn.auth_response_code) -m int 401 } !is_authentikoutpost
    http-request deny if protected-frontends !{ var(txn.auth_response_successful) -m bool } { var(txn.auth_response_code) -m int 403 } !is_authentikoutpost
    http-request redirect location %[var(txn.auth_response_location)] if protected-frontends !{ var(txn.auth_response_successful) -m bool } !is_authentikoutpost

    # Select backend based on services.map file or use backend no-match if not found.
    use_backend be_auth_request if protected-frontends is_authentikoutpost
    use_backend %[base,lower,map_beg(/etc/haproxy/services.map,no-match)]
    use_backend %[req.hdr(host),lower,map(/etc/haproxy/services.map,no-match)]

backend internal
    mode tcp
    server loopback-for-tls abns@internal send-proxy-v2

backend be_auth_request
    server proxy home:9000 check

listen be_auth_request_proxy
    bind :9000
    server be_auth_request home:9000 check

listen ldap_proxy
    bind :389
    server proxy home:389 check

backend ldap
    server be_auth_request home:389 check

# Normal Backends
backend no-match
    http-request deny deny_status 403

backend homedash
    server homedash home:81 check

backend hassio
    server hassio home:8123 check

backend downloads
    server downloads home:8200 check
backend prowlarr
    server prowlarr home:9696 check
backend radarr
    server radarr home:7878 check
backend sonarr
    server sonarr home:8989 check
backend arrs
    server arrs home:5055 check

# Backend for google to allow DNS resolution if using reCAPTCHA
backend captcha_verifier
    server captcha_verifier www.google.com:443 check

# Backend for crowdsec to allow DNS resolution
backend crowdsec
    server crowdsec *:8080 check

And services.map:

home.MYDOMAIN.com homedash
#
auth.home.MYDOMAIN.com be_auth_request
home.home.MYDOMAIN.com hassio
#
arrs.media.home.MYDOMAIN.com arrs
#
downloads.media.home.MYDOMAIN.com downloads
arrs.media.home.MYDOMAIN.com/prowlarr prowlarr
arrs.media.home.MYDOMAIN.com/radarr radarr
arrs.media.home.MYDOMAIN.com/sonarr sonarr

Besides that i’m also trying to get haproxy to pass HTTP Basic credentials stored in Authentik to sonar/radarr as explained here: Sonarr | authentik Although I imagine that getting the hassio auth working will create a lightbulb moment that will help me figure it out, but may as well mention it here anyway.

Feel free to correct anything that’s just blatantly wrong, as I said, new to the world of haproxy.

Thanks people!

1 post - 1 participant

Read full topic