HAProxy 2.5.1 occasional default backend selected instead of correct backend

I’ve been using HAProxy 1.7 to front Obsidian successfully for many years.
I have a couple of HAProxy servers which were running 2.5 fronting Obsidian in a production environment for months with no problems.
I recently upgraded them to 2.51 with no problems.

I’ve deployed 2 new HAProxy 2.51 servers in a development environment.
I see very strange behavior.
I’ve tested with many browsers from different locations.
I’ve even tried using HTTP 1.1 instead of 2.0 to no avail.

When it works correctly the logs look like this:

Feb  9 15:09:13 149.97.134.35:22159 [09/Feb/2022:15:08:42.499] main_ssl~ ssl_backend-obs/<NOSRV> 30601/-1/-1/-1/30600
302 90 - - LR-- 1/1/0/0/3 0/0 {} "GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb  9 15:09:13 149.97.134.35:22159 [09/Feb/2022:15:09:13.098] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 74/0/0/1/74
302 108 - - ---- 1/1/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"

tndci-obs.cloud.micropact.com is sent via 302 to the ssl_backend-obs and “/obsidian” is appended to the path.

When it fails, the logs look like this:

Feb  9 13:26:24 209.112.9.82:65220 [09/Feb/2022:13:26:23.879] main_ssl~ ssl_backend-vr/<NOSRV> 1/-1/-1/-1/191
302 86 - - LRNN 3/3/0/0/3 0/0 {} "GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb  9 13:26:24 209.112.9.82:65220 [09/Feb/2022:13:26:24.071] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 56/0/0/5/67
200 38818 - - --VN 3/3/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"

tndci-obs.cloud.micropact.com is sent to the default backend ssl_backend-vr and “/le5” is appended to the path.

Since this is HTTP 2.0 the URL is absolute and it clearly contains “tndci-obs”.

Frontend:

acl obs             hdr_dom(host)   tndci-obs.
use_backend ssl_backend-obs if obs

Backend:

backend ssl_backend-obs
balance     roundrobin
option httpchk GET /obsidian/
http-check disable-on-404
acl down nbsrv() eq 0
acl path_root  path /
http-response set-header Cache-Control no-cache,must-revalidate
http-response set-header Pragma no-cache
http-response set-header Expires 0
http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
http-request deny
redirect location /obsidian         if path_root !down

I assume the frontend occasionally fails to match.
How do I debug this?

2 posts - 2 participants

Read full topic