I have 3 frontend servers and 3 backend servers. The three frontends use haproxy, the three backends use openresty (nginx+lua).Two are with the same ISP in Europe, the third is with another ISP in North America. All use Debian 11.
The NA server is breaking. No POST requests are going through at all, they all time out with error 408 regardless of what the limit is. If it’s 5 seconds, 10 seconds, 30, 60, 120, it will timeout. The other two servers (with identical config files) do not have this issue with even a 5s timeout.
I noticed while looking for 408 errors that there were a ton of requests with as the URL. Upon examination, I am receiving dozens of these per second, but only in NA. Since all three frontends are for a multicast, errors should be common in all three. This is not the case.
The logs look like this.
Jan 8 13:59:14 ca haproxy[73212]: xxx:34464 [08/Jan/2023:13:59:14.831] https-in~ https-in/<NOSRV> -1/-1/-1/-1/0 400 0 - - CR-- 3442/3435/0/0/0 0/0 "<BADREQ>"
Jan 8 13:59:20 ca haproxy[73212]: xxx:56834 [08/Jan/2023:13:58:50.043] mati-in~ mati-in/<NOSRV> -1/-1/-1/-1/30001 408 0 - - cR-- 3498/5/0/0/0 0/0 "<BADREQ>"
Jan 8 13:59:20 ca haproxy[73212]: xxx:51904 [08/Jan/2023:13:59:20.165] https-in~ https-in/<NOSRV> -1/-1/-1/-1/10 0 0 - - PR-- 3507/3501/0/0/0 0/0 "<BADREQ>"
A well constructed 408, which reliably happens on any POST request, looks like this.
Jan 8 14:01:41 ca haproxy[73212]: xxx:43748 [08/Jan/2023:14:01:11.721] https-in~ https-in/<NOSRV> -1/-1/-1/-1/30001 408 304 - - cD-- 3703/3692/0/0/0 0/0 "POST https://foo.bar/threads/test.129102/add-reply HTTP/2.0"
I have tried:
- Ensuring TCP is permitted for posts 443 and 80.
- Increasing and decreasing timeouts.
- Adding and removing
http-keep-aliveandhttp-buffer-request. errorfile 408 /dev/null- Changing SSL ciphers.
- Enabling and disabling HTTP2.
The error is not browser dependent. I have tested Chromium forks and FireFox. It happens on mobile devices and desktop. It happens between ISPs. I have no clue what the issue is and I am running the latest stable version of haproxy.
1 post - 1 participant