Bizarre problem: HTTP~ -1 0 in HAProxy + 499 in NGINX

I have a bizarre problem, and I’m looking for a little help understanding the role that HAProxy is playing in it, if any.

Our setup is:

HAProxy → multiple web servers running NGINX/PHP-FPM → SQL/REDIS.

We have maxconn limits set in HAProxy, as well as rate limiting using the stick tables. This works really well. We haven’t had a successful DDoS attack against us in a long time.

A few days ago one of the sites we host started going down. Each time it happened, our monitoring tool sent us warnings in the exact same order:

1. That our REDIS server was maxing out its 1GB port with data being sent to the web servers in the backend;
2. That the web servers in the backend had run out of RAM;
3. That our site was down

After about 30 seconds, the backend servers would recover and the site would come back.

I looked for obvious signs of an external DDoS attack and found nothing. What I did find was the same IP address—the IP address of a logged in user—in between 10-50 identical lines just before the site started to go down. At first I discounted this, because I couldn’t see how it could possibly be the culprit. But, as the same thing kept happening, I kept noticing this same IP and the same pattern.

In HAProxy (which is the only public-facing server, and handles SSL; the rest are hidden within the LAN), the relevant entries in the log look like this (I’ve sanitized it):

Mar 25 16:46:23 proxy haproxy[11692]: [<OFFENDING IP>] [25/Mar/2023:16:46:23.138] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/} GET https://domain/page/ [HTTP/2.0] backend/server

In NGINX, on the web servers behind HAProxy, a relevant entry in the log looks like this:

10.0.1.5 - - [25/Mar/2023:16:46:23 -0400] "GET /page/ HTTP/1.1" 499 0 "https://domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" "<OFFENDING IP>"

(In this, 10.0.1.5 is the IP of the proxy and <OFFENDING IP> is the IP of the client out on the web.)

There are a few things I don’t understand here.

First off, HTTP~ -1 0 seems also to be what I see in the logs when HAProxy is stopping a request at the perimeter. If a client has made too many requests, for example, I see HTTP~ -1 0 in the lines after HAProxy has started blocking that client. But in this case, the requests are obviously getting through, because they’re creating equivalent 499 lines in the NGINX log. As such, my questions on this are:

1. In what circumstances does HAProxy add a HTTP~ -1 0 line in the log? It’s a little confusing to me because the lines in HAProxy that show the problematic requests being passed through to the backend are marked HTTP~ -1 0 and the lines in HAProxy after a client has been blocked by the rate limiting are marked HTTP~ -1 0. The only difference is that the ones that are passed through are marked backend/server whereas the ones that are being blocked are marked HTTP/<NOSRV>.
2. What should I assume these requests are? At the HAProxy level, they’re logged with HTTP~ -1 0. In NGINX, they’re logged with 499, which means that the client closed the connection. But, in this context, the “client” is actually HAProxy, right? What could explain this behavior?
3. If it’s obvious to anyone what this is, is there a straightforward way of stopping HAProxy from passing through these sorts of requests? I could be wrong, but it almost feels as if HAProxy is passing through requests that it knows are problematic. Why would it do that? If I were seeing 200 entries in the HAProxy log but 499 entries in the NGINX log, I’d assume there was an issue between HAProxy and NGINX. But the requests seem to arrive in a bad state.

The second question I have is whether it’s “normal” for a web server to receive these sorts of bursts. Here’s what they look like on HAProxy.

Mar 25 17:05:10 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:10.201] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:10 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:10.299] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:10 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:10.309] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:10 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:10.553] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:11 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:10.815] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-b
Mar 25 17:05:11 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:11.027] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:11 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:11.233] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:11 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:11.453] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:11 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:11.680] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:12 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:11.998] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:12 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:12.197] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:12 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:12.398] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:12 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:12.777] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:13 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:13.049] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-b
Mar 25 17:05:13 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:13.246] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] ricochet.com-internal/planet-a
Mar 25 17:05:13 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:13.449] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] HTTP/<NOSRV>
Mar 25 17:05:13 proxy haproxy[13550]: [<OFFENDING IP>] [25/Mar/2023:17:05:13.710] HTTP~ -1 0 {Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36|https://domain.com/referrer} GET https://domain.com/page [HTTP/2.0] HTTP/<NOSRV>

The odd thing is that, if I look around for the other behavior from that IP—which, again, belongs to a logged in user, not some random person out on the internet—it seems pretty normal. Should I conclude that this person has some malware on their computer that is attacking whatever site is open in their browser? Or is this more likely to be the product of some misconfiguration on our site? Or both? Looking through the logs, I can see a few other logged in IPs that generate a few HTTP~ -1 0 lines in HAProxy and 499 lines in NGINX, but nothing on the scale of this particular one.

Any advice anyone could give me would be very highly appreciated.

Thank you.

1 post - 1 participant

Read full topic