Backend NOSRV issue

@ghorio wrote:

Dear community,

Recently I have noticed an issue with our HAproxy serving backend on our nginx servers.
Basically we have haproxy in front of 2 nginx servers and every morning for like less than minute I am getting 503 NOSRV errors in haproxy logs, but at the same time fron nginx access logs I can see that there are requests going through with no issues. This is in the logs -

Dec  9 10:03:53 boost-2 haproxy[15199]: 80.232.255.170:59993 [09/Dec/2019:10:03:53.904] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 369/89/14/0/0 0/0 "GET /images/svg/days-recipe.svg HTTP/1.1"
Dec  9 10:03:53 boost-2 haproxy[15199]: 194.213.9.26:35652 [09/Dec/2019:10:03:53.917] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 371/91/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 87.110.50.14:57217 [09/Dec/2019:10:03:54.072] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 369/85/14/0/0 0/0 "GET /images/svg/days-recipe.svg HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 46.109.52.151:53813 [09/Dec/2019:10:03:54.115] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 366/84/14/0/0 0/0 "GET /example/example-url-1 HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 213.175.124.141:44632 [09/Dec/2019:10:03:54.186] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 365/86/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 81.198.123.2:53243 [09/Dec/2019:10:03:54.508] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 358/77/14/0/0 0/0 "GET /?_ga=2.48308399.820674435.1575875672-152970791.1575875669 HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 78.28.224.67:61335 [09/Dec/2019:10:03:54.705] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 367/80/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 212.3.196.55:39024 [09/Dec/2019:10:03:54.709] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 365/79/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 46.109.52.151:53812 [09/Dec/2019:10:03:54.775] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 355/76/14/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 83.99.167.240:53850 [09/Dec/2019:10:03:54.897] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 354/75/14/0/0 0/0 "GET /export/mixed/ HTTP/1.1"
Dec  9 10:03:54 boost-2 haproxy[15199]: 84.237.170.220:34939 [09/Dec/2019:10:03:54.956] frontend-http~ backend-example-com/<NOSRV> 0/-1/-1/-1/1 503 213 - - SC-- 362/84/14/0/0 0/0 "GET /example/example-url-2 HTTP/1.1"

And the config is as below

frontend frontend-http
  bind 222.155.222.242:80
  bind 222.155.222.243:80
  bind 222.155.222.242:443 ssl crt /etc/haproxy/ssl/certs/example.com.pem crt /etc/haproxy/ssl/certs/i.example.com.pem
  bind 222.155.222.243:443 ssl crt /etc/haproxy/ssl/certs/example.com.pem crt /etc/haproxy/ssl/certs/i.example.com.pem
  redirect scheme https code 301 if !{ ssl_fc }
  reqadd X-Forwarded-Proto:\ https if { ssl_fc }
  reqadd X-Forwarded-Port:\ 443 if { ssl_fc }
  reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
  reqadd X-Forwarded-Port:\ 80 if !{ ssl_fc }

  option forwardfor header X-GW-Client-IP
  http-request set-header X-GW-Client-IP %[src]

  acl host-img-example-com hdr_dom(host) -i -m reg ^(i.example.com)(:[0-9]+)?$
  acl host-www-example-com  hdr_dom(host) -i -m reg ^((www\.)?example.com)(:[0-9]+)?$
  acl host-example-com hdr_dom(host) -i -m reg ^(example.com)(:[0-9]+)?$
  use_backend backend-img-example-com if host-img-example-com
  use_backend backend-example-com if host-example-com


backend backend-example-com
  retries 3
  option redispatch
  option httpclose
  option forwardfor
  balance roundrobin
  option httpchk HEAD / HTTP/1.1\r\nHost:example.com
  # we use ssl cause backend checks generates sometiems output based on _backend_ was ssl or not
  server example-1  10.111.111.2:443 check ssl verify none
  server example-2  10.111.111.3:443 check ssl verify none

I would really appreciate any help/advice where should I begin to look at, thank you!

Posts: 1

Participants: 1

Read full topic