Hi good morning everyone, I hope everyone can help with this.
So apparently I have a WAF (cloudflare) and load balancer using haproxy stacks and my backends with request flow like client -> cloudflare -> haproxy -> backend server.
Currently I’m facing a condition where I experience intermittent error 403 NOSRV.
The configuration of Haproxy configuration is as follow:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
tune.bufsize 262144
tune.maxrewrite 32768
tune.maxaccept -1
user haproxy
group haproxy
daemon
defaults
log global
option httplog
no option checkcache
mode http
retries 2
option redispatch
option forwardfor except 127.0.0.1
maxconn 40000
timeout client 620s
timeout server 5m
timeout queue 60s
timeout connect 30s
timeout check 60s
timeout tunnel 1h
timeout tarpit 60s
timeout http-request 5m
option abortonclose
default-server inter 30s
default-server fastinter 5s
default-server downinter 5s
default-server maxconn 1500
default-server maxqueue 1500
default-server on-error fail-check
default-server slowstart 120s
default-server weight 1
fullconn 200000
frontend ssl
bind 0.0.0.0:443 ssl crt /etc/haproxy/ssl/nyem_digicert.pem ciphers ECDHE-RSA-AES256-GCM-SHA384:XXXXXXXXX
bind 0.0.0.0:80
mode http
tcp-request inspect-delay 5s
option httplog
option dontlognull
option accept-invalid-http-request
option forwardfor except 127.0.0.1
monitor-uri /healthcheck-uri
http-request add-header X-Forwarded-Proto https if { ssl_fc }
acl host_dev hdr_beg(host) -i dev.nyem.com
acl network_private src 10.104.0.0/24
acl path_order_trx path -i -m beg /api-nyem/order/transactions
acl path_order_nyem_trx path -i -m beg /api-nyem/order/nyem/transactions
acl path_order_callback path -i -m beg /api-nyem/order/callback/nyem
acl path_order path -i -m beg /api-ms/order/
http-request allow if path_order_trx
http-request allow if path_order_digiflazz_trx
http-request allow if path_order_callback
http-request deny if path_order !network_private
http-response set-header Cache-control no-cache,\ no-store,\ must-revalidate if { capture.req.uri -m beg /sw.js /sw.js.map }
# backends
use_backend order if path_order
use_backend dev_ws if host_dev
default_backend dev_ws
acl is_http hdr(X-Forwarded-Proto) eq http
redirect scheme https code 301 if is_http
backend dev_ws
mode http
server dev_ws 10.104.0.6:443 ssl verify none maxconn 100
backend order
option httpchk
http-check send meth GET uri /application/health
http-check expect status 200
http-request replace-path /api-nyem(.*) \1
mode http
server dev_order 10.104.0.24:8300 maxconn 100
So let’s say I have 2 API in the backend server:
- POST ‘https://dev.nyem.com/api-nyem/order/nyem/transactions’
- POST ‘https://dev.nyem.com/api-nyem/order/callback/nyem’
So I tried to simulate this CURL:
curl --location --request POST 'https://dev.nyem.com/api-nyem/order/nyem/transactions' \
--data '{
"nyem": "123131"
}'
And it always return perfectly fine with 200. However when I tried to add new endpoint which is POST 'https://dev.nyem.com/api-nyem/order/callback/nyem' and try to hit it with:
curl --location --request POST 'https://dev.nyem.com/api-nyem/order/callback/nyem' \
--data '{
"id": "12563858",
}'
It intermittently returning error like this:
200:
{"success":false,"code":"FAILED","message":"error","data":"error","serverTime":1694831245}
403:
<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>
I suspect this request comes from the haproxy because the request was logged in haproxy log with the following details:
200
Sep 16 02:00:56 dev-haproxy haproxy[1388152]: 162.158.162.93:52930 [16/Sep/2023:02:00:56.437] ssl~ order/dev_order 0/0/0/2/2 200 155 - - ---- 62/62/0/0/0 0/0 "POST /api-nyem/order/callback/nyem HTTP/1.1"
403:
Sep 16 02:00:57 dev-haproxy haproxy[1366652]: 162.158.162.53:60206 [16/Sep/2023:02:00:57.252] ssl~ ssl/<NOSRV> 0/-1/-1/-1/0 403 192 - - PR-- 115/115/0/0/0 0/0 {dev.nyem.com|PostmanRuntime/7.32.3|SG|178.128.58.144|80731d91cb4f4912-SIN|178.128.58.144} "POST /api-ms/order/callback/nyem HTTP/1.1"
Do you guys what is the root cause of the intermittent error while the first API works just fine? I’m already on my dead end on this one. Thanks
Oh yah this is my haproxy version:
HAProxy version 2.5.5-1ppa1~focal 2022/03/14 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.5.html
Running on: Linux 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64
1 post - 1 participant