We have deployed in Openshift an HAProxy to forward the traffic to upstream APIs.
The HAProxy is working fine and it can address thousands of request but, sometimes, it seems that there is an issue with a specific request. This request is a GET request where the payload is up to 10MB. Directly querying the upstream server works every time. The issue is never reproducible. if we try to perform the GET request going through the HAProxy it does not work all the times.
We created a bash script that is calling the same GET request with curl every 2 seconds, the script went ahead for ours and all the request where successful but suddenly curl started “hanging” Here the logs of a successful GET
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying XXX.XXX.XXX.XXX:5004...
* Connected to myhaproxy (XXX.XXX.XXX.XXX) port 5004 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
........
* Server certificate:
......
} [5 bytes data]
> GET /my/endpoint HTTP/1.1
> Host: myhost:5004
> User-Agent: curl/7.74.0
> Accept-Encoding: gzip,deflate
> Accept: application/json, application/*+json
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
....
* old SSL session ID is stale, removing
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< max-forwards: 20
< transfer-encoding: chunked
< server: Jetty(9.4.48.v20220622)
< content-type: application/json
< vary: Accept-Encoding
< content-encoding: gzip
<
{ [765 bytes data]
100 758 0 758 0 0 651 0 --:--:-- 0:00:01 --:--:-- 651
100 953k 0 953k 0 0 790k 0 --:--:-- 0:00:01 --:--:-- 790k
While this is the log when the issue happens for the very same endpoint
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying XXX.XXX.XXX.XXX:5004...
* Connected to myhaproxy (XXX.XXX.XXX.XXX) port 5004 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
......
* Server certificate:
......
} [5 bytes data]
> GET /my/endpoint HTTP/1.1
> Host: myhost:5004
> User-Agent: curl/7.74.0
> Accept-Encoding: gzip,deflate
> Accept: application/json, application/*+json
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
....
* old SSL session ID is stale, removing
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< max-forwards: 20
< transfer-encoding: chunked
< server: Jetty(9.4.48.v20220622)
< content-type: application/json
< vary: Accept-Encoding
< content-encoding: gzip
<
{ [765 bytes data]
100 198k 0 198k 0 0 94419 0 --:--:-- 0:00:02 --:--:-- 94375
100 357k 0 357k 0 0 112k 0 --:--:-- 0:00:03 --:--:-- 112k
100 357k 0 357k 0 0 87817 0 --:--:-- 0:00:04 --:--:-- 87796
100 357k 0 357k 0 0 70804 0 --:--:-- 0:00:05 --:--:-- 73744
100 357k 0 357k 0 0 59313 0 --:--:-- 0:00:06 --:--:-- 73744
100 357k 0 357k 0 0 51031 0 --:--:-- 0:00:07 --:--:-- 32359
100 357k 0 357k 0 0 44779 0 --:--:-- 0:00:08 --:--:-- 0
100 357k 0 357k 0 0 39891 0 --:--:-- 0:00:09 --:--:-- 0
100 357k 0 357k 0 0 35965 0 --:--:-- 0:00:10 --:--:-- 0
....
100 357k 0 357k 0 0 11728 0 --:--:-- 0:00:31 --:--:-- 0
100 357k 0 357k 0 0 11363 0 --:--:-- 0:00:32 --:--:-- 0
100 357k 0 357k 0 0 11021 0 --:--:-- 0:00:33 --:--:-- 0
100 357k 0 357k 0 0 10107 0 --:--:-- 0:00:36 --:--:-- 0
100 357k 0 357k 0 0 9835 0 --:--:-- 0:00:37 --:--:-- 0
....
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:47 --:--:-- 0
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:48 --:--:-- 0
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:49 --:--:-- 0
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:50 --:--:-- 0
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:54 --:--:-- 0
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:56 --:--:-- 0
100 357k 0 357k 0 0 44 0 --:--:-- 2:17:57 --:--:-- 0
As you can see it’s hanging forever.
This is the haproxy config
global
maxconn 4096
master-worker
log stdout len 8192 format raw local0 info
stats timeout 2m
default-path origin /opt/app-root/haproxy/
hard-stop-after 15s
tune.ssl.maxrecord 0
tune.ssl.lifetime 275
tune.ssl.cachesize 25000
tune.ssl.default-dh-param 2048
tune.bufsize 32768
tune.maxrewrite 10240
resolvers resolvcnf
parse-resolv-conf
hold valid 50s
hold nx 20s
hold other 20s
hold refused 20s
hold timeout 20s
hold obsolete 20s
resolve_retries 5
timeout resolve 3s
timeout retry 3s
defaults httpdefaults
mode http
log global
option httplog
timeout check 5s
timeout connect 5s
timeout client 30s
timeout server 90s
timeout client-fin 3s
timeout server-fin 90s
timeout http-request 120s
timeout http-keep-alive 60s
timeout tunnel 1s
compression algo gzip
compression type text/html text/plain application/json
log-format '.....'
In HAProxy I do not see any error logs. Any Idea what it could be or how to better investigate this issue?
1 post - 1 participant