gRPC + HAProxy and connection reset by peer

@armax wrote:

Hi folks,

I am looking for suggestions on how to understand an issue we’re observing in our environment where use have a couple of GRPC service replicas (active/backup) fronted by HAProxy (version 1.6.11). The haproxy configuration looks something like this:

listen stats
  maxconn 2000
  balance leastconn
  bind 127.0.0.1:8888
  mode http
  stats enable
  stats uri /
  stats hide-version
  stats refresh 5s

frontend myservers
  maxconn 2000
  bind 0.0.0.0:12000
  mode tcp
  default_backend myservers

backend myservers
  mode tcp
  balance roundrobin
  server server1 server1:12000 check port 12002 inter 1s
  server server2 server2:12000 check port 12002 inter 1s backup

The gRPC service is littered with traces like:

"transport: http2Server.HandleStreams failed to read frame: read tcp 10.64.37.24:12000-\u003e10.64.37.213:59012: read: connection reset by peer

I was able to pinpoint the presence of the traces to the fact that haproxy connections are not being closed with a FIN flag. Connections closed with a RST flag don’t seem to trigger the trace. It seems also that some connections are closed with the RST packet, others with the FIN flag.

While I can appreciate that closing with RST in legitimate, I wonder what is the source of the flapping and whether there’ is a way to stop it. Has anyone else experienced something of this nature?

Thanks in advance!!
Armando

Posts: 1

Participants: 1

Read full topic