Seamless Reloads don't work with systemd

@kustodian wrote:

I’m trying to make HAProxy 1.8.3 work on CentOS 7 and I want to enable seamless reloads. I’ve managed to make seamless reloads work if I run HAProxy in the shell, but it doesn’t want to work with systemd.

Here is the HAProxy version:

# haproxy -v
HA-Proxy version 1.8.3-205f675 2017/12/30
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Systemd unit file /usr/lib/systemd/system/haproxy.service:

[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
EnvironmentFile=/etc/sysconfig/haproxy
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
Type=notify

[Install]
WantedBy=multi-user.target

and the global section of the /etc/haproxy/haproxy.cfg looks like this:

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats expose-fd listeners

When I start haproxy everything looks fine:

[root@hap18 ~]# systemctl start haproxy.service
[root@hap18 ~]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-01-05 14:53:12 CET; 2s ago
  Process: 4004 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
  Process: 4003 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
  Process: 4038 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
 Main PID: 4039 (haproxy)
   CGroup: /system.slice/haproxy.service
           ├─4039 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           └─4041 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

Jan 05 14:53:12 hap18 systemd[1]: Starting HAProxy Load Balancer...
Jan 05 14:53:12 hap18 systemd[1]: Started HAProxy Load Balancer.

[root@hap18 ~]# ps auxf | grep haprox[y]
root      4039  0.0  0.2  76296  4344 ?        Ss   14:53   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy   4041  8.5  0.1 297808  2280 ?        Ssl  14:53   0:00  \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

then when I do a reload, this happens:

[root@hap18 ~]# systemctl reload haproxy.service
[root@hap18 ~]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-01-05 14:53:12 CET; 46s ago
  Process: 4058 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
  Process: 4057 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
  Process: 4038 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
 Main PID: 4039 (haproxy)
   CGroup: /system.slice/haproxy.service
           ├─4039 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats
           └─4060 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats

Jan 05 14:53:12 hap18 systemd[1]: Starting HAProxy Load Balancer...
Jan 05 14:53:12 hap18 systemd[1]: Started HAProxy Load Balancer.
Jan 05 14:53:56 hap18 systemd[1]: Reloaded HAProxy Load Balancer.
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145312 (4039) : Reexecuting Master process
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Failed to connect to the old process socket '/var/lib/haproxy/stats'
Jan 05 14:53:56 hap18 haproxy[4039]: [ALERT] 004/145356 (4039) : Failed to get the sockets from the old process!
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Former worker 4041 exited with code 0

[root@hap18 ~]# ps auxf | grep haprox[y]
root      4039  0.0  0.2  76296  4352 ?        Ss   14:53   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats
haproxy   4060 19.7  0.1 297808  2280 ?        Ssl  14:53   0:00  \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats

as you can see HAProxy logs that it cannot connect to the socket:

Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145312 (4039) : Reexecuting Master process
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Failed to connect to the old process socket '/var/lib/haproxy/stats'
Jan 05 14:53:56 hap18 haproxy[4039]: [ALERT] 004/145356 (4039) : Failed to get the sockets from the old process!
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Former worker 4041 exited with code 0

but it does everything else correctly because we can see that it spawned a new child process with adding -sf 4041 -x /var/lib/haproxy/stats where 4041 is the PID of the old process.

I tried many things:

• removed chroot
• set user user/group to root
• changed socket path
• changed socket permissions

but nothing helped.

The interesting thing is that if I run HAProxy from the shell, reload works (I just had to disable daemon mode in the config so that HAProxy logs everything into the console, and of course while working with systemd I tried with disabling daemon mode as well). Here are the steps to run it in the shell.

Run haproxy in tmux/screen:

[root@hap18 ~]# haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

then in a spearate session we list these processes:

[root@hap18 ~]# ps auxf | grep haprox[y]
root      4185  0.0  0.2  76296  4348 pts/1    S+   15:06   0:00      \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy   4186  0.0  0.1  76612  2024 ?        Ss   15:06   0:00          \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

Now we send that process the USR2 signal so that it starts the reload:

kill -USR2 $(pgrep -U 0 haproxy)

pgrep -U 0 haproxy finds the master HAProxy process.
Then if we go back to the tmux/screen session we can see this:

[root@hap18 ~]# haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
[WARNING] 004/150638 (4185) : Reexecuting Master process
[WARNING] 004/150741 (4185) : Former worker 4186 exited with code 0

If we list the haproxy processes again:

[root@hap18 ~]# ps auxf | grep haprox[y]
root      4185  0.0  0.2  76296  4348 pts/1    S+   15:06   0:00      \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4186 -x /var/lib/haproxy/stats
haproxy   4190  0.0  0.1  76612  2028 ?        Ss   15:07   0:00          \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4186 -x /var/lib/haproxy/stats

We can see that the reload was done and there weren’t any errors about not being able to read the socket.

Any ideas what could be the reason behind this? It has to be something related to systemd, but I’m not sure what could it be.

Posts: 1

Participants: 1

Read full topic