@scarey wrote:
Hi,
I’m noticing the configuration reload isn’t seamless when using dynamic service discovery via DNS. My simple test makes requests every second or so and I trigger a reload somewhere in the middle and there’re 8 seconds or so where services are unavailable. Is that the behavior you would expect? Is there some configuration I can use to minimize the downtime? Any help would be appreciated. If I run the same test with more static configuration the reload works very well.
Test output:
scarey@static:~/dev/data$ ./test.sh {"time":1529005368803}Thu Jun 14 19:42:48 UTC 2018 {"time":1529005370141}Thu Jun 14 19:42:50 UTC 2018 {"time":1529005371504}Thu Jun 14 19:42:51 UTC 2018 {"time":1529005372839}Thu Jun 14 19:42:52 UTC 2018 {"time":1529005374175}Thu Jun 14 19:42:54 UTC 2018 {"time":1529005375533}Thu Jun 14 19:42:55 UTC 2018 <html><body><h1>500 Internal Server Error</h1> An internal server error occured. </body></html> Thu Jun 14 19:42:56 UTC 2018 <html><body><h1>500 Internal Server Error</h1> An internal server error occured. </body></html> Thu Jun 14 19:42:58 UTC 2018 <html><body><h1>500 Internal Server Error</h1> An internal server error occured. </body></html> Thu Jun 14 19:42:59 UTC 2018 {"time":1529005380854}Thu Jun 14 19:43:00 UTC 2018 <html><body><h1>500 Internal Server Error</h1> An internal server error occured. </body></html> Thu Jun 14 19:43:02 UTC 2018 {"time":1529005383481}Thu Jun 14 19:43:03 UTC 2018 {"time":1529005384835}Thu Jun 14 19:43:04 UTC 2018 {"time":1529005386178}Thu Jun 14 19:43:06 UTC 2018 {"time":1529005387524}Thu Jun 14 19:43:07 UTC 2018 {"time":1529005388881}Thu Jun 14 19:43:08 UTC 2018 {"time":1529005390241}Thu Jun 14 19:43:10 UTC 2018HAProxy logs:
Jun 14 19:42:57 192.168.64.146 haproxy[9525]: Proxy haproxy-statistics started. Jun 14 19:42:57 192.168.64.146 haproxy[9525]: Proxy http-in started. Jun 14 19:42:57 192.168.64.146 haproxy[9525]: Proxy backend_secure_test-app6 started. Jun 14 19:42:57 192.168.64.146 haproxy[9525]: Proxy backend_secure_test-log4j2 started. Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Stopping proxy haproxy-statistics in 0 ms. Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Stopping frontend http-in in 0 ms. Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Stopping backend backend_secure_test-app6 in 0 ms. Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Stopping backend backend_secure_test-log4j2 in 0 ms. Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Proxy haproxy-statistics stopped (FE: 0 conns, BE: 0 conns). Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Proxy http-in stopped (FE: 65 conns, BE: 0 conns). Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Proxy backend_secure_test-app6 stopped (FE: 0 conns, BE: 0 conns). Jun 14 19:42:57 192.168.64.146 haproxy[11831]: Proxy backend_secure_test-log4j2 stopped (FE: 0 conns, BE: 65 conns). Jun 14 19:42:57 192.168.64.146 haproxy[19038]: backend_secure_test-app6/test-app6_secure_secure1 changed its FQDN from (null) to sqainfra-docker04.marathon.mesos by 'SRV record' Jun 14 19:42:57 192.168.64.146 haproxy[19038]: backend_secure_test-log4j2/test-log4j2_secure_secure1 changed its FQDN from (null) to sqainfra-docker03.marathon.mesos by 'SRV record' Jun 14 19:42:57 192.168.64.146 haproxy[19038]: backend_secure_test-log4j2/test-log4j2_secure_secure2 changed its FQDN from (null) to sqainfra-docker04.marathon.mesos by 'SRV record' Jun 14 19:42:57 192.168.64.146 haproxy[19038]: backend_secure_test-log4j2/test-log4j2_secure_secure3 changed its FQDN from (null) to sqainfra-docker05.marathon.mesos by 'SRV record' Jun 14 19:43:00 192.168.64.146 haproxy[19038]: backend_secure_test-app6/test-app6_secure_secure1 changed its IP from to 192.168.64.149 by DNS cache. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: backend_secure_test-log4j2/test-log4j2_secure_secure2 changed its IP from to 192.168.64.149 by DNS cache. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: backend_secure_test-log4j2/test-log4j2_secure_secure3 changed its IP from to 192.168.64.163 by DNS cache. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: backend_secure_test-log4j2/test-log4j2_secure_secure1 changed its IP from to 192.168.64.207 by localdns/dns1. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure1 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 1ms, status: 2/2 UP. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure2 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure2 is DOWN. 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure3 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:00 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure3 is DOWN. 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure4 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure4 is DOWN. 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure5 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure5 is DOWN. 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure6 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure6 is DOWN. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure7 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure7 is DOWN. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure8 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure8 is DOWN. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure9 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure9 is DOWN. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-app6/test-app6_secure_secure10 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:01 192.168.64.146 haproxy[19038]: Server backend_secure_test-app6/test-app6_secure_secure10 is DOWN. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure1 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 1ms, status: 2/2 UP. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure2 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 0ms, status: 2/2 UP. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure3 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 0ms, status: 2/2 UP. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure4 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure4 is DOWN. 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure5 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:02 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure5 is DOWN. 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure6 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure6 is DOWN. 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure7 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure7 is DOWN. 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure8 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure8 is DOWN. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure9 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure9 is DOWN. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Health check for server backend_secure_test-log4j2/test-log4j2_secure_secure10 failed, reason: Socket error, check duration: 0ms, status: 0/3 DOWN. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Server backend_secure_test-log4j2/test-log4j2_secure_secure10 is DOWN. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Jun 14 19:43:03 192.168.64.146 haproxy[19038]: Health check for server backend_secure_scarey-log4j2/scarey-log4j2_secure_secure1 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 1ms, status: 1/3 DOWN. Jun 14 19:43:13 192.168.64.146 haproxy[19038]: Health check for server backend_secure_scarey-log4j2/scarey-log4j2_secure_secure1 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 1ms, status: 2/3 DOWN.Thanks,
Steve
Posts: 1
Participants: 1