HAProxy terminates unexpectedly while executing mysql check

I am in a process of creating Lua script to check lag between two different mysql instances and to failover to the other instance when first the primary is unavailable or the replication lag is high. HAProxy is running as a Kubernetes deployment. I am facing random container restarts due to the following error once or twice in a day:

Thread 3 is about to kill the process.
  Thread 1 : id=0x7f073feaa200 act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
             stuck=0 prof=1 harmless=0 wantrdv=0
             cpu_ns: poll=10389659667 now=10396156807 diff=6497140
             curr_task=0x561b1122bc80 (task) calls=1 last=2598383951 ns ago
               fct=0x561b0feea590(process_stream) ctx=0x561b1122b830
             strm=0x561b1122b830 src=10.1.135.141 fe=tidb be=tidb dst=unknown
             rqf=908021 rqa=0 rpf=80000000 rpa=0 sif=EST,80020 sib=INI,30
             af=(nil),0 csf=0x561b1122bd30,9000
             ab=(nil),0 csb=(nil),0
             cof=0x561b11180280,80041300:PASS(0x561b1122b800)/RAW((nil))/tcpv4(22)
             cob=(nil),0:NONE((nil))/NONE((nil))/NONE(0)
             Current executing Lua from a stream analyser -- stack traceback:
                 [C]: in method 'execute'
                 /usr/local/etc/haproxy/lagcheck/lagcheck.lua:12: in function line 11
                 [C]: in global 'xpcall'
                 /usr/local/etc/haproxy/lagcheck/lagcheck.lua:53: in function line 20
 >Thread 2 : id=0x7f073fe86700 act=1 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
             stuck=1 prof=1 harmless=0 wantrdv=0
             cpu_ns: poll=8293383908 now=10461446588 diff=2168062680
             curr_task=0x7f0738017770 (task) calls=1 last=2168391846 ns ago
               fct=0x561b0feea590(process_stream) ctx=0x7f0738017320
             strm=0x7f0738017320 src=10.1.135.141 fe=tidb be=tidb dst=unknown
             rqf=908021 rqa=0 rpf=80000000 rpa=0 sif=EST,80020 sib=INI,30
             af=(nil),0 csf=0x7f07380172d0,9000
             ab=(nil),0 csb=(nil),0
             cof=0x7f0738017020,80041300:PASS(0x7f0738017250)/RAW((nil))/tcpv4(24)
             cob=(nil),0:NONE((nil))/NONE((nil))/NONE(0)
             Current executing Lua from a stream analyser --
             call trace(16):
             | 0x561b0ff5e27b [89 c5 85 c0 0f 85 db 01]: ha_thread_dump+0x24b/0x448
             | 0x561b0ff5e4e6 [48 8b 05 c3 6c 22 00 48]: debug_handler+0x66/0x10e
             | 0x7f0740645730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730
             | 0x561b0fe65062 [e9 79 ff ff ff 66 0f 1f]: hlua_ctx_init+0xb2/0x186
             | 0x561b0fe67871 [85 c0 0f 84 3f 01 00 00]: main+0x32211
             | 0x561b0ff00eed [85 c0 0f 84 8b 00 00 00]: sample_process+0x4d/0x127
             | 0x561b0ff01a0c [48 85 c0 74 3f 64 48 63]: sample_fetch_as_type+0x3c/0x9b
             | 0x561b0ff2fbaa [48 85 c0 0f 84 e4 15 00]: sess_build_logline+0x204a/0x3b39
             | 0x561b0fee349b [85 c0 75 20 48 8b 3d 62]: main+0xade3b
             | 0x561b0feec4c9 [85 c0 0f 85 b3 f2 ff ff]: process_stream+0x1f39/0x4f76
*>Thread 3 : id=0x7f073f533700 act=1 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
             stuck=1 prof=1 harmless=0 wantrdv=0
             cpu_ns: poll=7537145797 now=10026778437 diff=2489632640
             curr_task=0x7f07340460b0 (task) calls=1 last=2490903653 ns ago
               fct=0x561b0feea590(process_stream) ctx=0x7f0730047130
             strm=0x7f0730047130 src=10.1.136.232 fe=tidb be=tidb dst=unknown
             rqf=908021 rqa=0 rpf=80000000 rpa=0 sif=EST,80020 sib=INI,30
             af=(nil),0 csf=0x7f073002cee0,9000
             ab=(nil),0 csb=(nil),0
             cof=0x7f0730016940,80041300:PASS(0x7f0730016b70)/RAW((nil))/tcpv4(23)
             cob=(nil),0:NONE((nil))/NONE((nil))/NONE(0)
             Current executing Lua from a stream analyser --
             call trace(20):
 12   creationTimestamp: "2021-04-20T09:41:28Z"
             | 0x561b0ffc7607 [48 83 c4 10 5b 5d 41 5c]: wdt_handler+0x107/0x114
             | 0x7f0740645730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730
             | 0x561b0fe65062 [e9 79 ff ff ff 66 0f 1f]: hlua_ctx_init+0xb2/0x186
             | 0x561b0fe67871 [85 c0 0f 84 3f 01 00 00]: main+0x32211
             | 0x561b0ff00eed [85 c0 0f 84 8b 00 00 00]: sample_process+0x4d/0x127
             | 0x561b0ff01a0c [48 85 c0 74 3f 64 48 63]: sample_fetch_as_type+0x3c/0x9b
             | 0x561b0ff2fbaa [48 85 c0 0f 84 e4 15 00]: sess_build_logline+0x204a/0x3b39
             | 0x561b0fee349b [85 c0 75 20 48 8b 3d 62]: main+0xade3b
             | 0x561b0feec4c9 [85 c0 0f 85 b3 f2 ff ff]: process_stream+0x1f39/0x4f76
 >Thread 4 : id=0x7f073ed32700 act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
             stuck=1 prof=1 harmless=0 wantrdv=0
             cpu_ns: poll=8321293688 now=10915650748 diff=2594357060
             curr_task=0x7f0734017090 (task) calls=1 last=2643250396 ns ago
               fct=0x561b0feea590(process_stream) ctx=(nil)

             call trace(11):
             | 0x561b0ff5e27b [89 c5 85 c0 0f 85 db 01]: ha_thread_dump+0x24b/0x448
             | 0x561b0ff5e4e6 [48 8b 05 c3 6c 22 00 48]: debug_handler+0x66/0x10e
             | 0x7f0740645730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730
             | 0x561b0fe68222 [e9 c1 fd ff ff 66 0f 1f]: hlua_ctx_destroy+0x312/0x367
             | 0x561b0feedf81 [49 83 bf f0 00 00 00 00]: process_stream+0x39f1/0x4f76
[NOTICE] 111/144739 (1) : haproxy version is 2.2.13-5f3eb59
[NOTICE] 111/144739 (1) : path to executable is /usr/local/sbin/haproxy
[ALERT] 111/144739 (1) : Current worker #1 (7) exited with code 134 (Aborted)
[ALERT] 111/144739 (1) : exit-on-failure: killing every processes with SIGTERM
[WARNING] 111/144739 (1) : All workers exited. Exiting... (134)

1 post - 1 participant

Read full topic