Infinite loop in knot_rrset_to_wire_extra
Hello,
A kresd
in perfectly smooth production for months suddenly climbs to 100%CPU and stays there indefinitely. On this 4-CPU machine, we started 4 kresd
, and only one had the problem.
Gdb stack below:
(gdb) where
#0 0x00007f2ff0d86373 in ?? () from /lib/x86_64-linux-gnu/libknot.so.12
#1 0x00007f2ff0d86c7f in knot_rrset_to_wire_extra () from /lib/x86_64-linux-gnu/libknot.so.12
#2 0x00007f2ff0d85843 in knot_pkt_put_rotate () from /lib/x86_64-linux-gnu/libknot.so.12
#3 0x00007f2ff0dd9790 in ?? () from /lib/libkres.so.9
#4 0x00007f2ff0ddb03d in kr_resolve_finish () from /lib/libkres.so.9
#5 0x0000000000420f5a in ?? ()
#6 0x00000000004215a8 in ?? ()
#7 0x000000000041bcb1 in ?? ()
#8 0x0000000000416831 in ?? ()
#9 0x00007f2ff0ccb9ad in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#10 0x00007f2ff0ccda23 in ?? () from /lib/x86_64-linux-gnu/libuv.so.1
#11 0x00007f2ff0cbc714 in uv_run () from /lib/x86_64-linux-gnu/libuv.so.1
#12 0x000000000040de40 in ?? ()
#13 0x00007f2ff077cd0a in __scalbn (x=0, n=-733217408) at ../sysdeps/ieee754/dbl-64/wordsize-64/s_scalbn.c:35
#14 0x00007fff661c6628 in ?? ()
#15 0x00000004c005bb00 in ?? ()
#16 0x000000000040cfc0 in ?? ()
#17 0x00007f2ff0c03478 in ?? () from /lib/x86_64-linux-gnu/libgnutls.so.30
#18 0x0000000000000000 in ?? ()
(gdb)
Disassembly around the top address shows this tight loop:
0x00007f2ff0d86359: movzwl (%r14),%r14d
0x00007f2ff0d8635d: rol $0x8,%r14w
0x00007f2ff0d86362: add $0x4000,%r14w
0x00007f2ff0d86368: movzwl %r14w,%r14d
0x00007f2ff0d8636c: add %rdi,%r14
0x00007f2ff0d8636f: movzbl (%r14),%esi
=> 0x00007f2ff0d86373: and $0xffffffc0,%esi
0x00007f2ff0d86376: cmp $0xc0,%sil
0x00007f2ff0d8637a: jne 0x7f2ff0d861ef
0x00007f2ff0d86380: jmp 0x7f2ff0d86359
And a breakpoint set on the jne
escape address 0x7f2ff0d861ef
... is never hit. This confirms the above 10 instructions make up the "loop of death". Unfortunately we have no clue on the stimulus, though nothing of importance happened at the system level (disk, CPU, RAM).
Versions involved are displayed below.
root@dns6-gp:~# dpkg -l | grep knot
ii knot-dnsutils 3.1.7-cznic.1~bullseye amd64 DNS clients provided with Knot DNS (kdig, knsupdate)
ii knot-resolver 5.5.0-cznic.1 amd64 caching, DNSSEC-validating DNS resolver
ii knot-resolver-module-http 5.3.1-1 amd64 HTTP module for Knot Resolver
ii knot-resolver-release 1.9-1 all Knot Resolver official upstream repositories
ii libknot12:amd64 3.1.7-cznic.1~bullseye amd64 DNS shared library from Knot DNS