Skip to content
Snippets Groups Projects

XDP broken in kresd, since libknot 3.1

    • Closed Issue created by make @mahsiaoko

    There are some problem for us to get reply from knot resolver after we use xdp. The details are shown below:

    OS: Ubuntu 21.14
kernel: 5.13.0-35-generic
network card: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (driver i40e)
knot-resolver:5.5.0-cznic.1
knot:3.1.1-cznic.1

    kresd.conf

    -- SPDX-License-Identifier: CC0-1.0
-- vim:syntax=lua:set ts=4 sw=4:
-- Refer to manual: https://knot-resolver.readthedocs.org/en/stable/

-- Network interface configuration
net.listen('10.0.0.2', 53, { kind = 'xdp' })
--net.listen('10.0.0.2', 53, { kind = 'dns' })
--net.listen('127.0.0.1', 853, { kind = 'tls' })
--net.listen('127.0.0.1', 443, { kind = 'doh2' })
--net.listen('::1', 53, { kind = 'dns', freebind = true })
--net.listen('::1', 853, { kind = 'tls', freebind = true })
--net.listen('::1', 443, { kind = 'doh2' })

-- Load useful modules
modules = {
	'hints > iterate',  -- Allow loading /etc/hosts or custom root hints
	'stats',            -- Track internal statistics
	'predict',          -- Prefetch expiring/frequent records
}

-- Cache size
cache.size = 1024 * MB

    kresd@service

    # SPDX-License-Identifier: CC0-1.0
[Unit]
Description=Knot Resolver daemon
Documentation=man:kresd.systemd(7)
Documentation=man:kresd(8)
Wants=kres-cache-gc.service
Before=kres-cache-gc.service
Wants=network-online.target
After=network-online.target
Before=nss-lookup.target
Wants=nss-lookup.target

[Service]
Type=notify
Environment="SYSTEMD_INSTANCE=%i"
WorkingDirectory=/var/lib/knot-resolver
ExecStart=/usr/sbin/kresd -c /usr/lib/knot-resolver/distro-preconfig.lua -c /etc/knot-resolver/kresd.conf -n
ExecStopPost=/usr/bin/env rm -f "/run/knot-resolver/control/%i"
User=knot-resolver
Group=knot-resolver
CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN CAP_SYS_ADMIN CAP_SYS_RESOURCE
AmbientCapabilities=CAP_NET_RAW CAP_NET_ADMIN CAP_SYS_ADMIN CAP_SYS_RESOURCE
TimeoutStopSec=10s
WatchdogSec=10s
Restart=on-abnormal
LimitNOFILE=524288
Slice=system-kresd.slice

[Install]
WantedBy=kresd.target

    And after we start knot resolver through systemctl start kresd@1.service, and run dig @10.0.0.2 www.baidu.com, we cannot get reply from knot resolver

    root@master-ubuntu1:~# systemctl status kresd@1.service
● kresd@1.service - Knot Resolver daemon
     Loaded: loaded (/lib/systemd/system/kresd@.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-03-22 14:32:34 CST; 4s ago
       Docs: man:kresd.systemd(7)
             man:kresd(8)
   Main PID: 17028 (kresd)
      Tasks: 1 (limit: 629145)
     Memory: 21.2M
        CPU: 106ms
     CGroup: /system.slice/system-kresd.slice/kresd@1.service
             └─17028 /usr/sbin/kresd -c /usr/lib/knot-resolver/distro-preconfig.lua -c /etc/knot-resolver/kresd.conf -n

3月 22 14:32:34 master-ubuntu1 systemd[1]: Starting Knot Resolver daemon...
3月 22 14:32:34 master-ubuntu1 kresd[17028]: libbpf: Kernel error message: XDP program already attached
3月 22 14:32:34 master-ubuntu1 systemd[1]: Started Knot Resolver daemon.

    root@master-ubuntu1:~# dig @10.0.0.2 www.baidu.com

; <<>> DiG 9.16.15-Ubuntu <<>> @10.0.0.2 www.baidu.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items

    0
    Show labels

    No child items are currently assigned. Use child items to break down this issue into smaller parts.

      1

      When this merge request is accepted, this issue will be closed automatically.

      Activity

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          You seem to query from the same machine, but such a query won't be routed to the network interface that you configured with XDP. If you want to test that, you'll need to send packets from outside to the corresponding network card.

        • make
          make @mahsiaoko ·
          Author

          I have also tried to query from another machine, and I cannot get reply too.

          master@master-NF8480M5:~$ dig @10.0.0.2 www.baidu.com

; <<>> DiG 9.16.1-Ubuntu <<>> @10.0.0.2 www.baidu.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner
          • Your set up utilizes just a single XDP queue. Did you use ethtool to ensure that the interface only has one queue?
          • log_level('debug') config might show more about what's happening on the server side
        • make
          make @mahsiaoko ·
          Author
          • The server which I send a query from
          root@master-NF8480M5:~# ethtool -l enp173s0f1
Channel parameters for enp173s0f1:
Pre-set maximums:
RX:		0
TX:		0
Other:		1
Combined:	129
Current hardware settings:
RX:		0
TX:		0
Other:		1
Combined:	1

          The server which knot resolver run,

          root@master-ubuntu1:~# ethtool -l enp109s0f0
Channel parameters for enp109s0f0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		1
Combined:	129
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		1
Combined:	1

          But I still cannot get reply.

          • I also modified the configuration,
          -- SPDX-License-Identifier: CC0-1.0
-- vim:syntax=lua:set ts=4 sw=4:
-- Refer to manual: https://knot-resolver.readthedocs.org/en/stable/

-- Network interface configuration
net.listen('10.0.0.2', 53, { kind = 'xdp' })
net.listen('10.0.0.2', 53, { kind = 'dns' })
--net.listen('127.0.0.1', 853, { kind = 'tls' })
--net.listen('127.0.0.1', 443, { kind = 'doh2' })
--net.listen('::1', 53, { kind = 'dns', freebind = true })
--net.listen('::1', 853, { kind = 'tls', freebind = true })
--net.listen('::1', 443, { kind = 'doh2' })

-- Load useful modules
modules = {
        'hints > iterate',  -- Allow loading /etc/hosts or custom root hints
        'stats',            -- Track internal statistics
        'predict',          -- Prefetch expiring/frequent records
}

-- Cache size
cache.size = 1024 * MB

          But there will be a error,

          15:26:29 master-ubuntu1 kresd[20122]: libbpf: Kernel error message: XDP program already attached
15:26:29 master-ubuntu1 kresd[20122]: [net   ] bind to '10.0.0.2@53' (UDP): Permission denied
15:26:29 master-ubuntu1 kresd[20122]: [system] error while loading config: error occurred here (config filename:lineno is at the bottom, if config is involved):
                                              stack traceback:
                                                      [C]: in function 'listen'
                                                      /etc/knot-resolver/kresd.conf:7: in main chunk
                                              ERROR: net.listen() failed to bind (workdir '/var/lib/knot-resolver')
15:26:29 master-ubuntu1 systemd[1]: kresd@1.service: Main process exited, code=exited, status=1/FAILURE
          • And could you tell me how to config log_level('debug')
        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          log_level('debug') is a line that you can simply add to your configuration file (docs). The "permission denied" is suspicious; well verbose log might show a bit more, so I'd start with that.

        • make
          make @mahsiaoko ·
          Author

          I have added log_level('debug') to configuration file

          -- SPDX-License-Identifier: CC0-1.0
-- vim:syntax=lua:set ts=4 sw=4:
-- Refer to manual: https://knot-resolver.readthedocs.org/en/stable/

-- Network interface configuration
net.listen('10.0.0.2', 53, { kind = 'xdp' })
net.listen('10.0.0.2', 53, { kind = 'dns' })
--net.listen('127.0.0.1', 853, { kind = 'tls' })
--net.listen('127.0.0.1', 443, { kind = 'doh2' })
--net.listen('::1', 53, { kind = 'dns', freebind = true })
--net.listen('::1', 853, { kind = 'tls', freebind = true })
--net.listen('::1', 443, { kind = 'doh2' })

log_level('debug')

-- Load useful modules
modules = {
	'hints > iterate',  -- Allow loading /etc/hosts or custom root hints
	'stats',            -- Track internal statistics
	'predict',          -- Prefetch expiring/frequent records
}

-- Cache size
cache.size = 1024 * MB

          root@master-ubuntu1:/var/log# journalctl -u kresd@* -f
-- Journal begins at Fri 2022-03-11 11:21:12 CST. --
15:38:14 master-ubuntu1 systemd[1]: kresd@1.service: Main process exited, code=exited, status=1/FAILURE
15:38:14 master-ubuntu1 systemd[1]: kresd@1.service: Failed with result 'exit-code'.
15:38:14 master-ubuntu1 systemd[1]: Failed to start Knot Resolver daemon.
15:40:09 master-ubuntu1 systemd[1]: Starting Knot Resolver daemon...
15:40:09 master-ubuntu1 kresd[21614]: libbpf: Kernel error message: XDP program already attached
15:40:09 master-ubuntu1 kresd[21614]: [net   ] bind to '10.0.0.2@53' (UDP): Permission denied
15:40:09 master-ubuntu1 kresd[21614]: [system] error while loading config: error occurred here (config filename:lineno is at the bottom, if config is involved):
                                              stack traceback:
                                                      [C]: in function 'listen'
                                                      /etc/knot-resolver/kresd.conf:7: in main chunk
                                              ERROR: net.listen() failed to bind (workdir '/var/lib/knot-resolver')
15:40:09 master-ubuntu1 systemd[1]: kresd@1.service: Main process exited, code=exited, status=1/FAILURE
15:40:09 master-ubuntu1 systemd[1]: kresd@1.service: Failed with result 'exit-code'.
15:40:09 master-ubuntu1 systemd[1]: Failed to start Knot Resolver daemon.
15:48:46 master-ubuntu1 systemd[1]: Starting Knot Resolver daemon...
15:48:46 master-ubuntu1 kresd[21689]: libbpf: Kernel error message: XDP program already attached
15:48:46 master-ubuntu1 kresd[21689]: [net   ] bind to '10.0.0.2@53' (UDP): Permission denied
15:48:46 master-ubuntu1 kresd[21689]: [system] error while loading config: error occurred here (config filename:lineno is at the bottom, if config is involved):
                                              stack traceback:
                                                      [C]: in function 'listen'
                                                      /etc/knot-resolver/kresd.conf:7: in main chunk
                                              ERROR: net.listen() failed to bind (workdir '/var/lib/knot-resolver')
15:48:46 master-ubuntu1 systemd[1]: kresd@1.service: Main process exited, code=exited, status=1/FAILURE
15:48:46 master-ubuntu1 systemd[1]: kresd@1.service: Failed with result 'exit-code'.
15:48:46 master-ubuntu1 systemd[1]: Failed to start Knot Resolver daemon.

          As for 'permission denied',I have already started knot resolver as root.

          Edited by make
        • make
          make @mahsiaoko ·
          Author

          Also I have tried delete net.listen('10.0.0.2', 53, { kind = 'dns' }) from configuration file and start knot resolver, and then send a query from another machine,

          root@master-NF8480M5:~# dig @10.0.0.2 www.qq.com

; <<>> DiG 9.16.1-Ubuntu <<>> @10.0.0.2 www.qq.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
          root@master-ubuntu1:/etc/knot-resolver# journalctl -u kresd@* -f
-- Journal begins at Fri 2022-03-11 11:21:12 CST. --
16:31:39 master-ubuntu1 kresd[22014]: [resolv][22989.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:39 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:42 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:42 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'A' uid [37162.00]
16:31:42 master-ubuntu1 kresd[22014]: [iterat][37162.00]   'gitlab.nic.cz.' type 'A' new uid was assigned .01, parent uid .00
16:31:42 master-ubuntu1 kresd[22014]: [cache ][37162.01]   => satisfied by exact RRset: rank 060, new TTL 511186
16:31:42 master-ubuntu1 kresd[22014]: [iterat][37162.01]   <= rcode: NOERROR
16:31:42 master-ubuntu1 kresd[22014]: [resolv][37162.01]   AD: request classified as SECURE
16:31:42 master-ubuntu1 kresd[22014]: [resolv][37162.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:42 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:44 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:44 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'AAAA' uid [22989.00]
16:31:44 master-ubuntu1 kresd[22014]: [iterat][22989.00]   'gitlab.nic.cz.' type 'AAAA' new uid was assigned .01, parent uid .00
16:31:44 master-ubuntu1 kresd[22014]: [cache ][22989.01]   => satisfied by exact RRset: rank 060, new TTL 511184
16:31:44 master-ubuntu1 kresd[22014]: [iterat][22989.01]   <= rcode: NOERROR
16:31:44 master-ubuntu1 kresd[22014]: [resolv][22989.01]   AD: request classified as SECURE
16:31:44 master-ubuntu1 kresd[22014]: [resolv][22989.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:44 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:44 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:44 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'www.qq.com.' type 'A' uid [53164.00]
16:31:44 master-ubuntu1 kresd[22014]: [iterat][53164.00]   'www.qq.com.' type 'A' new uid was assigned .01, parent uid .00
16:31:44 master-ubuntu1 kresd[22014]: [cache ][53164.01]   => satisfied by exact CNAME: rank 060, new TTL 511225
16:31:44 master-ubuntu1 kresd[22014]: [iterat][53164.01]   <= rcode: NOERROR
16:31:44 master-ubuntu1 kresd[22014]: [iterat][53164.01]   <= cname chain, following
16:31:44 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'ins-r23tsuuf.ias.tencent-cloud.net.' type 'A' uid [53164.02]
16:31:44 master-ubuntu1 kresd[22014]: [valdtr][53164.01]   <= answer valid, OK
16:31:44 master-ubuntu1 kresd[22014]: [iterat][53164.02]   'ins-r23tsuuf.ias.tencent-cloud.net.' type 'A' new uid was assigned .03, parent uid .00
16:31:44 master-ubuntu1 kresd[22014]: [cache ][53164.03]   => satisfied by exact RRset: rank 060, new TTL 511187
16:31:44 master-ubuntu1 kresd[22014]: [iterat][53164.03]   <= rcode: NOERROR
16:31:44 master-ubuntu1 kresd[22014]: [valdtr][53164.03]   <= answer valid, OK
16:31:44 master-ubuntu1 kresd[22014]: [resolv][53164.03]   AD: request classified as SECURE
16:31:44 master-ubuntu1 kresd[22014]: [resolv][53164.03]   finished in state: 4, queries: 2, mempool: 32800 B
16:31:44 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:48 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:48 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'A' uid [37162.00]
16:31:48 master-ubuntu1 kresd[22014]: [iterat][37162.00]   'gitlab.nic.cz.' type 'A' new uid was assigned .01, parent uid .00
16:31:48 master-ubuntu1 kresd[22014]: [cache ][37162.01]   => satisfied by exact RRset: rank 060, new TTL 511180
16:31:48 master-ubuntu1 kresd[22014]: [iterat][37162.01]   <= rcode: NOERROR
16:31:48 master-ubuntu1 kresd[22014]: [resolv][37162.01]   AD: request classified as SECURE
16:31:48 master-ubuntu1 kresd[22014]: [resolv][37162.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:48 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:49 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:49 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'www.qq.com.' type 'A' uid [53164.00]
16:31:49 master-ubuntu1 kresd[22014]: [iterat][53164.00]   'www.qq.com.' type 'A' new uid was assigned .01, parent uid .00
16:31:49 master-ubuntu1 kresd[22014]: [cache ][53164.01]   => satisfied by exact CNAME: rank 060, new TTL 511220
16:31:49 master-ubuntu1 kresd[22014]: [iterat][53164.01]   <= rcode: NOERROR
16:31:49 master-ubuntu1 kresd[22014]: [iterat][53164.01]   <= cname chain, following
16:31:49 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'ins-r23tsuuf.ias.tencent-cloud.net.' type 'A' uid [53164.02]
16:31:49 master-ubuntu1 kresd[22014]: [valdtr][53164.01]   <= answer valid, OK
16:31:49 master-ubuntu1 kresd[22014]: [iterat][53164.02]   'ins-r23tsuuf.ias.tencent-cloud.net.' type 'A' new uid was assigned .03, parent uid .00
16:31:49 master-ubuntu1 kresd[22014]: [cache ][53164.03]   => satisfied by exact RRset: rank 060, new TTL 511182
16:31:49 master-ubuntu1 kresd[22014]: [iterat][53164.03]   <= rcode: NOERROR
16:31:49 master-ubuntu1 kresd[22014]: [valdtr][53164.03]   <= answer valid, OK
16:31:49 master-ubuntu1 kresd[22014]: [resolv][53164.03]   AD: request classified as SECURE
16:31:49 master-ubuntu1 kresd[22014]: [resolv][53164.03]   finished in state: 4, queries: 2, mempool: 32800 B
16:31:49 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:49 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:49 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'AAAA' uid [22989.00]
16:31:49 master-ubuntu1 kresd[22014]: [iterat][22989.00]   'gitlab.nic.cz.' type 'AAAA' new uid was assigned .01, parent uid .00
16:31:49 master-ubuntu1 kresd[22014]: [cache ][22989.01]   => satisfied by exact RRset: rank 060, new TTL 511179
16:31:49 master-ubuntu1 kresd[22014]: [iterat][22989.01]   <= rcode: NOERROR
16:31:49 master-ubuntu1 kresd[22014]: [resolv][22989.01]   AD: request classified as SECURE
16:31:49 master-ubuntu1 kresd[22014]: [resolv][22989.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:49 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:53 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:53 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'A' uid [37162.00]
16:31:53 master-ubuntu1 kresd[22014]: [iterat][37162.00]   'gitlab.nic.cz.' type 'A' new uid was assigned .01, parent uid .00
16:31:53 master-ubuntu1 kresd[22014]: [cache ][37162.01]   => satisfied by exact RRset: rank 060, new TTL 511175
16:31:53 master-ubuntu1 kresd[22014]: [iterat][37162.01]   <= rcode: NOERROR
16:31:53 master-ubuntu1 kresd[22014]: [resolv][37162.01]   AD: request classified as SECURE
16:31:53 master-ubuntu1 kresd[22014]: [resolv][37162.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:53 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:54 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:54 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'www.qq.com.' type 'A' uid [53164.00]
16:31:54 master-ubuntu1 kresd[22014]: [iterat][53164.00]   'www.qq.com.' type 'A' new uid was assigned .01, parent uid .00
16:31:54 master-ubuntu1 kresd[22014]: [cache ][53164.01]   => satisfied by exact CNAME: rank 060, new TTL 511215
16:31:54 master-ubuntu1 kresd[22014]: [iterat][53164.01]   <= rcode: NOERROR
16:31:54 master-ubuntu1 kresd[22014]: [iterat][53164.01]   <= cname chain, following
16:31:54 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'ins-r23tsuuf.ias.tencent-cloud.net.' type 'A' uid [53164.02]
16:31:54 master-ubuntu1 kresd[22014]: [valdtr][53164.01]   <= answer valid, OK
16:31:54 master-ubuntu1 kresd[22014]: [iterat][53164.02]   'ins-r23tsuuf.ias.tencent-cloud.net.' type 'A' new uid was assigned .03, parent uid .00
16:31:54 master-ubuntu1 kresd[22014]: [cache ][53164.03]   => satisfied by exact RRset: rank 060, new TTL 511177
16:31:54 master-ubuntu1 kresd[22014]: [iterat][53164.03]   <= rcode: NOERROR
16:31:54 master-ubuntu1 kresd[22014]: [valdtr][53164.03]   <= answer valid, OK
16:31:54 master-ubuntu1 kresd[22014]: [resolv][53164.03]   AD: request classified as SECURE
16:31:54 master-ubuntu1 kresd[22014]: [resolv][53164.03]   finished in state: 4, queries: 2, mempool: 32800 B
16:31:54 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:55 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:55 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'AAAA' uid [22989.00]
16:31:55 master-ubuntu1 kresd[22014]: [iterat][22989.00]   'gitlab.nic.cz.' type 'AAAA' new uid was assigned .01, parent uid .00
16:31:55 master-ubuntu1 kresd[22014]: [cache ][22989.01]   => satisfied by exact RRset: rank 060, new TTL 511173
16:31:55 master-ubuntu1 kresd[22014]: [iterat][22989.01]   <= rcode: NOERROR
16:31:55 master-ubuntu1 kresd[22014]: [resolv][22989.01]   AD: request classified as SECURE
16:31:55 master-ubuntu1 kresd[22014]: [resolv][22989.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:55 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:31:58 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:31:58 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'A' uid [44665.00]
16:31:58 master-ubuntu1 kresd[22014]: [iterat][44665.00]   'gitlab.nic.cz.' type 'A' new uid was assigned .01, parent uid .00
16:31:58 master-ubuntu1 kresd[22014]: [cache ][44665.01]   => satisfied by exact RRset: rank 060, new TTL 511170
16:31:58 master-ubuntu1 kresd[22014]: [iterat][44665.01]   <= rcode: NOERROR
16:31:58 master-ubuntu1 kresd[22014]: [resolv][44665.01]   AD: request classified as SECURE
16:31:58 master-ubuntu1 kresd[22014]: [resolv][44665.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:31:58 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
16:32:00 master-ubuntu1 kresd[22014]: [xdp   ] poll triggered, processing a batch of 1 packets
16:32:00 master-ubuntu1 kresd[22014]: [plan  ][00000.00] plan 'gitlab.nic.cz.' type 'AAAA' uid [22989.00]
16:32:00 master-ubuntu1 kresd[22014]: [iterat][22989.00]   'gitlab.nic.cz.' type 'AAAA' new uid was assigned .01, parent uid .00
16:32:00 master-ubuntu1 kresd[22014]: [cache ][22989.01]   => satisfied by exact RRset: rank 060, new TTL 511168
16:32:00 master-ubuntu1 kresd[22014]: [iterat][22989.01]   <= rcode: NOERROR
16:32:00 master-ubuntu1 kresd[22014]: [resolv][22989.01]   AD: request classified as SECURE
16:32:00 master-ubuntu1 kresd[22014]: [resolv][22989.01]   finished in state: 4, queries: 1, mempool: 32800 B
16:32:00 master-ubuntu1 kresd[22014]: [xdp   ] pushed a packet, ret = 0
        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          Ah, OK. I already had earlier suspicion that sending replies over XDP somehow regressed over time in kresd. I suspect that newer kernel versions triggered this, but overall I haven't looked into this much so far. (The important parts of XDP code are shared with knotd and there it apparently still works.)

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          I have no idea about the "permission denied" part, but perhaps that's unrelated to XDP and less important here.

        • make
          make @mahsiaoko ·
          Author

          So it is the newer linux kernel caused this? And may you recommend me a version of linux kernel that you have tested?

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          I'll have to get back to this issue later, as now I'm not confident that downgrading kernel will really help. But if you want to try, XDP testing of kresd was done mostly in kernel range 4.19 – 5.4 (those two branches are still maintained upstream).

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          Hmm, I quickly checked on our previous primary set up, with SW slightly updated in the meantime, of course, and it does not work. The failure to send XDP replies is the same (the non-XDP "permission denied" does not happen though). It's Ubuntu 18.04 with kernel 5.4 and current kresd 5.5.0 from our upstream repo.

          Edited by Vladimír Čunát
        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          No, kernel is not to blame here. It does work with older kresd+libknot combination on the very same set up.

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          The update of libknot from 3.0 to 3.1 is breaking this.

          EDIT: let me continue below, outside the thread.

          Edited by Vladimír Čunát
        • make
          make @mahsiaoko ·
          Author

          I installed older kresd+libknot combination, where the version of kresd is knot-resolver_5.3.1-1_amd64.deb and libknot is libknot-dev_3.0.10-cznic.1. It does not work.

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          I'm not sure what was wrong there, but one thing to note is that after major libknot changes you need to get rid of the old (or just different) BPF program on the interface(s). Reboot would work as well; I do:

          sudo ip link set dev eth1234 xdp off
        • Please register or sign in to reply
      • Vladimír Čunát
        Vladimír Čunát @vcunat ·
        Owner

        Also related part of our XDP docs:

        Note that you want to also keep the vanilla DNS line to service TCP and possibly any fallback UDP (e.g. from unclaimed queues).

      • Vladimír Čunát changed title from We cannot get reply from knot resolver after using xdp to XDP broken in kresd, since libknot 3.1

        changed title from We cannot get reply from knot resolver after using xdp to XDP broken in kresd, since libknot 3.1

      • Vladimír Čunát mentioned in merge request !1276 (merged)

        mentioned in merge request !1276 (merged)

        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          I wrote a patch that fixed XDP in my test set up. If you want to test it already, (preliminary) packages with instructions are at: https://software.opensuse.org/download.html?project=home%3ACZ-NIC%3Aknot-resolver-testing&package=knot-resolver

        • make
          make @mahsiaoko ·
          Author

          I installed the test version,

          root@master-ubuntu1:~# apt info knot-resolver
Package: knot-resolver
Version: 5.5.0.1647962986.1d9edb05-cznic.1
Status: install ok installed
Priority: optional
Section: net
Maintainer: Knot Resolver <knot-resolver@labs.nic.cz>
Installed-Size: 1,062 kB
Depends: adduser, dns-root-data, systemd, libc6 (>= 2.34), libcap-ng0 (>= 0.7.9), libdnssec8 (>= 3.1.0), libedit2 (>= 2.11-20080614-0), libgcc-s1 (>= 3.3.1), libgnutls30 (>= 3.7.0), libknot12 (>= 3.1.0), liblmdb0 (>= 0.9.9), libluajit-5.1-2 (>= 2.0.4+dfsg), libnghttp2-14 (>= 1.3.0), libstdc++6 (>= 5), libsystemd0 (>= 227), libuv1 (>= 1.7.0), libzscanner4 (>= 3.1.0)
Recommends: lua-basexx, lua-cqueues, lua-http, lua-psl
Suggests: knot-resolver-module-http
Breaks: libkres9 (<< 3.2.1-2)
Replaces: libkres9 (<< 3.2.1-2)
Homepage: https://www.knot-resolver.cz/

          But it seemd to don't solve this problem, I still cannot get response from knot resolver.

          Edited by make
        • Vladimír Čunát
          Vladimír Čunát @vcunat ·
          Owner

          Oh, I'm sorry. Somehow I put a part of the code on a wrong place, and surprisingly it worked "sometimes". That should be fixed now.

        • Please register or sign in to reply
      • Vladimír Čunát closed via merge request !1276 (merged)

        closed via merge request !1276 (merged)

      • Vladimír Čunát mentioned in commit d1988269

        mentioned in commit d1988269

      Please register or sign in to reply