Timeouts with XoQ (Zone transfer over QUIC)

Hi,

I'm trying out some of the new features in Knot DNS. I installed the 3.4.2-cznic.1~bookworm packages in a Debian sid VM but I'm running into timeouts when attempting to transfer a large zone over QUIC. The zone happens to be a catalog zone, but I'm not sure if that matters.

I can reliably replicate it with kdig -d +quic -t axfr [...] which transfers a small part of the zone but fails with something like:

[...]
;; WARNING: QUIC, peer took too long to respond
;; WARNING: QUIC, peer took too long to respond
;; WARNING: QUIC, peer took too long to respond
;; WARNING: QUIC, peer took too long to respond
;; WARNING: QUIC, failed to send
;; WARNING: can't receive reply from 198.18.53.2@853(QUIC)
[...]
;; DEBUG: print_data_xfr: null parameter
;; ERROR: failed to query server 198.18.53.2@853(UDP)
;; Received 151632 B (9 messages, 1622 records)
;; Time 2024-11-26 23:58:53 UTC
;; From 198.18.53.2@853(QUIC) in 4149.3 ms

Attached is the full timestamped output from kdig as well as the timestamped log output from knotd with quic: debug logging enabled.

The knot.conf is pretty basic but I do have these configuration settings (not sure if they're relevant) in the server block:

    tcp-max-clients: 100
    tcp-idle-timeout: 300
    tcp-io-timeout: 60000 # milliseconds
    tcp-remote-io-timeout: 60000 # milliseconds
    remote-pool-limit: 100
    remote-pool-timeout: 180

Thanks!

kdig.txt

knotd.txt

Also, it's sort of implied by the kdig -b in the full log but this is with kdig and knotd running on the same machine and the packets going over the loopback interface. No firewalls running, etc. So there shouldn't be any network issues.

Huh, it turned out to be mod-rrl. I had this in my knot.conf:

mod-rrl:
  - id: default
    rate-limit: 1000
    slip: 1

[...]

template:
  - id: default
    [...]
    global-module: mod-rrl/default
    [...]

That's very surprising, since I'd expect RRL to only be applied to UDP.

Ah, it looks like this is due to time-rate-limit in mod-rrl defaulting to enabled.

Maybe protolimit_start() needs a check against KNOTD_QUERY_FLAG_AUTHORIZED, similar to ratelimit_apply() so that authenticated AXFR/IXFR/NOTIFY transactions aren't limited?

Although, it still seems odd that this is breaking in-flight zone transfers. It seems like a zone transfer should be either allowed to proceed to completion, or prevented from starting, not broken in the middle.

It also seems that this breaks TCP zone transfers, I've managed to get TCP RSTs when mod-rrl is enabled.

That's why there is a mod-rrl whitelist. Add your other servers to whitelist.

Robert, sorry for the complications

I'm afraid it won't be easy to improve that. The time based limiting is per address/subnet and is common to all connection based transport protocols (TCP, TLS, QUIC). Another problem is that protolimit_start() is called at the earliest possible time, before ACL evaluation, so KNOTD_QUERY_FLAG_AUTHORIZED isn't set yet.

I agree with @bleve that whitelist is a reasonable solution.

I have pushed two improvements: 0b3beb26 and 16701a2f.

Ah, OK. Is there a way to disable the non-UDP rate-limiting performed by the new functionality in mod-rrl? Does time-rate-limit: 0 disable those limits? (It's not clear from the documentation if they're disable-able, but looking at the code it looks like the hook is only applied if (time_limit > 0).) I.e., if I were only concerned about forged source address attacks and wanted to only enable "classical" RRL.

Thanks!

Good catch! I will extend the doc.

Yes, you can disable the time (non-UDP) limiting completely.

However, ;; From 198.18.53.2@853(QUIC) in 4149.3 ms is pretty long! What's the XFR duration over TCP?

Without mod-rrl I can transfer this zone in about a second:

;; Received 28080000 B (1667 messages, 300003 records)
;; Time 2024-11-27 19:29:39 UTC
;; From 198.18.53.2@853(QUIC) in 1183.8 ms

;; Received 27576588 B (1667 messages, 300003 records)
;; Time 2024-11-27 19:29:50 UTC
;; From 198.18.53.2@53(TCP) in 898.9 ms

The above is with the output piped away. If I print the AXFR output to my terminal it takes twice as long :-)

;; Received 28080000 B (1667 messages, 300003 records)
;; Time 2024-11-27 19:30:33 UTC
;; From 198.18.53.2@853(QUIC) in 1939.5 ms

;; Received 27576588 B (1667 messages, 300003 records)
;; Time 2024-11-27 19:31:00 UTC
;; From 198.18.53.2@53(TCP) in 2064.9 ms

Interesting, thank you!

added bug label

closed via commit 04329a6e

mentioned in commit 04329a6e