knotd refuses to sync a secondary zone
I have the following setup:
- A primary server running debian bullseye (knot 3.0.5)
- A secondary server running the same Debian and knot version
Knot transfers most of my zones to the secondary dns without issues, except a few of them (all the reverse zones).
The main difference between working and non-working zones is that the reverse zones use mod synthrecord
to generate those entries that are undeclared on the primary zone file.
Logs for primary startup:
May 03 11:14:43 ns-primary systemd[1]: Starting Knot DNS server...
May 03 11:14:43 ns-primary knotc[1521904]: Configuration is valid
May 03 11:14:43 ns-primary knotd[1521905]: info: Knot DNS 3.0.5 starting
May 03 11:14:43 ns-primary knotd[1521905]: info: loaded configuration file '/etc/knot/knot.conf'
May 03 11:14:43 ns-primary knotd[1521905]: info: using reuseport for UDP
May 03 11:14:43 ns-primary knotd[1521905]: info: binding to interface 0.0.0.0@53
May 03 11:14:43 ns-primary knotd[1521905]: info: binding to interface ::@53
May 03 11:14:43 ns-primary knotd[1521905]: info: loading 17 zones
May 03 11:14:43 ns-primary knotd[1521905]: info: [10.168.192.in-addr.arpa.] zone will be loaded
May 03 11:14:43 ns-primary knotd[1521905]: info: [12.168.192.in-addr.arpa.] zone will be loaded
May 03 11:14:43 ns-primary knotd[1521905]: info: [21.168.192.in-addr.arpa.] zone will be loaded
May 03 11:14:43 ns-primary knotd[1521905]: info: [40.168.192.in-addr.arpa.] zone will be loaded
May 03 11:14:43 ns-primary knotd[1521905]: info: [41.168.192.in-addr.arpa.] zone will be loaded
[...]
May 03 11:14:43 ns-primary knotd[1521905]: info: [12.168.192.in-addr.arpa.] loaded, serial none -> 2022113000, 3129 bytes
May 03 11:14:43 ns-primary knotd[1521905]: info: [10.168.192.in-addr.arpa.] zone file parsed, serial 2023050200
May 03 11:14:43 ns-primary knotd[1521905]: info: [21.168.192.in-addr.arpa.] zone file parsed, serial 2022113000
May 03 11:14:43 ns-primary knotd[1521905]: info: [40.168.192.in-addr.arpa.] zone file parsed, serial 2021010433
May 03 11:14:43 ns-primary knotd[1521905]: info: [10.168.192.in-addr.arpa.] loaded, serial none -> 2023050200, 11429 bytes
May 03 11:14:43 ns-primary knotd[1521905]: info: [21.168.192.in-addr.arpa.] loaded, serial none -> 2022113000, 6757 bytes
May 03 11:14:43 ns-primary knotd[1521905]: info: [40.168.192.in-addr.arpa.] loaded, serial none -> 2021010433, 14961 bytes
May 03 11:14:43 ns-primary knotd[1521905]: info: [41.168.192.in-addr.arpa.] zone file parsed, serial 2020031100
May 03 11:14:43 ns-primary knotd[1521905]: info: [41.168.192.in-addr.arpa.] loaded, serial none -> 2020031100, 6866 bytes
[...]
May 03 11:14:43 ns-primary knotd[1521905]: info: server started in the foreground, PID 1521905
May 03 11:14:43 ns-primary knotd[1521905]: info: control, binding to '/run/knot/knot.sock'
May 03 11:14:43 ns-primary systemd[1]: Started Knot DNS server.
May 03 11:14:43 ns-primary knotd[1521905]: warning: [21.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:14:43 ns-primary knotd[1521905]: warning: [40.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:14:43 ns-primary knotd[1521905]: warning: [12.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:14:43 ns-primary knotd[1521905]: warning: [41.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:14:43 ns-primary knotd[1521905]: warning: [10.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:14:43 ns-primary knotd[1521905]: info: [21.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.13@53, serial 2022113000
May 03 11:14:43 ns-primary knotd[1521905]: info: [12.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.13@53, serial 2022113000
May 03 11:14:43 ns-primary knotd[1521905]: info: [40.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.13@53, serial 2021010433
May 03 11:14:43 ns-primary knotd[1521905]: info: [41.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.13@53, serial 2020031100
May 03 11:14:43 ns-primary knotd[1521905]: info: [10.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.13@53, serial 2023050200
May 03 11:14:43 ns-primary knotd[1521905]: info: [21.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.14@53, serial 2022113000
May 03 11:14:43 ns-primary knotd[1521905]: info: [12.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.14@53, serial 2022113000
May 03 11:14:43 ns-primary knotd[1521905]: info: [40.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.14@53, serial 2021010433
May 03 11:14:43 ns-primary knotd[1521905]: info: [41.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.14@53, serial 2020031100
May 03 11:14:43 ns-primary knotd[1521905]: info: [10.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.10.14@53, serial 2023050200
May 03 11:14:43 ns-primary knotd[1521905]: info: [company.com.] notify, outgoing, remote 192.168.21.38@53, serial 2023020621
The other side's version:
May 03 11:14:22 ns-secondary-hq systemd[1]: Starting Knot DNS server...
May 03 11:14:22 ns-secondary-hq knotc[17495]: Configuration is valid
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: Knot DNS 3.0.5 starting
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: loaded configuration file '/etc/knot/knot.conf'
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: using reuseport for UDP
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: binding to interface 0.0.0.0@53
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: binding to interface ::@53
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: loading 17 zones
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [10.168.192-in-addr.arpa.] zone will be loaded
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [12.168.192-in-addr.arpa.] zone will be loaded
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [21.168.192-in-addr.arpa.] zone will be loaded
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [40.168.192-in-addr.arpa.] zone will be loaded
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [41.168.192-in-addr.arpa.] zone will be loaded
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [company.com.] zone will be loaded
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: starting server
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [10.168.192-in-addr.arpa.] failed to parse zone file (not exists)
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [12.168.192-in-addr.arpa.] failed to parse zone file (not exists)
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [21.168.192-in-addr.arpa.] failed to parse zone file (not exists)
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [41.168.192-in-addr.arpa.] failed to parse zone file (not exists)
May 03 11:14:22 ns-secondary-hq knotd[17497]: info: [40.168.192-in-addr.arpa.] failed to parse zone file (not exists)
[...]
May 03 11:14:43 ns-secondary-hq knotd[17003]: warning: [10.168.192-in-addr.arpa.] AXFR, incoming, remote 192.168.60.20@53, server responded with error 'NOTAUTH'
May 03 11:14:43 ns-secondary-hq knotd[17003]: debug: [10.168.192-in-addr.arpa.] refresh, remote primary-ovh, address 192.168.60.20@53, failed (not allowed)
May 03 11:14:43 ns-secondary-hq knotd[17003]: warning: [10.168.192-in-addr.arpa.] refresh, remote primary-ovh not usable
May 03 11:14:43 ns-secondary-hq knotd[17003]: error: [10.168.192-in-addr.arpa.] refresh, failed (no usable master)
May 03 11:14:43 ns-secondary-hq knotd[17497]: debug: [company.com.] ACL, allowed, action notify, remote 192.168.60.20@49972, key none
May 03 11:14:43 ns-secondary-hq knotd[17497]: info: [company.com.] notify, incoming, remote 192.168.60.20@49972, serial 2023020621
[...]
May 03 11:15:30 ns-secondary-hq knotd[17003]: info: [company.com.] refresh, remote 192.168.60.20@53, remote serial 2023020621, zone is up-to-date
Given Debian bookworm is around the corner, I decided to take the plunge and upgrade the misbehaving secondary to Debian 12. But with Knot 3.2.6, same story, with slightly different log messages:
May 03 11:22:35 ns-secondary-hq systemd[1]: Starting knot.service - Knot DNS server...
May 03 11:22:35 ns-secondary-hq knotc[29899]: Configuration is valid
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: Knot DNS 3.2.6 starting
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: loaded configuration file '/etc/knot/knot.conf', mapsize 512 MiB
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: using UDP reuseport, incoming TCP Fast Open
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: binding to interface 0.0.0.0@53
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: binding to interface ::@53
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: loading 17 zones
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: [10.168.192-in-addr.arpa.] zone will be bootstrapped
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: [12.168.192-in-addr.arpa.] zone will be bootstrapped
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: [21.168.192-in-addr.arpa.] zone will be bootstrapped
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: [40.168.192-in-addr.arpa.] zone will be bootstrapped
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: [41.168.192-in-addr.arpa.] zone will be bootstrapped
May 03 11:22:35 ns-secondary-hq knotd[29901]: info: [company.com.] zone will be loaded
[...]
May 03 14:39:40 ns-secondary-hq knotd[31738]: warning: [41.168.192-in-addr.arpa.] AXFR, incoming, remote 192.168.60.20@53, server responded with error 'NOTAUTH'
May 03 14:39:40 ns-secondary-hq knotd[31738]: info: [41.168.192-in-addr.arpa.] refresh, remote primary-ovh, address 192.168.60.20@53, failed (not allowed)
May 03 14:39:40 ns-secondary-hq knotd[31738]: warning: [41.168.192-in-addr.arpa.] refresh, remote primary-ovh not usable
May 03 14:39:40 ns-secondary-hq knotd[31738]: error: [41.168.192-in-addr.arpa.] refresh, failed (no usable master), next retry at 2023-05-03T14:39:42+0000
May 03 14:39:40 ns-secondary-hq knotd[31738]: error: [41.168.192-in-addr.arpa.] zone event 'refresh' failed (no usable master)
Ah well. So I then even upgraded the primary server to 3.2.6:
May 03 11:44:54 ns-primary systemd[1]: Starting knot.service - Knot DNS server...
May 03 11:44:54 ns-primary knotc[1540091]: Configuration is valid
May 03 11:44:54 ns-primary knotd[1540092]: info: Knot DNS 3.2.6 starting
May 03 11:44:54 ns-primary knotd[1540092]: info: loaded configuration file '/etc/knot/knot.conf', mapsize 512 MiB
May 03 11:44:54 ns-primary knotd[1540092]: info: using UDP reuseport, incoming TCP Fast Open
May 03 11:44:54 ns-primary knotd[1540092]: info: binding to interface 0.0.0.0@53
May 03 11:44:54 ns-primary knotd[1540092]: info: binding to interface ::@53
May 03 11:44:54 ns-primary knotd[1540092]: info: loading 17 zones
May 03 11:44:54 ns-primary knotd[1540092]: info: [10.168.192.in-addr.arpa.] zone will be loaded
May 03 11:44:54 ns-primary knotd[1540092]: info: [12.168.192.in-addr.arpa.] zone will be loaded
May 03 11:44:54 ns-primary knotd[1540092]: info: [21.168.192.in-addr.arpa.] zone will be loaded
May 03 11:44:54 ns-primary knotd[1540092]: info: [40.168.192.in-addr.arpa.] zone will be loaded
May 03 11:44:54 ns-primary knotd[1540092]: info: [41.168.192.in-addr.arpa.] zone will be loaded
May 03 11:44:54 ns-primary knotd[1540092]: info: [company.com.] zone will be loaded
May 03 11:44:54 ns-primary knotd[1540092]: info: starting server
May 03 11:44:54 ns-primary knotd[1540092]: info: [12.168.192.in-addr.arpa.] zone file parsed, serial 2022113000
May 03 11:44:54 ns-primary knotd[1540092]: info: [12.168.192.in-addr.arpa.] loaded, serial none -> 2022113000, 3129 bytes
May 03 11:44:54 ns-primary knotd[1540092]: info: [21.168.192.in-addr.arpa.] zone file parsed, serial 2022113000
May 03 11:44:54 ns-primary knotd[1540092]: info: [41.168.192.in-addr.arpa.] zone file parsed, serial 2020031100
May 03 11:44:54 ns-primary knotd[1540092]: info: [10.168.192.in-addr.arpa.] zone file parsed, serial 2023050200
May 03 11:44:54 ns-primary knotd[1540092]: info: [21.168.192.in-addr.arpa.] loaded, serial none -> 2022113000, 6757 bytes
May 03 11:44:54 ns-primary knotd[1540092]: info: [41.168.192.in-addr.arpa.] loaded, serial none -> 2020031100, 6866 bytes
May 03 11:44:54 ns-primary knotd[1540092]: info: [10.168.192.in-addr.arpa.] loaded, serial none -> 2023050200, 11429 bytes
May 03 11:44:54 ns-primary knotd[1540092]: info: [40.168.192.in-addr.arpa.] zone file parsed, serial 2021010433
May 03 11:44:54 ns-primary knotd[1540092]: info: [40.168.192.in-addr.arpa.] loaded, serial none -> 2021010433, 14961 bytes
May 03 11:44:54 ns-primary knotd[1540092]: info: [company.com.] zone file parsed, serial 2023020621
May 03 11:44:54 ns-primary knotd[1540092]: info: [company.com.] loaded, serial none -> 2023020621, 24863 bytes
May 03 11:44:54 ns-primary knotd[1540092]: info: server started in the foreground, PID 1540092
May 03 11:44:54 ns-primary knotd[1540092]: info: control, binding to '/run/knot/knot.sock'
May 03 11:44:54 ns-primary systemd[1]: Started knot.service - Knot DNS server.
May 03 11:44:54 ns-primary knotd[1540092]: warning: [21.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:44:54 ns-primary knotd[1540092]: warning: [41.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:44:54 ns-primary knotd[1540092]: warning: [10.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:44:54 ns-primary knotd[1540092]: warning: [12.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
May 03 11:44:54 ns-primary knotd[1540092]: warning: [40.168.192.in-addr.arpa.] notify, outgoing, remote 192.168.21.38@53, server responded with error 'NOTAUTH'
The relevant configurations for a failing zone and a working zone:
remote:
- id: secondary-hq # ns1, ns-secondary1
address: 192.168.21.38@53
acl:
- id: acl_secondary-hq
address: 192.168.21.38
action: transfer
mod-synthrecord:
- id: reverse_local_net
type: reverse
prefix: dyn-
origin: local.company.com
ttl: 300
network: 192.168.10.0/24
reverse-short: on
template:
- id: companyprimary
storage: "/var/lib/knot/master"
file: "%s.zone"
notify: [ secondary-hq, secondary-old ]
acl: [ acl_secondary-hq, acl_secondary-old ]
zone:
- domain: company.com
template: companyprimary
notify: [ secondary-hq, secondary-old ]
acl: [ acl_secondary-hq, acl_secondary-old, tsig_update.acme-challenge ]
# Reverse zones
- domain: 10.168.192.in-addr.arpa
template: companyprimary
module: mod-synthrecord/reverse_local_net
notify: [ secondary-hq, secondary-old ]
acl: [ acl_secondary-hq, acl_secondary-old ]
The secondary server is similar:
remote:
- id: primary-ovh
address: 192.168.60.20@53
acl:
- id: acl_companysecondary
address: [ 192.168.60.20, 12.34.56.78 ]
action: [ transfer, notify ]
mod-synthrecord:
- id: reverse_local_net
type: reverse
prefix: dyn-
origin: local.company.com
ttl: 300
network: 192.168.10.0/24
reverse-short: on
template:
- id: companysecondary
storage: "/var/lib/knot/slave"
file: "%s.zone"
master: primary-ovh
acl: acl_companysecondary
zone:
# Slave zones
- domain: company.com
template: companysecondary
# local reverse
- domain: 10.168.192-in-addr.arpa.
template: companysecondary
module: mod-synthrecord/reverse_local_net
I have tried not declaring the synthrecord module in the secondary, but it made no difference. Any ideas? Is this a bug in knot, or am I doing something fundamentally wrong?