Error "refresh, failed (no usable master)" when master is present

Description

Slave server doesn't ask for updates after notify is being recived by it.
Error text: error: [example.com.] refresh, failed (no usable master)

The most strange part is that AXFR bootstrapping goes fine.
If zone files are deleted, then slave server is restarted, then files are recreated after notify event (or knotc zone-refresh).

Reproduction

Distro: Gentoo

Master version: 2.7.0
Slave version: 2.7.0

Steps:

1. update SOA record serial on the master
2. perform knotc zone-reload on the master
3. perform knot zone-notify on the master

Full slave log example:

info: [example.com.] notify, incoming, 123.123.123.123@35724: received, serial 2018080900
info: [example.com.] refresh, outgoing, 123.123.123.123@53: remote serial 2018080900, zone is outdated
warning: [example.com.] refresh, remote master not usable
error: [example.com.] refresh, failed (no usable master)

Additional info

Workaround: downgrading slave version to 2.5.7.
Version 2.6.8 is affected too.

More information can be provided on request.

changed the description

Thanks much for the report!

For us to start looking on it, additional info is needed:

• i assume the master is running Knot too. Is that correct?
• could you please provide pieces of your confiuration (and also the master configuration), related to XFR?
• could you please reproduce it with more verbose logging level and provide us with the log (possibly also from the master)?

log:
      - target: "stdout"
        any: "debug"

Thank you!

@eugene-bright any news?

Sorry for late response.

The master is running Knot 2.7.0.

Full master config:

server:
    listen: 0.0.0.0@53
    listen: ::@53

log:
  - target: syslog
    any: debug

key:
  - id: secret
    algorithm: hmac-sha256
    secret: ***

remote:
 - id: eugene2
   address: 51.15.99.23@53
   key: secret

acl:
 - id: accept_transfer
   key: secret
   # address: 51.15.99.23
   action: transfer

mod-rrl:
  - id: default
    rate-limit: 200 # Allow 200 resp/s for each flow
    slip: 2 # Every other response slips

template:
  - id: default
    storage: "/var/lib/knot/zones"
    global-module: mod-rrl/default # Enable RRL globally

zone:
  - domain: bright.gdn
    notify: eugene2
    acl: accept_transfer

zone:
  - domain: 6.8.0.d.0.7.4.0.1.0.0.2.ip6.arpa.
    # file: 1.0.0.0.6.8.0.d.0.7.4.0.1.0.0.2.ip6.arpa.zone
    notify: eugene2
    acl: accept_transfer

Full slave config:

server:
    listen: 0.0.0.0@53
    listen: ::@53

log:
  - target: syslog
    any: debug

key:
  - id: secret
    algorithm: hmac-sha256
    secret: ***

remote:
 - id: eugene1
   address: 217.182.90.28@53
   key: secret

acl:
 - id: accept_slave
   key: secret
   # address: 217.182.90.28
   action: [notify, update]

mod-rrl:
  - id: default
    rate-limit: 200 # Allow 200 resp/s for each flow
    slip: 2 # Every other response slips

template:
  - id: default
    storage: "/var/lib/knot/zones"
    global-module: mod-rrl/default # Enable RRL globally

zone:
  - domain: bright.gdn
    master: [eugene1]
    acl: accept_slave

zone:
  - domain: 6.8.0.d.0.7.4.0.1.0.0.2.ip6.arpa.
    master: [eugene1]
    acl: accept_slave

Scenario:

1. slave temporary updated to 2.7.0
2. bright.gdn.zone updated (sequence number bumped)
3. on master: knotc zone-reload

Master log:

Aug 21 02:08:41 eugene1.servers.bright.gdn knotd[18603]: info: control, received command 'zone-reload'
Aug 21 02:08:41 eugene1.servers.bright.gdn knotd[18603]: info: [6.8.0.d.0.7.4.0.1.0.0.2.ip6.arpa.] loaded, serial 2018081200 -> 2018081200                           
Aug 21 02:08:41 eugene1.servers.bright.gdn knotd[18603]: info: [bright.gdn.] loaded, serial 2018081402 -> 2018081403
Aug 21 02:08:41 eugene1.servers.bright.gdn knotd[18603]: info: [bright.gdn.] notify, outgoing, 51.15.99.23@53: serial 2018081403

Slave log:

Aug 21 02:08:41 eugene2.servers.bright.gdn knotd[20151]: info: [bright.gdn.] notify, incoming, 217.182.90.28@48916: received, serial 2018081403
Aug 21 02:08:41 eugene2.servers.bright.gdn knotd[20151]: info: [bright.gdn.] refresh, outgoing, 217.182.90.28@53: remote serial 2018081403, zone is outdated
Aug 21 02:08:41 eugene2.servers.bright.gdn knotd[20151]: warning: [bright.gdn.] refresh, remote eugene1 not usable
Aug 21 02:08:41 eugene2.servers.bright.gdn knotd[20151]: error: [bright.gdn.] refresh, failed (no usable master)

Hi Eugene, thanks much for provided detailed information!

It seems to me I am on the track of this issue, however I need a little more information from you to confirm this.

Could you please tell me what is the status (existence, permissions, contents (roughly)) of the /var/lib/knot/zones/keys directory on your slave server?

The KASP database is not only used for storing DNSSEC signing keys (which you don't use on your slave), but also some more zone metadata. It seems Knot has issues if this directory exists, but it's not accessible for knot user (or whichever user knotd is running under).

I guess deleting this directory will solve the problem for you. Anyway, more provided info will help us fixing the bug in Knot ;)

Thank you!

knotd is forcely runned as knot:knot from the beginning.

# systemctl cat knot                                                    
# /etc/systemd/system/knot.service                                                
# /usr/lib/systemd/system/knot.service
[Unit]
Description=Knot high-performance DNS Server
After=network.target

[Service]
ExecStart=/usr/sbin/knotd
ExecReload=/usr/sbin/knotc reload
ExecStop=/usr/sbin/knotc stop
PrivateTmp=true
User=knot
Group=knot
RuntimeDirectory=knot
RuntimeDirectoryMode=750
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Knot has read only permissions on keys directory.

# ls /var/lib/knot/zones/keys -ld                                       
drwxr-x--- 1 root root 40 июл  9 19:32 /var/lib/knot/zones/keys

# ls /var/lib/knot/zones/keys -l                                        
итого 48                                                                          
-rw-r----- 1 root root 45056 июл  9 19:32 data.mdb                                
drwxr-x--- 1 root root   176 июл  9 19:32 keys                                    
-rw-r----- 1 root root  8192 июл  9 19:32 lock.mdb

@eugene-bright what is your Linux distribution? The service file is not up-to-date for version 2.7.x.

I'm sorry, I overlooked that!

Gentoo Linux. This is a half-custom service file.

Someone should choose a path of experimentation and find such interesting configurations.
Just a moment, I'm checking behaviour with keys directory removed.

I would agree with Libor, you should fix the permissions or ownership (root->knot) of the files. Or purge the storage and let the server to create the files with proper attributes...

I've deleted keys directory.
All looks fine now. Many thanks!

But I still disappointed with misleading logging diagnostic. I know I can find the root course with strace, but found another way to workaround the case by downgrading slave version.
This mix of strange behaviours has blown my brain.

Ok, we will try to improve logging for this type of misconfiguration.

Could knot check all mandatory resources on start up and refuse to start without them?
For me the turndown is much better then possible malfunction in runtime.

I'm sorry as I can't provide patches right now. One day I will come to help with the code, I hope so.

I don't think that it's that easy. But we will see what is possible...

OK Eugene, many thanks for helping us find the cause, we will sure work on this issue. I'm also a little skeptical about "all mandatory resources".

Yeah. It's really hard to introduce 'resource' concept at the late stages.
The more time pass the more often I think about encapsulation of the program building blocks and separation of the machinery that configure and then combine them to the working application. It's the special case of Inversion of Control called Dependency Injection.
The Dependency is a perfect resource example. The building machinery is able to control all dependencies availability at the point of building of running application instance (by preparing file handlers as sub-dependencies for example).
It's hard to build such a system in pain C as it's not enforce DI concept.

added enhancement knot labels

mentioned in merge request !936 (merged)