Journal grows very big despite limits
We have a pair of identically configured servers, performing DNSSEC signing. Yesterday, one of them failed to re-sign some zones, with this error:
Sep 26 22:17:35 dnssigner4 knot[24932]: error: [77.in-addr.arpa.] zone event 'DNSSEC re-sign' failed (not enough space provided)
Upon investigation, I found that /var/lib/knot/journal
was about 20GB in size:
[root@dnssigner4 ~]# du -sh /var/lib/knot/journal
20G /var/lib/knot
This is the default configuration limit for the journal, so I was not surprised that there was no more space in the journal, and knot started to fail. But now, we come to the surprising issue. The other server, with an identical configuration, and the same zones defined, does not have this problem:
[root@dnssigner3 ~]# du -sh /var/lib/knot/journal
6.2G /var/lib/knot/journal
Additionally, kjournalprint
also shows that the zones are not using that much space in the journal:
[root@dnssigner3 ~]# kjournalprint /var/lib/knot/journal -dz
...
...
Occupied all zones together: 486096 KiB
[root@dnssigner4 ~]# kjournalprint /var/lib/knot/journal -dz
...
...
Occupied all zones together: 272248 KiB
So the mystery is how the journal on dnssigner4
ended up using so much disk space that it hit the limit.