doc: add statistics facility

doc/introduction.rst

@@ -48,6 +48,7 @@ Server features:
 * Automatic DNSSEC signing
 * PKCS #11 interface
 * Forward and reverse records synthesis
+* DNS request traffic statistics
 
 For more info and downloads see `www.knot-dns.cz <https://www.knot-dns.cz>`_.
 

doc/man/knot.conf.5in

@@ -440,6 +440,39 @@ A UNIX socket path where the server listens for control commands.
 Maximum time the control socket operations can take. Set 0 for infinity.
 .sp
 \fIDefault:\fP 5
+.SH STATISTICS SECTION
+.sp
+Periodic server statistics dumping.
+.INDENT 0.0
+.INDENT 3.5
+.sp
+.nf
+.ft C
+statistics:
+    timer: TIME
+    file: STR
+    append: BOOL
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
+.SS timer
+.sp
+A period after which all available statistics metrics will by written to the
+\fI\%file\fP\&.
+.sp
+\fIDefault:\fP not set
+.SS file
+.sp
+A file path of statistics output in the YAML format.
+.sp
+\fIDefault:\fP \fI\%rundir\fP/stats.yaml
+.SS append
+.sp
+If enabled, the output will be appended to the \fI\%file\fP
+instead of file replacement.
+.sp
+\fIDefault:\fP off
 .SH KEYSTORE SECTION
 .sp
 DNSSEC keystore configuration.
@@ -1233,6 +1266,252 @@ A module identifier.
 A path to the directory where the database is stored.
 .sp
 \fIRequired\fP
+.SH MODULE STATS
+.sp
+The module provides incoming query processing statistics.
+.sp
+\fBNOTE:\fP
+.INDENT 0.0
+.INDENT 3.5
+Leading 16\-bit message size over TCP is not considered.
+.UNINDENT
+.UNINDENT
+.INDENT 0.0
+.INDENT 3.5
+.sp
+.nf
+.ft C
+mod\-stats:
+  \- id: STR
+    request\-protocol: BOOL
+    server\-operation: BOOL
+    request\-bytes: BOOL
+    response\-bytes: BOOL
+    edns\-presence: BOOL
+    flag\-presence: BOOL
+    response\-code: BOOL
+    reply\-nodata: BOOL
+    query\-type: BOOL
+    query\-size: BOOL
+    reply\-size: BOOL
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
+.SS id
+.sp
+A module identifier.
+.SS request\-protocol
+.sp
+If enabled, all incoming requests are counted by the network protocol:
+.INDENT 0.0
+.IP \(bu 2
+udp4 \- UDP over IPv4
+.IP \(bu 2
+tcp4 \- TCP over IPv4
+.IP \(bu 2
+udp6 \- UDP over IPv6
+.IP \(bu 2
+tcp6 \- TCP over IPv6
+.UNINDENT
+.sp
+\fIDefault:\fP on
+.SS server\-operation
+.sp
+If enabled, all incoming requests are counted by the server operation. The
+server operation is based on message header OpCode and message query (meta) type:
+.INDENT 0.0
+.IP \(bu 2
+query \- Normal query operation
+.IP \(bu 2
+update \- Dynamic update operation
+.IP \(bu 2
+notify \- NOTIFY request operation
+.IP \(bu 2
+axfr \- Full zone transfer operation
+.IP \(bu 2
+ixfr \- Incremental zone transfer operation
+.IP \(bu 2
+invalid \- Invalid server operation
+.UNINDENT
+.sp
+\fIDefault:\fP on
+.SS request\-bytes
+.sp
+If enabled, all incoming request bytes are counted by the server operation:
+.INDENT 0.0
+.IP \(bu 2
+query \- Normal query bytes
+.IP \(bu 2
+update \- Dynamic update bytes
+.IP \(bu 2
+other \- Other request bytes
+.UNINDENT
+.sp
+\fIDefault:\fP on
+.SS response\-bytes
+.sp
+If enabled, outgoing response bytes are counted by the server operation:
+.INDENT 0.0
+.IP \(bu 2
+reply \- Normal response bytes
+.IP \(bu 2
+transfer \- Zone transfer bytes
+.IP \(bu 2
+other \- Other response bytes
+.UNINDENT
+.sp
+\fBWARNING:\fP
+.INDENT 0.0
+.INDENT 3.5
+Dynamic update response bytes are not counted by this module.
+.UNINDENT
+.UNINDENT
+.sp
+\fIDefault:\fP on
+.SS edns\-presence
+.sp
+If enabled, EDNS pseudo section presence is counted by the message direction:
+.INDENT 0.0
+.IP \(bu 2
+request \- EDNS present in request
+.IP \(bu 2
+response \- EDNS present in response
+.UNINDENT
+.sp
+\fIDefault:\fP off
+.SS flag\-presence
+.sp
+If enabled, some message header flags are counted:
+.INDENT 0.0
+.IP \(bu 2
+TC \- Truncated Answer in response
+.IP \(bu 2
+DO \- DNSSEC OK in request
+.UNINDENT
+.sp
+\fIDefault:\fP off
+.SS response\-code
+.sp
+If enabled, outgoing response code is counted:
+.INDENT 0.0
+.IP \(bu 2
+NOERROR
+.IP \(bu 2
+\&...
+.IP \(bu 2
+NOTZONE
+.IP \(bu 2
+BADVERS
+.IP \(bu 2
+\&...
+.IP \(bu 2
+BADCOOKIE
+.IP \(bu 2
+other \- All other codes
+.UNINDENT
+.sp
+\fBNOTE:\fP
+.INDENT 0.0
+.INDENT 3.5
+In the case of multi\-message zone transfer response, just one counter is
+incremented.
+.UNINDENT
+.UNINDENT
+.sp
+\fBWARNING:\fP
+.INDENT 0.0
+.INDENT 3.5
+Dynamic update response code is not counted by this module.
+.UNINDENT
+.UNINDENT
+.sp
+\fIDefault:\fP on
+.SS reply\-nodata
+.sp
+If enabled, NODATA pseudo RCODE (see RFC 2308, Section 2.2) is counted by the
+query type:
+.INDENT 0.0
+.IP \(bu 2
+A
+.IP \(bu 2
+AAAA
+.IP \(bu 2
+other \- All other types
+.UNINDENT
+.sp
+\fIDefault:\fP off
+.SS query\-type
+.sp
+If enabled, normal query type is counted:
+.INDENT 0.0
+.IP \(bu 2
+A (TYPE1)
+.IP \(bu 2
+\&...
+.IP \(bu 2
+TYPE65
+.IP \(bu 2
+SPF (TYPE99)
+.IP \(bu 2
+\&...
+.IP \(bu 2
+TYPE110
+.IP \(bu 2
+ANY (TYPE255)
+.IP \(bu 2
+\&...
+.IP \(bu 2
+TYPE260
+.IP \(bu 2
+other \- All other types
+.UNINDENT
+.sp
+\fBNOTE:\fP
+.INDENT 0.0
+.INDENT 3.5
+Not all assigned meta types (IXFR, AXFR,...) have their own counters,
+because such types are not processed as normal query.
+.UNINDENT
+.UNINDENT
+.sp
+\fIDefault:\fP off
+.SS query\-size
+.sp
+If enabled, normal query message size distribution is counted by the size range
+in bytes:
+.INDENT 0.0
+.IP \(bu 2
+0\-15
+.IP \(bu 2
+16\-31
+.IP \(bu 2
+\&...
+.IP \(bu 2
+272\-287
+.IP \(bu 2
+288\-65535
+.UNINDENT
+.sp
+\fIDefault:\fP off
+.SS reply\-size
+.sp
+If enabled, normal reply message size distribution is counted by the size range
+in bytes:
+.INDENT 0.0
+.IP \(bu 2
+0\-15
+.IP \(bu 2
+16\-31
+.IP \(bu 2
+\&...
+.IP \(bu 2
+4080\-4095
+.IP \(bu 2
+4096\-65535
+.UNINDENT
+.sp
+\fIDefault:\fP off
 .SH AUTHOR
 CZ.NIC Labs <http://www.knot-dns.cz>
 .SH COPYRIGHT

doc/man/knotc.8in

@@ -78,6 +78,10 @@ Stop the server if running.
 Reload the server configuration and modified zone files. All open zone
 transactions will be aborted!
 .TP
+\fBstats\fP [\fImodule\fP[\fB\&.\fP\fIcounter\fP]]
+Show global statistics counter(s). To print also counters with value 0, use
+force option.
+.TP
 \fBzone\-check\fP [\fIzone\fP\&...]
 Test if the server can load the zone. Semantic checks are executed if enabled
 in the configuration. (*)
@@ -137,6 +141,10 @@ Remove zone data within the transaction.
 \fBzone\-purge\fP \fIzone\fP\&...
 Purge zone data, file, journal, and timers.
 .TP
+\fBzone\-stats\fP \fIzone\fP [\fImodule\fP[\fB\&.\fP\fIcounter\fP]]
+Show zone statistics counter(s). To print also counters with value 0, use
+force option.
+.TP
 \fBconf\-init\fP
 Initialize the configuration database. (*)
 .TP

doc/man_knotc.rst

@@ -55,6 +55,10 @@ Actions
   Reload the server configuration and modified zone files. All open zone
   transactions will be aborted!
 
+**stats** [*module*\ [\ **.**\ *counter*\ ]]
+  Show global statistics counter(s). To print also counters with value 0, use
+  force option.
+
 **zone-check** [*zone*...]
   Test if the server can load the zone. Semantic checks are executed if enabled
   in the configuration. (*)
@@ -114,6 +118,10 @@ Actions
 **zone-purge** *zone*...
   Purge zone data, file, journal, and timers.
 
+**zone-stats** *zone* [*module*\ [\ **.**\ *counter*\ ]]
+  Show zone statistics counter(s). To print also counters with value 0, use
+  force option.
+
 **conf-init**
   Initialize the configuration database. (*)
 

doc/modules.rst

@@ -515,3 +515,33 @@ to the configuration file::
 
 .. NOTE::
    This module is not configurable.
+
+``stats`` — query statistics
+----------------------------
+
+The module extends server statistics with incoming DNS request and corresponding
+response counters, such as used network protocol, total number of responded bytes,
+etc (see :ref:`mod-stats<mod-stats>` for full list of supported counters).
+This module should be configured as the last module.
+
+Common statistics with default module configuration::
+
+    template:
+      - id: default
+        global-module: mod-stats
+
+Per zone statistics with explicit module configuration::
+
+    mod-stats:
+      - id: custom
+        edns-presence: on
+        query-type: on
+
+    template:
+      - id: default
+        module: mod-stats/custom
+
+.. NOTE::
+   Server initiated communication (outgoing NOTIFY, incoming \*XFR,...) is not
+   counted by this module.
+

doc/operation.rst

@@ -266,3 +266,24 @@ no differences will be created.
 If you want to refresh the slave zones, you can do this with::
 
     $ knotc zone-refresh
+
+.. _Statistics:
+
+Statistics
+==========
+
+The server provides some general statistics and optional query module statistics
+(see :ref:`mod-stats<mod-stats>`).
+
+Server statistics or global module statistics can be shown by::
+
+    $ knotc stats
+    $ knotc stats server             # Show all server counters
+    $ knotc stats mod-stats          # Show all mod-stats counters
+    $ knotc stats server.zone-count  # Show specific server counter
+
+Per zone statistics can be shown by::
+
+    $ knotc zone-stats example.com mod-stats
+
+To show all supported counters even with 0 value use the force option.

doc/reference.rst

@@ -506,6 +506,49 @@ Maximum time the control socket operations can take. Set 0 for infinity.
 
 *Default:* 5
 
+.. _statistics_section:
+
+Statistics section
+==================
+
+Periodic server statistics dumping.
+
+::
+
+  statistics:
+      timer: TIME
+      file: STR
+      append: BOOL
+
+.. _statistics_timer:
+
+timer
+-----
+
+A period after which all available statistics metrics will by written to the
+:ref:`file<statistics_file>`.
+
+*Default:* not set
+
+.. _statistics_file:
+
+file
+----
+
+A file path of statistics output in the YAML format.
+
+*Default:* :ref:`rundir<server_rundir>`/stats.yaml
+
+.. _statistics_append:
+
+append
+------
+
+If enabled, the output will be appended to the :ref:`file<statistics_file>`
+instead of file replacement.
+
+*Default:* off
+
 .. _Keystore section:
 
 Keystore section
@@ -1464,3 +1507,214 @@ dbdir
 A path to the directory where the database is stored.
 
 *Required*
+
+.. _mod-stats:
+
+Module stats
+============
+
+The module provides incoming query processing statistics.
+
+.. NOTE::
+   Leading 16-bit message size over TCP is not considered.
+
+::
+
+ mod-stats:
+   - id: STR
+     request-protocol: BOOL
+     server-operation: BOOL
+     request-bytes: BOOL
+     response-bytes: BOOL
+     edns-presence: BOOL
+     flag-presence: BOOL
+     response-code: BOOL
+     reply-nodata: BOOL
+     query-type: BOOL
+     query-size: BOOL
+     reply-size: BOOL
+
+.. _mod-stats_id:
+
+id
+--
+
+A module identifier.
+
+.. _mod-stats_request-protocol:
+
+request-protocol
+----------------
+
+If enabled, all incoming requests are counted by the network protocol:
+
+* udp4 - UDP over IPv4
+* tcp4 - TCP over IPv4
+* udp6 - UDP over IPv6
+* tcp6 - TCP over IPv6
+
+*Default:* on
+
+.. _mod-stats_server-operation:
+
+server-operation
+----------------
+
+If enabled, all incoming requests are counted by the server operation. The
+server operation is based on message header OpCode and message query (meta) type:
+
+* query - Normal query operation
+* update - Dynamic update operation
+* notify - NOTIFY request operation
+* axfr - Full zone transfer operation
+* ixfr - Incremental zone transfer operation
+* invalid - Invalid server operation
+
+*Default:* on
+
+.. _mod-stats_request-bytes:
+
+request-bytes
+-------------
+
+If enabled, all incoming request bytes are counted by the server operation:
+
+* query - Normal query bytes
+* update - Dynamic update bytes
+* other - Other request bytes
+
+*Default:* on
+
+.. _mod-stats_response-bytes:
+
+response-bytes
+--------------
+
+If enabled, outgoing response bytes are counted by the server operation:
+
+* reply - Normal response bytes
+* transfer - Zone transfer bytes
+* other - Other response bytes
+
+.. WARNING::
+   Dynamic update response bytes are not counted by this module.
+
+*Default:* on
+
+.. _mod-stats_edns-presence:
+
+edns-presence
+-------------
+
+If enabled, EDNS pseudo section presence is counted by the message direction:
+
+* request - EDNS present in request
+* response - EDNS present in response
+
+*Default:* off
+
+.. _mod-stats_flag-presence:
+
+flag-presence
+-------------
+
+If enabled, some message header flags are counted:
+
+* TC - Truncated Answer in response
+* DO - DNSSEC OK in request
+
+*Default:* off
+
+.. _mod-stats_response-code:
+
+response-code
+-------------
+
+If enabled, outgoing response code is counted:
+
+* NOERROR
+* ...
+* NOTZONE
+* BADVERS
+* ...
+* BADCOOKIE
+* other - All other codes
+
+.. NOTE::
+   In the case of multi-message zone transfer response, just one counter is
+   incremented.
+
+.. WARNING::
+   Dynamic update response code is not counted by this module.
+
+*Default:* on
+
+.. _mod-stats_reply-nodata:
+
+reply-nodata
+------------
+
+If enabled, NODATA pseudo RCODE (see RFC 2308, Section 2.2) is counted by the
+query type:
+
+* A
+* AAAA
+* other - All other types
+
+*Default:* off
+
+.. _mod-stats_query-type:
+
+query-type
+----------
+
+If enabled, normal query type is counted:
+
+* A (TYPE1)
+* ...
+* TYPE65
+* SPF (TYPE99)
+* ...
+* TYPE110
+* ANY (TYPE255)
+* ...
+* TYPE260
+* other - All other types
+
+.. NOTE::
+   Not all assigned meta types (IXFR, AXFR,...) have their own counters,
+   because such types are not processed as normal query.
+
+*Default:* off
+
+.. _mod-stats_query-size:
+
+query-size
+----------
+
+If enabled, normal query message size distribution is counted by the size range
+in bytes:
+
+* 0-15
+* 16-31
+* ...
+* 272-287
+* 288-65535
+
+*Default:* off
+
+.. _mod-stats_reply-size:
+
+reply-size
+----------
+
+If enabled, normal reply message size distribution is counted by the size range
+in bytes:
+
+* 0-15
+* 16-31
+* ...
+* 4080-4095
+* 4096-65535
+
+*Default:* off