Sphinx: Replace Sysrepo with YAML configuration file

parent 9e3c8571
Pipeline #69520 passed with stage
in 2 minutes and 24 seconds
......@@ -16,22 +16,23 @@ Configuration thread
====================
This is the master thread of DNS Probe. The configuration thread
loads the configuration from the Sysrepo datastore, initializes the network
loads the configuration from YAML file, initializes the network
ports for packet capture, and spawns worker threads and the export
thread. It also locks those threads to CPU's logical cores. For each
spawned thread, the configuration thread creates a two-way
communication link for sending configuration changes to the threads and
for listening to messages from the threads.
After the initial configuration is done, the thread polls the Sysrepo datastore for configuration changes and messages on the communication links.
After the initial configuration is done, the thread polls the remote
management API for configuration changes and messages on the communication links.
If a worker or export thread encounters an error, it sends a message
through the communication link to the configuration thread and the
configuration thread is then responsible for shutting down the rest of
the probe. If there's a change of probe's configuration in the Sysrepo
datastore, the configuration thread is alerted and it then distributes
the probe. If there's a change of probe's configuration via remote management
API, the configuration thread is alerted and it then distributes
the updated configuration to the rest of the probe's threads. It is also
responsible for periodic aggregation of runtime statistics from all
threads and sending them to the Sysrepo datastore. If time-based
threads and sending via the remote management API. If time-based
rotation of the output is enabled in probe's configuration, the
configuration thread takes care of the periodic timer and alerts all
threads when it's time to rotate the output.
......
This diff is collapsed.
......@@ -7,7 +7,7 @@ Storing exported data
DNS Probe supports storing the exported data either to local files or transferring them directly to a remote
location via secure network transfer using `TLS <https://tools.ietf.org/html/rfc8446>`_. This is determined
by the :ref:`location` option in Sysrepo configuration.
by the :ref:`location` option in YAML configuration file.
Local storage
-------------
......@@ -19,11 +19,11 @@ specified by :ref:`export-dir` option. The names of these files will have the fo
<prefix>yyyyMMdd.HHmmss.SSSSSS.p<proc_id>.<sufix>
The *<prefix>* is determined by :ref:`file-name-prefix` option in Sysrepo configuration. The
The *<prefix>* is determined by :ref:`file-name-prefix` option in YAML configuration file. The
*yyyyMMdd.HHmmss.SSSSSS* represents a UTC timestamp (microsecond precision) from when the output file was
first opened. *<proc_id>* is an internal identification of process (worker or export thread) which wrote
the output file. *<sufix>* is one of ``parquet``, ``cdns`` or ``cdns.gz`` based on the export format and
compression configured in Sysrepo.
compression configured in YAML file.
Export to remote location
-------------------------
......
......@@ -11,14 +11,6 @@ Glossary of Terms
System facility that is used for reading packets from a network interface card. Two backends are currently supported: `raw socket <https://man7.org/linux/man-pages/man7/packet.7.html>`_ (AF_PACKET) and `DPDK <https://www.dpdk.org/>`_.
dynamic configuration parameter
A configuration parameter that can be changed dynamically through Sysrepo and takes effect immediately. See :ref:`dynamic-conf-par`.
static configuration parameter
A configuration parameter that needs DNS Probe to be restarted in order to take effect. See :ref:`static-conf-par`.
RPC operation
Remote procedure call operation that is defined in a YANG module and activated via Sysrepo.
Remote procedure call operation that is activated via remote management API.
......@@ -77,8 +77,6 @@ Three alternative packages are available:
* ``dns-probe-dpdk`` uses the DPDK framework.
* ``dns-probe-collector`` is a collector for data exported from DNS Probe via the remote export feature.
Package installation also initializes the Sysrepo datastore with a default configuration, if no configuration is found.
Installation from source
========================
......@@ -89,6 +87,7 @@ distribution repositories:
- CMake, version at least 3.5
- Boost (C++ libraries)
- libpcap
- yaml-cpp
- OpenSSL (libssl-dev)
- DPDK (only for DPDK version)
......@@ -122,36 +121,6 @@ Apache Arrow packages can be installed on most distributions from Apache's own
and ``libparquet-dev`` packages or their equivalents in other distributions need
to be installed for successful compilation of DNS probe.
Sysrepo
-------
`Sysrepo <https://github.com/sysrepo/sysrepo>`_ provides a
configuration and management API. It uses the `libyang
<https://github.com/CESNET/libyang>`_ library that needs to be
installed first.
.. code:: shell
curl -L https://github.com/CESNET/libyang/archive/master.tar.gz > dl/libyang.tgz
mkdir build/libyang
tar -xf dl/libyang.tgz -C build/libyang --strip-components=1
mkdir -p build/libyang/build
cd build/libyang/build
cmake .. -DCMAKE_INSTALL_PREFIX="$DEP_DIR" -DCMAKE_BUILD_TYPE=Release -DGEN_LANGUAGE_BINDINGS=On -DGEN_CPP_BINDINGS=On -DGEN_PYTHON_BINDINGS=Off
make -j
make install
cd "$DEP_DIR"
curl -L https://github.com/sysrepo/sysrepo/archive/master.tar.gz > dl/sysrepo.tgz
mkdir build/sysrepo
tar -xf dl/sysrepo.tgz -C build/sysrepo --strip-components=1
mkdir -p build/sysrepo/build
cd build/sysrepo/build
cmake .. -DCMAKE_INSTALL_PREFIX="$DEP_DIR" -DCMAKE_BUILD_TYPE=Release -DGEN_LANGUAGE_BINDINGS=On -DGEN_CPP_BINDINGS=On -DGEN_PYTHON_BINDINGS=Off
make -j
make install
cd "$DEP_DIR"
C-DNS Library
-------------
......@@ -198,9 +167,3 @@ DNS Probe
cmake <GIT_REPO> -DCMAKE_INSTALL_PREFIX="$DEP_DIR" -DCMAKE_BUILD_TYPE=Release -DAF_PACKET_BACKEND=On -DDPDK_BACKEND=On -DBUILD_COLLECTOR=On
make -j
make install
Finally, YANG module containing the data model for DNS Probe and default configuration also needs to be installed:
.. code:: shell
sudo $DEP_DIR/bin/sysrepoctl -i <GIT_REPO>/data-model/cznic-dns-probe.yang
......@@ -19,7 +19,7 @@ Main features
* configurable export of data about DNS transactions in C-DNS [RFC8618]_ or `Apache Parquet <https://parquet.apache.org>`_ formats
* integrated configuration and management via `Sysrepo <https://www.sysrepo.org>`_; this also includes the possibility of using the standard NETCONF protocol [RFC6241]_
* configuration via `YAML <https://yaml.org/>`_ file; remote management API to be implemented in future versions
License
......
......@@ -16,8 +16,16 @@ For changes in software see `version descriptions <https://gitlab.nic.cz/adam/dn
- Segment
- Change description
* - **0.6**
- **1.5**
- :doc:`Architecture <Architecture>`, :doc:`Configuration <Configuration>`,
:doc:`Exported Data Schema <ExportedDataSchema>`, :doc:`Glossary <Glossary>`,
:doc:`Installation <Installation>`, :doc:`Overview <Overview>`, :doc:`Running DNS Probe <Running>`,
:doc:`Default YAML file <YAMLfile>`, :doc:`dns-probe-af manpage <manpages/dns-probe-af>`,
:doc:`dns-probe-dpdk manpage <manpages/dns-probe-dpdk>`
- Replace Sysrepo with YAML file to configure DNS Probe
* -
- **1.4**
- :doc:`YANG module <YANGmodule>`
- YANG module
- Fix default value for number of concurrent connections in tcp-table
* -
- **1.3**
......@@ -25,7 +33,7 @@ For changes in software see `version descriptions <https://gitlab.nic.cz/adam/dn
- Update pattern of exported file's names
* -
- **1.2**
- :doc:`Exported Data Schema <ExportedDataSchema>`, :doc:`YANG module <YANGmodule>`
- :doc:`Exported Data Schema <ExportedDataSchema>`, YANG module
- Add TCP RTT item to exported data schema
* -
- **1.1**
......@@ -35,15 +43,15 @@ For changes in software see `version descriptions <https://gitlab.nic.cz/adam/dn
- **1.0**
- :doc:`Architecture <Architecture>`, :doc:`Configuration <Configuration>`, :doc:`Installation <Installation>`,
:doc:`Exported Data Schema <ExportedDataSchema>`, :doc:`Data Collector <DataCollector>`,
:doc:`YANG module <YANGmodule>`, :doc:`Manual pages <manpages/dp-collector>`
YANG module, :doc:`Manual pages <manpages/dp-collector>`
- Add secure export to remote location
* -
-
- :doc:`Configuration <Configuration>`, :doc:`YANG module <YANGmodule>`
- :doc:`Configuration <Configuration>`, YANG module
- Fix description of "export-dir" item in YANG module from static to dynamic configuration
* -
-
- :doc:`Configuration <Configuration>`, :doc:`YANG module <YANGmodule>`, :doc:`Running DNS Probe <Running>`
- :doc:`Configuration <Configuration>`, YANG module, :doc:`Running DNS Probe <Running>`
- Integrate probe's command line parameters to Sysrepo configuration
* -
-
......@@ -51,15 +59,15 @@ For changes in software see `version descriptions <https://gitlab.nic.cz/adam/dn
- Update instructions for installation from packages
* -
-
- :doc:`Configuration <Configuration>`, :doc:`Installation <Installation>`, :doc:`YANG module <YANGmodule>`
- :doc:`Configuration <Configuration>`, :doc:`Installation <Installation>`, YANG module
- Add client IP anonymization
* -
-
- :doc:`Configuration <Configuration>`, :doc:`YANG module <YANGmodule>`
- :doc:`Configuration <Configuration>`, YANG module
- Add IP filtering to YANG module
* - **0.5**
- **1.1**
- :doc:`index <index>`, :doc:`Installation <Installation>`, :doc:`YANG module <YANGmodule>`,
- :doc:`index <index>`, :doc:`Installation <Installation>`, YANG module,
:doc:`Record Of Changes <RecordOfChanges>`
- Update GitLab URLs
* -
......
......@@ -2,12 +2,6 @@
References
**********
.. [RFC8040] Bierman, A.; Bjorklund, M.; Watsen, K. *RESTCONF Protocol*. `RFC 8040 <https://tools.ietf.org/html/rfc8040>`_, IETF, 2017. 137 p. ISSN 2070-1721.
.. [RFC7950] Bjorklund, M. (ed.). *The YANG 1.1 Data Modeling Language*. `RFC 7950 <https://tools.ietf.org/html/rfc7950>`_, IETF, August 2016. 217 p. ISSN 2070-1721.
.. [RFC8618] Dickinson J. et al. *Compacted-DNS (C-DNS): A Format for DNS Packet Capture*. `RFC 8618 <https://tools.ietf.org/html/rfc8618>`_, IETF 2019. 79 p. ISSN 2070-1721.
.. [RFC6241] Enns R. (ed.) et al. *Network Configuration Protocol (NETCONF)*. `RFC 6241 <https://tools.ietf.org/html/rfc6241>`_, IETF, 2011. 113 p. ISSN 2070-1721.
.. [RFC1035] Mockapetris P. *Domain Names - Implementation and Specification*. `RFC 1035 <https://tools.ietf.org/html/rfc1035>`_, IETF, 1987. 55 p.
......@@ -8,36 +8,45 @@ Running as systemd service
==========================
Installation packages include a *systemd* unit file
``dns-probe-<BACKEND>.service``, where ``<BACKEND>`` is either ``af``
``dns-probe-<BACKEND>@.service``, where ``<BACKEND>`` is either ``af``
or ``dpdk`` depending on the :term:`backend` that the package installs.
The *systemd* service can be run like this:
.. code:: shell
sudo systemctl start dns-probe-<BACKEND>.service
sudo systemctl start dns-probe-<BACKEND>@<instance-id>.service
Other ``systemctl`` subcommands can be used to stop, enable or restart the service.
The ``instance-id`` service parameter specifies what configuration to load from YAML configuration file at
``/etc/dns-probe-<BACKEND>/dns-probe.yml``.
With default configuration DNS Probe doesn't have any network interface to process traffic from configured.
User should therefore modify Sysrepo configuration before running the *systemd* service
for the first time. For example like this:
User should therefore modify the YAML configuration file before running the *systemd* service for the first
time. For example like this:
.. code-block:: shell
.. code-block:: yaml
sysrepocfg -E vim -m cznic-dns-probe
inst1:
interface-list:
- 'eth0'
- 'eth1'
log-file: '/var/log/dns-probe-af@inst1.log'
.. code-block:: xml
inst2:
interface-list:
- 'eth2'
log-file: '/var/log/dns-probe-af@inst2.log'
<dns-probe xmlns="https://www.nic.cz/ns/yang/dns-probe">
<interface-list>eth0</interface-list>
<interface-list>eth1</interface-list>
<log-file>/var/log/dns-probe-af.log</log-file>
</dns-probe>
This configuration will have ``inst1`` instance of DNS Probe process traffic from ``eth0`` and ``eth1``
interfaces and write its logs to `/var/log/dns-probe-af@inst1.log` file. ``inst2`` instance of DNS Probe
will process traffic from `eth2` and write its log to `/var/log/dns-probe-af@inst2.log` file. After this
modification is done the *systemd* service can be started as shown above with either ``inst1`` or ``inst2``
as ``instance-id`` service parameter. If desired both instances can be run at the same time via two
separate *systemd* services.
This configuration will have DNS Probe process traffic from `eth0` and `eth1` interfaces and
write its logs to `/var/log/dns-probe-af.log` file. After this modification is done
the *systemd* service can be started as usual.
For more information about configuring the YAML file see :doc:`Configuration <Configuration>` and
:doc:`Default YAML file<YAMLfile>`.
Running from command line
=========================
......@@ -58,7 +67,7 @@ For each :term:`backend`, one binary program and one shell script is installed.
The binary programs accept several command-line options described in their :ref:`manual pages <manpages>`.
The wrapper shell scripts accept the same options as the corresponding backend binary, and start the binary with these options. If the running binary program receives the :ref:`restart <rpc-restart>` operation through Sysrepo, it exits with return code 1. The wrapper script then starts the same binary again.
The wrapper shell scripts accept the same options as the corresponding backend binary, and start the binary with these options. If the running binary program receives the ``restart`` operation through remote management API, it exits with return code 1. The wrapper script then starts the same binary again.
For other codes returned by the binary, the wrapper script just exits and returns the same code.
......
.. _yaml-file:
*****************
Default YAML file
*****************
This section contains the complete default YAML configuration file that is used for DNS Probe.
It is also included in the project repository (`data-model/dns-probe.yml <https://gitlab.nic.cz/adam/dns-probe/blob/master/data-model/dns-probe.yml>`_) and packages.
.. code-block:: yaml
# Last revision: 2020-09-22
#
# Default instance configuration.
# This configuration is always loaded before other configuration specified by given instance's ID.
# DNS Probe contains default configuration values within itself so this file can be left empty
# if desired.
default:
# List of network interfaces to process traffic from in addition to interfaces passed
# with '-i' command line parameter.
interface-list: []
# List of PCAPs to process in addition to PCAPs passed with '-p' command line parameter.
pcap-list: []
# Indicates RAW PCAPs as input in 'pcap-list' or from command line with '-p' parameter.
# Might get overriden by '-r' command line parameter.
# MUST be set to 'false' if 'interface-list' or '-i' command line parameter are used.
raw-pcap: false
# Path (including file's name) to log file for storing logs. Might get overriden by '-l'
# command line parameter.
# By default logs are written to stdout.
log-file: ''
# This parameter is used for selecting CPU cores on which the application will be running.
coremask: 0x7
# List of allowed IPv4 addreses to process traffic from.
# By default all IPv4 addresses are allowed.
ipv4-allowlist: []
# List of IPv4 addresses from which to NOT process traffic.
# By default all IPv4 addresses are allowed.
ipv4-denylist: []
# List of allowed IPv6 addresses to process traffic from.
# By default all IPv6 addresses are allowed.
ipv6-allowlist: []
# List of IPv6 addresses from which to NOT process traffic.
# By default all IPv6 addresses are allowed.
ipv6-denylist: []
# List of ports used for identifying DNS traffic.
dns-ports:
- 53
# [SECTION] Items for configuration of exported data
export:
# Location for the storage of exported DNS records.
# Valid values are 'local' and 'remote'.
location: 'local'
# Directory for exported data.
export-dir: '.'
# IP address for remote export of DNS records.
remote-ip-address: '127.0.0.1'
# Transport protocol port number for remote export of DNS records.
remote-port: 6378
# Path (including file's name) to the CA certificate against which the remote server's
# certificate will be authenticated during TLS handshake.
# By default server's certificate will be authenticated against OpenSSL's default directory
# with CA certificates.
remote-ca-cert: ''
# Format of exported data.
# Valid values are 'parquet' and 'cdns'.
export-format: 'parquet'
# This sequence indicates which fields from the C-DNS standard schema are included in exported data.
# By default all fields available in DNS Probe are enabled as shown below.
cdns-fields:
- 'transaction_id'
- 'time_offset'
- 'query_name'
- 'client_hoplimit'
- 'qr_transport_flags'
- 'client_address'
- 'client_port'
- 'server_address'
- 'server_port'
- 'query_size'
- 'qr_dns_flags'
- 'query_ancount'
- 'query_arcount'
- 'query_nscount'
- 'query_qdcount'
- 'query_opcode'
- 'response_rcode'
- 'query_classtype'
- 'query_edns_version'
- 'query_edns_udp_size'
- 'query_opt_rdata'
- 'response_additional_sections'
- 'response_size'
- 'response_delay' # TCP RTT
# Maximum number of DNS records in one exported C-DNS block.
cdns-records-per-block: 10000
# Maximum number of C-DNS blocks in one exported C-DNS file.
cdns-blocks-per-file: 0
# Maximum number of Parquet records per file.
parquet-records-per-file: 5000000
# Common prefix of exported files' names.
file-name-prefix: 'dns_'
# Time interval after which the current export file is rotated.
timeout: 0
# Size limit for the export file. If the limit is exceeded, the export file is rotated.
# The value of 0 (default) means no size-based rotation.
file-size-limit: 0
# if this flag is true, the exported Parquet or C-DNS files will be compressed using GZIP.
# C-DNS willl be compressed explicitly with .gz sufix; Parquet files will be compressed
# internally due to the nature of the format.
file-compression: true
# Selection of packets to be stored in PCAP files, in addition to normal Parquet or C-DNS export.
# It's recommended to use this option only for testing purposes.
# Valid values are 'all', 'invalid', 'disabled'.
pcap-export: 'disabled'
# [SECTION] Configuration of client IP anonymization in exported data (Parquet or C-DNS).
# The optional PCAP export does NOT get anonymized!!!
ip-anonymization:
# If this flag is true, client IP addresses in exported data will be anonymized using
# Crypto-PAn prefix-preserving algorithm.
anonymize-ip: false
# Encryption algorithm to be used during anonymization of client IP addresses if enabled.
# Valid values are 'aes', 'blowfish', 'md5', 'sha1'.
encryption: 'aes'
# Path (including file's name) to the file with encryption key that is to be used for client
# IP anonymization if enabled. If the file doesn't exist, it is generated by the probe.
# The key needs to be compatible with the encryption algorithm set in the 'encryption' option
# above. User should generate the key using 'scramble-ips' tool installed by the cryptopANT
# dependency like this:
#
# scramble_ips --newkey --type=<encryption> <key-file>
key-path: 'key.cryptopant'
# [SECTION] Configuration of transaction table parameters.
transaction-table:
# Maximum number of entries in the transaction table.
# MUST be a power of 2.
max-transactions: 1048576
# Time interval after which a query record is removed from the transaction database if no
# response is observed.
# Value is in milliseconds.
query-timeout: 1000
# If this flag is true, DNS QNAME (if present) is used as a secondary key for matching
# requests with responses.
match-qname: false
# [SECTION] Configuration of TCP processing
tcp-table:
# Maximum number of concurrent TCP connections.
# MUST be a power of 2.
concurrent-connections: 131072
# Time interval after which a TCP connection is removed from the connection database
# if no data is received through that connection.
# Value is in milliseconds.
timeout: 60000
# Configuration for specific instances of DNS Probe (set by '-n' command line parameter).
# Only changes to default configuration need to be specified here.
#
# test1:
# interface-list:
# - 'lo'
#
# test2:
# interface-list:
# - 'enp0'
.. _yang-module:
***********
YANG module
***********
This section contains the complete YANG module *cznic-dns-probe* that is used for DNS Probe. It is also included in the project repository (`data-model/cznic-dns-probe.yang <https://gitlab.nic.cz/adam/dns-probe/blob/master/data-model/cznic-dns-probe.yang>`_) and packages.
::
module cznic-dns-probe {
yang-version 1.1;
namespace "https://www.nic.cz/ns/yang/dns-probe";
prefix dp;
import ietf-yang-types {
prefix yang;
}
import ietf-inet-types {
prefix inet;
}
organization
"CZ.NIC, z. s. p. o.";
contact
"Editor: Ladislav Lhotka
<mailto:lhotka@nic.cz>
Editor: Jan Dražil
<mailto:idrazil@fit.vutbr.cz>
Editor: Pavel Doležal
<mailto:pavel.dolezal@nic.cz>";
description
"This YANG module defines the data model for the DNS probe.
DNS Probe is a software tool that extracts DNS queries and
responses from network traffic (both UDP and TCP) and exports
records about DNS transactions in C-DNS or Apache Parquet
format.";
revision 2020-09-16 {
description
"Fix default value for number of concurrent connections in tcp-table";
}
revision 2020-09-04 {
description
"Add 'response_delay' bit to exported C-DNS fields";
}
revision 2020-08-24 {
description
"Add secure export to remote location";
}
revision 2020-08-17 {
description
"Add probe's command line parameters as Sysrepo configuration items.";
}
revision 2020-07-15 {
description
"Add IP anonymization";
}
revision 2020-07-09 {
description
"Add IP and port filtering";
}
revision 2020-06-09 {
description
"Initial revision.";
}
/* Data nodes */
container dns-probe {
description
"Configuration of DNS Probe.";
leaf-list interface-list {
type string;
description
"List of network interfaces to process traffic from in addition to
interfaces passed with '-i' command line parameter.
This is a static configuration parameter that is applied
only upon restarting the probe.";
}
leaf-list pcap-list {
type string;
description
"List of PCAPs to process in addition to PCAPs passed with
'-p' command line parameter.
This is a static configuration parameter that is applied
only upon restarting the probe.";
}
leaf raw-pcap {
type boolean;
default "false";
description
"Indicates RAW PCAPs as input in 'pcap-list' or from command line
with '-p' parameter. Might get overriden by '-r' command line
parameter.
MUST be set to 'false' if 'interface-list' or '-i' command line
parameter are used.
This is a static configuration parameter that is applied
only upon restarting the probe.";
}
leaf log-file {
type string;
description
"Path (including filename) to log file for storing logs.
Might get overriden by '-l' command line parameter.
By default logs are written to stdout.
This is a static configuration parameter that is applied
only upon restarting the probe.";
}
leaf coremask {
type uint64 {
range "7..max";
}
default "0x7";
description
"This parameter is used for selecting CPU cores where the
application will be running.
This is a static configuration parameter that is applied
only upon restarting the probe.";
}
leaf-list ipv4-allowlist {
type inet:ipv4-address-no-zone;
description
"List of allowed IPv4 addresses to process traffic from.
By default all IPv4 addresses are allowed.";
}
leaf-list ipv4-denylist {
type inet:ipv4-address-no-zone;
description
"List of IPv4 addresses from which to NOT process traffic.
By default all IPv4 addresses are allowed.";
}
leaf-list ipv6-allowlist {
type inet:ipv6-address-no-zone;
description
"List of allowed IPv6 addresses to process traffic from.
By default all IPv6 addresses are allowed.";
}