Dns Collector
A super-fast traffic collector and pre-processor for DNS queries, the first part of our DNS traffic analysis pipeline.
Released under GNU GPL v3 (see LICENSE
).
See also the public release blog post.
Contact Tomáš Gavenčiak (gavento@ucw.cz) with any questions.
Features
- Fast multithreaded processing (3 threads): up to 150 000 queries/s offline and offline (on i5 2.4 GHz, see benchmarks below).
- Matching requests to responses by (IPs, ports, transport, DNS ID), optionally also with QNAME. Matches the proposed draft.
- Reading capture files and live traces that libtrace reads, including kernel ringbuffer. Configurable packet filter.
- Pcap dumps of invalid packets with rate-limiting, compression and output file rotation.
- Configurable CBOR and CSV output (targeted at Impala/hadoop import, NOT RFC 4180 compatible) and optional binary CBOR output. Modular output allows easy implementation of other output formats.
- Automatic output file rotation and compression or other post-processing (any shell pipe command).
- Easy configuration via config files, periodic status and statistics logging, graceful shutdown on SIGINT.
- Well documented and clean code, C99, GNU GPL v3.
Extracted DNS features
- Timing of request and response (microsecond accuracy)
- Network information (addresses, ports, protocols, TTL, packet and payload sizes)
- DNS header information (query information, id, flags, rcode, opcode, RR counts)
- EDNS information (version, extended flags and rcode, DAU, DHU, N3U, NSID)
Not implemented
- More EDNS features: Client subnet, other options (needs manual EDNS traversal).
- TCP flow recoinstruction for reused streams (currently only single-query TCP connections are supported).
- IP(4/6) fragmented packet reconstuction, DNS via ICMP. However, IP fragmentation and DNS via ICMP are uncommon.
- Possible improvement: Delay the responses in the workflow (cca 10 us) to remove capture timing jitter (when response is mistakenly seen before request).
Building and installation
Run make
to compile the dns-collector
binary. Use USE_TCMALLOC=1 make
instead to build with tcmalloc.
Run make docs
to generate developer Doxygen documentation in docs/html
.
Linux packages are built in project GitLab CI and in OpenBuildServece repo.
Running
-
./dns-collector -i eth0 -oqueries.csv
runs a live trace with the default configuration. Ctrl+C (SIGINT) to gracefully stop. (See the config for automatical output rotation.) -
./dns-collector -o queries.csv capture1.pcap capture2.pcap ...
processes the given pcaps in the order given. -
./dns-collector -C dns-collector.conf
to use the given config file. Without any other parameters runs a live trace on the configured interface, may be combined with-i
and-o
overrides and offline pcap files as parameters.
See the distributed dns-collector.conf
for more options and their meaning.
NB: In offline mode (reading PCAPs), if no packets match the filter, no output file is produced. This is the only choice when the output filename depends on packet times, but this is not checked and applies also for fixed output filenames.
Requirements
These are Ubuntu and Debian package names, but should be similar in other distros.
- Clang or GCC build environmrnt, make.
-
libtrace3-dev
3.0.21+ (tested with 3.0.21 in Ubuntu Xenial to Artful, 3.0.18 from Ubuntu Trusty is not sufficient). -
libknot7
andlibknot-dev
2.6.x (tested with 2.6.4). Use Knot repositories for Debian and Ubuntu.
Optional
- Optionally tcmalloc (from package
libgoogle-perftools-dev
, tested with ver 2.4 and 2.5) for faster allocation and cca 20% speedup (to use, compile withUSE_TCMALLOC=1 make
)
Included
- LibUCW 6.5+ is included as a git subtree and built automatically.
- TinyCBOR 0.5+ is included as a git subtree and built automatically.
Memory footprint
On 64bit system, the basic memory usage is quite low (cca 30 MB). The only memory-intensive part is the matching window, which keeps all the packets of the past matching window in memory. Consider this when choosing the matching window.
The total usage is therefore approximately: (30 + 3 * T * Q) MB, where T is the matching window in seconds, and Q the request frequency in kq/s (thousand queries per second).
For example, 10 s matching window and 10 kqps requires 300 MB. In an extremal situation, 10 s matching window and 150 kqps would require 5 GB of memory.
NOTE: The current memory usage is 2 kB per request+response packet pair, including parsed packets data, allocation overhead and matching hash table. This could be likely reduced to some 500 B (in the optimistic case of most requests being matched) by early extraction of output-relevant data from the packets before matching (now done on output), but that would complicate the code structure and limit extensibility.
Benchmarks
Testing machine: knot-master
2x4 core Intel Xeon, 2.4 GHz. Debian 8.5. Online test packets are replayed from another machine via Intel 10G X520.
Offline processing
Cca 150 000 q/s on authoritative server data (Knot DNS server), 34% DNSSEC requests (very little other EDNS), 71% IPv6, 8% TCP.
Uses 2.5 cores without compression (good compression uses another 1 core). Varies with the selected field set.
Online processing
Running ont the same node (knot-master) as the Knot DNS server (Knot 2.3). Capturing via kernel ring buffer.
Packet statistics:
kq per sec | Knot | DnsCol |
---|---|---|
50 | OK | OK |
100 | OK | <1% lost |
150 | OK | 5%-10% lost |
... | OK | reading 150 kq/s |
300 | OK | reading 150 kq/s |
500 | <0.2% lost | reading 100-150 kq/s |
700 | 15% lost | reading 100-150 kq/s |
900 | 50% lost (see below) | reading 100-150 kq/s |
NOTE: In this setting, the knot server alone can handle up to 900 kq/s with <0.2% packet loss. However, the drop to 600 kq/s does not seem to come from dnscol CPU usage but rather the packet capture load on the kernel: the Knot speed drop is the same (to 600 kq/s) with dnscol cpulimited to just 0.5 CPU.
CBOR output
CBOR output has been introduced in 0.2 to adress some of the encoding problems of CSV: representing binary data and structured elements. CBOR format is defined in RFC 7049 and there are implementations for many common languages.
The CBOR output file is a sequence of independent CBOR items. This was preferred to a one large CBOR item per file to avoid having to load the entire file with most (non-streaming) parsers. The file structure resembles a CSV file with a header row:
The first item is an array of column names, corresponding to the present columns in order. Each subsequent item is array of attributes for one record. The meanings and types of the attributes are the same as for the CSV files below. The only exceptions are EDNS data with inner structure:
field name | format |
---|---|
req_edns_dau |
array of DAU numbers |
req_edns_dhu |
array of DHU numbers |
req_edns_n3u |
array of N3U numbers |
resp_edns_nsid |
raw NSID as a bytestring |
edns_client_subnet |
TODO |
edns_other |
TODO |
Compared to CSV, CBOR output uses cca 10% CPU (user time), is 30% smaller uncompressed and 5% smmaller gziped.
CSV output
Escaping
The output is primarily targetted for Impala import, so the CSV does NOT conform to RFC 4180. Instead of quoting, problematic characters are escaped with a backslash. Every record is one line - all newlines are escaped.
The escaped characters are: backslash (\\
), newline (\n
), the configured field separator (e.g. \,
),
all non-ASCII and most non-printable characters (\ooo
with octal notation).
"NULL" values are represented by an empty string in numeric fields and by \N
in string fields (as in Impala CSVs).
Fields
The fields prefixed by req_
always come from the request, resp_
from the response.
The other fields may generally come from either request or response, depending on
request or response presence and other context.
Some fields have a common configuration flag, for most the flag is the same as the field name.
field name | config flag name | type | comment |
---|---|---|---|
time |
. | DOUBLE | seconds of request (or response if no request) since Unix epoch, microsecond precision |
delay_us |
. | INT | request-response delay in microseconds |
req_dns_len |
. | INT | DNS paload length |
resp_dns_len |
. | INT | DNS paload length |
req_net_len |
. | INT | packet network length |
resp_net_len |
. | INT | packet network length |
client_addr |
. | STRING | e.g. "1.2.3.4" or "12:34::ef" |
client_port |
. | INT | |
server_addr |
. | STRING | e.g. "1.2.3.4" or "12:34::ef" |
server_port |
. | INT | |
net_proto |
. | INT | 17 for UDP, 6 for TCP |
net_ipv |
. | INT | 4 or 6 |
net_ttl |
. | INT | TTL or hoplimit |
req_udp_sum |
. | INT | only if present (and not 0) |
id |
. | INT | DNS ID |
qtype |
. | INT | |
qclass |
. | INT | |
opcode |
. | INT | |
rcode |
. | INT | |
resp_aa |
flags |
BOOLEAN | |
resp_tc |
flags |
BOOLEAN | |
req_rd |
flags |
BOOLEAN | |
resp_ra |
flags |
BOOLEAN | |
req_z |
flags |
BOOLEAN | |
resp_ad |
flags |
BOOLEAN | |
req_cd |
flags |
BOOLEAN | |
qname |
. | STRING | including the final dot, escaped by libknot (most non-basic-DNS characters as \ooo in octal) |
resp_ancount |
rr_counts |
INT | |
resp_arcount |
rr_counts |
INT | |
resp_nscount |
rr_counts |
INT | |
req_edns_ver |
edns |
INT | version, distinguish 0 and NULL ! |
req_edns_udp |
edns |
INT | |
req_edns_do |
edns |
BOOLEAN | |
resp_edns_rcode |
edns |
INT | extended part of rcode
|
req_edns_ping |
edns |
BOOLEAN | currently never present (ID conflict with DAU) |
req_edns_dau |
edns |
STRING | comma-separated numbers, e.g. "1,3,5" |
req_edns_dhu |
edns |
STRING | comma-separated numbers, e.g. "1,3,5" |
req_edns_n3u |
edns |
STRING | comma-separated numbers, e.g. "1,3,5" |
resp_edns_nsid |
edns |
STRING | escaped NSID |
edns_client_subnet |
edns |
STRING | TODO |
edns_other |
edns |
STRING | TODO |
Impala and Entrada import
See the wiki page on Impala and Entrada import for more details and SQL commands. Note that Impala can read gzip, lzo and bzip2 compressed CSV files transparently.
The table for all features (in the right order) is created by:
create table dnscol.csv_import (
time DOUBLE, -- convert to timestamp
delay_us INT,
req_dns_len INT,
resp_dns_len INT,
req_net_len INT,
resp_net_len INT,
client_addr STRING,
client_port INT,
server_addr STRING,
server_port INT,
net_proto INT,
net_ipv INT,
net_ttl INT,
req_udp_sum INT,
id INT,
qtype INT,
qclass INT,
opcode INT,
rcode INT,
resp_aa INT, -- convert to boolean
resp_tc INT, -- convert to boolean
req_rd INT, -- convert to boolean
resp_ra INT, -- convert to boolean
req_z INT, -- convert to boolean
resp_ad INT, -- convert to boolean
req_cd INT, -- convert to boolean
qname STRING,
resp_ancount INT,
resp_arcount INT,
resp_nscount INT,
req_edns_ver INT,
req_edns_udp INT,
req_edns_do INT, -- convert to boolean
resp_edns_rcode INT,
-- req_edns_ping INT, -- never present (conflict with DAU) (convert to boolean)
req_edns_dau STRING,
req_edns_dhu STRING,
req_edns_n3u STRING,
resp_edns_nsid STRING,
edns_client_subnet STRING,
edns_other STRING)
partitioned by (server STRING, batchname STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' ESCAPED BY '\\';
Potential improvements
- More EDNS features
- Country/ASN detection (e.g. MaxMind integration?)
- TCP SYN/ACK drops (and then better TCP integration)
Changelog
1.1 (2018-06-22)
- Better error messages and option defaults
- Option for QNAME matching
- Bugfixes, testsuite, CI tests
- Repo history cleanup, squash imported subtrees
- Add CI build and OBS repository
1.0 (2018-02-06)
- Improve multiple output types support
- Add CBOR output logging matching the CSV fields
- Add tinycbor as a git subtree
- Add simple CBOR-stream debugging utility
- Fix a flipped request/response bit bug
0.1.1 (2017-09-14)
- Fix backtrace output preventing core dump in upstream
- Updated distribution metadata and scripts, rename binaries
0.1 (2017-09-07)
- Initial release