This is essentially the opposite of limit-clients.lua.
This new script assigns one client IP address to a single output and
copies all packets for single client into the same output file.

Client assignment is pseudorandom and does not guarantee uniform
assignment. Clients typically do not generate equal load in any metric
(be it QPS or overall resources required to process client's query),
so there would be little point in complicating code for theoretical
uniformity.

Stats generated at the end provide some insight into uniformity of the
split on per-client and per-packet basis.

filter-dnsq: skip 'special' queries for *.dotnxdomain.net

Closes #25

See merge request !58

Petr Špaček authored 3 years ago and Tomas Krizek committed 3 years ago

By default, filter out queries for subdomains of dotnxdomain.net.
This is a 'special' measurement domain. Queries directed to it have
timestamps encoded in qname and replaying old queries results in
timeouts, not in a realistic traffic replay.

A new option -s can be used to keep the queries in the output if
desired.

The other domain - dashnxdomain.net - did not appear in any of my PCAPs
so for simplicity I omitted it from the filtering code.

Fixes: #25

Formerly, if malformed packets were requested in the output,
the script skipped DNS parsing because there was no point in doing so:
The malformed packets would have been included in the output anyway.

As preparation for a new feature which requires access to qname,
the script now tries to parse the packet even if user requested
inclusion of malformed packets in the output.

Even for tens of GBs of data overhead of doing this was negligible
compared to other processing, so I did not go to the trouble of
optimizing this further.

extract-clients: ability to write to stdout

See merge request !62

plot-packet-rate tweaks

See merge request !63

Formerly extra files were drawn using the same colors as files withing
the limit and it lead risk of misinterpreting the data.

Colors are combined with markers to create more combinations.
Obviously nobody can really read 40 combinations in detail because the
points will overlap, but it is sufficient to detect outliers and that's
what I'm after.

detect PCAP write errors

See merge request !60

This is useful when extract-clients is used in a pipeline like this:
mergecap | filter-dnsq | extract-clients | split-clients

output.null was removed from dnsjit, but users can use /dev/null
if needed so there is no need to complicate code with extra conditions.

plot-client-distribution: tweaks for many input files

See merge request !59

Colors are combined with hatch to create more combinations.
Obviously nobody can really read 40 combinations in detail because the
points will overlap, but it is sufficient to detect outliers and that's
what I'm after.

Formerly extra files were silently ignored.

ci: use debian 11

See merge request !57

extract-clients: reset UDP port numbers in output to 53

See merge request !56

Wireshark considers some source port numbers special and switches packet
parser to non-DNS protocol, despite destination port being 53.
Extractor now overrides both source and destination ports to 53 to make
it easier to analyze its output in Wireshark.

plot-response-rate: avoid white line on white background

See merge request !55

release v20210714

See merge request !54

http2() was part of dnssim since version 20210129 - and that is already
checked for.

plot-latency: ability to plot results as aggregated groups

See merge request !53

Each --group has user-supplied name, and line denoting average value,
and colored area denoting min/max range.

shotgun.output.dnssim

See merge request !52