ci: use black formatter instead of flake8

See merge request !68

Just automated changes, no manual intervention, no functionaly changes.

One code style to rule them all!

merge-chunks: allow disabling chunk randomization

See merge request !67

No manual changes and no behavior changes in this commit - simple code
style update from black.

Some use-cases require that the chunk merging script behaves more
predictably to better control the final PCAP. Add the option to use a
negative seed to turn off chunk randomization and just use them sorted
by file names.

plot-connections: set axis limits properly

See merge request !66

When set_xlim(), set_ylim() were called repeatedly on the same axes
after every plot, subsequent calls were ignored and only the first call
determined the axis limits.

This could result in charts where some data wouldn't be visible at all,
since it'd plotted beyond the smaller axis limit. To fix this, ensure
the axes limits are set only once -- when all the data is already
plotted.

split-clients: new tool to split large PCAP into multiple smaller ones

See merge request !61

This is essentially the opposite of limit-clients.lua.
This new script assigns one client IP address to a single output and
copies all packets for single client into the same output file.

Client assignment is pseudorandom and does not guarantee uniform
assignment. Clients typically do not generate equal load in any metric
(be it QPS or overall resources required to process client's query),
so there would be little point in complicating code for theoretical
uniformity.

Stats generated at the end provide some insight into uniformity of the
split on per-client and per-packet basis.

filter-dnsq: skip 'special' queries for *.dotnxdomain.net

Closes #25

See merge request !58

Petr Špaček authored 3 years ago and Tomas Krizek committed 3 years ago

By default, filter out queries for subdomains of dotnxdomain.net.
This is a 'special' measurement domain. Queries directed to it have
timestamps encoded in qname and replaying old queries results in
timeouts, not in a realistic traffic replay.

A new option -s can be used to keep the queries in the output if
desired.

The other domain - dashnxdomain.net - did not appear in any of my PCAPs
so for simplicity I omitted it from the filtering code.

Fixes: #25

Formerly, if malformed packets were requested in the output,
the script skipped DNS parsing because there was no point in doing so:
The malformed packets would have been included in the output anyway.

As preparation for a new feature which requires access to qname,
the script now tries to parse the packet even if user requested
inclusion of malformed packets in the output.

Even for tens of GBs of data overhead of doing this was negligible
compared to other processing, so I did not go to the trouble of
optimizing this further.

extract-clients: ability to write to stdout

See merge request !62

plot-packet-rate tweaks

See merge request !63

Formerly extra files were drawn using the same colors as files withing
the limit and it lead risk of misinterpreting the data.

Colors are combined with markers to create more combinations.
Obviously nobody can really read 40 combinations in detail because the
points will overlap, but it is sufficient to detect outliers and that's
what I'm after.

detect PCAP write errors

See merge request !60

This is useful when extract-clients is used in a pipeline like this:
mergecap | filter-dnsq | extract-clients | split-clients

output.null was removed from dnsjit, but users can use /dev/null
if needed so there is no need to complicate code with extra conditions.

plot-client-distribution: tweaks for many input files

See merge request !59

Colors are combined with hatch to create more combinations.
Obviously nobody can really read 40 combinations in detail because the
points will overlap, but it is sufficient to detect outliers and that's
what I'm after.

Formerly extra files were silently ignored.

ci: use debian 11

See merge request !57