Funnily default value 0 was not accepted when specified explicitly.
https://docs.python.org/3/library/random.html says that full int
is acceptable so let's allow it.

configs: generate additional response-rate chart with non-zero rcodes

See merge request !44

Generating charts is cheap so we generate variant with and also
without rcodes.

plot-response-rate: add ability to plot frequent rcodes

See merge request !43

--rcode now also accepts mnemonic names + extra value "other"

pylint started to complain about length of the main() function.

New option --rcodes-above-pct N allows to draw only rcodes which are more
frequent than N percent of all answers. Thus --rcodes-above-pct 0 draws
all rcodes which have non-zero number of samples.

Small values like --rcodes-above-pct 0.1 % can also filter noise like
"five REFUSED answers in the whole dataset" etc.

plot-response-rate: use distinct colors when plotting rcodes from a single JSON

See merge request !42

Commit ad584cef changed color selection
so each line from a single JSON uses the same color. That works fine
when plotting multiple data sets but makes it harder to deep-analyze a
single dataset, e.g. when plotting all rcodes from a single run.

This commit changes color selection once again:
- When plotting multiple JSON files, all lines from one file use the
  same color.
- When plotting a single JSON file, each rcode line uses different
  color. Each supported rcode has a stable color.

Each rcode line is still denoted with a symbol introduced in
ad584cef.

plot-response-rate: use the same color for rcode and summary lines

See merge request !41

Rcode lines use the same color as "summary" line.
Different rcodes are distinguished by markers derived from first letter
(s for SERVFAIL, n for NXDOMAIN ...) or rcode numbers for less common values.
NOERROR is displayed as 0 to distinguish it from n for NXDOMAIN.
Mathtext render of 0 also accidentally looks like O for Okay :-)

new option stop_after_s

See merge request !40

Ability to stop reading PCAP after specified number of seconds.
It depends on fact that extract-clients script renumbers packets
so first packet has timestamp 0.

Quirk:
Timestamps are checked only after they are already delayed by dnsjit
module. For this reason PCAP reading is in fact terminated when
time for the first packet _after_ the specified time limit passes.
In practice this would be noticeable only if delays between packets
were long.

replay: workaround for missing /32 or /128 support in Python ipaddress library

See merge request !39

Python issue #28577. Affects Ubuntu 20.04 which is often used in CI.

plot-client-distribution: fix handling of empty query count intervals

See merge request !38

Query count intervals with 0 clients in them were not handled properly.
E.g. if some clients were in interval [0-10) and [100-1000) while interval
[10-100) had no clients in it the old script has thrown an exception.

merge_chunks: utility to merge chunks on-the-fly

See merge request !37

Intended usage is to avoid generating PCAPs which are simple combination
of "base chunks".

Example:
When original PCAP is split into 100 chunks with 1 kQPS on average,
testing full range of 1k to 100k QPS formerly required generating 100
distincts PCAPs, wasting time and storage.
With this utility it is enough to generate "base chunks" and create
arbitrary multiplication on-the-fly.

Why Python?
- Lua cannot list content of directory, FFI solution would be
  unportable, and adding depedency just for that seems too much.
- BASH version of this script made me cry when I finished it.
- Python is already depedency of replay.py and it is used only to
  process mergecap arguments and python process terminates/replaces
  itself once its job is done.

Input improvements

See merge request !36

Petr Špaček authored 4 years ago and Tomas Krizek committed 4 years ago

In my tests mmap input is roughly 20 % faster than fpcap.
fpcap is used as fallback in case mmap does not work on selected file.
In practice it happens only when the input file is not "normal",
e.g. when it is a named pipe.

docs: add warning about scaling up traffic with mergecap

See merge request !35

Random seed improvements

See merge request !34

Options without short variant should use nil. It would blow up if two
options used empty string for short name.

extract-clients fixes

See merge request !32

Previously packets in all chunks had timestamp 0 and ignored original
time difference from the last packet in the previous chunk.

For steady traffic with non-negligible QPS it does not make significant
difference, but it leads to big imprecision if original traffic had
"gaps" in it.

After this change first chunk starts at time 0 and timing of all packets
is relative to timing of the very first packet in the first chunk.

Very long gaps between packets will generate empty chunks,
and these empty chunks will be deleted because they are not useful.
See discussion in MR !32.

This also changes client IP addresses by decrementing the value by one.
I think it does not matter and I consider this slightly prettier than
adding - 1 to print.

PCAP sanity checks

See merge request !31

Useful for investigating what sort of malformation is present.