Intended use is together with merge_chunks.py like this:
merge_chunks.py ... | cut-pcap.lua - /tmp/short.pcap 60

Stock editcap is designed to handle unsorted PCAPs and thus cannot stop
on encountering first packet with timestamp over limit.
This is very ineffective for processing large PCAPs generated by
extract-clients.lua because the mergecap + editcap pipeline keeps
processing all the "trailing" data, which can take really large time for
no benefit.

add tool to plot packet rate in PCAPs

See merge request !46

merge_chunks: relax overly strict check on --seed

See merge request !48

shotgun: log total number of processed packets

See merge request !49

plot-response-rate: ability to plot sum of specified rcodes

See merge request !47

This allows to compare number of `discarded` with something meaningful.
It's much harder to pass this number to JSON but it is good enough as
stop-gap measure.

Funnily default value 0 was not accepted when specified explicitly.
https://docs.python.org/3/library/random.html says that full int
is acceptable so let's allow it.

configs: generate additional response-rate chart with non-zero rcodes

See merge request !44

Generating charts is cheap so we generate variant with and also
without rcodes.

plot-response-rate: add ability to plot frequent rcodes

See merge request !43

--rcode now also accepts mnemonic names + extra value "other"

pylint started to complain about length of the main() function.

New option --rcodes-above-pct N allows to draw only rcodes which are more
frequent than N percent of all answers. Thus --rcodes-above-pct 0 draws
all rcodes which have non-zero number of samples.

Small values like --rcodes-above-pct 0.1 % can also filter noise like
"five REFUSED answers in the whole dataset" etc.

plot-response-rate: use distinct colors when plotting rcodes from a single JSON

See merge request !42

Commit ad584cef changed color selection
so each line from a single JSON uses the same color. That works fine
when plotting multiple data sets but makes it harder to deep-analyze a
single dataset, e.g. when plotting all rcodes from a single run.

This commit changes color selection once again:
- When plotting multiple JSON files, all lines from one file use the
  same color.
- When plotting a single JSON file, each rcode line uses different
  color. Each supported rcode has a stable color.

Each rcode line is still denoted with a symbol introduced in
ad584cef.

plot-response-rate: use the same color for rcode and summary lines

See merge request !41

Rcode lines use the same color as "summary" line.
Different rcodes are distinguished by markers derived from first letter
(s for SERVFAIL, n for NXDOMAIN ...) or rcode numbers for less common values.
NOERROR is displayed as 0 to distinguish it from n for NXDOMAIN.
Mathtext render of 0 also accidentally looks like O for Okay :-)

new option stop_after_s

See merge request !40

Ability to stop reading PCAP after specified number of seconds.
It depends on fact that extract-clients script renumbers packets
so first packet has timestamp 0.

Quirk:
Timestamps are checked only after they are already delayed by dnsjit
module. For this reason PCAP reading is in fact terminated when
time for the first packet _after_ the specified time limit passes.
In practice this would be noticeable only if delays between packets
were long.

replay: workaround for missing /32 or /128 support in Python ipaddress library

See merge request !39

Python issue #28577. Affects Ubuntu 20.04 which is often used in CI.

plot-client-distribution: fix handling of empty query count intervals

See merge request !38

Query count intervals with 0 clients in them were not handled properly.
E.g. if some clients were in interval [0-10) and [100-1000) while interval
[10-100) had no clients in it the old script has thrown an exception.

merge_chunks: utility to merge chunks on-the-fly

See merge request !37

Intended usage is to avoid generating PCAPs which are simple combination
of "base chunks".

Example:
When original PCAP is split into 100 chunks with 1 kQPS on average,
testing full range of 1k to 100k QPS formerly required generating 100
distincts PCAPs, wasting time and storage.
With this utility it is enough to generate "base chunks" and create
arbitrary multiplication on-the-fly.

Why Python?
- Lua cannot list content of directory, FFI solution would be
  unportable, and adding depedency just for that seems too much.
- BASH version of this script made me cry when I finished it.
- Python is already depedency of replay.py and it is used only to
  process mergecap arguments and python process terminates/replaces
  itself once its job is done.

Input improvements

See merge request !36

Petr Špaček authored 4 years ago and Tomas Krizek committed 4 years ago

In my tests mmap input is roughly 20 % faster than fpcap.
fpcap is used as fallback in case mmap does not work on selected file.
In practice it happens only when the input file is not "normal",
e.g. when it is a named pipe.

docs: add warning about scaling up traffic with mergecap

See merge request !35

Random seed improvements

See merge request !34

Options without short variant should use nil. It would blow up if two
options used empty string for short name.