It helps to determine where to look in the packet.

Usable ONLY as sanity check, does not have any value for real benchmarking.

No manual changes and no behavior changes in this commit - simple code
style update from black.

Some use-cases require that the chunk merging script behaves more
predictably to better control the final PCAP. Add the option to use a
negative seed to turn off chunk randomization and just use them sorted
by file names.

This is essentially the opposite of limit-clients.lua.
This new script assigns one client IP address to a single output and
copies all packets for single client into the same output file.

Client assignment is pseudorandom and does not guarantee uniform
assignment. Clients typically do not generate equal load in any metric
(be it QPS or overall resources required to process client's query),
so there would be little point in complicating code for theoretical
uniformity.

Stats generated at the end provide some insight into uniformity of the
split on per-client and per-packet basis.

Petr Špaček authored 3 years ago and Tomas Krizek committed 3 years ago

By default, filter out queries for subdomains of dotnxdomain.net.
This is a 'special' measurement domain. Queries directed to it have
timestamps encoded in qname and replaying old queries results in
timeouts, not in a realistic traffic replay.

A new option -s can be used to keep the queries in the output if
desired.

The other domain - dashnxdomain.net - did not appear in any of my PCAPs
so for simplicity I omitted it from the filtering code.

Fixes: #25

Formerly, if malformed packets were requested in the output,
the script skipped DNS parsing because there was no point in doing so:
The malformed packets would have been included in the output anyway.

As preparation for a new feature which requires access to qname,
the script now tries to parse the packet even if user requested
inclusion of malformed packets in the output.

Even for tens of GBs of data overhead of doing this was negligible
compared to other processing, so I did not go to the trouble of
optimizing this further.

This is useful when extract-clients is used in a pipeline like this:
mergecap | filter-dnsq | extract-clients | split-clients

output.null was removed from dnsjit, but users can use /dev/null
if needed so there is no need to complicate code with extra conditions.

Wireshark considers some source port numbers special and switches packet
parser to non-DNS protocol, despite destination port being 53.
Extractor now overrides both source and destination ports to 53 to make
it easier to analyze its output in Wireshark.

Intended use is together with merge_chunks.py like this:
merge_chunks.py ... | cut-pcap.lua - /tmp/short.pcap 60

Stock editcap is designed to handle unsorted PCAPs and thus cannot stop
on encountering first packet with timestamp over limit.
This is very ineffective for processing large PCAPs generated by
extract-clients.lua because the mergecap + editcap pipeline keeps
processing all the "trailing" data, which can take really large time for
no benefit.

Funnily default value 0 was not accepted when specified explicitly.
https://docs.python.org/3/library/random.html says that full int
is acceptable so let's allow it.

Intended usage is to avoid generating PCAPs which are simple combination
of "base chunks".

Example:
When original PCAP is split into 100 chunks with 1 kQPS on average,
testing full range of 1k to 100k QPS formerly required generating 100
distincts PCAPs, wasting time and storage.
With this utility it is enough to generate "base chunks" and create
arbitrary multiplication on-the-fly.

Why Python?
- Lua cannot list content of directory, FFI solution would be
  unportable, and adding depedency just for that seems too much.
- BASH version of this script made me cry when I finished it.
- Python is already depedency of replay.py and it is used only to
  process mergecap arguments and python process terminates/replaces
  itself once its job is done.

Options without short variant should use nil. It would blow up if two
options used empty string for short name.

Previously packets in all chunks had timestamp 0 and ignored original
time difference from the last packet in the previous chunk.

For steady traffic with non-negligible QPS it does not make significant
difference, but it leads to big imprecision if original traffic had
"gaps" in it.

After this change first chunk starts at time 0 and timing of all packets
is relative to timing of the very first packet in the first chunk.

Very long gaps between packets will generate empty chunks,
and these empty chunks will be deleted because they are not useful.
See discussion in MR !32.

This also changes client IP addresses by decrementing the value by one.
I think it does not matter and I consider this slightly prettier than
adding - 1 to print.

Useful for investigating what sort of malformation is present.

This is order of magnitude faster than dpkt implementation.

Avoid creating long unique strings which trigger GC and slow down the
loop. Shorter non-unique string are deduplicated, leading to less GC and
faster processing.