|
|
# Technical tools/libs choices
|
|
|
# Related libs, tools and projects
|
|
|
|
|
|
|
|
|
## Parsing DNS messages in C
|
|
|
## DNS parsers in C
|
|
|
|
|
|
### libknot
|
|
|
* [knot gitlab](https://gitlab.labs.nic.cz/labs/knot), [packet API header](https://gitlab.labs.nic.cz/labs/knot/blob/master/src/libknot/packet/pkt.h)
|
... | ... | @@ -16,19 +15,61 @@ |
|
|
|
|
|
### others
|
|
|
* wdns, resolver libs, ... - mostly very simple parsing API
|
|
|
* wireshark - based on ASN.1 grammar descriptions, probably not fast (also, ASN.1 is kinda ugly)
|
|
|
* wireshark - based on ASN.1 grammar descriptions, probably slow (also, ASN.1 is kinda ugly)
|
|
|
|
|
|
|
|
|
## Packet capture, stream reconstruction
|
|
|
|
|
|
### libPCAP (tcpdump)
|
|
|
* [web](http://www.tcpdump.org/), [manual pages](http://www.tcpdump.org/manpages/pcap.3pcap.html)
|
|
|
* BSD licence
|
|
|
* Basic structure and functionality (need to parse/skip headers yourself, switch on multiple capture layers), probably slower than libtrace (?)
|
|
|
|
|
|
### libTrace
|
|
|
* [web](http://research.wand.net.nz/software/libtrace.php), [wiki](http://wand.net.nz/trac/libtrace/wiki/UserDocumentation), [API docs](http://research.wand.net.nz/software/libtrace-docs/html/libtrace_8h.html)
|
|
|
* GPLv2 licensed
|
|
|
* Multiple packet sources (can do compressed PCAP, mutiple live traces at once and linux ring-buffers)
|
|
|
* can parse/skip headers (for multiple layers), nice API
|
|
|
|
|
|
### TcpTrace
|
|
|
* [web](http://www.tcptrace.org/download.html) - last release in 2003
|
|
|
* can reconstruct Tcp stream from pcap
|
|
|
* minus: old, ugly code base, unmaintained, [csv ported to github](https://github.com/blitz/tcptrace)
|
|
|
* miuns: program-like structure, global variables ...
|
|
|
|
|
|
## Data serialization
|
|
|
### Wireshark/EPAN
|
|
|
* glib-based library, core of wireshark, [doxygen](https://www.wireshark.org/docs/wsar_html/epan/index.html)
|
|
|
* ASN1 and analysis based - what about speed?
|
|
|
* minus: glib
|
|
|
|
|
|
### libNIDS
|
|
|
* [web](http://libnids.sourceforge.net/), [github import](https://github.com/korczis/libnids)
|
|
|
* IP defrag and TCP reassembly lib
|
|
|
* minus: last dev in 2010, created in 2003 based on linux kernel 2.0.x
|
|
|
* minus: IPv4 only
|
|
|
|
|
|
### libNtoH
|
|
|
* [github](https://github.com/sch3m4/libntoh/)
|
|
|
* IP4+6 and TCP reassembly, quite new, stability?
|
|
|
|
|
|
### DPDK
|
|
|
* [web](http://dpdk.org/), libs (incl. low-level) for fast packet processing
|
|
|
* [ip fragment reassembly](http://dpdk.org/doc/guides/prog_guide/ip_fragment_reassembly_lib.html) - both IPv4 and IPv6
|
|
|
|
|
|
### standard netinet headers
|
|
|
* `netinet/in.h`, `netinet/ip.h`, `netinet/ip6.h` - manual header matching, partial (best effort) defragmentation
|
|
|
|
|
|
|
|
|
## Data serialization libraries
|
|
|
|
|
|
What to consider:
|
|
|
* Speed
|
|
|
* Compact (beware: wasteful "string" field names in encoded structs)
|
|
|
* Stable in C (needs good implementation!), Java, then JS, C++, Python
|
|
|
* Compact (beware: wasteful "string" field names in JSON-encoded structs)
|
|
|
* Stable in C (needs good implementation!), then JS, C++, Python, Java
|
|
|
* Accepted by the community, tools
|
|
|
* Dynamic vs static typing (schema-less and JSON-like harder to read with static languages (C, C++, Java))
|
|
|
|
|
|
(Compactness numbers from [here](https://github.com/eishay/jvm-serializers/wiki) )
|
|
|
Compactness numbers from [here](https://github.com/eishay/jvm-serializers/wiki).
|
|
|
|
|
|
### CBOR
|
|
|
[Concise Binary Object Representation web](http://cbor.io/), [RFC 7094](http://tools.ietf.org/html/rfc7049)
|
... | ... | @@ -90,51 +131,9 @@ What to consider: |
|
|
* compact: probably very bad
|
|
|
|
|
|
|
|
|
## Data compression
|
|
|
* fast stream: [snappy](https://google.github.io/snappy/) (100+ MB/s, used in hadoop and elsewhere)
|
|
|
* for entropy encoding inspiration: [text compression comparison](http://mattmahoney.net/dc/text.html) - too slow for our purpose
|
|
|
* good encoding (ProtoBuf etc.) will make compression harder and less needed
|
|
|
|
|
|
## Network packet capture
|
|
|
|
|
|
### Mirror UDP stream to/from port 53 (both dirs) to some port of collector
|
|
|
* plus: UDP defrag and header parsing done by kernel, can use recvmsg, libuv or similar
|
|
|
* minus: UDP only
|
|
|
## Similar projects
|
|
|
|
|
|
### TcpTrace
|
|
|
* [web](http://www.tcptrace.org/download.html) - last release in 2003
|
|
|
* can reconstruct Tcp stream from pcap
|
|
|
* minus: old, ugly code base, unmaintained, [csv ported to github](https://github.com/blitz/tcptrace)
|
|
|
* miuns: program-like structure, global variables ...
|
|
|
|
|
|
### Wireshark/EPAN
|
|
|
* glib-based library, core of wireshark, [doxygen](https://www.wireshark.org/docs/wsar_html/epan/index.html)
|
|
|
* ASN1 and analysis based - what about speed?
|
|
|
* minus: glib
|
|
|
|
|
|
### libNIDS
|
|
|
* [web](http://libnids.sourceforge.net/), [github import](https://github.com/korczis/libnids)
|
|
|
* IP defrag and TCP reassembly lib
|
|
|
* minus: last dev in 2010, created in 2003 based on linux kernel 2.0.x
|
|
|
* minus: IPv4 only
|
|
|
|
|
|
### libNtoH
|
|
|
* [github](https://github.com/sch3m4/libntoh/)
|
|
|
* IP4+6 and TCP reassembly, quite new, stability?
|
|
|
|
|
|
### DPDK
|
|
|
* [web](http://dpdk.org/), libs (incl. low-level) for fast packet processing
|
|
|
* [ip fragment reassembly](http://dpdk.org/doc/guides/prog_guide/ip_fragment_reassembly_lib.html) - both IPv4 and IPv6
|
|
|
|
|
|
### standard netinet headers
|
|
|
* `netinet/in.h`, `netinet/ip.h`, `netinet/ip6.h` - manual header matching, partial (best effort) defragmentation
|
|
|
|
|
|
|
|
|
# Existing tools, libs and formats
|
|
|
|
|
|
A comparison of some similarly aimed projects.
|
|
|
|
|
|
## DnsTap (format+tools)
|
|
|
### DnsTap (format+tools)
|
|
|
* [DnsTap web](http://dnstap.info/) by Robert Edmonds from [farsightsec](https://github.com/farsightsec)
|
|
|
* Capture within DNS server process (impl. for Unbounded, Knot)
|
|
|
* Logging with a [dnstap ProtoBuf](https://github.com/dnstap/dnstap.pb/blob/master/dnstap.proto)
|
... | ... | @@ -145,19 +144,19 @@ A comparison of some similarly aimed projects. |
|
|
* frame stream [fstrm](https://github.com/farsightsec/fstrm) for reliable frame dropping under load
|
|
|
* DNS parser [ldns](http://www.nlnetlabs.nl/projects/ldns/), [parser usage in dnstap-ldns](https://github.com/dnstap/dnstap-ldns/blob/master/host2str.c)
|
|
|
|
|
|
## DNSCap (tool)
|
|
|
### DNSCap (tool)
|
|
|
* [dnscap web](https://www.dns-oarc.net/tools/dnscap), [dnscap git](https://github.com/verisign/dnscap)
|
|
|
* captures DNS packets (query+response), output in pcap, basic filtering options
|
|
|
* no query/response matching
|
|
|
* inspiration for simple pcap/parsing?
|
|
|
* does not do defragmentation / tcp stream reconstruction (see [source comments](https://github.com/verisign/dnscap/blob/3f3468f0c9ed7d2d554a23813b769b9b2924eaf1/dnscap.c#L1797))
|
|
|
|
|
|
## DNS Stats Collector (tool)
|
|
|
### DNS Stats Collector (tool)
|
|
|
* [dsc git](https://github.com/DNS-OARC/dsc)
|
|
|
* captures packets, basic DNS parsing, counting stats, plus some XML and graph presentation
|
|
|
* inspiration for simple pcap/parsing?
|
|
|
|
|
|
## DNSTable, nmsg etc. from Farsightsec (format/lib/tool group)
|
|
|
### DNSTable, nmsg etc. from Farsightsec (format/lib/tool group)
|
|
|
* [dnstable](https://github.com/farsightsec/dnstable) - file based tables for DNS domain information (not queries)
|
|
|
* not directly useful, but very fast indexed storage of DNS records
|
|
|
* [blog entry on storage](https://www.farsightsecurity.com/Blog/20151028-ziegast-realtime-dnsdb/)
|
... | ... | @@ -170,7 +169,7 @@ A comparison of some similarly aimed projects. |
|
|
* could nmsgtool or libnmsg be useful? probably not if we store captured data in some DB
|
|
|
* [ncap](https://www.dns-oarc.net/tools/ncap) - obsoloete DNS-only capture format, non-extensible
|
|
|
|
|
|
## Zendesk DDoS detection (solution)
|
|
|
### Zendesk DDoS detection (solution)
|
|
|
* [Slides from RIPE](https://ripe71.ripe.net/presentations/42-zendesk-ddos.pdf) (info from Jan and Petr)
|
|
|
* metrics based solution - no DNS inspection, much lower data flows
|
|
|
* based on: [FastNetMon](https://github.com/pavel-odintsov/fastnetmon) (metrics from traffic), [InfluxDB](https://influxdata.com/time-series-platform/influxdb/) (time-series data DB), [Morgoth](http://docs.morgoth.io/) (time-series anomaly detection)
|
... | ... | @@ -179,25 +178,8 @@ A comparison of some similarly aimed projects. |
|
|
* Idea of "exceptional fingeprints" (time window aggregations/statistics compared to previously seen windows)
|
|
|
* written in Go, not very well documented
|
|
|
|
|
|
## DNS packet deduplication (from Jan Vcelak)
|
|
|
### DNS packet deduplication (from Jan Vcelak)
|
|
|
* [gitlab](https://gitlab.labs.nic.cz/knot/smoke-tests)
|
|
|
* simple pcap (TCP+UDP) parsing - but no defragmentation
|
|
|
* deduplication via nice HAT-trie (inspiration?)# Stores for captured and prefiltered data
|
|
|
|
|
|
*Tomas:* I am looking for (prefiltered) packet data stores with good speed and C api - may be very different from the main processing databases.
|
|
|
|
|
|
* A NoSQL comparison (Cassandra, Couchbase, HBase, MongoDB; but only 100 byte packets): [PDF](http://www.datastax.com/wp-content/themes/datastax-2014-08/files/NoSQL_Benchmarks_EndPoint.pdf)
|
|
|
|
|
|
## LMDB
|
|
|
* [lmdb web](http://symas.com/mdb/), [docs](http://symas.com/mdb/doc/), [github](https://github.com/LMDB/lmdb)
|
|
|
* mmaped DB, key-value (unique keys by default), very fast
|
|
|
* idea/question: use for request/response matching? bad: value rewrites fast, but accumulate on-disk size :(
|
|
|
* what would be useful (unique?) keys?
|
|
|
* plus: indexable individual packets
|
|
|
* minus: not compressed
|
|
|
|
|
|
## Protobuf file
|
|
|
* Protobuf with metainfo and stream of query PBs
|
|
|
* small overhead (2 bytes/message), super fast :)
|
|
|
* plus: optionally compress (snappy, lzo, xz, ...)
|
|
|
* minus: sequential - messages not indexable |
|
|
\ No newline at end of file |