Skip to content
Snippets Groups Projects

libs and projects

Last edited by Tomáš Gavenčiak

Related libs, tools and projects

DNS parsers in C

libknot

ldns

others

  • wdns, resolver libs, ... - mostly very simple parsing API
  • wireshark - based on ASN.1 grammar descriptions, probably slow (also, ASN.1 is kinda ugly)

Packet capture, stream reconstruction

libPCAP (tcpdump)

  • web, manual pages
  • BSD licence
  • Basic structure and functionality (need to parse/skip headers yourself, switch on multiple capture layers), probably slower than libtrace (?)

libTrace

  • web, wiki, API docs
  • GPLv2 licensed
  • Multiple packet sources (can do compressed PCAP, mutiple live traces at once and linux ring-buffers)
  • can parse/skip headers (for multiple layers), nice API

TcpTrace

  • web - last release in 2003
  • can reconstruct Tcp stream from pcap
  • minus: old, ugly code base, unmaintained, csv ported to github
  • miuns: program-like structure, global variables ...

Wireshark/EPAN

  • glib-based library, core of wireshark, doxygen
  • ASN1 and analysis based - what about speed?
  • minus: glib

libNIDS

  • web, github import
  • IP defrag and TCP reassembly lib
  • minus: last dev in 2010, created in 2003 based on linux kernel 2.0.x
  • minus: IPv4 only

libNtoH

  • github
  • IP4+6 and TCP reassembly, quite new, stability?

DPDK

standard netinet headers

  • netinet/in.h, netinet/ip.h, netinet/ip6.h - manual header matching, partial (best effort) defragmentation

Data serialization libraries

What to consider:

  • Speed
  • Compact (beware: wasteful "string" field names in JSON-encoded structs)
  • Stable in C (needs good implementation!), then JS, C++, Python, Java
  • Accepted by the community, tools
  • Dynamic vs static typing (schema-less and JSON-like harder to read with static languages (C, C++, Java))

Compactness numbers from here.

CBOR

Concise Binary Object Representation web, RFC 7094

  • type: binary JSON (with extras), no schema, dynamic types
  • Clib: libcbor nice, streaming, simple refcounting for cleanup, (allocates a lot?)
  • Clib: tinycbor no streaming, a bit too simple
  • plus: versatile format (streaming arrays, tags)
  • minus: field names in structs as literal strings (wasteful)
  • speed: probably ok
  • compact: bad, ok (386) with numbered attributes

Protobuf

Protocol buffers

  • type: schema and code generators, static types
  • Clib: protobuf-c - TODO: test (API/speed)
  • Clib: protobluff - TODO: test (API/speed)
  • minus: only contrib C libraries
  • plus: streaming arrays/maps (via repeated fields)
  • plus: C/C++/Java struct/class generation (typewise-easy parsing)
  • plus: used by dnstap DNS logs
  • speed: probably good
  • compact: great (239)

BSON

Binary JSON web

  • type: binary JSON, no schema
  • Clib: mongo libbson - nice, own allocation, easier construction (BCON)
  • plus: arrays/maps streaming (unknown array size in advance)
  • minus: field names in structs as strings (wasteful)
  • plus: used natively in MongoDB (nice C api)
  • mongoDB: max 16MB BSON chunks, GridFS for more (fragmented, ...)
  • speed: probably ok
  • compact: bad (495), better (?) with numbered attributes

MessagePack

MessagePack web

  • type: binary JSON, no schema, dynamic
  • Clib: msgpack-c, C docs - parses only whole trees (problems with bigger data?), mempools, refcounts
  • minus: no streaming of maps/arrays (size known in advance)
  • minus: field names in structs as literal strings (wasteful)
  • speed: probably ok
  • compact: bad, great (233) with numbered attributes

Thrift

Thrift

  • type: schema and code generators, static types
  • Clib: c_glib - Glib based!
  • minus: bad C library support (only GLib!!!) (bad docs)
  • minus: RPC focus ...
  • speed: probably ok
  • compact: ok (349)

Cap'n'proto

Cap'n'proto

  • type: schema and code generators, static types
  • Clib: c-capnproto - only writes!
  • minus: bad C library support (only writing)
  • speed: probably great
  • compact: probably very bad

Similar projects

DnsTap (format+tools)

  • DnsTap web by Robert Edmonds from farsightsec
  • Capture within DNS server process (impl. for Unbounded, Knot)
  • Logging with a dnstap ProtoBuf
    • larger messages (not optimized for size, cca 50+ bytes vs 20 possible otherwise, would compression help?)
    • seems to store compatible information, but only raw DNS message
    • does not specify storage (either frame stream (below) or just ProtoBuf messages)
  • implementation using:

DNSCap (tool)

  • dnscap web, dnscap git
  • captures DNS packets (query+response), output in pcap, basic filtering options
  • no query/response matching
  • inspiration for simple pcap/parsing?
  • does not do defragmentation / tcp stream reconstruction (see source comments)

DNS Stats Collector (tool)

  • dsc git
  • captures packets, basic DNS parsing, counting stats, plus some XML and graph presentation
  • inspiration for simple pcap/parsing?

DNSTable, nmsg etc. from Farsightsec (format/lib/tool group)

  • dnstable - file based tables for DNS domain information (not queries)
  • nmsg - format/library for storing various network message types (based on protobufs)
    • common container for various data types, some tools to transfer, merge, convert, compress, ... data
    • based on ProtoBufs, basic protocols such as: DNS record, DNS query, HTTP, email, generic packet, ...
    • blog posts: intro, nmsgtool, format
    • their DNS query ProtoBuf does not quite match our needs, we could add our own
    • the data storage is inefficient (extra information cca 30 bytes/packet)
    • could nmsgtool or libnmsg be useful? probably not if we store captured data in some DB
  • ncap - obsoloete DNS-only capture format, non-extensible

Zendesk DDoS detection (solution)

  • Slides from RIPE (info from Jan and Petr)
  • metrics based solution - no DNS inspection, much lower data flows
  • based on: FastNetMon (metrics from traffic), InfluxDB (time-series data DB), Morgoth (time-series anomaly detection)
  • Morgoth as an inspiration for anomaly detection?
    • "lossy event counting" algorithm
    • Idea of "exceptional fingeprints" (time window aggregations/statistics compared to previously seen windows)
    • written in Go, not very well documented

DNS packet deduplication (from Jan Vcelak)

  • gitlab
  • simple pcap (TCP+UDP) parsing - but no defragmentation
  • deduplication via nice HAT-trie (inspiration?)# Stores for captured and prefiltered data

Comments

    Please register or sign in to add a comment.