use common, stream-oriented data format
The current data formats are giant JSON dictionaries and that causes two major problems:
- Processing it using standard libraries has outrageous memory requirements (see #41)
- Premature termination causes incomplete output (see #31)
- It produces very very big files, especially because of the latency tables - which are often filled with zeroes
- It is unique to us and not shared.
Working with other tools to standardize on something common could solve most of these issues and as a bonus allow us the same post-processing for different tools.
See https://github.com/DNS-OARC/flamethrower/issues/99 . I know Knot DNS's kxdpgun is experimenting with this format.