... | @@ -17,6 +17,23 @@ Some compression is necessary, even fastest compression smoothes major differenc |
... | @@ -17,6 +17,23 @@ Some compression is necessary, even fastest compression smoothes major differenc |
|
* [xz](http://tukaani.org/xz/), [file format](http://tukaani.org/xz/format.html), [liblzma git](http://git.tukaani.org/?p=xz.git;a=tree;f=src/liblzma/api;hb=HEAD) - slow, best compression, independent frames and an embedded block index, xz format and cmdline tools
|
|
* [xz](http://tukaani.org/xz/), [file format](http://tukaani.org/xz/format.html), [liblzma git](http://git.tukaani.org/?p=xz.git;a=tree;f=src/liblzma/api;hb=HEAD) - slow, best compression, independent frames and an embedded block index, xz format and cmdline tools
|
|
* for entropy encoding inspiration: [text compression comparison](http://mattmahoney.net/dc/text.html) - too slow for our purpose
|
|
* for entropy encoding inspiration: [text compression comparison](http://mattmahoney.net/dc/text.html) - too slow for our purpose
|
|
|
|
|
|
|
|
## CBOR output comparison (Sinodun dumpcap tool)
|
|
|
|
|
|
|
|
**Data:** akuma.20150130.12 (177MB pcap, 1 hour, 441278 queries, 405051 responses)
|
|
|
|
|
|
|
|
### Basic data in CBOR
|
|
|
|
|
|
|
|
Size baseline (100%): 6183444 for cbor-chunk.xz-9
|
|
|
|
| type | plain | lz4-9 | xz-0 | xz-2 | xz-6 | xz-9 |
|
|
|
|
| --- | --- | --- | --- | --- | --- | --- |
|
|
|
|
| cbor-chunk | 308% | 168% 0.8s | 120% 1.8s | 116% 5.1s | 100% 11.6s | 100% 12.5s |
|
|
|
|
| cbor-simple | 755% | 182% 1.5s | 137% 2.2s | 128% 5.6s | 110% 30.3s | 109% 32.8s |
|
|
|
|
| cbor-table | 611% | 173% 1.2s | 134% 2.0s | 124% 5.2s | 108% 23.9s | 107% 26.7s |
|
|
|
|
|
|
|
|
### Extended data in CBOR
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Experimental data rates for various storage
|
|
## Experimental data rates for various storage
|
|
|
|
|
|
Data source: `akuma.20150106.145000.018146` - 37426 queries (pairs)
|
|
Data source: `akuma.20150106.145000.018146` - 37426 queries (pairs)
|
... | | ... | |