Skip to content

JSON output + msgdiff/diffrepro/diffsum toolchain refactoring

Tomas Krizek requested to merge machine-readable-output into master

TODO:

  • order fields by count
  • take report.json from env dir by default

Following changes were made:

  • orchestrator, msgdiff, diffrepro and diffsum now store data to a JSON file, which contains:
    • stats (start/end time, total answers, queries) from orchestrator
    • diffs (query is referred to its ID) from msgdiff
    • reproducibility data (optional) from diffrepro
    • summary (optional, re-generateable) from diffsum
    • since queries are reffered to by ID, original LMDB with prepared queries should be stored along with the JSON file
  • diffsum output format changed slightly (e.g. others_disagree is now referred to as upstream_unstable)
  • diffrepro was integrated into the toolchain; diffsum can now generate reports with reproducibility data
  • tabulate pip dependency added (to simplify and avoid reinventing the wheel)

Diffsum (no reproducibility data)

== Global statistics
duration                   683 s
queries                    10000
answers                    10000   100.00 % of queries

== Differences statistics
upstream unstable             68    0.68 % of answers (ignoring)
target disagrees             195    1.96 % of not ignored answers

== Target Disagreements
+--------------+---------+-------------------+
| Field        |   Count |   % of mismatches |
|--------------+---------+-------------------|
| timeout      |     132 |             67.69 |
| answerrrsigs |      59 |             30.26 |
| rcode        |       4 |              2.05 |
+--------------+---------+-------------------+

== Field "timeout" mismatch statistics
+------------+---------+---------+-------------------+
| Expected   | Got     |   Count |   % of mismatches |
|------------+---------+---------+-------------------|
| answer     | timeout |     128 |             65.64 |
| timeout    | answer  |       4 |              2.05 |
+------------+---------+---------+-------------------+

== Field "rcode" mismatch statistics
+------------+----------+---------+-------------------+
| Expected   | Got      |   Count |   % of mismatches |
|------------+----------+---------+-------------------|
| NOERROR    | SERVFAIL |       4 |              2.05 |
+------------+----------+---------+-------------------+

== Field "answerrrsigs" mismatch statistics
+------------+-------+---------+-------------------+
| Expected   | Got   |   Count |   % of mismatches |
|------------+-------+---------+-------------------|
| RRSIG(A)   |       |      59 |             30.26 |
+------------+-------+---------+-------------------+

== Field "timeout", mismatch "answer != timeout" query details
  Count  Query
      1  google.com.bd. A
      1  sq.cn. A
      1  shihuo.cn. A
      1  pengpeng.com. A
      1  eroshare.com. A
      1  kaixin001.com. A
      1  gmx.at. A
      1  lbj.tw. A
      1  ucl.ac.uk. A
      1  kintera.org. A
    ...  omitting 118 queries

== Field "timeout", mismatch "timeout != answer" query details
  Count  Query
      1  dilidili.com. A
      1  xiamp4.com. A
      1  lxwxw.com. A
      1  artww.com. A

== Field "rcode", mismatch "NOERROR != SERVFAIL" query details
  Count  Query
      1  gry-online.pl. A
      1  google.pl. A
      1  goldenline.pl. A
      1  rozklad-pkp.pl. A

== Field "answerrrsigs", mismatch "RRSIG(A) != " query details
  Count  Query
      1  parimatch.com. A
      1  arabseed.com. A
      1  adf.ly. A
      1  mgm.gov.tr. A
      1  ptcl.net.pk. A
      1  ouo.io. A
      1  docusign.net. A
      1  videomega.tv. A
      1  videowood.tv. A
      1  smallseotools.com. A
    ...  omitting 49 queries

Diffsum (with reproducibility data)

== Global statistics
duration                   683 s
queries                    10000
answers                    10000   100.00 % of queries

== Differences statistics
upstream unstable            107    1.07 % of answers (ignoring)
not 100% reproducible         43    0.43 % of answers (ignoring)
target disagrees             113    1.15 % of not ignored answers

== Target Disagreements
+--------------+---------+-------------------+
| Field        |   Count |   % of mismatches |
|--------------+---------+-------------------|
| timeout      |      72 |             63.72 |
| answerrrsigs |      41 |             36.28 |
+--------------+---------+-------------------+

== Field "timeout" mismatch statistics
+------------+---------+---------+-------------------+
| Expected   | Got     |   Count |   % of mismatches |
|------------+---------+---------+-------------------|
| answer     | timeout |      71 |             62.83 |
| timeout    | answer  |       1 |              0.88 |
+------------+---------+---------+-------------------+

== Field "answerrrsigs" mismatch statistics
+------------+-------+---------+-------------------+
| Expected   | Got   |   Count |   % of mismatches |
|------------+-------+---------+-------------------|
| RRSIG(A)   |       |      41 |             36.28 |
+------------+-------+---------+-------------------+

== Field "timeout", mismatch "answer != timeout" query details
  Count  Query
      1  google.com.bd. A
      1  sq.cn. A
      1  jiemian.com. A
      1  eroshare.com. A
      1  brndrm.com. A
      1  kaixin001.com. A
      1  cps.com.cn. A
      1  gmx.at. A
      1  lbj.tw. A
      1  cooks.com. A
    ...  omitting 61 queries

== Field "timeout", mismatch "timeout != answer" query details
  Count  Query
      1  lxwxw.com. A

== Field "answerrrsigs", mismatch "RRSIG(A) != " query details
  Count  Query
      1  parimatch.com. A
      1  arabseed.com. A
      1  mgm.gov.tr. A
      1  ptcl.net.pk. A
      1  docusign.net. A
      1  smallseotools.com. A
      1  baykoreans.com. A
      1  texas.gov. A
      1  webtransfer.com. A
      1  fossbytes.com. A
    ...  omitting 31 queries
Edited by Tomas Krizek

Merge request reports