diff for reports with support for thresholds
Once we have machine readable output (resolver-benchmarking#12), the next step is to write semantic diff for reports (output from diffsum
).
It should have ability to display diff and also to evaluate the diff according to configured thresholds/conditions. Values should support specification as % of (answers agreed by other resolvers)/(of mismatches)/absolute value (including zero)
Thresholds should include:
-
number of mismatches for new field: e.g. number of mismatches in field opcode
which was not present at all in the old data is allowed to have at most value X -
number of mismatches for new pair of mismatching values: e.g. rcode
field in the new report contains mismatch pair(REFUSED,NOTIMP)
which was previously not present: -
increase in previously recorded mismatches can be at most X -
maximum number of new domain names referenced in the report is X
Example
Old report
== Global statistics
queries 907607
answers 907607 100.00 % of queries
others agree 901240 99.30 % of answers (ignoring 0.70 % of answers)
target diagrees 2409 0.27 % of matching answers from others
== Field count % of mismatches
answerrrsigs 1611 67 %
rcode 607 25 %
answertypes 160 7 %
flags 31 1 %
[...]
== Field "flags" mismatch ('QR RD RA AD', 'QR RD RA') query details
nofreezingmac.click. DS 1 mismatches
[...]
New report
== Global statistics
queries 907607
answers 907607 100.00 % of queries
others agree 901240 99.30 % of answers (ignoring 0.70 % of answers)
target diagrees 2500 0.28 % of matching answers from others
== Field count % of mismatches
answerrrsigs 1611 67 %
rcode 607 25 %
answertypes 160 7 %
opcode 90 4 %
flags 32 1 %
[...]
== Field "flags" mismatch ('QR RD RA AD', 'QR RD RA') query details
nofreezingmac.click. DS 2 mismatches
[...]
Diff
== Global statistics
queries 907607
answers 907607 100.00 % of queries
others agree 901240 99.30 % of answers (ignoring 0.70 % of answers)
target diagrees 2500 0.28 % of matching answers from others +91, +0.01 %
== Field count % of mismatches
answerrrsigs 1611 67 %
rcode 607 25 %
answertypes 160 7 %
opcode 90 4 % +90, +4 %
flags 32 1 % +1, +0 %
[...]
== Field "flags" mismatch ('QR RD RA AD', 'QR RD RA') query details
nofreezingmac.click. DS 2 mismatches +1
[...]
== New domains with mismatching answers (1 total)
now.invalid.opcode.example. +90