Skip to content

diff for reports with support for thresholds

Once we have machine readable output (resolver-benchmarking#12), the next step is to write semantic diff for reports (output from diffsum).

It should have ability to display diff and also to evaluate the diff according to configured thresholds/conditions. Values should support specification as % of (answers agreed by other resolvers)/(of mismatches)/absolute value (including zero)

Thresholds should include:

  • number of mismatches for new field: e.g. number of mismatches in field opcode which was not present at all in the old data is allowed to have at most value X
  • number of mismatches for new pair of mismatching values: e.g. rcode field in the new report contains mismatch pair (REFUSED,NOTIMP) which was previously not present:
  • increase in previously recorded mismatches can be at most X
  • maximum number of new domain names referenced in the report is X

Example

Old report

== Global statistics
queries            907607
answers            907607    100.00 % of queries
others agree       901240     99.30 % of answers (ignoring 0.70 % of answers)
target diagrees      2409      0.27 % of matching answers from others

== Field          count    % of mismatches
answerrrsigs          1611      67 %
rcode                  607      25 %
answertypes            160       7 %
flags                   31       1 %

[...]
== Field "flags" mismatch ('QR RD RA AD', 'QR RD RA') query details
nofreezingmac.click. DS         1 mismatches
[...]

New report

== Global statistics
queries            907607
answers            907607    100.00 % of queries
others agree       901240     99.30 % of answers (ignoring 0.70 % of answers)
target diagrees      2500      0.28 % of matching answers from others

== Field          count    % of mismatches
answerrrsigs          1611      67 %
rcode                  607      25 %
answertypes            160       7 %
opcode                  90       4 %
flags                   32       1 %

[...]
== Field "flags" mismatch ('QR RD RA AD', 'QR RD RA') query details
nofreezingmac.click. DS         2 mismatches
[...]

Diff

== Global statistics
queries            907607
answers            907607    100.00 % of queries
others agree       901240     99.30 % of answers (ignoring 0.70 % of answers)
target diagrees      2500      0.28 % of matching answers from others           +91, +0.01 %


== Field          count    % of mismatches
answerrrsigs          1611      67 %
rcode                  607      25 %
answertypes            160       7 %
opcode                  90       4 %          +90, +4 %
flags                   32       1 %           +1, +0 %

[...]
== Field "flags" mismatch ('QR RD RA AD', 'QR RD RA') query details
nofreezingmac.click. DS         2 mismatches    +1
[...]

== New domains with mismatching answers (1 total)
now.invalid.opcode.example.                    +90
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information