diffsum.rst 2.07 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Diffsum
=======

Usage
-----

.. code-block:: console

   $ ./diffsum.py "${DIR}"    # basic example
   $ ./diffsum.py --help      # for more info


Description
-----------

Differences computed by ``msgdiff.py`` can be translated into text report using
tool ``diffsum.py`` which computes summary based on that comparison's results.

The report uses the following terms:

- *upstream unstable* represents queries, where the servers other than
  ``target`` haven't received the same answer, thus the source (upstream) of
  these queries is considered unstable. These queries aren't counted further
  towards the other statistics.
- *not reproducible* appears in cases where ``diffrepro.py`` tool was used
  to attempt to reproduce the measured differences. In case the difference
  doesn't match exactly the one before, the query is ignored in further
  processing.
- *target disagreements* refers to cases, when there's a difference
  between the answer from ``target`` server and the others server, and the
  other servers agree on the answer (there is no difference between them).
Tomas Krizek's avatar
Tomas Krizek committed
32 33
  These are the most interesting cases that are analyzed further.
- *manually ignored* is the number of queries which were omitted from the
34 35
  report by using `--without-ref-failing` or `--without-ref-unstable` along
  with a reference statistics file
36 37 38 39 40 41 42 43 44 45

The summary evaluates how many *target disagreements* there were in particular
*fields* (or ``criteria``), and what did these mismatches look like. It produces
both textual output and machine readable file (``*.json``).


Notes
-----

* If you adjust the ``field_weights``, just re-order the fields. Don't remove
Tomas Krizek's avatar
Tomas Krizek committed
46
  them, otherwise there will be issues if such field is ever encountered when
47 48 49 50 51 52 53
  producing the summary.
* In case you update respdiff and ``diffsum.py`` doesn't work, check the
  changelog. If a new field was added, adjust your config accordingly.
* Redirect *stdout* of the command to a text file in case you want to keep the
  textual report for future reference.
* If you want a comprehensive list of mismatched queries in the text report,
  use ``-l 0`` argument.