automatic mismatch verification/reproducibility testing

Some mismatches might be transient (e.g. a random packet loss) and others might be caused by a bug. A tool which will attempt to verify reproducibility of particular mismatch would be very useful. Proof-of-concept is called diffrepro but it needs a lot of work to make it easy to use.

Roungh ideas:

• Provide hooks allowing user to put in custom scripts to flush cache or restart daemon. This should allow us to determine if the mismatch was caused by some bad state in cache or if it is reproducible with empty cache.
• Dump state of cache before flush for further inspection
• Output reproducible and irreproducible failures for further processing, e.g. for statistical evaluation that domain example.com. is suffering long-term instability and thus can be excluded from comparison using live data.
• Capture verbose logs from reproduction attempts to reproducible issues can be investigated right away.