Commit 331979d0 authored by Štěpán Balážik's avatar Štěpán Balážik Committed by Tomas Krizek
Browse files

docs: py.test runner

parent 3cd8f3cf
......@@ -27,6 +27,8 @@ Deckard requires following software to be installed:
- dnspython_ - DNS library for Python
- Jinja2_ - template engine for generating config files
- PyYAML_ - YAML parser for Python
- pytest_ - testing framework for Python, used for running the test cases
- pytest-xdist_ - module for pytest for distributed testing
- custom C libraries (installed automatically, see below)
For convenient use it is strongly recommended to have a C compiler, Git, and ``make`` available.
......@@ -88,3 +90,5 @@ Happy testing.
.. _`libfaketime`: https://github.com/wolfcw/libfaketime
.. _`modified socket_wrapper`: https://gitlab.labs.nic.cz/labs/socket_wrapper
.. _`original socket_wrapper`: https://cwrap.org/socket_wrapper.html
.. _`pytest`: https://pytest.org/
.. _`pytest-xdist`: https://pypi.python.org/pypi/pytest-xdist
......@@ -9,7 +9,7 @@ There are four components in play:
- Deckard itself (test orchestrator)
- binary under test (your own)
- configuration for the binary (generated by Deckard from *template*, i.e. ``.j2`` file)
- configuration for the binary (generated by Deckard from *template* and *YaML configuration*, i.e. ``.j2`` and ``.yaml`` files)
- environment description and test data (Deckard *scenario*, i.e. ``.rpl`` file)
It is easy to run tests if everything is already prepared and running tests gets harder
......@@ -19,13 +19,15 @@ Let's start with the easiest case:
First run
---------
Before we start, please note that Deckard depends on couple of modified C libraries.
Easiest way to run Deckard is using one of the prepared Shell scripts in Deckard repository (``{kresd,unbound,pdns}_run.sh`` for Knot Resolver, Unbound and PowerDNS Recursor respectively).
Please note that Deckard depends on a couple of modified C libraries.
These will be automatically downloaded and compiled on first run, so do not be surprised when you see
output from Git and C compiler:
.. code-block::
$ make
$ ./kresd_run.sh
Submodule 'contrib/libfaketime' (https://github.com/wolfcw/libfaketime.git) registered for path 'contrib/libfaketime'
Submodule 'contrib/libswrap' (https://gitlab.labs.nic.cz/labs/socket_wrapper.git) registered for path 'contrib/libswrap'
[...]
......@@ -34,75 +36,114 @@ output from Git and C compiler:
[ 50%] Building C object src/CMakeFiles/socket_wrapper.dir/socket_wrapper.c.o
[...]
[100%] Built target socket_wrapper
For details see `README <../README.rst>`_.
Deckard uses `pytest` to generate and run the tests as well as collect the results.
Output is therefore generated by `pytest` as well (``.`` for passed test, ``F`` for failed test and ``s`` for skipped test) and will look something like this:
.. code-block::
$ ./kresd_run.sh
........s...s...s....................ssss...s.ss.............ssss..s..ss [ 24%]
ssss.....sssssssssssssss.sssss.......ssssss.ss...s..s.ss.sss.s.s........ [ 49%]
.............ssss....................................................... [ 73%]
........................................................................ [ 98%]
.... [100%]
229 passed, 62 skipped in 76.50 seconds
.. note:: There is a lot of tests skipped because we run them with query minimization both on and off and some of the scenarios work only with query minimization on (or off respectively). For details see `Scenario guide#Configuration <scenario_guide.rst#configuration-config-end>`_.
Time elapsed which is printed by `py.test` is often not acurate (or even negative). `py.test` is confused about our time shifting shananigans done with ``libfaketime``. We can overcome this by using ``-n`` command line argument. See below.
Using an existing scenarios and configuration template
------------------------------------------------------
In the simplest case, the scenario and configuration template for given binary already exist
and you just need to run them.
Most typically Deckard is not executed directly but through auxiliary scripts which supply parameters and run series of tests. For Knot Resolver, Unbound, and PowerDNS Recursor you can use scripts [run.sh]_ from Deckard repo. Following example shows script ``kresd_run.sh`` which executes tests on binary ``kresd`` (which must be in ``PATH``).
Command line arguments
----------------------
As mentioned above we use `py.test` to run the tests so all possible command line arguments for the ``*run.sh`` scripts can be seen by running ``py.test -h`` in the root of Deckard repository.
Here is a list of the most useful ones:
- ``-n number`` – runs the testing in parallel with ``number`` of processes (this requires `pytest-xdist` to be installed)
- ``-k EXPRESSION`` – only run tests which match the given substring expression (e.g. ``./kresd_run -k "world_"`` will only run the scenarios with `world_` in their file name.
- ``--collectonly`` – only print the names of selected tests, no tests will be run
- ``--log-level DEBUG`` – print all debug information for failed tests
- ``--scenarios path`` – specifies where to look for `.rpl` files (``sets/resolver`` is the default)
YaML configuration
------------------
All ``*_run.sh`` scripts internally call the ``run.sh`` script and pass command line arguments to it. For example:
.. code-block::
$ ./kresd_run.sh
=== Testing WITH query minimization ===
[...]
[100%] Built target socket_wrapper
[ OK ] sets/resolver/iter_badglue.rpl
[ OK ] sets/resolver/iter_cname_badauth.rpl
[...]
=== Testing WITHOUT query minimization ===
[...]
[ OK ] sets/resolver/iter_badglue.rpl
[ OK ] sets/resolver/iter_cname_badauth.rpl
[...]
# running ./kresd_run.sh -n 4 -k "iter_" will result in running
./run.sh --config configs/kresd.yaml -n 4 -k "iter_"
Return code will be 0 if all tests passed, non-zero otherwise.
As you can see, path to YaML configuration file is passed to ``run.sh``. You can edit one of the prepared ones stored in `config/` or write your own.
.. [run.sh] See scripts ``kresd_run.sh``, ``unbound_run.sh``, and ``pdns_run.sh`` in Deckard repo.
Commented contents ``kresd.yaml`` follow:
.. code-block:: yaml
Tweaking script execution
^^^^^^^^^^^^^^^^^^^^^^^^^
Behavior of ``*_run.sh`` scripts can be modified using following environment variables:
programs:
- name: kresd # path to binary under test
binary: kresd
additional: # list additional parameters for binary under test (e.g. path to configuration files)
- -f
- "1" # CAUTION: All parameters must be strings.
templates:
- template/kresd.j2 # list of Jinja2_ template files to generate configuration files
configs:
- config # list of names of configuration files to be generated from Jinja2_ templates
- ``DAEMON`` - path to binary under test
- ``ADDITIONAL`` - additional parameters for binary under test (e.g. path to configuration files)
- ``CONFIG`` - colon-separated list of names of configuration files to be generated from Jinja2_ templates
- 'configs' files will be generated from respective files in 'templates' list
- i.e. the first file in 'configs' list is the result of processing of the first file from 'templates' list and so on
- generated files are stored in a new working directory created by Deckard for each binary
- ``CONFIG`` files will be generated from respective files in ``TEMPLATE`` list
- i.e. the first file in ``CONFIG`` list is the result of processing of the first file from ``TEMPLATE`` list and so on
- generated files are stored in a new working directory created by Deckard for each binary
Most often it is sufficient to use these files for basic configuration changes. Read next section for details about config file templates.
- ``TEMPLATE`` - colon-separated list of Jinja2_ template files to generate configuration files
- ``TESTS`` - path to scenario file or directory with scenario files to be recursively processed
- ``MAKEFLAGS`` - scripts internally use ``make`` so all relevant options can be used
Running multiple binaries
^^^^^^^^^^^^^^^^^^^^^^^^^
You can specify multiple programs to run in the YaML configuration. Deckard executes all binaries using parameters from the file. This is handy for testing interoperability of multiple binaries, e.g. when one program is configured as DNS recursor and other program is using it as forwarder.
- feel free to set environment variable ``MAKEFLAGS`` to run tests in parallel, e.g. ``export MAKEFLAGS="-j4"``
The YAML file contains **ordered** list of binaries and their parameters. Deckard will send queries to the binary listed first.
Commented default values taken from ``unbound_run.sh`` follow:
.. code-block:: yaml
.. code-block:: bash
programs:
- name: forwarding # name of this Knot Resolver instance
binary: kresd # kresd is first so it will receive queries from Deckard
additional: []
templates:
- template/kresd_fwd.j2 # this template uses variable IPADDRS['recursor']
configs:
- config
- name: recursor # name of this Unbound instance
binary: unbound
additional:
- -d
- -c
- unbound.conf
templates:
- template/unbound.j2
- template/hints_zone.j2 # this template uses variable ROOT_ADDR
configs:
- unbound.conf
- hints.zone
- ta.keys
# Path to daemon
DAEMON="unbound"
In this setup it is necessary to configure one binary to contact the other. IP addresses assigned by Deckard at run-time are accessible using ``IPADDRS`` `template variables`_ and symbolic names assigned to binaries in the YAML file. For example, template ``kresd_fwd.j2`` can use IP address of binary named ``recursor`` like this:
# Additional parametes for binary: configuration file is in working directory
ADDITIONAL="-d -c unbound.conf"
.. code-block:: lua
# Template file names
TEMPLATE="template/unbound.j2:template/hints_zone.j2"
policy.add(policy.all(policy.FORWARD("{{IPADDRS['recursor']}}")))
When all preparations are finished, run Deckard using following syntax:
# Config file names: generated respectively from templates above
CONFIG="unbound.conf:hints.zone:ta.keys"
.. code-block:: bash
# Run all tests in directory "sets/resolver"
TESTS="sets/resolver"
$ ./run.sh --config path/to/config.yaml
Most often it is sufficient to use these variables for basic configuration changes. Read next section for details about config file templates.
Using an existing scenarios with custom configuration template
......@@ -198,23 +239,6 @@ It's okay if you don't use all of the variables, but expect some tests to fail.
then the DNSSEC tests will not work properly.
Running custom templates
^^^^^^^^^^^^^^^^^^^^^^^^
Custom templates can be used in the same way as templates listed in existing [run.sh]_ scripts. During template development it might be handy to use ``make`` variables for quick prototyping:
.. code-block:: bash
make \
TESTS="sets/resolver" \
DAEMON="unbound" \
ADDITIONAL="-d -c unbound.conf" \
TEMPLATE="template/unbound.j2:template/hints_zone.j2" \
CONFIG="unbound.conf:hints.zone:ta.keys"
(These are the default values for Unbound.)
Debugging scenario execution
----------------------------
Output from a failed test looks like this:
......@@ -222,31 +246,27 @@ Output from a failed test looks like this:
.. code-block::
$ ./kresd_run.sh
[...]
[ FAIL ] sets/resolver/iter_cname_cache.rpl
sets/resolver/iter_cname_cache.rpl step 50 line 283, "rcode": expected 'NOERROR', got 'SERVFAIL' in the response:
id 12540
opcode QUERY
rcode SERVFAIL
flags QR RD
edns 0
payload 4096
;QUESTION
ns.bla.nl. IN AAAA
;ANSWER
;AUTHORITY
;ADDITIONAL
Traceback (most recent call last):
File "/home/pspacek/pkg/deckard/git/pydnstest/test.py", line 25, in run
test_callback(name, args, config)
File "/home/pspacek/pkg/deckard/git/deckard.py", line 290, in play_object
server.play(prog_under_test_ip)
File "/home/pspacek/pkg/deckard/git/pydnstest/testserver.py", line 198, in play
self.scenario.play({'': (subject_addr, 53)})
File "/home/pspacek/pkg/deckard/git/pydnstest/scenario.py", line 788, in play
raise Exception('%s step %d %s' % (self.file, step.id, str(e)))
In this example, the test step ``50`` in scenario ``sets/resolver/iter_cname_cache.rpl`` is failing. The binary under test did not produce expected answer, so either the test scenario or binary is wrong. If we were debugging this example, we would have to open file ``iter_cname_cache.rpl`` on line ``283`` and use our brains :-)
=========================================== FAILURES ===========================================
_____ test_passes_qmin_off[Scenario(path='sets/resolver/val_ta_sentinel.rpl', qmin=False)] _____
[...]
E ValueError: val_ta_sentinel.rpl step 212 char position 15875, "rcode": expected 'SERVFAIL',
E got 'NOERROR' in the response:
E id 54873
E opcode QUERY
E rcode NOERROR
E flags QR RD RA AD
E edns 0
E payload 4096
E ;QUESTION
E _is-ta-bd19.test. IN A
E ;ANSWER
E _is-ta-bd19.test. 5 IN A 192.0.2.1
E ;AUTHORITY
E ;ADDITIONAL
pydnstest/scenario.py:888: ValueError
In this example, the test step ``212`` in scenario ``sets/resolver/val_ta_sentinel.rpl`` is failing with query-minimisation off. The binary under test did not produce expected answer, so either the test scenario or binary is wrong. If we were debugging this example, we would have to open file ``val_ta_sentinel.rpl`` on character postition ``15875`` and use our brains :-).
Tips:
......@@ -254,78 +274,14 @@ Tips:
- network traffic from each binary is logged in PCAP format to a file in working directory
- standard output and error from each binary is logged into log file in working directory
- working directory can be explicitly specified in environment variable ``SOCKET_WRAPPER_DIR``
- environment variable ``VERBOSE=1`` forces extra verbose logging, including logs from all binaries and packets handled by Deckard
- command line argument ``--log-level DEBUG`` forces extra verbose logging, including logs from all binaries and packets handled by Deckard
Writting own scenarios
----------------------
See `the scenario guide <scenario_guide.rst>`_.
Direct Deckard invocation
-------------------------
.. warning:: Direct Deckard invocation is typically used only for development. The command line interface is not stable!
Usually Deckard is invoked using ``make`` or even higher-level scripts like [run.sh]_. The main reason is that each Deckard invocation requires ``LD_PRELOAD`` variable set to custom versions of C libraries which are used for environment simulation.
If you really have to, you can run Deckard directly. Single scenario can be executed in two distinct modes named ``one`` and ``multiple``:
- ``one`` - read all parameters from command-line and run one binary
- ``multiple`` - read parameters from YAML file and run multiple binaries
For all the details please see built-in help, run ``deckard.py --help`` to see it.
One binary mode (``one``)
^^^^^^^^^^^^^^^^^^^^^^^^^
This mode runs one binary with parameters specified on command line and is the most common usage of Deckard. It is used by supplied [run.sh]_ scripts (indirectly through ``make``). Use ``MAKEFLAGS`` to see what the [run.sh]_ script is executing:
.. code-block:: bash
$ MAKEFLAGS="--dry-run" ./kresd_run.sh
LD_PRELOAD=".../contrib/..." deckard.py sets/resolver/world_mx_nic_www.rpl one kresd template/kresd.j2 config --
Multiple binaries mode (``multiple``)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In this mode Deckard reads YAML configuration file and executes all binaries using parameters from the file. This is handy for testing interoperability of multiple binaries, e.g. when one program is configured as DNS recursor and other program is using it as forwarder.
The YAML file contains **ordered** list of binaries and their parameters. Deckard will send queries to the binary listed first.
.. code-block:: yaml
programs:
- name: forwarding # name of this Knot Resolver instance
binary: kresd # kresd is first so it will receive queries from Deckard
additional: []
templates:
- template/kresd_fwd.j2 # this template uses variable IPADDRS['recursor']
configs:
- config
- name: recursor # name of this Unbound instance
binary: unbound
additional:
- -d
- -c
- unbound.conf
templates:
- template/unbound.j2
- template/hints_zone.j2 # this template uses variable ROOT_ADDR
configs:
- unbound.conf
- hints.zone
- ta.keys
In this setup it is necessary to configure one binary to contact the other. IP addresses assigned by Deckard at run-time are accessible using ``IPADDRS`` `template variables`_ and symbolic names assigned to binaries in the YAML file. For example, template ``kresd_fwd.j2`` can use IP address of binary named ``recursor`` like this:
.. code-block:: lua
policy.add(policy.all(policy.FORWARD("{{IPADDRS['recursor']}}")))
When all preparations are finished, run Deckard using following syntax:
.. code-block:: bash
$ LD_PRELOAD=".../contrib/..." deckard.py --loglevel INFO scenario.rpl multiple config.yml
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment