Snippets Groups Projects

GitLab now enforces expiry dates on tokens that originally had no set expiration date. Those tokens were given an expiration date of one year later. Please review your personal access tokens, project access tokens, and group access tokens to ensure you are aware of upcoming expirations. Administrators of GitLab can find more information on how to identify and mitigate interruption in our documentation.

Self sign-up has been disabled due to increased spam activity. If you want to get access, please send an email to a project owner (preferred) or at gitlab(at)nic(dot)cz. We apologize for the inconvenience.

Compare History

nsfarm/lxd: remove exclusive devices and define all devices in image

Karel Koci authored 3 years ago

This removes concept of exclusive device. The only use of it was to pass
exclusive access to network interface but that is not essentially
required as it is even more versatile to use macvlan as thus we can
easily spawn multiple containers to simulate network.
The only known use for physical device pass trough and thus exclusive is
Wi-Fi. It won't be possible to use macvlan for it. At the same time this
is not an issue as it is not expected that we are going to be reusing
this interface in single tests run multiple times over and over. In the
end there is no need to automatically suspend containers to steal
devices as it has been implemented (and in reality not finished).

The introduced device management now required all devices to be defined
in image as attributes. This gives image definition control over name of
this device in container. It is up to container user to assign
appropriate real device for it. This is done using device map that is
simply pair of attribute and real device specifier. This concept can be
in future expanded to even encode additional configuration if have need
for it.

4f971f15

Name	Last commit	Last update
docs
imgs
nsfarm
selftests
tests
.gitignore
README.md
conftest.py
pytest.ini
requirements.txt
targets.example.ini
tool.sh

NSFarm (Network System testing Farm)

NSFarm is system testing tool for routers. It is intended as external device testing tool but it does not do exactly blackbox testing. Its target is to do testing of complete software stack of router.

Requirements

NSFarm utilizes tools present on standard Linux based PC. It is based on Python3, pytest, pexpect and LXD.

You need following software and its dependencies:

Python3 (>=3.6)
pytest (>=5.0)
pytest-html (>=2.0)
pexpect
pyserial
pylxd

You can install all Python libraries using pip3 install -r requirements.txt. It is suggested to either install them to system or to use virtualenv.

You should also make sure that LXD is up and running on your system and is correctly configured. Please see appropriate documentation on how to configure it.

In terms of hardware you need target router board of course. It has to be connected with appropriate USB-to-TTL converter and with at least one Ethernet port for WAN. The USB-to-TTL convert has to have CST and RST signals. For more info about hardware connection please see hardware wiring.

Usage

To run tests you have to have configuration file. That file defines available target boards tests can be run on. It is in INI file format. Please see targets.example.ini for available options.

To run full test sweat it is as easy as running pytest in root directory of NSFarm. You have to provide configured target with -T switch.

pytest -T omnia

You should read tests writing guide to see how you can write more tests and/or to understand current ones.

Print live logs

NSFarm logs during its execution everything it can. This can later be used to diagnose test failures. It is also nice to see how your tests progress so you can enable live logging on console when tests are running. That can be done as follow:

pytest -T omnia --log-cli-level=DEBUG

You can choose any Python logging compatible level.

In rule of thumb the levels used in NSFarm are as follow:

DEBUG: reports of NSFarm state changes and in general reports to be used when NSFarm behavior is diagnosed.
INFO: on this level all console communication is logged.
WARNING: some non-standard behavior proceeds (like waiting for other instance and so on).
ERROR: non-standard situation encountered (like concurrent instance is running).

Non-standard behavior and situations noted here are meant as something that is outside of tests scope but affects testing. The common effect is that tests execution can't proceed and NSFarm has to wait for something to complete.

HTML report

Using pytest-html it is possible to generate nice HTML report of test run.

pytest -T omnia --self-contained-html --html=report.html

You can change report.html to any other name or path.

NSFarm utility

nsfarm is not only library but serves at the same time as utility Python program. It implements some common operations you might want to use.

To invoke NSFarm utility you can use script tool.sh. Use ./tool.sh -h to see documentation and abilities of this utility.

How it works

TBD