One todo is about not working flush but that was resolved by
84e6788e and thus no longer  relevant.

The second one is about DNS to be specified to ISP but that is already
set to local ISP's address.

This test was written on old version of LXD API and wasn't updated.

This can potentially take a while thanks to slow connection or
repo.turris.cz load. Two minutes should be enough time to download
medkit and do preparation.

This changes check from carrier being up to just interface being up.
That is actually the correct check we want to do. The carrier might be
down if we are connected directly to board (not trough switch) and thus
this test might fail even if everything is all right. In reality we want
to check that interface is set to be up and thus is going to be up once
carrier is up. This information is encoded in flags as lowest bit (this
was discovered by checking flag value rather not from documentation). So
this test now only checks if this bit is set or not.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

This allows better logging control when we run nsfarm tool directly.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

This changes the order of setup. The original pretty much relied on
configuration change happening before service actually started and
system booted. This of course can't be ensured and is pretty fragile.

In general we can change file access or any file content without waiting
for system boot but once we want to communicate with services or to
access the Internet we need to wait for system to actually boot.

The clean effect here is the need to reload networking service once we
modify interfaces after boot. This should have always been there.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

These tests aim to check if firewall is well configured in such a way
attackers can be caught by Sentinel.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

This way we can have easy access to ISP container as well as to WAN
configuration.
This is required as we have to know WAN IPv4 address in some generic
way. We could define constant but this way it is prepared for standard
ISP being based rather on DHCP in the future.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

This is only alias for sending ^C but this way it is way less cryptic in
tests them self.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

The idea behind this is that we have more descriptive names of marks and
thus it should be easier to understand why some tests are marked with
given combination of boards as they are.
At the same time it should be easier to go trough them at one location
once we add new boards. The new board would have to be selectively added
to appropriate marks and all tests using these marks would be
automatically handled. Of course that does not solve it fully but still
it reduces the burden on introducing new boards.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

This removes concept of exclusive device. The only use of it was to pass
exclusive access to network interface but that is not essentially
required as it is even more versatile to use macvlan as thus we can
easily spawn multiple containers to simulate network.
The only known use for physical device pass trough and thus exclusive is
Wi-Fi. It won't be possible to use macvlan for it. At the same time this
is not an issue as it is not expected that we are going to be reusing
this interface in single tests run multiple times over and over. In the
end there is no need to automatically suspend containers to steal
devices as it has been implemented (and in reality not finished).

The introduced device management now required all devices to be defined
in image as attributes. This gives image definition control over name of
this device in container. It is up to container user to assign
appropriate real device for it. This is done using device map that is
simply pair of attribute and real device specifier. This concept can be
in future expanded to even encode additional configuration if have need
for it.

We do not need pytest-html for normal functionality. It is just nice
plugin we can use to generate human readable report to go alongside with
pytest native xml and log output.
We also support only pytest-html with minimal version 2.1.0 and thus all
this is included only if pytest-html is at least of that version.

The plugin is removed by this from requirements.txt as this file lists
necessary Python packages and this is now only optional.

Karel Koci authored 3 years ago and Karel Koci committed 3 years ago

The tests added in df0bf8d057735d316fc35f36928b2875b39e3f87 made it
necessary to start using deploy mark. The deploy mark is used on tests
needed to pass for deploy to proceed. These should be minimal necessary
tests and should be fast enough.
The reasoning why we need it here is because in most cases it should be
enough the check in the test_no_wan that runs fast. The additional
slower but more general test TestNoInternetAccess should cover only
corner case and thus is not essentially required as a blocker for
deploy.

Karel Koci authored 3 years ago and Karel Koci committed 3 years ago

This adds additional test that tries if router really does not try to
receive address from DHCP even if not configured. This is even more
powerful test but thanks to need to blindly wait it can take some
considerable amount of time and thus does not replace test_no_wan but
complements it.

Karel Koci authored 3 years ago and Karel Koci committed 3 years ago

This checks if there is no configuration fo WAN after clean boot. This
is intended as default that foces user to go immediately trough first
setup guide and configure router and primarily the administration
password.

Josef Schlehofer authored 3 years ago and Karel Koci committed 3 years ago

I tried to run pip command on Debian Bullseye and it failed with
following output:

Collecting pytest>=5.0 (from -r requirements.txt (line 1))
  Could not find a version that satisfies the requirement pytest>=5.0 (from -r requirements.txt (line 1)) (from versions: 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.4.0, 2.4.1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.8.5, 2.8.6, 2.8.7, 2.9.0, 2.9.1, 2.9.2, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 3.3.0, 3.3.1, 3.3.2, 3.4.0, 3.4.1, 3.4.2, 3.5.0, 3.5.1, 3.6.0, 3.6.1, 3.6.2, 3.6.3, 3.6.4, 3.7.0, 3.7.1, 3.7.2, 3.7.3, 3.7.4, 3.8.0, 3.8.1, 3.8.2, 3.9.1, 3.9.2, 3.9.3, 3.10.0, 3.10.1, 4.0.0, 4.0.1, 4.0.2, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.3.0, 4.3.1, 4.4.0, 4.4.1, 4.4.2, 4.5.0, 4.6....

Version 2.0.x does not support hook - pytest_html_report_title.
It fails with:

INTERNALERROR> pluggy.manager.PluginValidationError: unknown hook 'pytest_html_report_title' in plugin <module 'conftest' from '/foo/nsfarm/conftest.py'>

Karel Koci authored 3 years ago and Karel Koci committed 3 years ago

This explicitly enables DNSSEC validation and adds additional option to
conform to RFC1035.
It also uses named.ca file shipped as part of package as root hints.
We can safely left out `listen-on` as `listen-on-v6` uses dual-stack and
thus listens on IPv4 as well as on IPv6.

And the last but the most important change is disable of IPv6. This
solves issues on IPv4 only network but IPv6 once we begin to support
IPv6 we should allow disable/removal of this line.

Karel Koci authored 3 years ago and Karel Koci committed 3 years ago

This fixes commit: 57f0b251
It added Bind as ISP's DNS resolver but open in firewall was missing.

This only installs Sentinel components and checks if appropriate
services started.

This reason for this is that we use at some point serial console only to
get logs. We do not process it further. There is no reader for serial
console at most of time.
This change allows disable of serial console input and serial console
logs are thus only logged without further propagation.

This checks not only if services we want are running but also if they
are enabled.
The real reason for this is not to cover issue of disabled but running
service as that is highly unlikely. This rather covers services that
spawn actually no process but still we have to check if they are enabled
and thus executed in some manner. There is a lot of such services.

It should be enough to just download index for now. It checks only if we
are able to access HTTP, nothing more.

The issue was invalid condition as well as that if all input was
consumed that linebuf stayed set to old (already consumed) value.

This is negation of board mark. This should help us to limit tests not
only to explicit list of boards but also do exclusion list.

The immediate usage of this is in DNS resolver processes check.

The idea of having one module called toolbox is simply to share common
short but still little bit complicated functions in multiple tests.

This also implements first test using this function. It checks if all
services we expect to run after router start are actually running.

It is common that tests are run with logs printed to terminal. It helps
developer to see what is actually happening.
The problem is that log printing interferes with terminal output. It
should be safe to set level of root logger such it prints no messages
for time of mterm execution.

The message was being printed almost right on prepare method enter. The
problem is that it was called before parent image prepare method was
called. In effect this generated messages in reverse order. It also made
it pretty much impossible to identify which container exactly failed to
prepare as warnings for all images were printed even before any of them
started actually preparing. This moves it after parent prepare method
call and thus makes it so prepare messages are printed in order and
right before actual work is being performed.

This also gives us the option to include full container name we use for
bootstrapping.

Originally mterm was using just semicolon but bare semicolon is invalid
in shell so it prints error. This instead passes decision of command to
be used to specific Cli implementations.

The nsfarm library tools can be invoked by: python3 -m nsfarm
This is pretty simple but it is not directly visible, the documentation
has to be investigated to found this out, as well as not exactly short.
Having this short script should do us no harm.

This should prevent hopefully failures when network is not yet up in the
client container.

It tests only if hardware type is correctly reported and if appropriate
serial number is returned.

This adds new requirement for configuration and that is serial number.
We need this to verify that crypto-wrapper correctly returns appropriate
number.

This improves XML and HTML reports. It is minor expansion with required
info: tested branch and target.

The only possible execution right now is with serial and wan present.
In reality we need lan1 as well but we have to get around that because
CPU only Mox.
This simply removes parameters and marks for appropriate fixtures.
The effect is just less marks as well as better tests reports as they no
longer report None parameters.

It seems that pytest_html is now included automatically and including it
second time here produces warning.

This reduces user's overhead to launch tests. It uses targets filtering.
This makes target specification optional.
At the moment user can specify that he wants to run tests on specific
board. In the future we should add filters for additional configured
board features, such as presence of Wi-Fi cards or USB storage device.

This not only moves target config parsing to nsfarm library but also it
changes it to be implementation defined not just access to configparser.

This should suppress mod of the day and other messages just to make logs
little bit less duplicate.