This does not check anything other than that all root USB hubs are
visible in lsusb. At the same time this should cover any error where USB
driver gets miss configured or broken.

The minimal requirement is that lspci should list SoC.

The broken PCI can also facilitate by not identified SoC PCI device.

Not every board has reasonable (huge as possible) TFTP block and window
size set. Thus the TFTP performance is hurt. Setting it explicitly makes
TFTP performance reasonable on all boards (cca 1.7MB/s).

The original idea of exit_code was to allow flexibility of handling exit
code. The true is that most of our uses were to set it to None to
disable the default handler to get exit code. This use case now covers
check attribute instead. Any other use can be covered by getting exit
code from run method and doing anything with it later on.

The check attribute was selected to be consistent with standard
subprocess.run.

This covers a known issue in the Turris Mox. Sometimes the U-Boot is
unable to bring up the ethernet device. The message is "bad rx status"
that repeats. The reboot of the board helps this implements that we
attempt the boot again if we encounter this issue.

This allows to also match additional expressions alongside of the
prompt. The idea is that we can use it to detect failure early and act
on it.

Sometimes the removal of IP from the set fails and that signals that for
some reason the set was either flushed or that IP address was just
simply removed from it. The only possible culprit is the
sentinel-dynfw-client and thus we disable it for duration of dynfw
firewall tests.

We sometimes want to change state of some services on the router. That
can be either to stop it temporally for test or start it. This
implements setup for exactly that.

The original 10 seconds were just too short time considering that
sometimes we have to reload newtork configuration before we proceede to
the next step in the guide. The tests that do that such as interfaces
were failing because of that. The one minit is reasonable time we can
give it before we consider it a failed state.

The nc does not return "succeeded" but rather "open" now, so relying on
exit code is much better.

This fixes invalid tests for password. We can use the complete password
and let crypt to parse it on its own without us doing it wrongly.

The check for output in test_attack_blocked had no effect. This changes
that and adds assert.

The sentinel-nikola is with Turris OS 5.3 relaced by sentinel-fwlogs.

This requires some explanation specially because I was unable to locate
the root cause of the weirdness. The initial find was in selftests. The
selftests were passing on their own but once they were included in one
test run with tests the test cli.test_shell.test_txt[OpenWrt] started
failing but in setup of the fixture shell. It seemed that the new shell
session was forked but not exactly correctly connected. The shell in the
end was unable to read anything. At the same time there was output from
previous pexpect instances (thanks to logging). So it seems like somehow
the pexpect instances were messed up. When I added log message before
return it turned out that not on every test it was called and log (where
there should have been busybox banner) confirmed that. The "fix" by
changing return to yield was discovered by random as I switched it
around to add also log message for teardown. That suddenly started
working and issue disappeared. I tested a multiple combination and I
can't found out what triggers this behavior. I really think that this is
a bug in Pytest. It for sure handles fixtures that are just functions
and not generators slightly differently. Just to be sure I switched all
returns in fixtures to yields.

There is a bug in (old) stock version of U-Boot's trusted firmware that
causes stall. The fix is to remove phy property from dtb. We do that
only on the old U-Boots and keep it as it is on new ones. This is
consistent with boot script behavior.

The WAN tests configure and reload network settings to test various
Internet access methods. The issue here is simply that after that is
done and configuration is reverted it takes some time before the
Internet access is reestablished from DHCP. Thus this twists existing
code to allow call to generic function for waiting for uplink in an
appropriate moment.

The next button text now contains space so xpath no longer matched. The
new xpath does not rely on text but is too general now. Unfortunately
there is probably no other way around this.

The reForis now allows guide skip immediately when password is set. Thus
the test_skip_after_password no longer fails.

This changes configuration for client container from static IP to DHCP.
The reason is because we spawn multiple clients in parallel (especially
with selenium) and having them all on same static IP does not work. At
the same time we want to have the primary client container on static IP
so when we test DHCP we won't break our primary connection to the
router.
Thus this converts the client-dhcp to client-static and makes client a
DHCP.

This also cleanups the client-static (previously client-dhcp) bootstrap
script. There is no need to install software that is already installed
in client.

This prevents unnecessary removals of Sentinel as the tests are going
trough different modules in sentinel test package.

This adds a way to select ISP and client container when booting board
using tool. The defaults are used as expected but any other image can be
specified instead. To disable the empty argument can be provided
instead.

To implement this in more generic way this also adds a helper method to
image to identify if it is child of some other image. This is after that
used in dedicated function in nsfarm.setup.uplink that provides a
selection of best uplink setup for given ISP container.

This fixes some of the warnings pydocstyle produces.

The common development workflow is to run selftest after changes. Such
significant change is for example update of base container. The issue is
that not every container is used in selftests and thus not every issue
is discovered early before tests on board are executed. It is possible
to run `./tool.sh lxd bootstrap -a` to check but that is easy to forget.
On the other hand this forces bootstrap of all images even if they are
not used in tests and thus possibly unnecessary work. That is a good
trade off for making sure that all needed images can be prepared.

This generalizes boot routine using FIT images. The only place we have
to correctly modify or create are its files at the boot container. The
rest should be generic.

Unfortunately not every board supports the U-Boot FIT images. Thus this
also introduces (hopefully only temporally) legacy boot that can be
enabled for targets.

This allows easy bootup to the specified branch. It provides a way to
access the board trough the client container and a way for board to
access the Internet.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

This does not address tests fixes. This only makes sure that Mox can
boot in NSFarm.

This also fixes bug in './tool.sh lxd bootstrap' that was discovered as
part of this update.

Jan Miksik authored 3 years ago and Karel Koci committed 3 years ago

Added DHCPv4 essential test that will be run
as part of deploy
Added static lease tests

This allows not only strings but also pathlib.PurePosixPath as that is a
common format we are using to work with paths.

This is rather about typing. The code itself works regardless thanks to
pathlib.PurePosixPath implementation.

This still sets the root password but that is only to unlock the
account. The key is used instead to authenticate and access the board.

This should increase tests resilience as we can now spawn new shell
instance for every test and thus taint of shell from one test can't
influence others.

This of course needs changes in existing tests and that is not part of
this code. The update of existing code and removal of client_board
fixture in favor of board_access should be done in the future. For now
client_board is documented as obsolete and not suggested for the new
code.

The disadvantage of this change is that every fixture that wants to do a
simple call (such as uci or cat) has to create its own connection and
that can get spammy pretty fast. The solution for this is the new
fixture fixture_board_access. This provides one shell instance. In
reality it provides shell instance that can reconnect. This is due to
issues when client container would change its IP address (for example
when we change LAN addressing).

These setups are handy in general. They created files and diretories
that are automatically removed on revert.