1. 15 Apr, 2021 8 commits
    • Karel Koci's avatar
      tests: move target specification to top level · 8cc886cb
      Karel Koci authored
      This allows to run selftests not on all available targets but only on
      specified one. This way we do not pollute the results with test of any
      other target.
      
      This also improves marks initialization. Marks with limited effect
      (those are lan1 and lan2) are defined in runtime. This makes their
      definition and effect at the same level and thus we should be able to
      better manage them.
      8cc886cb
    • Karel Koci's avatar
      selftests: add simple tests for cli shell · db1b9aab
      Karel Koci authored
      This is simple test on OpenWrt as well as on Alpine of nsfarm's Shell
      abstraction.
      db1b9aab
    • Karel Koci's avatar
      tests: use ping instead of sleep to check for network readiness · cae250c2
      Karel Koci authored
      There is an issue that although software reports interface being up a
      few initial frames won't pass. The original "fix" for it was to just
      wait a moment but that brakes depending on hardware we are running tests
      on. Although it is not ideal we instead use ping. Pinging our gateway
      that is our ISP container should be enough to make sure that connection
      is established.
      At the same time this is not in any way required for DHCP. The reason is
      because to get IP address we have to communicate with ISP and thus
      connection is established for sure. This means that any sleep is nor any
      other wait is required there.
      cae250c2
    • Karel Koci's avatar
      tests: wait for container boot or network on container spawn · 734abfcd
      Karel Koci authored
      This is causing issues because containers in some cases are not starting
      fast enough. We should always make sure that container is prepared for
      tests and for exactly that purpose wait scripts available.
      734abfcd
    • Karel Koci's avatar
      nsfarm/board/omnia: fix invalid wan interface · 089093f1
      Karel Koci authored
      The Omnia wan interface is eth2 not eth0.
      089093f1
    • Karel Koci's avatar
      tests/conftest: solve some todos · 38077ba8
      Karel Koci authored
      One todo is about not working flush but that was resolved by
      84e6788e and thus no longer  relevant.
      
      The second one is about DNS to be specified to ISP but that is already
      set to local ISP's address.
      38077ba8
    • Karel Koci's avatar
      tests/test_bootup: fix invalid container setup for TestNoInternetAccess · 4f4f933a
      Karel Koci authored
      This test was written on old version of LXD API and wasn't updated.
      4f4f933a
    • Karel Koci's avatar
      nsfarm/board: image prepare can timeout sometimes as we are downloading medkit · bc7f6512
      Karel Koci authored
      This can potentially take a while thanks to slow connection or
      repo.turris.cz load. Two minutes should be enough time to download
      medkit and do preparation.
      bc7f6512
  2. 13 Apr, 2021 13 commits
    • Karel Koci's avatar
      selftests/target: improve check for interface being up · 1d74c1dc
      Karel Koci authored
      This changes check from carrier being up to just interface being up.
      That is actually the correct check we want to do. The carrier might be
      down if we are connected directly to board (not trough switch) and thus
      this test might fail even if everything is all right. In reality we want
      to check that interface is set to be up and thus is going to be up once
      carrier is up. This information is encoded in flags as lowest bit (this
      was discovered by checking flag value rather not from documentation). So
      this test now only checks if this bit is set or not.
      1d74c1dc
    • Karel Koci's avatar
      nsfarm: add logging configuration for nsfarm tool · bf74a307
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This allows better logging control when we run nsfarm tool directly.
      bf74a307
    • Karel Koci's avatar
      imgs/base-alpine: improve bootstrap script · e189836d
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This changes the order of setup. The original pretty much relied on
      configuration change happening before service actually started and
      system booted. This of course can't be ensured and is pretty fragile.
      
      In general we can change file access or any file content without waiting
      for system boot but once we want to communicate with services or to
      access the Internet we need to wait for system to actually boot.
      
      The clean effect here is the need to reload networking service once we
      modify interfaces after boot. This should have always been there.
      e189836d
    • Karel Koci's avatar
      tests/sentinel: add dynfw-client and minipot tests · 1cee8784
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      These tests aim to check if firewall is well configured in such a way
      attackers can be caught by Sentinel.
      1cee8784
    • Karel Koci's avatar
      tests: replace basic_isp fixture with two fixtures · 0b605356
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This way we can have easy access to ISP container as well as to WAN
      configuration.
      This is required as we have to know WAN IPv4 address in some generic
      way. We could define constant but this way it is prepared for standard
      ISP being based rather on DHCP in the future.
      0b605356
    • Karel Koci's avatar
      nsfarm/cli: add dedicated method to send ctrl+c character · 75a10b2c
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This is only alias for sending ^C but this way it is way less cryptic in
      tests them self.
      75a10b2c
    • Karel Koci's avatar
      tests/mark: add set of simple marks based on board mark · 7dde2147
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      The idea behind this is that we have more descriptive names of marks and
      thus it should be easier to understand why some tests are marked with
      given combination of boards as they are.
      At the same time it should be easier to go trough them at one location
      once we add new boards. The new board would have to be selectively added
      to appropriate marks and all tests using these marks would be
      automatically handled. Of course that does not solve it fully but still
      it reduces the burden on introducing new boards.
      7dde2147
    • Karel Koci's avatar
      nsfarm/lxd: remove exclusive devices and define all devices in image · 4f971f15
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This removes concept of exclusive device. The only use of it was to pass
      exclusive access to network interface but that is not essentially
      required as it is even more versatile to use macvlan as thus we can
      easily spawn multiple containers to simulate network.
      The only known use for physical device pass trough and thus exclusive is
      Wi-Fi. It won't be possible to use macvlan for it. At the same time this
      is not an issue as it is not expected that we are going to be reusing
      this interface in single tests run multiple times over and over. In the
      end there is no need to automatically suspend containers to steal
      devices as it has been implemented (and in reality not finished).
      
      The introduced device management now required all devices to be defined
      in image as attributes. This gives image definition control over name of
      this device in container. It is up to container user to assign
      appropriate real device for it. This is done using device map that is
      simply pair of attribute and real device specifier. This concept can be
      in future expanded to even encode additional configuration if have need
      for it.
      4f971f15
    • Karel Koci's avatar
      tests: just rename files to be consistent · b36a51fc
      Karel Koci authored
      b36a51fc
    • Karel Koci's avatar
      conftest: make pytest-html optional · e7360410
      Karel Koci authored
      We do not need pytest-html for normal functionality. It is just nice
      plugin we can use to generate human readable report to go alongside with
      pytest native xml and log output.
      We also support only pytest-html with minimal version 2.1.0 and thus all
      this is included only if pytest-html is at least of that version.
      
      The plugin is removed by this from requirements.txt as this file lists
      necessary Python packages and this is now only optional.
      e7360410
    • Karel Koci's avatar
      tests: start using deploy mark · 34197d04
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      The tests added in df0bf8d057735d316fc35f36928b2875b39e3f87 made it
      necessary to start using deploy mark. The deploy mark is used on tests
      needed to pass for deploy to proceed. These should be minimal necessary
      tests and should be fast enough.
      The reasoning why we need it here is because in most cases it should be
      enough the check in the test_no_wan that runs fast. The additional
      slower but more general test TestNoInternetAccess should cover only
      corner case and thus is not essentially required as a blocker for
      deploy.
      34197d04
    • Karel Koci's avatar
      test_bootup: check not only for config for disabled network on boot · b592db4b
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This adds additional test that tries if router really does not try to
      receive address from DHCP even if not configured. This is even more
      powerful test but thanks to need to blindly wait it can take some
      considerable amount of time and thus does not replace test_no_wan but
      complements it.
      b592db4b
    • Karel Koci's avatar
      test-bootup: check if router does not specify WAN access after boot · 7888e362
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This checks if there is no configuration fo WAN after clean boot. This
      is intended as default that foces user to go immediately trough first
      setup guide and configure router and primarily the administration
      password.
      7888e362
  3. 12 Apr, 2021 1 commit
    • Josef Schlehofer's avatar
      README: prefer pip3 instead of pip · 32a51b55
      Josef Schlehofer authored and Karel Koci's avatar Karel Koci committed
      I tried to run pip command on Debian Bullseye and it failed with
      following output:
      
      Collecting pytest>=5.0 (from -r requirements.txt (line 1))
        Could not find a version that satisfies the requirement pytest>=5.0 (from -r requirements.txt (line 1)) (from versions: 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.4.0, 2.4.1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.8.5, 2.8.6, 2.8.7, 2.9.0, 2.9.1, 2.9.2, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 3.3.0, 3.3.1, 3.3.2, 3.4.0, 3.4.1, 3.4.2, 3.5.0, 3.5.1, 3.6.0, 3.6.1, 3.6.2, 3.6.3, 3.6.4, 3.7.0, 3.7.1, 3.7.2, 3.7.3, 3.7.4, 3.8.0, 3.8.1, 3.8.2, 3.9.1, 3.9.2, 3.9.3, 3.10.0, 3.10.1, 4.0.0, 4.0.1, 4.0.2, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.3.0, 4.3.1, 4.4.0, 4.4.1, 4.4.2, 4.5.0, 4.6.0, 4.6.1, 4.6.2, 4.6.3, 4.6.4, 4.6.5, 4.6.6, 4.6.7, 4.6.8, 4.6.9, 4.6.10, 4.6.11)
      No matching distribution found for pytest>=5.0 (from -r requirements.txt (line 1))
      32a51b55
  4. 10 Apr, 2021 1 commit
    • Josef Schlehofer's avatar
      requirements: bump pytest-html to 2.1.x version · 5d9d1ced
      Josef Schlehofer authored
      Version 2.0.x does not support hook - pytest_html_report_title.
      It fails with:
      
      INTERNALERROR> pluggy.manager.PluginValidationError: unknown hook 'pytest_html_report_title' in plugin <module 'conftest' from '/foo/nsfarm/conftest.py'>
      5d9d1ced
  5. 30 Mar, 2021 17 commits
    • Karel Koci's avatar
      imgs/isp-common: improve bind configuration · ac47a771
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This explicitly enables DNSSEC validation and adds additional option to
      conform to RFC1035.
      It also uses named.ca file shipped as part of package as root hints.
      We can safely left out `listen-on` as `listen-on-v6` uses dual-stack and
      thus listens on IPv4 as well as on IPv6.
      
      And the last but the most important change is disable of IPv6. This
      solves issues on IPv4 only network but IPv6 once we begin to support
      IPv6 we should allow disable/removal of this line.
      ac47a771
    • Karel Koci's avatar
      imgs/isp-common: fix access to DNS · a93ff988
      Karel Koci authored and Karel Koci's avatar Karel Koci committed
      This fixes commit: 57f0b251
      It added Bind as ISP's DNS resolver but open in firewall was missing.
      a93ff988
    • Karel Koci's avatar
      tests/sentinel: add basic tests for sentinel services · 06d5ea25
      Karel Koci authored
      This only installs Sentinel components and checks if appropriate
      services started.
      06d5ea25
    • Karel Koci's avatar
      nsfarm: allow and disable input read for serial console · 3c7282dd
      Karel Koci authored
      This reason for this is that we use at some point serial console only to
      get logs. We do not process it further. There is no reader for serial
      console at most of time.
      This change allows disable of serial console input and serial console
      logs are thus only logged without further propagation.
      3c7282dd
    • Karel Koci's avatar
      tests/test_bootup: check if services we want are enabled · 748f9db2
      Karel Koci authored
      This checks not only if services we want are running but also if they
      are enabled.
      The real reason for this is not to cover issue of disabled but running
      service as that is highly unlikely. This rather covers services that
      spawn actually no process but still we have to check if they are enabled
      and thus executed in some manner. There is a lot of such services.
      748f9db2
    • Karel Koci's avatar
      tests/test_bootup: add check for accessbile web interface · 1912c20a
      Karel Koci authored
      It should be enough to just download index for now. It checks only if we
      are able to access HTTP, nothing more.
      1912c20a
    • Karel Koci's avatar
      nsfarm/cli: fix delayed logging of PexpectLogging · 4ab73125
      Karel Koci authored
      The issue was invalid condition as well as that if all input was
      consumed that linebuf stayed set to old (already consumed) value.
      4ab73125
    • Karel Koci's avatar
      tests: add not_board mark · cebecdc2
      Karel Koci authored
      This is negation of board mark. This should help us to limit tests not
      only to explicit list of boards but also do exclusion list.
      
      The immediate usage of this is in DNS resolver processes check.
      cebecdc2
    • Karel Koci's avatar
      nsfarm/toolbox: add check for running service · 5909ade1
      Karel Koci authored
      The idea of having one module called toolbox is simply to share common
      short but still little bit complicated functions in multiple tests.
      
      This also implements first test using this function. It checks if all
      services we expect to run after router start are actually running.
      5909ade1
    • Karel Koci's avatar
      nsfarm/mterm: suppress logging · 71785f37
      Karel Koci authored
      It is common that tests are run with logs printed to terminal. It helps
      developer to see what is actually happening.
      The problem is that log printing interferes with terminal output. It
      should be safe to set level of root logger such it prints no messages
      for time of mterm execution.
      71785f37
    • Karel Koci's avatar
      nsfarm/lxd: improve warning about image bootstrap · f3f0e88f
      Karel Koci authored
      The message was being printed almost right on prepare method enter. The
      problem is that it was called before parent image prepare method was
      called. In effect this generated messages in reverse order. It also made
      it pretty much impossible to identify which container exactly failed to
      prepare as warnings for all images were printed even before any of them
      started actually preparing. This moves it after parent prepare method
      call and thus makes it so prepare messages are printed in order and
      right before actual work is being performed.
      
      This also gives us the option to include full container name we use for
      bootstrapping.
      f3f0e88f
    • Karel Koci's avatar
      nsfarm/mterm: fix invalid semicolon command in Shell · 93573aef
      Karel Koci authored
      Originally mterm was using just semicolon but bare semicolon is invalid
      in shell so it prints error. This instead passes decision of command to
      be used to specific Cli implementations.
      93573aef
    • Karel Koci's avatar
      tool.sh: add script to access more easilly nsfarm library tools · de5188d5
      Karel Koci authored
      The nsfarm library tools can be invoked by: python3 -m nsfarm
      This is pretty simple but it is not directly visible, the documentation
      has to be investigated to found this out, as well as not exactly short.
      Having this short script should do us no harm.
      de5188d5
    • Karel Koci's avatar
      tests/network/test_lan: wait for network before continuing · b0828c9b
      Karel Koci authored
      This should prevent hopefully failures when network is not yet up in the
      client container.
      b0828c9b
    • Karel Koci's avatar
      test_cryptowrapper: add some basic tests for crypto-wrapper · 8cc46caa
      Karel Koci authored
      It tests only if hardware type is correctly reported and if appropriate
      serial number is returned.
      
      This adds new requirement for configuration and that is serial number.
      We need this to verify that crypto-wrapper correctly returns appropriate
      number.
      8cc46caa
    • Karel Koci's avatar
      tests: improve reports · 466e943b
      Karel Koci authored
      This improves XML and HTML reports. It is minor expansion with required
      info: tested branch and target.
      466e943b
    • Karel Koci's avatar
      tests: remove serial and wan markers · 65c8fea6
      Karel Koci authored
      The only possible execution right now is with serial and wan present.
      In reality we need lan1 as well but we have to get around that because
      CPU only Mox.
      This simply removes parameters and marks for appropriate fixtures.
      The effect is just less marks as well as better tests reports as they no
      longer report None parameters.
      65c8fea6