unify packaging tests

Currently, there are two different implementations of packaging tests:

• distro/tests

  • depend on external repos (currently OBS)
  • use VMs, because we need to test systemd integration
  • are pretty much exactly what the user would use
  • primarily test default config

• tests/packaging

  • use docker containers
  • added in !892 (merged) and !947 (merged)
  • test if resolver starts up with given config
  • can't detect certain type of failures (crash shortly after startup, e.g. due to TA update)
  • test loading of optional modules

I think tests/packaging are redundant and they add additional maintenance burden (see my opinion in !947 (comment 154887)). However, they do test that all supported modules can be loaded in config. This functionality should be moved and unified with distro/tests as additional test, so we don't have to maintain two separate implementations of these tests.

added tests label

Yes, tests should be independent on particular environment, be it metal, VM, container, or whatever else.

As tests are written in ansible, it should be just a matter of running it on correct target.

I see 3 major use cases here:

• interactive CLI: during development etc, now handled by distro/tests/test-distro.sh
• VM: tests are run on a virtual machine (we have local vagrant option or equivalent in OBS)
• container: tests are run in a (docker) container, this is now handled by tests/packaging but should reuse the same ansible roles

EDIT: remove tasks made obsolete by @tkrizek's comment

I'm not sure I'd bother with being able to run just a single test or running them locally... The distro/tests require a package, which is currently built in OBS (and that takes a while). Once you have the package, running all tests is really quick.

Let's keep in mind the primary goal of these tests. They are functional test that verify the package works out of the box on all distros, in the ways the user would use them. This is also the reason systemd is important. I wouldn't implement the container use case at all.

I'm not sure I'd bother with being able to run just a single test or running them locally...

That's for developers - for example I'm missing this when writing new test and the time of testing one code iteration is slow because all tests are run and also it's hard to parse all the ansible output, I'd strongly prefer to be able to run just and only the code I'm testing as opposed wasting my cycles with running (and parsing output of) ALL THE CODE.

Nonetheless, that's an optional feature.

Let's keep in mind the primary goal of these tests. They are functional test that verify the package works out of the box on all distros, in the ways the user would use them. This is also the reason systemd is important. I wouldn't implement the container use case at all.

Correct. I'd consider implementing container tests only if they provided significantly faster response time to code changes (again dev scenario), but the maintenance burden is high, so if there is no explicit need for them, it's definitely better not to bother and use VMs which are slower but closer to real machines people will use our packages on.

mentioned in commit 4c4453c1

mentioned in commit f412c411

mentioned in merge request !1118 (merged)

added packaging label

mentioned in merge request !1178 (merged)

mentioned in issue knot-resolver-manager#5 (closed)

For the record, there's now a 3rd way (proof-of-concept) to run packaging tests using apkg & LXC - hopefully, this is a transitional phase and all packaging tests will eventually be unified to use this solution.

Details: !1178 (comment 215301)

mentioned in merge request !1193 (closed)

Possible use-case to keep in mind: Executing our config test suite as part of the packaging tests.

The main requirement of our config test suite is that kresd has to be installed, which is why it can't be executed during packaging itself (at least in rpm). See !1193 (comment 220325)

We've decided to try reusing Debian autopkgtests format and integrate in apkg test - see apkg#65.

mentioned in commit b87c5f6a

mentioned in commit e47fe5f7

mentioned in commit 1d5b1b9d

mentioned in commit 126d11b8

mentioned in commit cc5a42e5

mentioned in commit d9bb8832

mentioned in issue #744 (closed)

mentioned in commit 8d79236b

mentioned in commit 0c93eb66

mentioned in commit eb4e0ba4

mentioned in commit 69b35f9f

mentioned in commit 06bbae75

mentioned in commit 4d629f46

mentioned in commit 3e20377b

mentioned in commit fd94b77c

mentioned in commit e1db5363

mentioned in commit 87d57ead

mentioned in commit 2c0f3483

mentioned in merge request !1341 (merged)

mentioned in commit 384864e1

mentioned in commit 08b77d9a

mentioned in commit e140dd46

mentioned in commit bbec8667

mentioned in commit 44dced37

mentioned in commit 9241dd1a

mentioned in commit 99b14e09

mentioned in commit ec1265fd

mentioned in commit 9c35e502

mentioned in commit 336735ca

mentioned in commit 3441c352

mentioned in commit 2a3a13f8

mentioned in commit 646a69ec

This is finally resolved by !1341 (merged) after a long journey to get there with apkg 🥳

closed