Base integration tests on proper packages

I think if you want to test with real packages, the process to create them should probably look something like:

clone main kresd repo (master?), where manager would be added as submodule
fetch and checkout manager submodule to the version you intend to test
run packaging process (apkg build-dep, apkg build) in kresd repo to create development packages of kresd -- apkg support should be merged before the end of month (knot-resolver!1178 (merged))

At first I thought that you suggestion is overkill. But I changed my mind after working for a bit on !4 (closed) and now I think, that you are right. The situation wouldn't be simplified by ignoring the build system merging problem. So let's figure out how to solve #7 (closed) first and then continue with this issue.

mentioned in merge request !4 (closed)

When working on !4 (closed), I realized that there are problems worth talking about a bit more...

The problem, this is trying to solve

The integration tests now work symbolically like this - code -> shitty copy-pasted shell script -> running system. This would change the situation into code -> package build system -> running system. Therefore, we would be testing not only the manager itself, but the build system as well. The test system would then exactly match the state end users would see. This makes errors found in integration tests much more representative of the problems encountered in real-world.

What we want

It must be possible to run integration tests on every distro. It should be as easy as possible in order to make it approachable by any developer after installing just a few dependencies.

(Due to this reasoning, we run integration tests in containers.)

Problems, this introduces

This will put another step in between the actual tests and the tested code. Building packages takes time and it would effectively double the time it takes to run the tests.
Ideally, we would be able to build all packages without dependencies on the outside system. That's sadly not true. Therefore, for every package we build, we need a container with an appropriate distro (and appropriate version). Without this, we would break the goal of making integration tests runnable on any system.
Let's assume we will use containers for building packages on any base system. That wouldn't however mean we can just ignore native builds. We would like to merge packaging of kresd with packaging of the manager and forcing usage of containers onto kresd developers is ?probably? undesirable. Effectively, we can create nice polished system which handles everything automatically using containers. And then somebody would still have to build natively with all dependency problems native builds introduce, completely sidestepping all the work made to make it simple.

Possible solutions

We can just close this issue and keep the current state with shell scripts configuring the test systems. Packaging would be solved natively, without containers, effectively making it impossible for developers to test it out on their machines.
We can implement packaging within containers. Due to expected similarity between container distros and the real distros, the same code should "just work" natively. This would make integration test time much longer.
1. The containerized packaging could work for kresd as well, wrapping it's build system as well and effectively making it possible to build all packages just about anywhere.
2. The containerized packaging could be made just for manager's needs with integration tests.

My opinion

I think that solution 2.1 would be the best. It's effectively the same amount of work as 2.2 while giving the benefits to more people.

On the other hand, I feel like I am trying to make sense of a super-complex world of packaging and breaking assumptions and procedures that have been tried and tested. It feels really stupid to me, that I need several systems just to build different versions of almost the same compressed archive. But I guess that's how it works now. If the current state is desirable, then the only solution is 1 - just don't do integration tests on packages.

I am now undecided. At first, merging packaging and integration tests seemed like a good idea. But doing it might not be worth it at all. It might be a lot of work for not much gain.

@tkrizek @jruzicka What are your takes on this?

Packaging Tests Rule

Testing the entire distribution pipeline as opposed to ignoring packaging aspect adds value beyond measure.

Let's definitely do that if possible 🦾

This is related to unification of kresd packaging tests: knot-resolver#612 (closed)

Direct vs Containers

Ideally, tests should be independent on the particular testing machine - invocation should be the same in container, VM, and bare metal.

A developer should be able to run a single (or few) command(s) to reproduce CI run. For example apkg is using pytest for this so reproducing a failed CI run is a matter of:

pytest ci/tests

on a configured system.

Testing in containers is a de-facto standard for today's CI but we must focus on making tests system-agnostic in order to allow manual reproduction as well as support different testing setups.

We shouldn't force containers as a test requirement, only as a supported option.

System Setup vs Test Run

In order to address all mentioned problems, I suggest we split the problem of testing into 2 distinct steps:

system setup: different on each system (local / container / VM)
test run: should be the same command regardless of host system

When designing an universal testing system, these MUST be treated as separate problems.

Requirements Installation

The most import part of system setup is to install requirements for the software in question to build, install, and run.

When using apkg, these requirements can be defined in native distro package format (Requires and BuildRequires) in distro/pkg and then installed on-demand:

apkg build-dep

Furthermore, apkg provides a command to install system-specific packaging-related requires:

apkg system-setup

When requirements are correctly defined in upstream packaging, these commands should prepare any system (local, container, VM) for packaging.

After requirements are in place, packages can be created:

apkg build

Integration tests can be ran after testing packages are installed.

To sum up, with apkg the system setup boils down to:

apkg system-setup
apkg build-dep
apkg build | apkg install

This enables automatic CI image creation as well as local reproducibility.

With apkg providing means to install requirements on local machine as well as in containers, there is no need for complex layers of image/container/VM management.

Integration vs Packaging Tests

I see Packaging Tests as a subset of Integration Tests. For most users, packages are primary means of distribution and thus in order to truly test SW integration, packaging MUST be tested as well.

Packaging is indeed super-complex but that's because it tries to solve the super-complex problem of SW distribution in the wilderness. It's in fact so complex and important that it should be tested.

apkg is designed to help bring packaging testing into standard CI pipelines without suicides.

However, including packaging testing changes the flow of CI from:

system setup -> build -> test (project-local)

to

system setup -> pkg build -> pkg install -> pkg test (system-global)

Package build is a superset of normal build, but both flows can be tested in parallel or mixed together as desired, although it's kind of redundant.

BIRD CI tests both paths.

Local vs Gloal (System-wide) testing

I think the biggest challenge is to write integration tests in a way that allows invocation from local repo as well as on real system with packages installed globally.

For example, kresd packaging tests are ansible-based and could in theory be ran on localhost, but they include global commands such as systemctl restart which shouldn't be called when testing kresd locally without it being installed.

Supporting both flows in one test would require significant effort and I haven't discovered any good tools for this, so I currently incline towards strict separation of local tests such as unit tests and global tests such as integration/packaging tests.

I think integration (global) tests should assume the tested software is installed (usually but not necessarily via a package) and test accordingly as a developer would on a normal system with the software package installed.

I think the issue here comes from trying to do everything at once. I believe we need to split the concerns between integration tests and packaging tests.

integration tests

a quick and painless way to run tests by the developer to check the code works
can be more comprehensive and do more granular testing
don't need to care in which environment they run - just run it locally after running kresd installation
the only difficult part is to figure out how to use the systemd (without breaking the machine)
testing is done from git repo and it uses (installed) code

packaging tests

these are for release engineers, packagers and CI systems
their purpose is not to test a feature works (that is covered by integration tests), but that it works everywhere (i.e. on all supported platforms, distros, ...)
the environment they run in should be as close as possible to pristine environment a used would have
the source for this test is a package - the one a user would consume

In terms of coverage and comprehensiveness: integration tests >= packaging tests

The entire testing process is complex and many parties are involved, so let's clearly separate the responsibilities here.

@vsraier: integration tests only. I don't think you should worry about packaging tests, or packaging. Or having the whole container podman setup - at least definitely not in this repo.
@tkrizek / knot resolver team: packaging tests. And I mean with real kresd packages that you simply can't produce from this repo and we need #7 (closed) first. I won't release 6.0 without writing and running these checks anyway, so trying to do them here probably just duplicates the work that will have to be done elsewhere.
@jruzicka: the apkg toolchain that makes this simpler to do (quickly build packages), and perhaps eventually "normalizing" the packaging test framework into apkg feature

Next steps

simplify integration tests to just run the checks on the current machine
#7 (closed) to figure out how to integrate manager into kresd repo
figure out how to unify kresd packaging tests knot-resolver#612 (closed)
add packaging test coverage of manager inside kresd repo
use our LXC CI & apkg to build&run all the packaging tests

I think the issue here comes from trying to do everything at once. I believe we need to split the concerns between integration tests and packaging tests. [...] The entire testing process is complex and many parties are involved, so let's clearly separate the responsibilities here.

Thanks. This is probably the most important point I see. Clearly separating what everybody does would help immensely. You are right that I am probably trying to solve everything at once.

@vsraier: integration tests only. I don't think you should worry about packaging tests, or packaging. Or having the whole container podman setup - at least definitely not in this repo. [...]

Getting rid of the containers could be a bit of a problem. I am kind of scared of changing the managers code just to allow for local testing, because that could lead to differences in behavior when run locally and globally. I think that testing everything locally the way it will be run in production is much safer.

However, thinking out loud... In order to run it locally without containers, we would have to:

switch to using systemd in user mode (not sure about how the behavior differs, but it should not be that much)
switch from upstream kresd systemd unit files to modified copies
change the default used configuration of kresd
automate configuration of user services, effectively touching the host system outside of the boundaries of the repository

Hmm, that might be less of a problem than I thought it would be... It even looks that it should work regardless of the host distro.

So let's assume the integration tests run locally as well as the manager. We would dump the whole container setup. That would mean we can totally ignore packaging as there is only trivial amount of system setup (copying config files to something like .config/systemd). Effectively, it should simplify everything a lot as there would not be an inherent need for complex system setup automation.

Ok, I see that this approach could have more upsides than downsides. In this case, the next steps would be:

implement necessary changes for running locally
automate local system setup
change existing integration tests so that they all run locally
update CI once again, so that it matches this local workflow

And I wouldn't continue working on:

packaging (it would not be needed for development)
testing on different versions of dependencies, that would be left for further packaging tests
this issue exactly would become pointless

Do you agree with the steps as I've laid them out?

I think that approach is fine. I'm not very familiar with systemd's user mode -- but if it'd be possible to run the integration tests the same way either with --user or without it, they could be possibly re-used in the future as packaging tests.

And I wouldn't continue working on:

packaging (it would not be needed for development)

Agreed, packaging belongs to kresd repo.

testing on different versions of dependencies, that would be left for further packaging tests

Ok, I'd suggest you target the oldest versions/distro for your integration testing to avoid nasty surprises once we start testing it everywhere with packaging tests.

testing on different versions of dependencies, that would be left for further packaging tests Ok, I'd suggest you target the oldest versions/distro for your integration testing to avoid nasty surprises once we start testing it everywhere with packaging tests.

I agree. That's essentially the way the project is setup from the start. Sadly, I've already managed to run into a problem, where manager run fine on python 3.6 and 3.7, but not on anything newer. This one was my mistake, but it still took me a while to notice. However, I think that there is no other solution than run the whole integration tests once fully packaged. That should hopefully catch most of the problems.

By the way, everything is ready for a new way of executing packaging tests in kresd. This pipeline is executed for every MR. It builds packages and then installs them to run some tests (for every supported distro, currently amd64 only). The configuration looks like this.

Sounds good to me!

I think the issue here comes from trying to do everything at once.

Definitely, we need to correctly reduce/factor into smaller sub-problems. Otherwise a hydra will be born...

I believe we need to split the concerns between integration tests and packaging tests.

Agreed, but there are multiple ways to define these and their relation... for example both unit and integration tests could be run as a part of a packaging test which would make packaging tests a superset of integration (and unit) tests. OTOH packaging tests could also be run as a step of integration test reversing the relation to a subset. They can be completely independent too.

The definitions you provide are clear and meaningful so I say let's go with those and see how the overall structure develops.

In terms of coverage and comprehensiveness: integration tests >= packaging tests

That's generally the case in the wilds. I think ideally packaging tests = integration tests on packages but that might not be viable due to reliability, resource, and maintenance concerns. Ambition aside, having any packaging tests at all is already a great success so let's start small and see where it goes.

perhaps eventually "normalizing" the packaging test framework into apkg feature

That'd be worth a separate test framework/runner project which apkg can fully support.

Let's keep things simple each doing one thing well, feature creep was the downfall of rdopkg. apkg's job is to build packages from upstream sources and that's about it.

However, I've been thinking about such test framework/runner for quite some time... gitlab-runner and github-runner are the opposite of simple and they're very specific to their native environment. Something like that but generic, universal, and modular would be truly a game changer.

perhaps eventually "normalizing" the packaging test framework into apkg feature

That'd be worth a separate test framework/runner project which apkg can fully support.

Let's keep things simple each doing one thing well, feature creep was the downfall of rdopkg. apkg's job is to build packages from upstream sources and that's about it.

However, I've been thinking about such test framework/runner for quite some time... gitlab-runner and github-runner are the opposite of simple and they're very specific to their native environment. Something like that but generic, universal, and modular would be truly a game changer.

The described testing tool might be in a way the thing I was trying to develop. What I've created is general - the integration test tool could very likely work with completely different codebases and it does not have that much dependencies other than Python+Podman. It runs easily locally as well as in CI.

So, if we stick to the steps I wrote somewhere in the thread above, this "tool" would get scrapped. If you want to create something similar, this might be a starting point or at least another attempt from whose mistakes we could learn.

I ended up with a tool that is configured like this. Please note, that the README.md in the directory above is outdated and does not match how the tool actually works. And adding packaging tests to it was pretty much the point of this issue. What you see is pretty much a second version of the testing tool - the first had plenty of issues and I almost completely rewrote it.

The described testing tool might be in a way the thing I was trying to develop. What I've created is general - the integration test tool could very likely work with completely different codebases and it does not have that much dependencies other than Python+Podman. It runs easily locally as well as in CI.

That's what I thought, you were developing the manager AND a (nice) container test runner at once which to me sounds more risky than simply providing (local) test code and leave the testing to existing testing systems.

I am kind of scared of changing the managers code just to allow for local testing, because that could lead to differences in behavior when run locally and globally.

That's indeed an extra complexity, but also extra test coverage and TBH it seems less scary to me than relying on containers for testing entirely. Supporting that is awesome, requiring it is not. But maybe I'm too old-school expecting things to be modular so I can arrange them any way I want as opposed to getting a single monolith.

So, if we stick to the steps I wrote somewhere in the thread above, this "tool" would get scrapped. If you want to create something similar, this might be a starting point or at least another attempt from whose mistakes we could learn.

I like the idea of the test runner, I secretly hoped you'd split it into a separate project eventually :) Being able to run something like test debian-10 fedora-34 and getting the results of the testing through local containers would be very convenient, but that's a significant amount of effort put in something that's not strictly required thanks to GitLab CI and other existing container/VM tools. I'd be definitely interested in using such tool and contributing, but not in developing it as a core dev.

Consider dumping the testing code into a separate repo for later (re)use if you don't want to throw away your work on it or maybe add a tag on latest commit where it's still present so that we can continue later if needed. Or just purge it with fire, your call :)

Created a repository https://gitlab.nic.cz/knot/containerized-testing-tool with the testing scripts from here. Might be useless, might be not. We'll see.

mentioned in issue #6 (closed)

mentioned in issue #10 (closed)

Closing this issue as the main topic is no longer relevant. As we discussed, the integration testing infra will be replaced by something simpler. See #10 (closed) for progress on that.

closed

Base integration tests on proper packages

Child items

Activity

The problem, this is trying to solve

What we want

Problems, this introduces

Possible solutions

My opinion

Packaging Tests Rule

Direct vs Containers

System Setup vs Test Run

Requirements Installation

Integration vs Packaging Tests

Local vs Gloal (System-wide) testing

integration tests

packaging tests

Next steps

Admin message

Admin message

Base integration tests on proper packages

Child items

Linked items

Related merge requests

Activity

The problem, this is trying to solve

What we want

Problems, this introduces

Possible solutions

My opinion

Packaging Tests Rule

Direct vs Containers

System Setup vs Test Run

Requirements Installation

Integration vs Packaging Tests

Local vs Gloal (System-wide) testing

integration tests

packaging tests

Next steps