RFE: apkg test

Implement a packaging testing command apkg test based on Debian autopkgtest.

apkg test should:

• install testing requirements
• run all tests
• present test result in a nice readable format

A mechanism to define and select proper test requirements is needed. In autopkgtests this is done through debian/tests/control file, but that's distro-specific, we need to efficiently define requirements for all supported distros.

changed milestone to %v0.4

added enhancement tests labels

mentioned in issue #5 (closed)

mentioned in issue knot/knot-resolver#612 (closed)

It might also useful to be able to run some pre-setup script, e.g. to set up external repositories for the required packages.

After some discussion, I withdraw this proposal. Provisioning of the system (incl. proper repository setup) should happen outside apkg.

Tests with exit code 77 should be marked as skipped instead of failed. meson does the same thing and it seems to be some GNU convention.

I'd expect that typically apkg would not run individual tests directly but rather through some system like meson which ensures skipping, scheduling, reporting the result summary, etc.

assigned to @jruzicka

Perhaps installation of test dependencies could be handles by a separate command. E.g. apkg test-dep, similar to apkg build-dep

mentioned in issue knot/knot-resolver#713 (closed)

test dependencies specification

In Debian autopkgtest, test dependencies are specified using a special debian/tests/control file like this one from Knot DNS:

Tests: kdig
Depends:
 ca-certificates,
 knot-dnsutils,

Tests: authoritative-server
Depends:
 findutils,
 knot,
 knot-dnsutils,

apkg test should reuse this syntax by providing a default tests/control file with per-distro templating as well as a way to specify per-distro tests control file consistent with template selection mechanism from distro/pkg.

So far I'm considering two possible layouts for storing per-distro test control files:

control files named per-distro (compact)

tests
├── control   <- default control
├── deps
│   ├── arch  <- control for arch
│   ├── deb   <- control for debian systems
│   ├── el-7  <- control for defined apkg distro alias el-7
│   └── rpm   <- control for rpm systems
├── test-a
└── test-b

control files in per-distro dirs (consistent)

tests
├── control
├── deps
│   ├── arch
│   │   └── control
│   ├── deb
│   │   └── control
│   ├── el-7
│   │   └── control
│   └── rpm
│       └── control
├── test-a
└── test-b

Advantages at the cost of extra dir level:

• structure consistent with distro/pkg
• control files are actually named control
  • easier to understand nature of the file at first glance
  • can easily diff against debian/tests
  • auto syntax highlighting in text editor (for Debian control files)
  • ability to override other test files than control per-distro in one place in the future

Which layout do you prefer?

Do you have better suggestions for the name of distro/tests/deps dir?

• I don't think there needs to be a default control file. The test dependencies will almost always have different names across distros.
• The second solution is probably more flexible in case we'd want to extend the functionality in the future (e.g. support fedora's standard test interface)
• deps/ dir name ideas: config/, meta/, files/?

If the test is defined via the control file, is it possible to have additional files that are part of the test?

There might be use-cases where it could make sense to have additional non-test files, e.g. to set-up the environment and then source it from the actual tests to avoid duplication of code.

If no default file got provided, symlinks could be easily used instead.

I don't think there needs to be a default control file. The test dependencies will almost always have different names across distros.

If no default file got provided, symlinks could be easily used instead.

That's true, but people may still want to define all test deps in one file using templating conditionals like {% if distro.match('fedora') %} and diff that against downstream debian/tests/control to keep them in sync easily as opposed to having multiple copies. This approach is supported throughout apkg so I'd like to have that option in tests as well, but I wonder how useful it is as an implicit default

Another fairly big advantage for default control would be that simple cp -r ~/debian/foo/debian/tests distro/tests would result in a working configuration on some debian systems without any effort in the spirit of try and ask for forgiveness and trying to be useful by default without configuration. Other distros would be probably (but not neccessarily for simple projects with few commod deps) failing to install deps and thus requiring either templating in distro/tests/control or per-distro dirs as illustrated above. Both approaches have their use cases, and they can be freely combined at will.

The second solution is probably more flexible in case we'd want to extend the functionality in the future.

It definitely is, distro/tests/extra/$DISTRO/* can be copied on top of distro/tests/ base in a temporary testing dir to get a fair degree of per-distro control/freedom.

So we're all in favor of consistent solution with subdirs (including me) - thanks!

If the test is defined via the control file, is it possible to have additional files that are part of the test?

I think I can skip the future RFE and implement that directly

As that's the case, we need to name the distro/tests/$THIS dir generally enough to fit future description. Regarding that:

• deps is too specific as dirs can contain files unrelated to test deps
• meta is too generic, not to mention the word got Zucc'd by Facebook recenty
• files is too generic
• config is close, but still too specific to my taste, as in deps

Other options:

• override - more accurate but cryptic and too long
• extra - short and kind of accurate we're talking about "extra per-distro test files to include on top of shared ones in distro/tests"

I probably like extra the most, followed by config. Let me know if you have any other revelations ;)

Another thing to consider is whether templating shloud be enabled by default for:

• base test files distro/tests
• extra test files from distro/tests/extra

control file itself should certainly be templated for reasons described above, but for the test files themselves it might be better to simply copy them over without templating for maximum compatibility with autopkgtests in the wild (IOW being able to copy them without escaping with {% raw %}) and enable templating on-demand, but that kinda clashes with distro/pkg default of templating unless disabled

people may still want to define all test deps in one file using templating conditionals

Wouldn't it make sense to enable this on apkg-templating level? If rpm can be expanded to cover all rpm-based distros, perhaps all could be expanded to all distros? (when no other more specific dir is available)

whether templating should be enabled by default

• distro/tests: no
  • there is no counterpart in distro/pkg - everything there is in distro-specific subdirectories
  • I wouldn't make it more complex that it needs to be - if you need special sauce, shell can do it (or anything else in the executable file)
• distro/tests/extra: yes, to make it consistent with distro/pkg and enable the one-control-file-to-run-them-all use-case

Current WiP in test branch is working well, needs some more work & testing before MR.

mentioned in commit 7187b1b9

mentioned in merge request !108 (merged)

mentioned in commit 899c89cc

mentioned in commit 95daad71

mentioned in commit 8daad9ae

mentioned in commit 3695977a

closed via commit 3695977a

closed via merge request !108 (merged)

mentioned in commit 3bbf490b