RFE: Multiple Upstream Tarballs support

I'm dealing with a package where a few files are added that are not shipped by upstream. Building a source package, the same files need to be present e.g. in distro/pkg/deb and distro/pkg/rpm, making it quite error prone to maintain (imagine you add a new one, you might forget to add a copy in the other location/s).

Things like symlinking don't seem to work (not surprised), is there a way to share files between packaging styles from a common location?

One thing that I consider working with is MUT (Multiple Upstream Tarball) for debs and mulitple sources for rpms, will see if apkg does the right thing to get me there.

The documentation suggests apkg make-archive accepts multiple lines from the make-archive script, one per line with the first one being the main archive. What actually happens in apkg/commands/make_archive.py:make_archive is different - the last line is used and all other lines are completely ignored.

Looking at the rest of the code along the way to a source package being available, everything accepts one or more archive files but will only operate on the first item in the list, that means they are about half way there.

For MUTs to work, either those extra archive files would encode the component part in the filename somehow so apkg could extract it (I'd be ok with that) or apkg would need to be given that mapping somehow else.

The latter would probably come from having make-archive output codified further as:

• first line for {package name}-v{version}.{extension} as documented already
• followed with lines like {component name}.{extension} or maybe even component {component_name} {filename} to allow lines like signature ... later

You're right, make_archive behavior isn't consistent with docs. I'll try to fix that for %v0.5 .

I've tried to make support for multiple files easier in the future as you've noticed from the lists being passed around, but in the case of archives it was mainly intended for signatures and other extra files, not MUT.

apkg assumes a single project resulting in a single archive and a single source package which gets rid of quite some complexity.

I'm not sure MUT support is worth the hassle as it needed to be implemented in all pkgstyles

Would you please elaborate on a common example of MUT you're expecting to face?

I'm in the middle of testing a patch for MUT support, I will let you know once I've finished a first take. RPM already supports it (Source: tags and macros) and I didn't actually have to do anything to apkg/pkgstyles/rpm.py for things to work, not sure about others yet.

Preliminary support for debs is here https://git.symas.net/symas-public/apkg/-/tree/MUTs, it switches to treating the first line as the main archive to match documentation, followed by others (which are not expected to follow the name-version structure).

I've tried to be non-invasive where it comes to the archive_paths list, leaving it to you to decide whether any changes to internal structures are wanted/needed for signature support etc. I can look into what would be needed to have Nix/Arch pkgstyles updated if you want.

Cool!

I currently see this error on your MUTs branch:

make_archive_script finished successfully but the archive
listed doesn't exist:

[MUTs c8ab9e2] DROP: update __version__ = 0.4.3.dev30+448af9a

and self tests are broken as well with similar outputs:

make_archive_script finished successfully but the archive
listed doesn't exist:

apkg-example-templates-v1.0

so something is amiss.

I suggest running self tests with tox:

tox -e self

And generally make sure the entire tox run (includes linters) is green:

tox

Let me know when you're happy with your changes and I'll unleash CI on them which tests on several projects so it should catch regressions.

As long as the change is backward compatible, I'm happy to merge.

The failures are mostly coming from the tests themselves:

• one make-archive script runs tar with -v
• various make-archive versions try to commit (and git commit also emits stuff on stdout unless -q is passed)

That I can deal with, however I also get a strange make-archive failure where it thinks /ci files are changed somehow during test and I have no idea what's happening. Would you be able to help there?

This is the output of git diff-index HEAD from the failing make-archive:

:120000 100644 5f7cb5621de93d3fcd7201e2098228dfd4b3911a 0000000000000000000000000000000000000000 T      CHANGELOG.md
:120000 100755 0eff50d0620bdfa258e79f6ee1215188e61895a4 0000000000000000000000000000000000000000 T      ci/images/full/build.sh
:120000 100755 696576ca749ef3b7dfcf615118fd0b77b4b3c172 0000000000000000000000000000000000000000 T      ci/images/full/push.sh
:120000 100755 3f710b61b9cb490f686242b000060c4a4983cec7 0000000000000000000000000000000000000000 T      ci/images/full/update.sh
:120000 100755 696576ca749ef3b7dfcf615118fd0b77b4b3c172 0000000000000000000000000000000000000000 T      ci/images/systemd/push.sh
:120000 100755 3f710b61b9cb490f686242b000060c4a4983cec7 0000000000000000000000000000000000000000 T      ci/images/systemd/update.sh
:120000 100755 0eff50d0620bdfa258e79f6ee1215188e61895a4 0000000000000000000000000000000000000000 T      ci/images/test/build.sh
:120000 100755 696576ca749ef3b7dfcf615118fd0b77b4b3c172 0000000000000000000000000000000000000000 T      ci/images/test/push.sh
:120000 100755 3f710b61b9cb490f686242b000060c4a4983cec7 0000000000000000000000000000000000000000 T      ci/images/test/update.sh

I've updated the branch to deal with the above two, the third only seems to happen on CentOS 7 and I also have some locale issues there.

Haven't run self-test on Debian yet except on Debian testing (actually a testing/sid hybrid), where things seem to break because distro version is not detected (Invalid version: 'n/a') and don't think I know how to address that.

Ok, tracked the test failures down - if the hostname is not set to something git finds meaningful (which happens on throwaway test systems) git commit commands will refuse the committer/author and make-archive and tests/self/test_self.py suffer. This is fixed now.

Thanks, I've opened !165 (closed) and ran integration CI on it which revealed a problem with this approach.

This breaks compatibility with existing make-archive scripts, because they print lots of stuff to stdout.

Only last line is considered so that the make_archive_script can freely print to stdout.

Backward compatibility with existing make-archive scripts MUST be retained.

I need to think about this but the path of least resistance would be to provide alternative make_archive_script output format which is clearly distinguishable from the current archive-on-last-line.

As I'd hate to restrict users stdout printing (arbitrary annoying restriction resulting in extra work), we simply need a clear separator of script output and the resulting archives lines.

For example 'archives:\n' line could be sufficient separator:

creating archive...
some other stdout from the script...
archives:
pkg/archives/dev/foo-2.3.tar.xz
pkg/archives/dev/foo-2.3.tar.xz.asc
pkg/archives/dev/foo-component-2.3.tar.xz

What do you think?

I'd say any output from the script that's not meant to be parsed should go to stderror, like with any other UNIX tools.

You are already adding compat = 4 so we could retain the 'only last line is considered' as a documented behaviour for compat <= 3, which I considered was how you planned the compat level to be used in the first place and keep in line with documentation going forward.

In general I'm against things like ^archives:$ as a marker because it gets you into a game of cat and mouse trying to document and stress to users that they have to be in control of their stdout, when they might as well do the right thing the first time around (and keep legacy behaviour in for older compat levels).

Yes what you suggest makes sense, even though I'm not sure requiring users to redirect all stdout to stderr (otherwise weird error about a $RANDOM_LINE_OF_TEXT not found) is less stressful than allowing them to print what they want as long as they finish stdout in a required format.

Speaking of stress, retaining current behavior for compat <= 3 is a worst-ish case scenario which would force users to update their make-archive scripts on compat update. No apkg update so far was so disruptive IIRC. Especially unwanted given existing users gain nothing from this.

If this was truly necessary to have a nice 1.0 behavior, I'm willing to consider but I'd much prefer a backward and forward compatible solution which doesn't require arbitrary changes to working projects.

At least, should it come to this, compat level allows for a relatively graceful upgrade, yay.

Your other comment about getting this set up properly (tests, etc.) has a point. I'll see how I can get this sorted slowly.

As for context this is the situation I'm facing:

• we have a packaging repo for multiple distros where we have to do packaging 1 distro -> 1 branch, ending up with 9 branches, it was an improvement over the previous situation already but try and spot where they have diverged in terms of included files
• apkg allows us to convert it to a single branch and a split only within rpm/deb, a massive win, except rpm is still the odd one out (as ever), so you end up with something like 27 "sources" (I don't think I pushed a version with them in pkg/rpm directly), so the issue with missed files remains
• The latest state of the packaging with the proposed changes is here

The end result is much more manageable and I couldn't find a different way of achieving it, if you spot one that deals with the "add a file and forget to add it everywhere", I'm all up for going that route.

I terms of your comments here:

Yes what you suggest makes sense, even though I'm not sure requiring users to redirect all stdout to stderr (otherwise weird error about a $RANDOM_LINE_OF_TEXT not found) is less stressful than allowing them to print what they want as long as they finish stdout in a required format.

It is only git commit that seems to emit to stdout as part of normal operation. Presumably because it is not considered "porcelain". Most things only print to stderr. And you did document that it is the first line that matters so people should have been thinking about this.

BTW, not sure you've noticed, but the existing code ignores lines starting with # already so if you really want to give something like progress updates on stdout, you can.

Speaking of stress, retaining current behavior for compat <= 3 is a worst-ish case scenario which would force users to update their make-archive scripts on compat update. No apkg update so far was so disruptive IIRC. Especially unwanted given existing users gain nothing from this.

If this was truly necessary to have a nice 1.0 behavior, I'm willing to consider but I'd much prefer a backward and forward compatible solution which doesn't require arbitrary changes to working projects.

TBH, I would hope running git commit in make-archive is the exception, not the norm. For everything else there is #.

Your other comment about getting this set up properly (tests, etc.) has a point. I'll see how I can get this sorted slowly.

Awesome

1 distro -> 1 branch, ending up with 9 branches

That's exactly the scenario apkg tries to prevent.

apkg allows us to convert it to a single branch and a split only within rpm/deb, a massive win

I'm very happy to hear it works as intended in the wild!

except rpm is still the odd one out (as ever), so you end up with something like 27 "sources"

I couldn't agree more, rpm is always special in annoying ways.

The latest state of the packaging with the proposed changes is here

O yeah, that looks significantly more manageable that 9 branches :)

I like how you're using templating, for example

{%- if distro.match("debian <= 10") -%}
  4.3.0
{%- elif distro.match("debian <= 11", "ubuntu <= 20.04") -%}
  4.5.0
{%- elif distro.match("debian <= 12", "ubuntu <= 22.04") -%}
  4.6.0
{%- else -%}
  4.6.2
{%- endif %}

is fairly readable even to apkg/jinja stranger (especially compared to rpm macros) and with #76 (closed) you can make that {{ deb_standards_version }} through templates if you want.

Similarly, conditional requires are incredibly useful as they prevent file duplicities:

{% if distro.match("rhel <= 7") -%}
Requires:	cracklib >= 2.9.0
{%- else -%}
Requires:	cracklib >= 2.9.6
{%- endif %}

Finally, this is interesting usage:

{%- if '~' in version -%}
symas-openldap ({{ version }}-0{{ deb_series }}1) UNRELEASED; urgency=medium

  * Local/release candidate build of symas-openldap for {{ deb_codename }}

 -- Symas Support <support@symas.com>  {{ now }}
{%- elif 0 -%}
Commented explanation
{%- endif %}

You wanted a custom behavior and you were able to accomplish it through templating without having to touch apkg code - awesome!

It's inconvenient that some standard tools (such as dch in this case) can't be used with templated sources, but such is the price for automation.

if you spot one that deals with the "add a file and forget to add it everywhere", I'm all up for going that route

I'm afraid that would require more abstractions and complexity which would in the end make things even worse than having to manually add things to multiple places (as you would without apkg just in even more places).

The differences between individual pkgstyles / package managers / systems are so great that creating an abstract unified interface over all of them would involve insane amounts of work to implement and maintain.

Most things only print to stderr.

Including apkg unless it's printing operation output. So yeah I guess it's reasonable.

BTW, not sure you've noticed, but the existing code ignores lines starting with # already so if you really want to give something like progress updates on stdout, you can.

Yup, not to mention they can go nuts in stderr.

TBH, I would hope running git commit in make-archive is the exception, not the norm. For everything else there is #.

Fair enough :)

1 distro -> 1 branch, ending up with 9 branches

That's exactly the scenario apkg tries to prevent.

Yes, multiply that by about a dozen packages that our overlay "distro" provides and you can see why I got excited when I learned about apkg about a year ago ;)

apkg allows us to convert it to a single branch and a split only within rpm/deb, a massive win

I'm very happy to hear it works as intended in the wild!

We're not live yet, but the end is in sight - thank you so much for writing this tool!

The latest state of the packaging with the proposed changes is here

Finally, this is interesting usage:

[...]

You wanted a custom behavior and you were able to accomplish it through templating without having to touch apkg code - awesome!

It's inconvenient that some standard tools (such as dch in this case) can't be used with templated sources, but such is the price for automation.

The use case was to be able to draft the changelog content but with an overriding need to prevent accidentally "releasing" a version (multiple times even). AFAIK apkg lets you emit the templated file? Then piping it to dch for linting etc. should be easy anyway.

if you spot one that deals with the "add a file and forget to add it everywhere", I'm all up for going that route

I'm afraid that would require more abstractions and complexity which would in the end make things even worse than having to manually add things to multiple places (as you would without apkg just in even more places).

The differences between individual pkgstyles / package managers / systems are so great that creating an abstract unified interface over all of them would involve insane amounts of work to implement and maintain.

Now I hope you can see why I settled on MUTs as a way out, there is good support in deb/rpm infrastructure for them already (and in Arch and probably Nix as well) and they can be integrated into apkg relatively painlessly - except for an additional restriction to those who (unknowingly) misused the current make-package parser - in my view a compat level bump is the appropriate response, AFAIK Debian have been doing things in a similar way too.

We can still decide to change what the right make-package output should look like, I considered a config level option to have it treated as JSON or something instead but those things are not easy to output by a shell script (e.g. string escaping)... But still toying with something like:

# the first non-comment line is still the main archive, no change here
upstream-name-and-version.ext
# followed by <type> then something type-specific
signature N-V-R-archive.ext.asc
component the-components-name archive.ext
signature archive.ext.asc

Let me know what you think

We're not live yet, but the end is in sight - thank you so much for writing this tool!

It is my great pleasure that apkg is useful to others, thanks for sharing <3

It could've been yet another pile of scripts but I invested great effort in making it reusable and accessible.

Each new user brings in some fixes and enhancements for his/hers use case so I'm eager in cooperating and thus making apkg better for everyone while creating a little community.

The use case was to be able to draft the changelog content but with an overriding need to prevent accidentally "releasing" a version (multiple times even). AFAIK apkg lets you emit the templated file? Then piping it to dch for linting etc. should be easy anyway.

Yeah it's possible with some extra effort:

cat $(apkg srcpkg -d debian-12 --render-template)/changelog 

We can still decide to change what the right make-package output should look like

Absolutely, you might even discover some new requirements when you implement MUT.

MUT can be integrated into apkg relatively painlessly - except for an additional restriction to those who (unknowingly) misused the current make-package parser

My only worry was introduction of unneeded complexity/maintenance burden, but as you say the change should be fairly painless when implemented right.

I considered a config level option to have it treated as JSON or something instead but those things are not easy to output by a shell script (e.g. string escaping)

TOML is easier to render from shell script but it's verbose and not very good for representing this kind of data.

Let me know what you think

# the first non-comment line is still the main archive, no change here
upstream-name-and-version.ext
# followed by <type> then something type-specific
signature N-V-R-archive.ext.asc
component the-components-name archive.ext
signature archive.ext.asc

Yeah I was thinking about adding some information about file type like that which would enable smooth extensions in the future (i.e. new file type) and also serve as an extra check that make_archive_script output is correct.

The first line can be implicit type archive for single-archive cases and backward compat, but to maintain file_type file line structure, we can allow explicitly specifying

archive upstream-name-and-version.ext

In fact, requiring this for compat >= 4 would work as a little sanity check resulting in better messages if make_archive_script doesn't produce expected output for any reason (doesn't start each stdout line with known file type).

component the-components-name archive.ext

This works fine but it might be easier to keep two-column file_type file format for all lines, so this could be:

component:the-components-name archive.ex

Not sure about that, just thinking

We're using symlinks for this in Knot DNS with no problems, why would they not work?

changed milestone to %v0.5

mentioned in merge request !165 (closed)

OK, I made some thinking and researching so let me sum up what needs to be done to implement this.

First, please note that I don't personally have any use for MUT in any projects I maintain so I don't have an incentive to actively develop this - I'm happy to help (@mistotebe or anyone else interested in this feature) with reviewing and integrating your code until its merged, but I don't have cycles to develop this feature myself.

requirements: example project, tests, docs

As no current projects or tests involve MUT, a working example project must be included in examples/ (presumably examples/multiple-sources or similar human-readable) which tests MUT functionality for pkgstyles where it's supported.

Additionally, tests over this example must be included in tests/self/test_multiple_sources.py or equivalent.

This will ensure the feature stays functional through further apkg development.

Documentation should properly describe new behavior (this time ) as well as the old one for compat <= 3 if backward compat is retained.

pkgstyle support

.deb and .rpm packages support MUTs natively, so at least deb and rpm pkgstyles should be supported.

backward compatibility

As described in #101 (comment 295995) I'd prefer not to force existing users to update their make-archive scripts, but that would result in supporting two output formats in parallel which is bound to be confusing and painful.

It might be better to solve this properly before 1.0 at the cost of one-time make-archive script update on compat level bump. Current last-line-only behavior would be retained for compat <= 3 projects with compat >= 4 defaulting to one-source-per-line.

I'll do more research on this, but generally whatever needs to be done without breaking setups.

added enhancement label

changed title from Sharing files/directories across packaging styles to RFE: Multiple Upstream Tarballs support

changed milestone to %v0.6

I have made some progress on cleaning up the branch to address your comments, the code is here:

• updated format (in new compat level 5)
• old behaviour is retained in compat < 5
• documentation
• tests

I am slightly at a loss at how to deal with the tests, this only affects make-archive and srcpkg. The former is relatively easy to test, the latter, not sure. On most platforms, apkg can generate the source package on its own, but on rpms rpmbuild is required and it might not be available. Any thoughts?

I had a brief look at the code and it looks good. I'll have a deeper look once you're done/happy with your changes.

As of tests, existing make_archive_scripts (those in examples/ as well as scripts/make-archive.sh) should be updated to the new compat level so that they can be used as an up-to-date reference and get proper testing through existing tests.

There should be a new example and test for MUT use case, which you already have in your branch.

I also suggest adding one more simple example project with old compat = 4 to ensure the backwards compat works as well. Something like examples/legacy-make-archive and associated test.

Getting closer except it turns out I don't know how to handle the --file-list and input files when passed to apkg srcpkg. Can I rely on the compat level of the project at this point?

The infiles handling is somewhat strange from there onwards so not sure what the intention is.

I have now finished the tests (deb and rpm), rebased on current master and the branch is ready for review. There weren't many srcpkg tests so I had to improvise a little.

mentioned in merge request !175