RFE: Multiple Upstream Tarballs support
I'm dealing with a package where a few files are added that are not shipped by upstream. Building a source package, the same files need to be present e.g. in distro/pkg/deb
and distro/pkg/rpm
, making it quite error prone to maintain (imagine you add a new one, you might forget to add a copy in the other location/s).
Things like symlinking don't seem to work (not surprised), is there a way to share files between packaging styles from a common location?
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Link issues together to show that they're related. Learn more.
When these merge requests are accepted, this issue will be closed automatically.
Activity
- Author Developer
One thing that I consider working with is MUT (Multiple Upstream Tarball) for debs and mulitple sources for rpms, will see if apkg does the right thing to get me there.
- Author Developer
The documentation suggests
apkg make-archive
accepts multiple lines from the make-archive script, one per line with the first one being the main archive. What actually happens inapkg/commands/make_archive.py:make_archive
is different - the last line is used and all other lines are completely ignored.Looking at the rest of the code along the way to a source package being available, everything accepts one or more archive files but will only operate on the first item in the list, that means they are about half way there.
For MUTs to work, either those extra archive files would encode the
component
part in the filename somehow so apkg could extract it (I'd be ok with that) or apkg would need to be given that mapping somehow else.The latter would probably come from having
make-archive
output codified further as:- first line for
{package name}-v{version}.{extension}
as documented already - followed with lines like
{component name}.{extension}
or maybe evencomponent {component_name} {filename}
to allow lines likesignature ...
later
- first line for
- Owner
You're right,
make_archive
behavior isn't consistent with docs. I'll try to fix that for %v0.5 .I've tried to make support for multiple files easier in the future as you've noticed from the lists being passed around, but in the case of archives it was mainly intended for signatures and other extra files, not MUT.
apkg
assumes a single project resulting in a single archive and a single source package which gets rid of quite some complexity.I'm not sure MUT support is worth the hassle as it needed to be implemented in all pkgstyles
Would you please elaborate on a common example of MUT you're expecting to face?
- Author Developer
I'm in the middle of testing a patch for MUT support, I will let you know once I've finished a first take. RPM already supports it (Source: tags and macros) and I didn't actually have to do anything to
apkg/pkgstyles/rpm.py
for things to work, not sure about others yet. - Author Developer
Preliminary support for debs is here https://git.symas.net/symas-public/apkg/-/tree/MUTs, it switches to treating the first line as the main archive to match documentation, followed by others (which are not expected to follow the name-version structure).
I've tried to be non-invasive where it comes to the
archive_paths
list, leaving it to you to decide whether any changes to internal structures are wanted/needed for signature support etc. I can look into what would be needed to have Nix/Arch pkgstyles updated if you want. - Owner
Cool!
I currently see this error on your
MUTs
branch:make_archive_script finished successfully but the archive listed doesn't exist: [MUTs c8ab9e2] DROP: update __version__ = 0.4.3.dev30+448af9a
and self tests are broken as well with similar outputs:
make_archive_script finished successfully but the archive listed doesn't exist: apkg-example-templates-v1.0
so something is amiss.
I suggest running self tests with
tox
:tox -e self
And generally make sure the entire tox run (includes linters) is green:
tox
Let me know when you're happy with your changes and I'll unleash CI on them which tests on several projects so it should catch regressions.
As long as the change is backward compatible, I'm happy to merge.
- Author Developer
The failures are mostly coming from the tests themselves:
- one
make-archive
script runstar
with-v
- various
make-archive
versions try to commit (andgit commit
also emits stuff on stdout unless-q
is passed)
That I can deal with, however I also get a strange
make-archive
failure where it thinks/ci
files are changed somehow during test and I have no idea what's happening. Would you be able to help there?This is the output of
git diff-index HEAD
from the failingmake-archive
::120000 100644 5f7cb5621de93d3fcd7201e2098228dfd4b3911a 0000000000000000000000000000000000000000 T CHANGELOG.md :120000 100755 0eff50d0620bdfa258e79f6ee1215188e61895a4 0000000000000000000000000000000000000000 T ci/images/full/build.sh :120000 100755 696576ca749ef3b7dfcf615118fd0b77b4b3c172 0000000000000000000000000000000000000000 T ci/images/full/push.sh :120000 100755 3f710b61b9cb490f686242b000060c4a4983cec7 0000000000000000000000000000000000000000 T ci/images/full/update.sh :120000 100755 696576ca749ef3b7dfcf615118fd0b77b4b3c172 0000000000000000000000000000000000000000 T ci/images/systemd/push.sh :120000 100755 3f710b61b9cb490f686242b000060c4a4983cec7 0000000000000000000000000000000000000000 T ci/images/systemd/update.sh :120000 100755 0eff50d0620bdfa258e79f6ee1215188e61895a4 0000000000000000000000000000000000000000 T ci/images/test/build.sh :120000 100755 696576ca749ef3b7dfcf615118fd0b77b4b3c172 0000000000000000000000000000000000000000 T ci/images/test/push.sh :120000 100755 3f710b61b9cb490f686242b000060c4a4983cec7 0000000000000000000000000000000000000000 T ci/images/test/update.sh
- one
- Author Developer
I've updated the branch to deal with the above two, the third only seems to happen on CentOS 7 and I also have some locale issues there.
Haven't run self-test on Debian yet except on Debian testing (actually a testing/sid hybrid), where things seem to break because distro version is not detected (
Invalid version: 'n/a'
) and don't think I know how to address that. - Author Developer
Ok, tracked the test failures down - if the hostname is not set to something git finds meaningful (which happens on throwaway test systems)
git commit
commands will refuse the committer/author and make-archive andtests/self/test_self.py
suffer. This is fixed now. - Owner
Thanks, I've opened !165 (closed) and ran integration CI on it which revealed a problem with this approach.
This breaks compatibility with existing
make-archive
scripts, because they print lots of stuff to stdout.Only last line is considered so that the
make_archive_script
can freely print to stdout.Backward compatibility with existing
make-archive
scripts MUST be retained.I need to think about this but the path of least resistance would be to provide alternative
make_archive_script
output format which is clearly distinguishable from the current archive-on-last-line.As I'd hate to restrict users stdout printing (arbitrary annoying restriction resulting in extra work), we simply need a clear separator of script output and the resulting archives lines.
For example 'archives:\n' line could be sufficient separator:
creating archive... some other stdout from the script... archives: pkg/archives/dev/foo-2.3.tar.xz pkg/archives/dev/foo-2.3.tar.xz.asc pkg/archives/dev/foo-component-2.3.tar.xz
What do you think?
- Author Developer
I'd say any output from the script that's not meant to be parsed should go to stderror, like with any other UNIX tools.
You are already adding
compat = 4
so we could retain the 'only last line is considered' as a documented behaviour forcompat <= 3
, which I considered was how you planned the compat level to be used in the first place and keep in line with documentation going forward. - Author Developer
In general I'm against things like
^archives:$
as a marker because it gets you into a game of cat and mouse trying to document and stress to users that they have to be in control of their stdout, when they might as well do the right thing the first time around (and keep legacy behaviour in for older compat levels). - Owner
Yes what you suggest makes sense, even though I'm not sure requiring users to redirect all stdout to stderr (otherwise weird error about a
$RANDOM_LINE_OF_TEXT
not found) is less stressful than allowing them to print what they want as long as they finish stdout in a required format.Speaking of stress, retaining current behavior for
compat <= 3
is a worst-ish case scenario which would force users to update theirmake-archive
scripts oncompat
update. Noapkg
update so far was so disruptive IIRC. Especially unwanted given existing users gain nothing from this.If this was truly necessary to have a nice 1.0 behavior, I'm willing to consider but I'd much prefer a backward and forward compatible solution which doesn't require arbitrary changes to working projects.
At least, should it come to this,
compat
level allows for a relatively graceful upgrade, yay.Edited by Jakub Ružička - Author Developer
Your other comment about getting this set up properly (tests, etc.) has a point. I'll see how I can get this sorted slowly.
As for context this is the situation I'm facing:
- we have a packaging repo for multiple distros where we have to do packaging 1 distro -> 1 branch, ending up with 9 branches, it was an improvement over the previous situation already but try and spot where they have diverged in terms of included files
- apkg allows us to convert it to a single branch and a split only within rpm/deb, a massive win, except rpm is still the odd one out (as ever), so you end up with something like 27 "sources" (I don't think I pushed a version with them in
pkg/rpm
directly), so the issue with missed files remains - The latest state of the packaging with the proposed changes is here
The end result is much more manageable and I couldn't find a different way of achieving it, if you spot one that deals with the "add a file and forget to add it everywhere", I'm all up for going that route.
I terms of your comments here:
Yes what you suggest makes sense, even though I'm not sure requiring users to redirect all stdout to stderr (otherwise weird error about a
$RANDOM_LINE_OF_TEXT
not found) is less stressful than allowing them to print what they want as long as they finish stdout in a required format.It is only
git commit
that seems to emit to stdout as part of normal operation. Presumably because it is not considered "porcelain". Most things only print to stderr. And you did document that it is the first line that matters so people should have been thinking about this.BTW, not sure you've noticed, but the existing code ignores lines starting with
#
already so if you really want to give something like progress updates on stdout, you can.Speaking of stress, retaining current behavior for
compat <= 3
is a worst-ish case scenario which would force users to update theirmake-archive
scripts oncompat
update. Noapkg
update so far was so disruptive IIRC. Especially unwanted given existing users gain nothing from this.If this was truly necessary to have a nice 1.0 behavior, I'm willing to consider but I'd much prefer a backward and forward compatible solution which doesn't require arbitrary changes to working projects.
TBH, I would hope running git commit in make-archive is the exception, not the norm. For everything else there is
#
. - Owner
Your other comment about getting this set up properly (tests, etc.) has a point. I'll see how I can get this sorted slowly.
Awesome
1 distro -> 1 branch, ending up with 9 branches
That's exactly the scenario
apkg
tries to prevent.apkg allows us to convert it to a single branch and a split only within rpm/deb, a massive win
I'm very happy to hear it works as intended in the wild!
except rpm is still the odd one out (as ever), so you end up with something like 27 "sources"
I couldn't agree more,
rpm
is always special in annoying ways.The latest state of the packaging with the proposed changes is here
O yeah, that looks significantly more manageable that 9 branches :)
I like how you're using templating, for example
{%- if distro.match("debian <= 10") -%} 4.3.0 {%- elif distro.match("debian <= 11", "ubuntu <= 20.04") -%} 4.5.0 {%- elif distro.match("debian <= 12", "ubuntu <= 22.04") -%} 4.6.0 {%- else -%} 4.6.2 {%- endif %}
is fairly readable even to
apkg
/jinja
stranger (especially compared torpm
macros) and with #76 (closed) you can make that{{ deb_standards_version }}
through templates if you want.Similarly, conditional requires are incredibly useful as they prevent file duplicities:
{% if distro.match("rhel <= 7") -%} Requires: cracklib >= 2.9.0 {%- else -%} Requires: cracklib >= 2.9.6 {%- endif %}
Finally, this is interesting usage:
{%- if '~' in version -%} symas-openldap ({{ version }}-0{{ deb_series }}1) UNRELEASED; urgency=medium * Local/release candidate build of symas-openldap for {{ deb_codename }} -- Symas Support <support@symas.com> {{ now }} {%- elif 0 -%} Commented explanation {%- endif %}
You wanted a custom behavior and you were able to accomplish it through templating without having to touch
apkg
code - awesome!It's inconvenient that some standard tools (such as
dch
in this case) can't be used with templated sources, but such is the price for automation.if you spot one that deals with the "add a file and forget to add it everywhere", I'm all up for going that route
I'm afraid that would require more abstractions and complexity which would in the end make things even worse than having to manually add things to multiple places (as you would without
apkg
just in even more places).The differences between individual pkgstyles / package managers / systems are so great that creating an abstract unified interface over all of them would involve insane amounts of work to implement and maintain.
Most things only print to stderr.
Including
apkg
unless it's printing operation output. So yeah I guess it's reasonable.BTW, not sure you've noticed, but the existing code ignores lines starting with
#
already so if you really want to give something like progress updates on stdout, you can.Yup, not to mention they can go nuts in stderr.
TBH, I would hope running git commit in make-archive is the exception, not the norm. For everything else there is #.
Fair enough :)
- Author Developer
1 distro -> 1 branch, ending up with 9 branches
That's exactly the scenario
apkg
tries to prevent.Yes, multiply that by about a dozen packages that our overlay "distro" provides and you can see why I got excited when I learned about apkg about a year ago ;)
apkg allows us to convert it to a single branch and a split only within rpm/deb, a massive win
I'm very happy to hear it works as intended in the wild!
We're not live yet, but the end is in sight - thank you so much for writing this tool!
The latest state of the packaging with the proposed changes is here
Finally, this is interesting usage:
[...]
You wanted a custom behavior and you were able to accomplish it through templating without having to touch
apkg
code - awesome!It's inconvenient that some standard tools (such as
dch
in this case) can't be used with templated sources, but such is the price for automation.The use case was to be able to draft the changelog content but with an overriding need to prevent accidentally "releasing" a version (multiple times even). AFAIK apkg lets you emit the templated file? Then piping it to dch for linting etc. should be easy anyway.
if you spot one that deals with the "add a file and forget to add it everywhere", I'm all up for going that route
I'm afraid that would require more abstractions and complexity which would in the end make things even worse than having to manually add things to multiple places (as you would without
apkg
just in even more places).The differences between individual pkgstyles / package managers / systems are so great that creating an abstract unified interface over all of them would involve insane amounts of work to implement and maintain.
Now I hope you can see why I settled on MUTs as a way out, there is good support in deb/rpm infrastructure for them already (and in Arch and probably Nix as well) and they can be integrated into apkg relatively painlessly - except for an additional restriction to those who (unknowingly) misused the current make-package parser - in my view a compat level bump is the appropriate response, AFAIK Debian have been doing things in a similar way too.
We can still decide to change what the right make-package output should look like, I considered a config level option to have it treated as JSON or something instead but those things are not easy to output by a shell script (e.g. string escaping)... But still toying with something like:
# the first non-comment line is still the main archive, no change here upstream-name-and-version.ext # followed by <type> then something type-specific signature N-V-R-archive.ext.asc component the-components-name archive.ext signature archive.ext.asc
Let me know what you think
- Owner
We're not live yet, but the end is in sight - thank you so much for writing this tool!
It is my great pleasure that
apkg
is useful to others, thanks for sharing <3It could've been yet another pile of scripts but I invested great effort in making it reusable and accessible.
Each new user brings in some fixes and enhancements for his/hers use case so I'm eager in cooperating and thus making
apkg
better for everyone while creating a little community.The use case was to be able to draft the changelog content but with an overriding need to prevent accidentally "releasing" a version (multiple times even). AFAIK apkg lets you emit the templated file? Then piping it to dch for linting etc. should be easy anyway.
Yeah it's possible with some extra effort:
cat $(apkg srcpkg -d debian-12 --render-template)/changelog
We can still decide to change what the right make-package output should look like
Absolutely, you might even discover some new requirements when you implement MUT.
MUT can be integrated into apkg relatively painlessly - except for an additional restriction to those who (unknowingly) misused the current make-package parser
My only worry was introduction of unneeded complexity/maintenance burden, but as you say the change should be fairly painless when implemented right.
I considered a config level option to have it treated as JSON or something instead but those things are not easy to output by a shell script (e.g. string escaping)
TOML is easier to render from shell script but it's verbose and not very good for representing this kind of data.
Let me know what you think
# the first non-comment line is still the main archive, no change here upstream-name-and-version.ext # followed by <type> then something type-specific signature N-V-R-archive.ext.asc component the-components-name archive.ext signature archive.ext.asc
Yeah I was thinking about adding some information about file type like that which would enable smooth extensions in the future (i.e. new file type) and also serve as an extra check that
make_archive_script
output is correct.The first line can be implicit type
archive
for single-archive cases and backward compat, but to maintainfile_type file
line structure, we can allow explicitly specifyingarchive upstream-name-and-version.ext
In fact, requiring this for
compat >= 4
would work as a little sanity check resulting in better messages ifmake_archive_script
doesn't produce expected output for any reason (doesn't start each stdout line with known file type).component the-components-name archive.ext
This works fine but it might be easier to keep two-column
file_type file
format for all lines, so this could be:component:the-components-name archive.ex
Not sure about that, just thinking
Edited by Jakub Ružička - Please register or sign in to reply
- OwnerResolved by Jakub Ružička
We're using symlinks for this in Knot DNS with no problems, why would they not work?
-
Last reply by Jakub Ružička
- Jakub Ružička changed milestone to %v0.5
changed milestone to %v0.5
- Jakub Ružička mentioned in merge request !165 (closed)
mentioned in merge request !165 (closed)
- Owner
OK, I made some thinking and researching so let me sum up what needs to be done to implement this.
First, please note that I don't personally have any use for MUT in any projects I maintain so I don't have an incentive to actively develop this - I'm happy to help (@mistotebe or anyone else interested in this feature) with reviewing and integrating your code until its merged, but I don't have cycles to develop this feature myself.
requirements: example project, tests, docs
As no current projects or tests involve MUT, a working example project must be included in
examples/
(presumablyexamples/multiple-sources
or similar human-readable) which tests MUT functionality for pkgstyles where it's supported.Additionally, tests over this example must be included in
tests/self/test_multiple_sources.py
or equivalent.This will ensure the feature stays functional through further
apkg
development.Documentation should properly describe new behavior (this time
) as well as the old one forcompat <= 3
if backward compat is retained.pkgstyle support
.deb
and.rpm
packages support MUTs natively, so at leastdeb
andrpm
pkgstyles should be supported.backward compatibility
As described in #101 (comment 295995) I'd prefer not to force existing users to update their
make-archive
scripts, but that would result in supporting two output formats in parallel which is bound to be confusing and painful.It might be better to solve this properly before 1.0 at the cost of one-time
make-archive
script update on compat level bump. Current last-line-only behavior would be retained forcompat <= 3
projects withcompat >= 4
defaulting to one-source-per-line.I'll do more research on this, but generally whatever needs to be done without breaking setups.
- Jakub Ružička added enhancement label
added enhancement label
- Jakub Ružička changed title from Sharing files/directories across packaging styles to RFE: Multiple Upstream Tarballs support
changed title from Sharing files/directories across packaging styles to RFE: Multiple Upstream Tarballs support
- Jakub Ružička changed milestone to %v0.6
changed milestone to %v0.6
- Author Developer
I have made some progress on cleaning up the branch to address your comments, the code is here:
- updated format (in new compat level 5)
- old behaviour is retained in compat < 5
- documentation
- tests
I am slightly at a loss at how to deal with the tests, this only affects
make-archive
andsrcpkg
. The former is relatively easy to test, the latter, not sure. On most platforms, apkg can generate the source package on its own, but on rpmsrpmbuild
is required and it might not be available. Any thoughts?Edited by Ondřej Kuzník - Owner
I had a brief look at the code and it looks good. I'll have a deeper look once you're done/happy with your changes.
As of tests, existing
make_archive_script
s (those inexamples/
as well asscripts/make-archive.sh
) should be updated to the newcompat
level so that they can be used as an up-to-date reference and get proper testing through existing tests.There should be a new example and test for MUT use case, which you already have in your branch.
I also suggest adding one more simple example project with old
compat = 4
to ensure the backwards compat works as well. Something likeexamples/legacy-make-archive
and associated test. - Author Developer
Getting closer except it turns out I don't know how to handle the
--file-list
and input files when passed toapkg srcpkg
. Can I rely on thecompat
level of the project at this point?The
infiles
handling is somewhat strange from there onwards so not sure what the intention is. - Author Developer
I have now finished the tests (deb and rpm), rebased on current master and the branch is ready for review. There weren't many srcpkg tests so I had to improvise a little.
- Jakub Ružička mentioned in merge request !175
mentioned in merge request !175