Since 7f064e57 we do deep clones by
default, so -d (deep clone) doesn't make sense anymore

Paul Spooren authored 3 years ago and Michal Vasilek committed 2 years ago

Both $(AUTORELEASE) and $(PKG_SRC_VERSION) (from luci.git) use the Git
log to determine releases and package timestamps.

Feeds are shallow cloned by default, resulting in an incomplete Git log
and therefore different local package versions than offered upstream.

This commits sets the default feeds to use `src-git-full` to solve that.

Signed-off-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: Josef Schlehofer <josef.schlehofer@nic.cz>
[apply it for Turris OS feeds.conf; modify compile_pkgs and
documentation in generate_common]

Karel Koci authored 2 years ago and Simon Borek committed 2 years ago

The syntax used in the Bash scripts here requires at minimum Bash 5.0.
This checks for it and reports reasonable error message instead of
failure.

There are two discovered issues with older versions of bash.

The first issue is the change of empty array expansion in Bash 4.4
resulted in empty arrays not being considered as undefined anymore. We
depended on this heavily and thus we can't easily go bellow that
version.

The second issue is that syntax `[[ -v "foo[key]" ]` works only in Bash
5.0 and newer. This checks if specific key is in the associative array.
The replacement could be `[ "${foo[key]:+_}" ]` in some cases but this
only works if value is also nonempty. Thus it makes sense to just the
more clear and reliable syntax and such depend on Bash 5.0.

This only cover and fixes error message printed when invalid command is
passed to compile_pkgs script.

The first issue is unbound variable access for _updated_mirrors. This
requires new function that checks correctly that feed name is a key in
the associative array.
The second issue in the same function is that _updated_mirrors was never
actually set after feed was updated.

Another issue discovered and introduced during review is that
_git_mirror_lock can't call functions because in case of mirror it
spawns subshell where functions are not available. This simply reverts
all _git usages after _git_mirror_lock to git.

The another issue discovered was that repositories cloned using
git_checkout function were not actually using mirror. At the same time
the mirror URL construction is now placed in single function that can be
called from two locations it is used in.

There was also an issue with feed_url_replace function that was
invalidly handling URLs that had no reference. The effect was that the
URL for feed without any reference was handled by the first case and
result was something like https://example.org;https://example.org for
https://example.org

Another fix for mirrors was invalid detection of existence of the
mirror. The wrong variable was used and also the initial clone was
placed in the invalid directory.
There was also an issue if git mirror directory itself was missing. This
is fixed by simply creating it unconditionally.

And the last issue was with OpenWrt expecting full git history. This
actually does not break build but produces nasty warnings from git
commands that are run in script getver.sh that is called from top level
makefile. The fix here is to mask CLONE_DEEP when getting OpenWrt
repository.

Karel Koci authored 4 years ago and Karel Koci committed 3 years ago

There are two changes in this commit. First most noticable is removal of
all updater lists.

The second but more significant change is pretty much reformat of all
generate scripts. It extracts common git and configuration handling from
compile_pkgs to generate_common.
The advantage of this is of course code sharing but also that other
generate scripts now get features such as local git mirror.

This reverts commit 0c0b95f3.

The feature, which was introduced, was added to Git in version 2.28.
Thus unfortunately, it is not present in Git version like 2.20.1, which
is used in distributions like Debian Strech. It throws the following
error message: error: unknown option `initial-branch=master'.

New versions of git complain that the initial branch name is not
specified. It suggests specification in config but in our case the
better option is to not rely on user and just suppress this warning by
explicitly specifying master.

This makes stat little bit more resilient. It also should make it more
readable as we first prepare data and then insert it in formated output.

Step pkgsrepo moves packages stat checks away so it is required to first
generate stats and after that do move.

This is just stylistic change. It removes dirname and instead relies on
string operation. There should be no way of having in $0 anything else
other than some valid path. That is last part has to be at least
/name_of_file. This means that we do not have to rely on dirname and we
can safely just remove anything after last slash (including that slash).
We user dirname only for paths in sub-directories.

The hash character denotes hash over branch. Hash is prefixed by it. The
code was trying to remove if as suffic instead of prefix so it ended up
checking out invalid reference with hash char included.

This simple typo caused forice-make to run only in fallback mode. That
is always building in single thread with verbose output.

There is problem with repo_checkout as using it with newer version of
turris-build does not aways provide what you want. You should
effectivelly match turris-build version with target version when using
repo_checkout but that is not easily possible. It is easier to just drop
it and provide utility script that checkouts turris-build repository to
appropriate version and modified feeds.conf file. This script is going
to be added by subsequent commit.

This also optimizes corner case when we are cloning from mirror. We do
not have to fetch everything when we are cloning from mirror as git can
fetch just single commit. The problem is that this is not available when
commit is being fetched from servers. This change just tweaks condition
so that if mirror is used then hash fetch is used and not general fetch.

We in the end have only one place where we use those variables. It is
cleaner to have them specified directly by appropriate tool.

Running make multiple times for builds with last debug fallback run is
handy not even for calls available trough compile_pkgs. For example
building single selected package is not possible using compile_pkgs and
that script is not intended to even do it. The clean solution because of
that is to just simply extract logic for forced make run to separate
script that can be used without compile_pkgs,

White characters were invalidly being ignored in sed used to extract
OpenWrt hash from URL. This lead to checkout failing.

Setting -e in hashbang is not a good idea as users might just run it by
some other interpreter and that way removing (without noticing it) that
option.

This checks generated config after Kconfig pass if it still contains
preferences we have in files in configs directory.

This also drops some of the options in our config that are obsolete or
invalid.

CONFIG_VERSION_REPO is overwritten by setting pushed dynamically in
config by compile_pkgs so it makes no sense to have it here.

CONFIG_BUSYBOX_DEFAULT_CROND seems to be impossible to change and is
also default value that we later override with
CONFIG_BUSYBOX_CONFIG_CROND.

All PACKAGE options are not necessary. They are all enabled to 'm' but
getting them to 'y' would require a lot of works with gain that build
would in some cases failed instead of passed without that package. Not
worth it.

CONFIG_DEFAULT_TARGET_OPTIMIZATION is not applies so it is dropped here
without replacement. Same as one occurrence of CONFIG_CPU_TYPE.

This was renamed in HBL to report.

Karel Koci authored 5 years ago and Karel Koci committed 5 years ago

This locks git mirror directory when we are updating mirrors. This
should solve race condition with multiple builds failing to update
mirrors and failing that way completely.

Only update is protected. The expectation is that initialization is
happenning only once and is protected by non-existence of directory. If
there is going to be problem with initialization as well then we can add
locking as well.

This was colliding with OpenWrt's own configuration push and was kind of
hack. Replacement is to use OpenWrt's approach. That is same but in
OpenWrt configuration. To ensure that configuration is used by OpenWrt
KConfig in OWrt tree has to be added.
The new approach also simplifies some stuff because we can let KConfig
to do configuration for us. There is no need to specify specific
options to configuration in this repository. An example can be Turris
Omnia LEDS support (file target/linux/mvebu/image/Config.in):

config KERNEL_LEDS_OMNIA
	bool "Build Turris Omnia LEDS support to kernel"
	depends TARGET_mvebu
	default y if TARGET_mvebu_cortexa9_DEVICE_cznic_turris-omnia
	default n

This adds option CONFIG_KERNEL_LEDS_OMNIA that is copied to kernel
configuration as CONFIG_LEDS_OMNIA. It is available only for
TARGET_mvebu as it is part of patch for that target. In default it is
set to no. No is copied to kernel configuration as "# CONFIG_LEDS_OMNIA
is not set" and that ensures that for other boards kernel compiles as it
knows how to set this option. The default for Turris Omnia board is yes
and that inserts correct "CONFIG_LEDS_OMNIA=y" to kernel config.

There are various changes done to configuration on top of just general
cleanup.

All constants from compile_pkgs were moved to configs as it makes sense
that they are in one location not in script.

The settings to use specific version of toolchain was removed. Now we
use upstream default toolchain version which prevents problems with some
packages not being compiled because of old toolchain is used.

All options settings support and capabilities of target were dropped as
they are overriden by target setting non the less so they make no sense.
These should be modified in OpenWrt target not in configuration.

All package settings were dropped with exception of essential packages
and package disablements.
The reason why essential packages were not dropped is to ensure that all
failed build won't be deployed. With future tests done as part of build
this file can be dropped.
And lastly we have to preserve disablement of packages to not build
them as intended.

All DEFAULT options wer...

This was missed in previous commit cd222783

This can be shared between multiple scripts so move it to centralized
file.

This moves version file generation right after clean. This ensures that
we really report OpenWrt hash and not our patch commit that is created
to extend ignore list.

This also uses different git command that has lower overhead and is
intended more for this kind of operation.

This replaces automatic hash insertion by our hash. The reason is that
default is hash of tip but we patch our feed so that hash is
effectively nonsense. This uses remote hash instead.

This does the same thing as git status but is safe for script usage. It
just goes from tip (HEAD) trough all commit hashes and requests git to
name them using symbolic link. We limit that to only references from
origin so default remote references. This is to get commit that is in
upstream instead of commit hash of tip that is patch.

This reverts commit 8da67619.

We do not use this and I consider board specific patching as a hack.

Karel Koci authored 5 years ago and Karel Koci committed 5 years ago

This is not clean patching and we do not pretend it is so just ignore
those warnings.

Karel Koci authored 5 years ago and Karel Koci committed 5 years ago

This is not clean patching and we do not pretend it is so just ignore
those warnings.

Karel Koci authored 5 years ago and Karel Koci committed 5 years ago

This is not clean patching and we do not pretend it is so just ignore
those warnings.

Disabled packages add unnecessary complexity and introduce no big gain.
Furthermore they might and do hide if package is fixed by upstream.

There are two reason why package can be disabled. Either id does not
compile because it is broken and in that case we should build it in hope
that it will be fixed. Or it might be because it just does not apply on
given board or architecture and in that case that package should be
marked as incompatible with that architecture in package itself not
somewhere outside.

This adds support for applying patches only for a specific board.
This should be a temporary solution for a regression on Mox (patch for
Omnia multi CPU DSA broke Mox Peridot-Topaz configuration).

Signed-off-by: Marek Behún <marek.behun@nic.cz>

Upstream renamed it to config.buildinfo.

They also introduced new file feeds.buildinfo. We could use that instead
of our git-hash generated file.

Configuration override options was misleading. In reality those are
options you most probably don't want to mess too much with so let's call
them rather advanced options.

This also moves PUBLISH_BRANCH in example configuration file to
section: don't mess with. This is because now we provide it from
defaults and user just should not mess with it unless he know what he is
doing.

OpenWRT commit to be used is specified in feeds.conf file with rest of
feeds now so it makes not sense to overwrite it. This is just confusing
without any real gain in here. You can always modify feeds.conf file to
achieve the same.

Configuration override options was misleading. In reality those are
options you most probably don't want to mess too much with so let's call
them rather advanced options.

This also moves PUBLISH_BRANCH in example configuration file to
section: don't mess with. This is because now we provide it from
defaults and user just should not mess with it unless he know what he is
doing.

OpenWRT commit to be used is specified in feeds.conf file with rest of
feeds now so it makes not sense to overwrite it. This is just confusing
without any real gain in here. You can always modify feeds.conf file to
achieve the same.