compile_pkgs: speedup compilation by caching built host tools and toolchain
In order to speed up subsequent builds try to cache built tools and toolchain. The resulting tarball is stored in the dl directory and has a name in format: turris-build-cache-BRANCH-TARGET-BOARD-MD5SUM.tar.zst where BRANCH is hb?/crashlab (determined from $GIT_BRANCH, and if empty, from git rev-parse --abbrev-ref HEAD. If this returns something different from hb?/crashlab, cached version won't be used), TARGET is tools or toolchain, BOARD is turris1x, omnia or mox, MD5SUM is md5 checksum of the content of .config and the TARGET directory (tools or toolchain), so any change in these will force a rebuild. I.e. if a patch is added/remove/renamed for automake, all tools will be rebuilt. If .config changes, both tools and toolchain will be rebuilt (this is because we can't determine in a simple way whether the .config settings changed for tools/toolchain). When a build of tools/toolchain is forced (the cached tarball name has different MD5SUM or is nonexistent), the old tarball are removed from dl directory and compilation is invoked. After the compilation unneeded sources are removed to make the tarball smaller. This includes all regular files in build_dir/host (or build_dir/toolchain) except files starting with ".built" or ".prepared" and files with name ".configured" - this is done to make OpenWRT's build system think that the sources are there and has not changed, so it won't try to rebuild. Afterwards the build_dir/TARGET_DIR and staging_dir/TARGET_DIR are packed and stored. On subsequent run if the tarball with given name exists, instead of compiling again the cached version will be used. Testing showed that toolchain tarball has around 370 MB and tools tarball around 40 MB. Zstd compression is used because of speed (cca 5 seconds to compress 1.2 GB of toolchain to 370 MB).
Please register or sign in to comment