README.adoc 9.9 KB
Newer Older
1
= Turris OS building scripts
2
3
4
5
6
7

Turris OS is based on OpenWrt. This repository contains script and additional
patches to convert OpenWrt tree to Turris OS one.

OpenWrt consists of core repository and additional packages feeds defined in `feeds.conf`.

8
== Repository layout
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

`compile_pkgs`:: script to prepare OpenWrt based source tree and to compile
  Turris OS in it. The final result are Turris OS packages.
`generate_lists`:: this script generates lists for Turris updater (updater-ng).
  Those are exposed alongside of Turris OS packages and control updater's
  behavior.
`generate_medkit`:: script creating Turris medkit (factory image) from Turris OS
  packages and Turris updater's lists.
`configs/`:: this directory contains configurations inserted to generated OpenWrt
  configuration. There are subdirectories for every Turris router and directory
  `common` that applies on all of them.
`feeds.conf`:: this file defines which feeds and in what version are about to be
  used for Turris OS packages compilation (OpenWrt based build). There is also
  non-standard commented line for OpenWrt used by `compile_pkgs` script to get
  OpenWrt source tree.
`patches`:: this directory contains all patches applied on both base OpenWrt tree
  as well as on any feed. It contains additional directories named after feed.
  Given patch is applied on a feed of the same name as is name of directory it is in
  relative to `patches/`. There can optionally be additional directories named after
  the feeds to sort patches to groups. Please read
  link:patches/README.adoc[readme file] in `patches` directory for more in depth
  explanation.
`NEWS`:: this is Turris OS changelog. It is parsed and used in git tags or in
  package informing users about changes.
`helpers/`:: directory containing libraries and utility scripts for top level
  scripts and some less important scripts as well.
35

36
== Requirements
37

38
39
40
41
42
43
44
45
46
47
48
You need to have a GNU/Linux distribution to be able to compile Turris OS and packages.

To install all dependencies for Debian distribution, you can use:

----
apt install ca-certificates git build-essential zlib1g-dev gawk libssl-dev subversion unzip libncurses-dev wget python python3 file rsync gcc-multilib g++-multilib libc6-dev-i386
----

For other distributions, we suggest to look at
https://openwrt.org/docs/guide-developer/build-system/install-buildsystem[OpenWrt
documentation].
49

50
== Getting started
51

52
53
54
55
56
57
58
59
60
Create an empty directory you want to build Turris OS in, enter that directory
and from it call `compile_pkgs` script. In most basic use-case, this should be
enough.

WARNING: *It will delete all previous content of the current directory!*

You can repeat individual steps of the compilation separately, check with the
following command `compile_pkgs --help` for a list of available commands,
generic options, and their brief descriptions.
61

62
Before you started you need to choose what version of Turris OS you want to
63
64
65
66
build. There are development versions in branches named `hbk`, `hbl` and `hbd`
(see link:WORKFLOW.adoc[workflow]). Then there are tagged final releases. Those
are in format of for example `v4.0.0` for Turris OS 4.0.0 or `v4.1.2` for Turris
OS 4.1.2.
67
68

To get to appropriate version you have to run command like this:
69
70
`git checkout hbk` where in this case this takes you to development branch for
Turris OS fixup releases.
71

72
73
It is advised to build the latest release version as with that you are most
likely to get to the end. To get the latest version you can do:
74
`git checkout "$(git tag | sort -V | tail -1)"`.
75

76
== Advanced usage
77

78
79
80
81
=== OpenWrt build system preparation

Sometimes you want to just get OpenWrt build system. That is possible with
the following command:
82

83
84
85
----
./compile_pkgs prepare -t omnia
----
86

87
88
Where `-t` means `-target`, so the only valid values are `turris1x`, `omnia`,
`mox`.
89

90
WARNING: Be aware *it removes previous content of current directory*!
91

92
93
94
95
This gives you fully patched OpenWrt build system. The "patched" here is
important. You should not use `scripts/feeds` script for fetching feeds as that
can remove local patches. Instead to refetch feeds you can use
`./compile_pkgs repatch_feeds`.
96

97
=== Building packages
98

99
To build a package you have to first prepare build directory. That can be
100
achieved by running in target directory.
101
102
103
----
./compile_pkgs prepare_tools -t omnia
----
104

105
106
Where `-t` means `-target`, so the only valid values are `turris1x`, `omnia`,
`mox`.
107

108
WARNING: Be aware *it removes previous content of current directory*!
109

110
Once, you have compiled required tools, you can build packages using this command:
111

112
113
114
115
116
----
make package/name/compile
----

The `name` is package name, which you want to compile.
117
118
119

You can also clean single package just by replacing `compile` with `clean`.

120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
The resulting package is placed in directory `bin/packages/ARCH/REPO` where the
`ARCH` is target architecture specific string and the `REPO` is the name of
package's source repository.

Occasionally, you can meet cases, where you need to run
`make package/name/compile` with parameters `-j1 V=s` or `-j1 V=sc` to be able
to see, what went wrong.

=== Generate lists

The lists are rules for Turris packages manager called Updater-ng. They are
processed using M4 macro language. To generate the lists you have to run:

----
./generate_lists generated_lists
----

The output is placed to directory `generated_lists` (feel free to use some other
path).

WARNING: Be aware *it removes previous content of current directory*!
141
142


143
=== Generate medkits
144

145
146
Generating medkits is using Turris package manager Updater-ng and using for
signing. 
147

148
149
150
151
Medkits are always generated against https://repo.turris.cz[official Turris
repositories]. There is no easy way to generate medkits against locally compiled
packages and generated lists. This also means that you don't have to generate
lists nor compile packages to generate medkits.
152

153
Requirements to be able to compile Updater-ng and usign.
154

155
156
157
----
apt install cmake liblua5.1-0-dev libevent-dev libarchive-dev libcurl4-openssl-dev liburiparser-dev uthash-dev
----
158

159
160
If you installed all the requirements for Updater-ng, you can now use following
command:
161

162
163
164
----
 generate_medkit -t omnia
----
165

166
167
Where `-t` means `--target`, so the only valid values are `turris1x`, `omnia`,
`mox`.
168

169

170
== Development and maintenance
171

172
=== Configuration
173

174
Default config for OpenWrt is put in `configs` directory either in directory
175
`common` or in a directory named after the target board. Files in those
176
177
directories get merged (common ones first) and will form the final `.config`
file. Some parts of `compile_pkgs` might modify it further.
178

179
=== Release notes
180

181
There is a `NEWS` file in this directory that contains the version number and after
182
it lines with new features in the release. This file is used to generate
183
`turris-version` package. `compile_pkgs` script reads it, finds the newest
184
version and from notes underneath it creates the package with release notes.
185

186
=== Releasing new version
187
188
189
190
191
192
193
194
195
196
197
198
199
200

When new version is released to Snails (see workflow for explanation) then new
commit with hashes has to be created and pushed. This commit have to be tagged
with appropriate version tag in format `vVERSION` where VERSION is released
version. To make this all simple and to correctly set always all configurations we
have script `helpers/new_release.sh`. Run this script after release from
turris-build project root directory and it is going to automatically detect,
commit and tag new version. You should review commit and tag it created for you
and then push it using `git push --tags`.

Script `helpers/new_release.sh` can generate various errors and warnings. It is
advised to run it even before release in `verify` mode to review possible problems
with release.

201
=== Forking new release
202
203
204
205
206
207
208
209
210
211
212
213
214

According to workflow new releases are forked from parent branch. During this
process care should be taken to tweak defaults to appropriate values. Following
list should be taken as a checklist for new release branch.

. Set `PUBLISH_BRANCH` in `defaults.sh`. Master branch should always be set to
  `hbd` and release branches should be set to `hbs`.
. Set branches in `feeds.conf`. You should append string like this:
  `;openwrt-18.06`. This has to be done for all OpenWRT feeds as well for OpenWRT
  it self. URL used by `compile_pkgs` is specified as first line in `feeds.conf`
  and with exception of first column it has same format as feeds. Note that that
  line is intentionally commented out as that is not feed.

215
=== Patching
216
217

If you need to patch OpenWRT or any of the feeds, put a git formatted patch
218
into the respective subdirectory of patches directory. In patches directory, there
219
220
is openwrt directory and directory for each feed we are patching. Inside those
directories, patches are also divided into subdirectories like `to-upstream`,
221
`wip` or `hack` to somehow distinguish between the quality of patches.
222

223
==== Rebasing patches
224

225
Sometimes patches no longer apply cleanly and need to be rebased. The most
226
227
228
229
230
231
232
233
234
simple way to do that, given that we have everything in git, is the following:

--------------------------------------------------------------------------------
find . -name '*.rej' -delete                # Clean all existing reject files
git am --reject .../patches/.../xyz.patch   # Try hard to apply patch
find . -name '*.rej'                        # List all failed chunks
--------------------------------------------------------------------------------

After those commands, we'll  get our git tree in partially merged state and
235
with few `.rej` files containing failed chunks. Actually, `compile_pkgs` script
236
by default cleans up checkout directory and applies patches with `git am
237
--reject` so if the build fails, checkout ends up in the state similar to what can
238
be achieved by those commands. Files with `.rej` extension needs to be merged
239
manually and afterward, we need to call `git add` on newly patched files. Once
240
241
242
all conflicts are resolved, `git am --continue` will create real commit that we
were trying to add using `git am`. Now all that is left is to export it using
`git format-patch -1` and overwrite patch stored in it of _build_ repository.