Verified Commit 04dd2d52 authored by Karel Koci's avatar Karel Koci 🤘
Browse files

docs: update documentation and remove no longer valid one

These documentation is kind of old and although partially correct they
also say a lot of stuff that are no longer true. We should replace them
with more accurate documentation.
parent cf313897
Pipeline #71821 failed with stages
in 6 minutes and 14 seconds
Updater internals
=================
This document describes how the updater would work under the hood.
Execution phases
----------------
There are distinct phases of the execution, each one having a
different task.
Reading of the current state
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Information about what is already installed, about stored flags, etc.
are loaded.
Parsing of the lua scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~
The main lua configuration script is loaded and run. From it, further
lua scripts are obtained and executed.
All the repositories and intentions to install packages are described.
Downloading the repository indices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The repository index files are downloaded or obtained from disk and
parsed, providing list of packages each.
Planning
~~~~~~~~
The intentions what shall be installed is put together with the
repository indices. All the dependencies are resolved, which produces
list of packages to install or remove.
Optionally, if the configurantion requires it, user is notified and
further action is postponed. In such case, the plan is stored for
future use.
Downloading
~~~~~~~~~~~
The needed packages are downloaded.
Unpacking
~~~~~~~~~
Packages are unpacked to a temporary location. This may be in
permanent storage (because of the transactions), but it is not the
live system yet.
Merging
~~~~~~~
Once the files are all ready and synced, they start to be moved into
the right place. This is interspersed with running pre-* and post-*
scripts and hooks.
At the end of this phase, any extra files from previous or uninstalled
packages are removed.
Save of status
~~~~~~~~~~~~~~
The flags and status of installed packages are stored.
Components
----------
These are the pieces that form the updater. This does not list all the
small things, like package objects, there's not much sense in
designing them yet.
Also, there's going to be a lot of glue in between tying it together.
There's nothing much to be said about it in a high-level design
document.
Event loop
~~~~~~~~~~
Some parts of the program need to wait for multiple events. This'll do
the waiting in some generic way. We may need to be able to wait for a
set of events to complete and then finish the event loop.
Process manager
~~~~~~~~~~~~~~~
It allows running other processes and pass input and output with them.
URI manager
~~~~~~~~~~~
This part is able to obtain the data for an URI. It also can do
validation of the content.
The downloading may work in a parallel way, downloading multiple URIs
at once.
State manager
~~~~~~~~~~~~~
This allows reading and writing of the flags and package state data.
It is likely going to be just a set of functions.
Logger
~~~~~~
A place where log messages are sorted and output to several possible
locations.
Journal
~~~~~~~
The journal is some persistent data structure allowing for recovery in
case of power outage or something. It needs to be careful about writes
and flushes to the file. Also, it may need to handle broken end of the
journal file.
Sandbox
~~~~~~~
This part prepares execution environments for the configuration
scripts.
Planner
~~~~~~~
This part resolves the dependencies and produces a plan of action.
Parts by language
-----------------
Likely, the updater would be combination of C and lua code. Almost all
of the glue code is likely to be in lua. From the components above,
many are likely to end up in C:
* The event loop
* The process manager
* The URI manage.
* The journal.
Lua seems better for:
* State manager
* Sandbox
The logger might turn easier in either of the languages. The planner
will start in lua, but it is possible there'll be some
performance-critical parts needing the power of C.
Opkg on-disc structures
=======================
In case we decide not to use opkg as the backend (which is somewhat
problematic with the old updater and doesn't allow some nice features,
like transactions), we need to migrate from its data structures.
There are two possibilities. Either repeatedly call opkg to provide
the information (eg. +opkg list-installed+, +opkg files <package>+).
The other possibility is to decode its on-disk data structures.
It would also be possible to use something backwards compatible with
the opkg structures (keep them up to date and use them, but add some
more information to other files, like journal).
Here's a list of files discovered to be used by opkg to store the
database of installed packages.
Global files
------------
There's +/usr/lib/opkg/status+. This file lists all the installed
packages. Each one is represented by a block of email-like headers,
listing some basic info about the package. This is the local status of
the package ‒ when it has been installed, if it is installed as a
dependency or requested by a user, etc. The block could be different
from router to router even for the same package, unlike the files in
the following list.
Per-package files
-----------------
There are some files for each installed package in the
+/usr/lib/opkg/info+ directory. The files are named after the package
names, suffixed by the type of the file (eg. +package-name.type+). The
types are following:
control::
This is the description block taken vanilla from the package. The
same info is also found in the repository index list.
list::
Lists the files owned by the package, one file per line.
conffiles::
Lists the config files owned by the package (files that shall not be
overwritten by an upgrade of the package). Note that config files
are also listen in +list+ file. The file is present only if the
package has some config files.
preinst::
postinst::
prerm::
postrm::
Executable scripts that are run in the right moment of installation
process. They are present only if the package contains them.
It looks like some of the files are taken directly from the
control.tar.gz of the package.
Requirements
============
Here's a list of the things the new updater needs or should support.
The how of doing so is in a different documents, but there may be some
hints about it here.
User configuration flexibility
------------------------------
The user should be able to express wishes about the working of the
updater. This includes at least:
* Require a user list to be installed, as currently.
* Require a package to be installed.
* Revoke the requirement wishes from above.
* Keep a package on current or specific version.
* Block package from being installed.
Server-side configuration
-------------------------
As with the user, the same set of commands must be allowed from the
server.
There needs to be some kind of priority system to resolve conflicts
between the user and server configuration. We need to be able to
specify that user may override the server on some packages but not
with others.
Vanilla repositories
--------------------
The updater should be able to work with vanilla repositories that are
produced by the OpenWRT build root. The configuration instructions
must be provided by a separate channel.
Multiple repositories
---------------------
We need to be able to support multiple repositories, including
third-party repositories. The repositories need some priorities in
case of package colliding. It should be possible to force a package
from a given repository in case of collision and override the
priorities on per-package basis.
Multiple repositories also support multiple compiled branches used in
parallel.
Security
--------
Both the configurations and repositories need to be secured somehow.
This is likely to be some kind of cryptographic signing. The keys and
required checks need to be configurable on per-source basis ‒ some
repositories are deploy ones, with higher level of security, some are
testing ones with lower level, some might live on trusted LAN at the
user's side. And some might even be local files.
It must support off-site signing somehow as well, for the extra-secure
repositories.
Dependencies
------------
We need flexible dependency system. It must be able to depend on
packages from any repository, from the same repository, it must be
possible to specify versions. We also need some kind of conditional
dependencies (like any-of, or at-most-one-of), negative dependencies
(when two packages fight).
From the above requirements, it seems the depgraph needs to be
computed on the router.
Attention-less updates
----------------------
It needs to be able to solve the usual problems without the need of
user solving anything.
Early-warnings
--------------
It should be possible to configure the system to give an early warning
about what will be done and give the user some time to react. The time
should be zero by default and configurable, but there should be some
kind of limit for given update (eg. critical security updates could
mandate to be installed in shorter time).
Resilience against interruptions & FS problems
----------------------------------------------
We need to not screw the system completely in case the system is
rebooted during an update. We also need to ensure the updates are
synced properly, including the metadata, so the system doesn't get
broken in case the FS is not commited or something.
Also, the updater itself should be able to survive a nuclear war
against its own files, libraries, etc (possibly by statically linking
it, having it backed up in two places).
An option to mark a package extra critical may be needed to enable
some special handling.
Install transactions
--------------------
Install all the packages in single step to ensure nothing breaks
because of changed versions (like libopenssl and openssl-utils, which
need to be of the same version all the time, otherwise nothing works).
Handle special update requirements
----------------------------------
Sometimes, an update requires some special handling in addition to
installing packages. It may be installing the packages in some
specific order (though the transactions might mitigate this need), the
router may need to be rebooted between two updates or some action like
changing uboot parameters at specific time might be needed.
Maybe some kind of additional pre-install, post-install,
version-conditioned hooks and order requirements added into the
configuration/upgrade plan might work.
ABI changes, reverse „dependencies“, …
--------------------------------------
We may have a need to re-install all packages depending on a library
because of ABI change or something. We need to be specify this.
Configuration migrations
------------------------
Sometimes, we need to change an option, or add a new one, to an
existing configuration. Currently, this requires user attention, since
opkg doesn't overwrite config files if they are modified.
Reinstall commands
------------------
Sometimes, it is needed to reinstall a package. Have an ability to
perform some task (like reinstalling a package) once, but not each
time, after it is published.
Individual parts of configuration
---------------------------------
Make it possible to amend the commands or required package lists on
per-router basis from the server side (eg. solutions for specific
problems happening at user's ISP or something like that).
Hook support
------------
It would be cool to have ability to easily extend the functionality
somehow. The idea is, for example, to snapshot a btrfs volume before
installing the updates.
Test coverage
=============
Just writing tests without knowing what code is or isn't tested is like shooting
blindly. For this reason we have to generate and check test coverage informations
regularly.
Generating coverage information
-------------------------------
You can run all tests or any combination of tests to record coverage. But before
doing so you should clean whole project as coverage have to be explicitly enabled.
Also running valgrind tests is not recommended.
To run test with enabled coverage you should pass `COV=y` to make call. So running
all tests can be done by calling: `make COV=y test test-sys`
To generate website with coverage info you have to run `make COV=y coverage`.
To record coverage and generate website for all tests run:
make clean
make COV=y test test-sys
make COV=y coverage
Testing of the updater
======================
Every coder makes mistakes and the updater might turn out to be a
larger project. Therefore, some automatic tests that may spot errors
and problems may be useful.
But testing updater automatically in the full is problematic, since we
need a whole system to do it. While we might want to do something like
that in some Turris-global way, we need a way to test more scenarios.
Pure unit tests
---------------
Some parts are easy to separate and test independently. This is
definitely the case of the planner, which would be fed with objects
describing the situation and would produce list of actions to do.
The state manager might be used in this manner, provided it may be
configured to use different path to the files than the default.
The process manager and URI manager may be tested in very similar way.
The URI manager needs access to network and it needs a server with
preset files somewhere. We probably don't want to bundle a server with
the source code.
Logger is likely to be testable separately in a limited way.
Mocking parts of functionality
------------------------------
Certain tests may need some lower-level routines to be replaced. We
may want to run the whole updater and see what would have been done
without actually replacing files or downloading packages.
Replacing lua code is simple. We can just assign to the correct global
variable and overwrite the function or object stored there.
The C functions might be mocked as well, but in a harder way. The
general approach could be to link all the .o modules except the ones
to be replaced, and linking a replacement .o for such modules. This
way the test would need to be in its own binary, but reusing a lot of
code.
Testing libraries
-----------------
It seems lunit might work for lua code (http://www.mroth.net/lunit/).
Check for C code looks usable too (https://libcheck.github.io/check/).
Also, some tests might be assisted with shell scripts checking the
output, produced files, etc.
We probably want to include the tools as git submodules instead of
requiring them being installed.
Guidelines
----------
First, any code being written should be either tested somehow or at
least considered why writing tests for it would be impossible or too
much work, or the tests would test only trivial matters (like that
after assigning to a variable the value is stored there).
Also, as the tests depend on extra libraries and need to compile
additional binaries, they should be in a separate directory and be
compiled only as a dependency of `make test`.
The testing „framework“ should be initialized during the starting
phases of implementation.
This document describes files and structured shared with Opkg. This is primarily
recherche of Opkg to ensure compatibility with it.
== Opkg on-disc structures
In case we decide not to use opkg as the backend (which is somewhat problematic
with the old updater and doesn't allow some nice features, like transactions), we
need to migrate from its data structures.
There are two possibilities. Either repeatedly call opkg to provide the
information (eg. `opkg list-installed`, `opkg files <package>`). The other
possibility is to decode its on-disk data structures.
It would also be possible to use something backwards compatible with the opkg
structures (keep them up to date and use them, but add some more information to
other files, like journal).
Here's a list of files discovered to be used by opkg to store the database of
installed packages.
=== Global files
There's `/usr/lib/opkg/status`. This file lists all the installed packages. Each
one is represented by a block of email-like headers, listing some basic info about
the package. This is the local status of the package ‒ when it has been installed,
if it is installed as a dependency or requested by a user, etc. The block could be
different from router to router even for the same package, unlike the files in the
per-package files.
It is also important to note that single package can be noted there multiple times
with different version.
Expected fields are:
Package::
Specified name of package.
Version::
This is package version.
Depends::
Comma separated list of dependencies. Every dependency is name of package.
Optional version can also be specified. That is done in format `
Recommends::
In general optional dependencies. This is not used in OpenWrt's packages (as of
OpenWrt 19.07).
Suggests::
In general optional extensions. This is not used in OpenWrt's packages (as of
OpenWrt 19.07).
Provides::
Comma separated list of packages aliases. Optional version can be specified the
same way as in case of `Depends`.
Replaces::
List of comma separated packages this package replaces. This is not used in
OpenWrt's packages (as of OpenWrt 19.07).
Conflicts::
Comma separated list of conflicting packages. Optional version can be specified
the same way as in case of `Depends`.
Status::
This is space separated triplet. Description is in following subsection.
Essential::
The only acceptable value is `yes`. This marks package as essentially required.
Removal of such package should not be allowed.
Architecture::
This is package's architecture.
Conffiles::
This is multiline (space indented) where every line is space separated pair of
path and MD5 sum.
Installed-Time::
Time of package installation. This is decimal number of time since epoch.
Auto-Installed::
The only acceptable value is `yes`. This is used to label packages as installed
only because they are dependency of some other package.
Alternatives::
Space separated list of alternatives. Alternatives are essentially links.
Multiple packages can specify same alternative target and the one with highest
priority wins target. One alternative is triplet separated by `:`. First field
is priority. Second field is link path. Third field is link target.
==== Status triplet
First field specifies wanted state:
unknown::
Wanted state is effectively unknown.
install::
Package should be installed. This is the only value commonly expected in
`/usr/lib/opkg/status` file as there is commonly no reason to store in index
any other state.
deinstall::
Package should be deinstalled but not removed from system.
purge::
Package should be removed from system.
Second field contains comma separated list of flags:
ok::
This is default flag. This flag should be used if you do not want to use any
other flag. It is ignored.
hold::
This package should not be upgraded to newer version.
noprune::
Old packages from this package should not be removed but rather appended to
this package.
prefer::
This version of package is preferred.
obsolete::
This package is older than some other that replaces it.
user::
This package is requested by user.
And the last third field contains status:
not-installed::
The package is not in general installed.
unpacked::
Files of package are present in file-system.
half-configured::
Package is half-configured. This seems to be unused by Opkg at the moment (as
of 2020-08-31).
installed::
State when package is installed and configured. This is the desired state.
half-installed::
Package is in some way installed only partially. This seems to be unused by
Opkg at the moment (as of 2020-08-31).
config-files::
Package is not installed but config files are present and preserved in system.
This seems to be set but accepted by Opkg at the moment (as of 2020-08-31).
post-inst-failed::
Post-install of package failed. Such package is installed but can be
potentially in broken state. This seems to be unused by Opkg at the moment (as
of 2020-08-31).
removal-failed::
Removal failed in any way. Package is not fully removed. Files can be present
in system. This seems to be unused by Opkg at the moment (as of 2020-08-31).
=== Per-package files
There are some files for each installed package in the
`/usr/lib/opkg/info` directory. The files are named after the package
names, suffixed by the type of the file (eg. `package-name.type`). The
types are following:
control::
This is the description block taken vanilla from the package. Some of the info
is also found in the repository index list. It has email-like header format.
list::
Lists the files owned by the package, one file per line.
conffiles::
Lists the config files owned by the package (files that shall not be overwritten
by an upgrade of the package). Note that config files are also listen in `list`
file. The file is present only if the package has some config files.
preinst::
postinst::
prerm::
postrm::
Executable scripts that are run in the right moment of installation process.
They are present only if the package contains them.
There can be also other files. Those are used commonly by scripts. Opkg does not
use them but it manages them.
=== Package control files
`control` file contains additional fields that are not stored in
`/usr/lib/opkg/status` file. It is expected to be at least:
Section::
Name of section package is included in.
Maintainer::
Identified of maintainer of given package.
Source::
Path to package in in OpenWrt's build tree (relative to `packages` directory).
SourceName::
Name of source package. That is name used to build given package. This can be
different to `Package` value.
License::
License of packaged software.
LicenseFiles::
Path to license files in source. These files are not in general present in
resulting package so they are not accessible in running system.
Description::
This can be multi-line user readable description of package.
Tags::
This is list of package tags. This is not in OpenWrt's packages (as far as
OpenWrt 19.07).
Require-User::