turris-maintain: Network restart leaves net-related services in a broken state
I noticed this a while back when attempting to change network-related settings within Foris — if you change any such settings, Foris will restart the networking stack via /usr/bin/maintain-network-restart
which in turn calls /etc/init.d/network restart
. Upon doing so, certain services on the router - at least OpenVPN and networking in LXC containers - are now non-functional, requiring different manual restart strategies to get them working again. This becomes a significant problem if you're relying on them when you make the change (eg you're connected via OpenVPN or your DNS served by process in LXC). Likewise, it's easily forgotten if you change wifi settings and forget that this restarts the whole networking stack, which then flows on.
Here's my breakdown so far:
-
OpenVPN - the service remains up and connectable from a client, but traffic does not route. Looking into this, the
tun_turris
network interface on the router loses its IP addresses as shown inifconfig tun_turris
after the network restart occurs. OpenVPN requires a restart to bring it back online. -
LXC containers - the containers remain up but processes inside the container can't connect out (even to the the Turris Omnia host) and likewise nothing can connect in, even if the network is bridged. The state of the container shows interfaces still up with IP addresses but an empty routing table (eg output from
route
is empty) after the network restart occurs. A full restart of the container (eglxc-stop
&lxc-start
) corrects the interface/routing problem.
In short, there should be some strategy in place for restarting services that depend on the network after it restarts. I'm unfamiliar with specifics of OpenWrt's init system, but that might be an option if it support dependencies. Otherwise, adding to turris-maintain
's scripts directly (if no other code is directly restarting the network) or else associating hook scripts to be trigged when /etc/init.d/network
gets run.
As a workaround, I've amended /usr/bin/maintain-network-restart
with the following at the end of the file (eg after network restart):
time.sleep(5) # Allow time for network to come online; OpenVPN needs this
subprocess.call(['/etc/init.d/openvpn', 'restart'])
subprocess.call(['/usr/bin/lxc-stop', '--name', 'mydnsserver'])
subprocess.call(['/usr/bin/lxc-start', '--name', 'mydnsserver'])
"
This way, at least when I make changes to LAN/WAN, OpenVPN or WiFi in Foris, all my services come back up okay and my network is functional.