Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • Turris OS packages Turris OS packages
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 97
    • Issues 97
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 8
    • Merge requests 8
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • TurrisTurris
  • Turris OS
  • Turris OS packagesTurris OS packages
  • Issues
  • #705
Closed
Open
Issue created Dec 04, 2020 by Glenn Strauss@gstraussContributor

lighttpd: reduce memory usage of foris and reforis

lighttpd: reduce memory usage of foris and reforis

[I do not have permission to create a fork of turris-os-packages to be able to submit a merge request, so I am creating an issue here.]

On my 512 MB Mox, over 160 MB resident memory is being used by foris, reforis, foris-ws, and foris-controller. That is a significant amount of memory, especially given the occasional Turris OS upgrade issues sometimes due to running out of memory. (These processes also use a significant amount of CPU time, but that is another issue for another day)

foris and reforis both can be configured to use CGI rather than staying resident as FastCGI backends to lighttpd. However, that is not the default configuration.

I would like to propose a small change to foris and reforis configurations which employ lighttpd min-procs => 0 and start up FastCGI backends upon demand, then kill them after an idle period. The default idle period is 60 seconds, but in the below patch I configure 180 seconds (3 minutes), so that if you access foris or reforis web pages, the FastCGI backend will stick around or a few minutes as you navigate pages, but will be killed after you are done. The initial page load takes a few seconds, but after logging in, navigation is quick since the backend processes are still running until idle for 3 minutes.

Quick win: This small change immediately saves over 50 MB of resident memory.

diff --git a/web/foris/foris/files/lighttpd-dynamic-conf b/web/foris/foris/files/lighttpd-dynamic-conf
index 3a27bc422..a5f2dcfac 100644
--- a/web/foris/foris/files/lighttpd-dynamic-conf
+++ b/web/foris/foris/files/lighttpd-dynamic-conf
@@ -75,6 +75,8 @@ case $SERVER in
                echo "                          \"socket\" => \"/tmp/fastcgi.foris-$APP.socket\","
                echo "                          \"bin-path\" => var.foris-$APP.bin + var.foris-$APP.flags,"
                echo '                          "check-local" => "disable",'
+               echo '                          "idle-timeout" => 180,'
+               echo '                          "min-procs" => 0,'
                echo '                          "max-procs" => 1,'
 if [ -n "$SENTRY_DSN" ]; then
                echo '                          "bin-environment" => ('
diff --git a/web/reforis/reforis/files/reforis-lighttpd-dynamic.sh b/web/reforis/reforis/files/reforis-lighttpd-dynamic.sh
index 9064148af..032077108 100644
--- a/web/reforis/reforis/files/reforis-lighttpd-dynamic.sh
+++ b/web/reforis/reforis/files/reforis-lighttpd-dynamic.sh
@@ -45,6 +45,8 @@ case $SERVER in
                echo "                          \"socket\" => \"/tmp/fastcgi.reforis.socket\","
                echo "                          \"bin-path\" => var.reforis.bin,"
                echo '                          "check-local" => "disable",'
+               echo '                          "idle-timeout" => 180,'
+               echo '                          "min-procs" => 0,'
                echo '                          "max-procs" => 1,'
                echo '                          "bin-environment" => ('
 if [ -n "$SENTRY_DSN" ]; then

On my Turris Mox (Turris OS 5.1.4) (hbt) which was last rebooted 18 days ago, please note the memory use and CPU usage of foris, reforis, et al. The resident memory used by these programs is ~ 115x that used by lighttpd. (Yes, that's not 10x, that is OVER 100x) kresd is the top memory user on my system as expected, but foris processes are the next 7 largest.

 2757 root      20   0  124.7m  34.8m   0.0   7.2   2:44.96 S /usr/bin/kresd --noninteractive -c /tmp/kresd.config /tmp/kresd -a 0.0.0.0#53 -a ::#53
 3898 root      20   0   42.3m  32.2m   0.0   6.6   1908:39 S /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.plain --controller-id 0000000D30007440
 4476 root      20   0   41.4m  31.2m   0.0   6.4 137:20.58 S /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.plain --controller-id 0000000D30007440
 4356 root      20   0   37.1m  28.2m   0.0   5.8 120:59.20 S /usr/bin/python3 /usr/bin/reforis
 4355 root      20   0   33.8m  24.6m   0.0   5.1 122:22.64 S /usr/bin/python3 /usr/bin/foris -s flup -a config -b mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain --mqtt-controller-id 0000000D30007440
 3218 root      20   0   26.3m  17.0m   0.0   3.5  76:45.56 S /usr/bin/python3 /usr/bin/foris-ws -a ubus filesystem --host 127.0.0.1 --port 9080 mqtt --mqtt-host localhost --mqtt-port 11883 --mqtt-passwd-file /etc/fosquitto/credentials.plain
 4178 root      20   0   30.6m  15.3m   0.0   3.2  19:12.81 S /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.plain --controller-id 0000000D30007440
 4151 root      20   0   29.4m  12.1m   0.0   2.5  19:12.66 S /usr/bin/python3 /usr/bin/foris-controller -b openwrt -C /var/run/foris-controller-client.sock mqtt --host localhost --port 11883 --passwd-file /etc/fosquitto/credentials.plain --controller-id 0000000D30007440
[...]
 3916 root      20   0    5.7m   1.4m   0.0   0.3   4:45.91 S /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
Assignee
Assign to
Time tracking