manager: datamodel: max_workers configuration

Issue follows the comment in !1280 (merged).

Basically, we're not sure if it's a good idea to allow max-workers to be configured, capping the max number of workers. It could confuse users with workers option that determines the requred number of workers.

Currently, the maximum number of workers is also capped during validation by 10 workers per CPU.

added manager label

mentioned in merge request !1280 (merged)

Perhaps someone should describe the intentions here, as they're not clear to me at all.

changed the description

assigned to @vsraier

Let me explain this a bit... Why does this issue exist?

This is a snippet from supervisord's config:

[program:kresd]
process_name=%(group_name)s%(process_num)d
numprocs=8
redirect_stderr=false
stdout_logfile=logs/kresd%(process_num)d.log
stderr_logfile=logs/kresd%(process_num)d.log
directory=/home/vasek/Work/knot-resolver/manager/etc/knot-resolver/runtime
command=/home/vasek/Work/knot-resolver/manager/.install_kresd/sbin/kresd -c kresd%(process_num)d.conf -n
autostart=false
autorestart=true
stopsignal=TERM
killasgroup=true
startsecs=10
environment=SYSTEMD_INSTANCE="%(process_num)d",X-SUPERVISORD-TYPE=notify

On the 3rd line, you can see numprocs=8. That tells supervisord to internally prepare 8 instances which we can start later. If we would want to start more, we can't just call startProcess("kresd:kresd8"). We can start only instances numbered 0-7.

And sadly, we can't set numprocs=100000. Interactions with supervisord would become extremely slow. Especially asking for state of processes, because then it would list all 100000 processes to tell us they are stopped.

So currently, the number in numprocs is configurable and the user is not allowed to start more processes without restarting everything.

---

Recently, I've tried to get rid of this configuration option and configure it dynamically based on demand. There is however a major issue with that - supervisord does not apply the config until all instances of the particular config are stopped. So I've been trying to figure out a workaround for this limitation. But all options are somewhat sketchy and I don't like them. The easiest and most honest option to me seems to leave the configuration option available for power users and make the defaults such, that nobody is likely to run into this limitation. Do you agree or should I really try to remove it?

---

Additional to the configuration described above, manager also checks that you are not starting stupid amount of workers. It's not a problem to drop this validation, it's there just as a sanity check trying to stop accidental mistakes. If you set the number of workers to high, kresd+manager effectively become a sophisticated fork bomb.

I see, OK.

Default: 256 sounds high enough to cover all except extreme deployments. Sounds good if the overhead there is still low enough for small devices. Anyway, we can easily adjust such details when the need arises. I can imagine that the default would depend on detected number of CPUs (during startup), too.

My idea is somehing like this:

• Get number of CPUs during the manager startup.
• MAX_WORKERS = cpus * 10 (max 10 workers per CPU), If number of CPUs is not somehow available use preconfigured value, e.g. 256
• Use MAX_WORKER as default in configuration schema for max-workers option.
• User can configure max-workers in yaml to overwrite the default value.

Sounds good to me. I'll do that.

mentioned in commit 51ee4f7b

mentioned in merge request !1330 (merged)

closed