Split the server-side daemon
Due to python limitation called GIL (global interpreter lock), the daemon can't use all the available cores to handle clients. Also, this makes it act oddly (in the bad way) when under load ‒ it starts dropping connections because the communication thread is slowed down by the background worker threads submitting data to database.
We want to be able to split the daemon into multiple precesses, each handling a subset of clients. This way we can spawn multiple to take advantage of the CPUs and in future start them on multiple machines.
While the handling of clients is mostly independent, there are few issues to think of:
- Events happening at the same time ‒ counts, bandwidth and possibly others have batches synchronized across all the clients.
- Coordinating of sniff batches.
- Making sure a client is connected to at most one daemon and making sure a reconnection doesn't produce inconsistencies in the DB (eg. the logout must be written before the login).
This seems to be possible to do with a coordination daemon that'd hold the timeouts, client status, etc and inform the master daemons.