can't reliably fetch stats when using SO_REUSEPORT

I'm using knot resolver with systemd, and want to use the stats module + http module to fetch stats in prometheus format.

My problem is that if I start more that one instance (kresd@1, kresd@2, …), stats fetching requests are distributed among the instances and returns only the stats from the answering instance.

I can't get a reliable way to fetch the stats in such configuration.

Workaround: I can fetch and aggregate individual workers stats from the controls sockets, but the control socket is very unreliable (it is not able to properly parse 2 successives queries properly and often try to interpret them as a single query).

Hi @jddupas and welcome. You are tight that SO_REUSEPORT makes HTTP endpoint unsuitable for gathering stats, we will think about it. In meanwhile let's have a look at the control socket.

Please post the code you use to gather stats from control socket, we will investigate where could be a problem.

@amrazek This is something to consider in your future work.

Ok. Actually, I'm trying to access the socket directly using python code.

The code is rather simple.

Opening a unix socket. Writing and reading from that socket. My problem is that when I send 2 queries one after an other like this:

sock.send(b"__binary\n")
sock.send(b"stats.list()\n")

Some times, in the knot-resolver side, libuv will only report one event containing the 2 queries (that is call tty_process_input with a single buffer containing "__binary\nstats.list()\n")

Unfortunately, tty_process_input strongly assumes that a buffer contains a single whole query (which look rather unsafe knowing how low-level IO routines have the bad habit to rarely truncate buffer where we expect them to do it).

I tried to mitigate it by trying to disable socket buffer and add flush on the python side, but I'm not sure these settings are properly honored.

sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 0)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 0)

Right. The module for prometheus only tries to handle this when multiple instances are started via the -f parameter and not separately (which is what we do in our systemd packages).

Thanks for finding this problem with delimitation of commands in the control socket

added bug label

Yeah, this is certainly a bug!

Here is a workaround: Do it the other way around and write out from the instances elsewhere, e.g. to a per-instance file, and then read it from there. Yes, it is ugly, sorry about that!

Example Lua code suitable for kresd config:

modules.load('stats')

-- determine own pid
local statfile = io.open('/proc/self/stat')
local mypid = tostring(statfile:read('*n'))
statfile:close()
statfile = nil

-- write current stats
local function write_stats()
	statfile = io.open ('/tmp/stats.' .. mypid, 'w')
	statfile:write(table_print(stats.list()))
	statfile:close()
end

-- write stats every 1 second
event.recurrent(1000, write_stats)

We are starting a project to overhaul user interface so you can look forward to bright future ... but it will take couple months or so. Thank you for patience!

assigned to @amrazek

Thanks for the workaround. I finally managed to get a somewhat reliable solution by adding a sleep() between the 2 commands in my python code.

By working on this, I discovered an important point that you may want to take into account when working on this:

When exporting to prometheus (at least) you can't aggregate the multiple workers stats simply by adding all the values and creating series with the results.

Prometheus expect that counters are monotonic, and can detect and handle counter reset. But if you aggregate all workers stats into a single counter, if a single worker restart, it break monotonicity, but does not properly reset the counter, which may result in inconsistent series in prometheus.

For instance, instead of creating a single anwser_total series containing the sum of all worker stats:

   answer_total    5208

you have to create a series per worker (using labels) like this:

   answer_total{worker="1"}    2563
   answer_total{worker="2"}    2645

In the case above, if worker 1 restart and worker 2 keep running, the first counter will be properly reset, and the second still be monotonic.

FYI it is possible to store "previous" stats and export only differences from each individual process, conceptually:

Working example (can be generalized):

stats_prev = stats.list()
function get_stat_increment()
        local stats_now = stats.list()
        local doh_increment = (stats_now['request.doh'] - stats_prev['request.doh'])
        local dot_increment = (stats_now['request.dot'] - stats_prev['request.dot'])
        local internal_increment = (stats_now['request.internal'] - stats_prev['request.internal'])
        local answer_c_increment = (stats_now['answer.cached'] - stats_prev['answer.cached'])
        local total_increment = (stats_now['request.total'] - stats_prev['request.total'] - internal_increment) 
        stats_prev = stats_now
        return {doh=doh_increment,dot=dot_increment,total=total_increment,answer_c=answer_c_increment}    

assigned to @ljezek and unassigned @amrazek

mentioned in merge request !1000 (merged)

For reference, the issue with receiving multiple commands in one syscall should be solved since release 5.1.1: https://gitlab.labs.nic.cz/knot/knot-resolver/-/merge_requests/991. Newlines now work as command separators. Sending very long "strings" over a single connection still probably won't be reliable, but I don't think your use case is anywhere near that.