daemon: lower EDNS buffer size to 1232

Closes #538 (closed), #300 (closed).

changed milestone to %5.0.0

added easyfix label

The main question is when to merge this. The date of DNS Flag Day 2020 hasn't even been decided yet, though e.g. PowerDNS have switched already IIRC and generally no issues are expected.

• write a NEWS entry (probably early for this ATM)

added 1 commit

• 0d662250 - fixup! daemon: lowered EDNS buffer size to 1232

Compare with previous version

marked as a Work In Progress from 0d662250

I just peeked at the test results from respdiff. A lof of fwd tests timed out, but one that did finish showed huge increase in target_disagreements. (This also explains the other jobs timing out, since diffrepro takes quite a while)

ERROR    target_disagreements: threshold exceeded! sample: 8694 / 100.00% vs threshold: 724 / 98.00%

I wouldn't release this in 5.0. Let's wait until the other vendors are close to releasing version with updated default EDNS buffer size.

removed milestone

added 163 commits

• 0d662250...4a8d6bfb - 161 commits from branch master
• e96dbecf - daemon: lowered EDNS buffer size to 1232
• 5751b037 - docs: EDNS bufsize = 1232

Compare with previous version

assigned to @ljezek

I was thinking about this for a while and I think we have to split EDNS buffer size for clients and for upstream. This split will enable better comparison in respdiff, right now it is full of noise because of TC bit etc.

So the next step is to separate current net.bufsize() to two values:

• net.bufsize_upstream() for communication with other servers
• net.bufsize_downstream() for communication with clients

Maybe we will rename this but I do not have a better idea at the moment.

TODO:

• split net.bufsize()
• retest in respdiff with downstream value 4096 and upstream 1232
• think about backwards compatibility - should we remove net.bufsize() or keep it so it can set both values at once?
• if we keep it, what it should return if the values for upstream and downstream differ?
• document new functions

@tkrizek Any opinion on backwards compatibility in Lua config?

Letting it set both values sounds good to me, should be easy enough and good for compatibility. The get doesn't seem so important; EDIT: it's also possible to return two values (so the second one is simply ignored on normal assignment).

An alternative approach to two functions is to let net.bufsize() accept one or two parameters (positional); EDIT: that would sound consistent with the two-valued return, too.

added usability label and removed easyfix label

net.bufsize() that sets both values sounds good. It should return two values, in all cases. The first value should be the one that is more relevant to the user (probably the downstream one?), so it doesn't impact backward compatibility.

net.bufsize(both) and net.bufsize(downstream, upstream) seem a bit better than two additional functions.

added 242 commits

• 5751b037...6552d471 - 239 commits from branch master
• 756dc36d - daemon: lowered EDNS buffer size to 1232
• bd95122d - docs: EDNS bufsize = 1232
• 8d2b53a5 - net: Split of the EDNS buffer size into upstream and downstream

Compare with previous version

resolved all threads

added 1 commit

• 935fa915 - WIP: test new upstream bufsize in CI

Compare with previous version

added 30 commits

• 935fa915...87253ef7 - 26 commits from branch master
• a64e79e3 - daemon: lowered EDNS buffer size to 1232
• 9c441080 - docs: EDNS bufsize = 1232
• b45c516e - net: Split of the EDNS buffer size into upstream and downstream
• 3dc76c19 - WIP: test new upstream bufsize in CI

Compare with previous version

TODO: Split into two MRs - one with new net.bufsize() command and second which changes defaults.

mentioned in merge request !1026 (merged)

Split net.bufsize() moved into !1026 (merged)

changed title from WIP: daemon: lowered EDNS buffer size to 1232 to WIP: daemon: lower EDNS buffer size to 1232

changed milestone to %5.2.0

added 156 commits

• 9c441080...88e1b1ee - 154 commits from branch master
• 465a3120 - daemon: lower EDNS buffer size to 1232
• 4500a90f - docs: EDNS bufsize = 1232

Compare with previous version

added 1 commit

• 84e642e9 - ci/respdiff: keep bufsize towards respdiff at 4K

Compare with previous version

Respdiff waiting for respdiff!90 (merged)

Otherwise I think it's ready, though it will be nice to do some larger respdiff runs after the above is merged (merge needed to avoid this MR skewing comparisons).

added 8 commits

• 84e642e9...adccb464 - 5 commits from branch master
• b7bd3fd5 - daemon: lower EDNS buffer size to 1232
• 6c0b8a8d - docs: EDNS bufsize = 1232
• 46c87605 - ci/respdiff: keep bufsize towards respdiff at 4K

Compare with previous version

mentioned in issue #300 (closed)

added 42 commits

• 46c87605...a5c0c695 - 39 commits from branch master
• 87a5e1bc - daemon: lower EDNS buffer size to 1232
• e7db68a6 - docs: EDNS bufsize = 1232
• ee2dfe0f - ci/respdiff: keep bufsize towards respdiff at 4K

Compare with previous version

Currently, respdiff sets the downstream bufsize to 4096 and keeps the upstream bufsize as-is - which means 1232 for this MR, but 4096 for our reference sample running on master.

The current state is that fwd-unbound jobs fails to finish in time on this MR. When I ran it manually, I got the following results.

== Global statistics
duration                    4842           seconds                                        
queries                   616341                                                          
answers                   616341  100.00 % of queries                                     

== Differences statistics
manually ignored               0    0.00 % of answers (ignoring)                          
upstream unstable           6278    1.02 % of answers (ignoring)                          
not 100% reproducible          0    0.00 % of answers (ignoring)                          
target disagrees           35678    5.85 % of not ignored answers                         

== Target Disagreements
Field           Count    % of mismatches
------------  -------  -----------------
rcode           35643              99.90
flags              15               0.04
timeout            12               0.03
answertypes         6               0.02
answerrrsigs        2               0.01

== Field "timeout" mismatch statistics
Expected    Got        Count    % of mimatches
----------  -------  -------  ----------------
answer      timeout       12              0.03

== Field "rcode" mismatch statistics
Expected    Got         Count    % of mimatches
----------  --------  -------  ----------------
NOERROR     SERVFAIL    31317             87.78
NXDOMAIN    SERVFAIL     4322             12.11
SERVFAIL    NOERROR         2              0.01
NOERROR     NXDOMAIN        1              0.00
NXDOMAIN    NOERROR         1              0.00

j._report.txt

j._report.json

It's interesting that kresd takes a lot of time before answering SERVFAIL now:

Compared to the current situation on master:

added 96 commits

• ee2dfe0f...5b474f2b - 93 commits from branch master
• 2665b56f - daemon: lower EDNS buffer size to 1232
• 0f9da06a - docs: EDNS bufsize = 1232
• 6857b7f0 - ci/respdiff: keep bufsize towards respdiff at 4K

Compare with previous version

resolved all threads

unmarked as a Work In Progress

approved this merge request

enabled an automatic merge when the pipeline for 6857b7f0 succeeds

canceled the automatic merge

unapproved this merge request

There's still some issue with iter.tls6 test. The number of SERVFAILs is sligthly increased by about 100-150 (to about 2k differences total). It is reproducible, against current master, when tests are executed at the same time, even with decreased number of jobs. For reference, this is my attempt with just 128 parallel jobs (half of the throughput currently used in CI).

added 7 commits

• 6857b7f0...2852f81c - 4 commits from branch master
• 6e288cf4 - daemon: lower EDNS buffer size to 1232
• 1d747f77 - docs: EDNS bufsize = 1232
• 3904435d - ci/respdiff: keep bufsize towards respdiff at 4K

Compare with previous version

added 1 commit

• bc2fbf00 - DROP: turn off gnutls logging

Compare with previous version

marked as a Work In Progress

assigned to @tkrizek and unassigned @ljezek

added 14 commits

• bc2fbf00...74237b5b - 11 commits from branch master
• af5ddacc - daemon: lower EDNS buffer size to 1232
• 8a50589b - docs: EDNS bufsize = 1232
• eadc0515 - ci/respdiff: keep bufsize towards respdiff at 4K

Compare with previous version

resolved all threads

unmarked as a Work In Progress

approved this merge request

unassigned @tkrizek

approved this merge request

mentioned in commit 43b40577

merged