performance problem because of shared cache

I am making benchmarks for a project. And sending 10M queries to resolvers for test.

• Every test starts with cold start.
• Opening 8 process.
• measuring %core, pps, elapsed miliseconds, and download Mbps.

I founded an interesting result.

Opening 8 process with shared cache at the same folder (/var/cache/knot-resolver) vs 8 process with different cache folders

results look like these values (approximately)

• each core (every 8 cores)
• % 60 - %99
• pps
• 20000- 30000
• elapsed miliseconds
• 500 - 300
• download Mbps
• 100 - 160

conclusion: using shared cache slows down performance dramatically. Is there a way to fix this problem?

changed the description

We haven't observed such a slow-down. 8 is rather a low count. Still, performance is affected by so many things.

Anyway, first make sure that the cache is on a tmpfs and not a real disk; otherwise the disk I/O often causes issues.

I'm not convinced by your results, I think many important info is missing, e.g.:

• how many of the responses are NOERROR vs SERVFAIL vs REFUSED etc.?
• what data are you benchmarking with? using single query will yield drastically different results than using a data set with unique queries

Benchmarking resolver performance isn't simple, but I can recommend a tool we've created for this purpose if you're interested in realistic benchmarks: https://dns-shotgun.readthedocs.io

• I totally agree with you, benchmarking resolver is so hard. I will check dns-shotgun
• I have also tested tmpfs cache vs real disc. there is a difference but not so much.
• I am testing with a 8 Rx and Tx queue len (there is not way to make it upper in my environment). and starting every process for listening 1 different ip.
• Every query is unique
• also every query will result NOERROR.
• 1M query retried one more because of response timeout.
• I am getting these numbers from my benchmark software that I have written with libuv
• I accidentally discovered this. when I have started 8 docker image on different listen ips.
• I am following dropped udp packets.
• using sysctl values below

net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216
net.core.wmem_default = 4194304
net.core.rmem_default = 4194304
net.ipv4.ip_local_port_range = 1024 65535
 fs.file-max = 100000000
 fs.nr_open = 100000000

I will try with dns-shotgun. but probably you have a performance check environment. And you knot how dns-shotgun works better than me, please make a comprasion with shared cache and non shared cache.

I am trying to find out,what I am doing something wrong.

If you're using docker, beware of their overhead for networking and also for cache-sharing not working unless you share the PID namespace.

No I am not using docker. I realized these status when I tested with docker. I was expecting a low performance. But got high performance.and I wondered why? and I started 8 kresd process with different kresd.conf files with cache.open(512*MB,'llmd:///tmp/cache/differentForEachProcess')

Thanks for the more detailed description. I might run some performance tests later to compare these two cases.

Every query is unique

I think this probably favors the non-shared cache scenario, since sharing the cache in such case provides very little benefit. However, in real DNS traffic once the cache is warmed up, I'd say 90-95% of queries will be answered from cache.

Right, that might be the main cause. LMDB design isn't good with many parallel write attempts.