dns-crawler-controller never stop, queue information wrong
Problem
Somehow the dns-crawler-controller
never finish all jobs in the queue, or the worked doesn't receive new information from the queue. Based on the information the queue is also empty but the number of jobs is still listed in the dns-crawler-controller
.
dns-crawler-controller
The below part is repeated all the time, so I expect some jobs need to be finished.
...
[2022-05-25 17:06:08] 9207/9940
[2022-05-25 17:06:13] 9207/9940
[2022-05-25 17:06:19] 9207/9940
[2022-05-25 17:06:24] 9207/9940
[2022-05-25 17:06:29] 9207/9940
rq info
But from rq
it looks like nothing todo:
default | 0
1 queues, 0 jobs total
0 workers, 1 queues
Updated: 2022-05-25 17:07:10.203003
dns-crawler-workers
All worker unsubscribe because nothing is todo based on the queue information.
17:07:32 Unsubscribing from channel rq:pubsub:crawl-01.XXXX-100
Environment
I've three dns-crawler-workers
on three different nodes with different IPs. All are connected to a public Redis instance on which also the controller
is running.
My redis.conf
is really minimal:
daemonize yes
pidfile /var/db/redis/redis.pid
logfile /var/log/redis/redis.log
maxmemory 2G
protected-mode no
save ""
stop-writes-on-bgsave-error no
I've configured a timeouts.jobs
with 15 seconds, which I could also see is reached sometimes on the workers
.
What I expect
I expect that the queue and jobs are handled correctly so the controller
really finish at some point in time. I would love that the timeout are somehow reported in the results document for the domain - or somehow anywhere that I could reschedule it.