Skip to content
Snippets Groups Projects

Abort traffic replay if servers are unresponsive

Open Nicki Křížek requested to merge 32-crash-on-nonresponding-server into master
All threads resolved!

If the target server doesn't send any reply within the configured time limit (when some are expected), abort the traffic replay. This most commonly happens due to either incorrect configuration or server crash. In the case of premature exit, ensure that existing statistics are collected.

The default (baked-in) server timeout limit is configured to be 16 seconds. This has proven to be the right magic number in the development of related tools (respdiff). It is unlikely that this limit would have to be reconfigured or disabled, so no way to adjust or turn off this limit is provided until a reasonable use-case is presented.

Closes #32

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • requested review from @isc-pspacek

  • Initially, I hoped to extend this to also be able to handle signals such as SIGINT. However, that would be quite messy and complicated -- python script would have to propagate the signal to the lua script, and then the real fun begins. Lua being the minimal language that it is, it provides no means to handle signals and while there are some pieces of code which might work, they aren't packaged anywhere and last development happened 10 years ago, as is customary in the lua space...

    TL;DR I'm not going to bother with this. It'd be a nice feature, but it's not worth going down this rabbit hole (for me, anyway).

  • added 1 commit

    Compare with previous version

    • Resolved by Petr Špaček

      It looks like it's almost working :-)

      Sometimes it can hang on shutdown:

      $ pcap/generate-const-qps.lua -q 128 | ./replay.py -s 127.0.0.1 -c configs/tcp.toml -r - -T3
      2024-03-05 15:31:06,552     INFO  Output directory: /var/tmp/shotgun/1709649067
      2024-03-05 15:31:06,552     INFO  Thread distribution:
      2024-03-05 15:31:06,552     INFO    (main): 1 thread(s)
      2024-03-05 15:31:06,552     INFO    TCP: 2 thread(s)
      2024-03-05 15:31:06,561     INFO  Configuration sucessfully created
      2024-03-05 15:31:06,561     INFO  Firing shotgun...
      TCP-02 notice: set target to 127.0.0.1 port 53
      TCP-02 notice: transport set to TCP
      TCP-01 notice: set target to 127.0.0.1 port 53
      TCP-01 notice: transport set to TCP
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:       256; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:       384; answers:         0; discarded:         0; ongoing:       352
      TCP-02 notice: total processed:       512; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:       640; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:       768; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:       896; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1024; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1152; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1280; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1408; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1536; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1664; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1792; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      1920; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      2048; answers:         0; discarded:         0; ongoing:       256
      TCP-01 notice: total processed:         0; answers:         0; discarded:         0; ongoing:         0
      TCP-02 notice: total processed:      2176; answers:         0; discarded:         0; ongoing:       256
      TCP-02 critical: target server not responding
      core notice: dnssim error detected, finishing up
      core notice: processed 2240 packets from input PCAP
      TCP-01 notice: results exported to /var/tmp/shotgun/1709649067/data/TCP/TCP-01.json
      TCP-02 notice: results exported to /var/tmp/shotgun/1709649067/data/TCP/TCP-02.json
      core fatal: dnssim error caused premature abort
    • Petr Špaček Nicki Křížek Last reply by Petr Špaček
  • Nicki Křížek mentioned in merge request !97

    mentioned in merge request !97

  • Petr Špaček resolved all threads

    resolved all threads

  • Petr Špaček requested review from @ostava and removed review request for @isc-pspacek

    requested review from @ostava and removed review request for @isc-pspacek

  • It seems to work but I can't properly review implementation. @ostava Can you have a look, please?

Please register or sign in to reply