Unified the approach to fd set events inspection.

There was also a small error when a TCP master socket was removed,
the next one was skipped regardless of it's activity level.