Count inactivity since last poll time.

This is a backport from packet-api-rewrite branch (sort of).
The problem is when we add a socket to the set and then do a lengthy
operation (transfer answer), it may be sweeped at the end without a
chance of querying.