Proto flow gatherer
Currently we have an adhoc data structure that reads from the guts daemon and modifies some internal data structures (and performs something like 10 different conversions of data structures on the way).
Rip the part that keeps the track of them out. This'll keep the data in a unified form. Each proto slice would contain (similar to flow slices in #34 (closed)):
- Current statistics, if any are available.
-
Rc<RefCell<Headers>>
. - Some other auxiliary data, possibly.
Furthermore, make sure we can have multiple lookup keys at once (eg. a flow ID in guts, flow ID in suricata, the (ip, ip, proto, port, port) tuple). Think about how we can produce them as we add information and if we want to keep them in a map or multiple.
The processing will stay similar, but headers may be added at any time. This may generate new keys in the map. Make sure they are removed once we drop the flow.
One option around these keys would be:
- Have a register of key types (each one simply lists some interesting columns)
- When the columns are added, the keys are computed and remembered. We need some kind of mapping from the new column to the relevant columns.
- The keys are then put into the map.
- Once removing the flow, go through the storage of computed keys and drop all of them.
When we want to add info to a flow, parametrize it by at least one key.
What do we want to do if we would put a duplicate key into the map? Merge them together?
Also note we get rid of some conversions when we produce the flow ‒ we already have the statistic and headers ready. And we don't produce new copies of the headers.