CERT
Software Assurance Secure Systems Organizational Security Coordinating Response Training
Child pages
  • Removing duplicate flows
Skip to end of metadata
Go to start of metadata

In the case of a malfunctioning flow sensor that is producing duplicate flow entries, the rwdedupe tool provides a means of removing the duplicate flow data.

Do NOT use rwtuc as a means to apply UNIX tools to flow data to perform this operation, as in:

  rwcut myflows.raw >myflows.txt
  sort myflows.txt | uniq >tempflows.txt
  rwtuc tempflows.txt >mynewflows.raw

This is MUCH slower than using the binary flow-manipulation tools, and won't handle large volumes of data.

Rather, do it this way:

  rwdedupe --stime-delta=1 myflows.raw | rwsort --fields=1-9 >mynewflows.txt
  • No labels