Monitoring for Large-Scale Networks

YAF

yafMeta2Pcap

NAME

yafMeta2Pcap - YAF PCAP Metadata File Parser and PCAP Creator

SYNOPSIS

    yafMeta2Pcap [--pcap-meta-file META_FILE_PATTERN]
                 [--pcap PCAP_FILE_PATTERN]
                 [--caplist CAP_FILE_NAME]
                 [--metalist META_FILE_NAME]
                 [--out OUTPUT_SPECIFIER]
                 [--hash FLOW_KEY_HASH]
                 [--time FLOW_START]
                 [--etime FLOW_END]
                 [--packets NUM_PACKETS]
                 [--yaf-program PATH_TO_YAF]
                 [--verbose]

DESCRIPTION

yaf(1) has several PCAP export options available. yaf will create a pipe-delimited file that contains information to assemble a PCAP file for any particular flow. yafMeta2Pcap parses the metadata file created by yaf and creates a PCAP file for the given flow identified by the flow key hash and start time.

yafMeta2Pcap has the ability to read IPFIX from getFlowKeyHash(1) on stdin to avoid typing the flow key hash and start time on the command line. yafMeta2Pcap will accept the first 2 records from getFlowKeyHash. If the first 2 records have the same start time, yafMeta2Pcap will sum the values for packetTotalCount to determine the total number of packets to search for (to account for uniflows). yafMeta2Pcap will only search for the first hash and start time it receives. See below for examples for using IPFIX input.

OPTIONS

The following options are required for yafMeta2Pcap to create the PCAP file.

--pcap-meta-file META_FILE_PATTERN

META_FILE_NAME is the path to the pcap-meta-file as created by yaf. META_FILE_NAME is a file glob pattern, which must be escaped or quoted to prevent the shell expansion if using wildcards. Files that match this pattern will be processed by yafMeta2Pcap. Alternatively, you can use --metalist to provide a text file with an ordered list of pcap-meta-files created by yaf. Alternatively, you can set the YAF_PCAP_META_FILE environment variable to the full path to the pcap-meta-file created by yaf to avoid including it to the command line.

--pcap PCAP_FILE_NAME

PCAP_FILE_NAME is the path to the PCAP file that was given to yaf to create the pcap-meta-file. PCAP_FILE_NAME is a file glob pattern, which must be escaped or quoted to prevent shell expansion if using wildcards. The option is only required if full path is not available in the PCAP metadata file. yafMeta2Pcap will open subsequent PCAPs only when the third column in the PCAP-meta-file changes. For example, if the third column is 0, yafMeta2Pcap will open the first PCAP given to --pcap. Alternatively, you can use --caplist to give a text file with an ordered list of PCAP files.

--caplist CAP_FILE_NAME

Instead of listing multiple PCAP files, use CAP_FILE_NAME as an ordered newline-delimited list of pathnames to PCAP dumpfiles as read by yaf. Alternatively, you can set the YAF_CAPLIST environment variable to the full path of the CAP_FILE_NAME to avoid adding it to the command line.

--metalist META_FILE_NAME

Instead of listing multiple PCAP metadata files, use META_FILE_NAME as an ordered newline-delimited list of pathnames to PCAP metadata files as created by yaf. Alternatively, you can set the YAF_PCAP_META_LIST environment variable to the META_FILE_NAME to avoid adding it to the command line.

--out OUTPUT_SPECIFIER

The full path of the PCAP file that will be created containing just the packets from the given flow. If this option is not present, yafMeta2Pcap will simply write the filename(s) of the PCAP(s) that contained the flow with the given FLOW_KEY_HASH. yafMeta2Pcap will overwrite any existing file at OUTPUT_SPECIFIER.

--hash FLOW_KEY_HASH

The flow key hash of the flow as created by getFlowKeyHash. Use getFlowKeyHash with the flow's 5-tuple and vlan to calculate the flow key hash. A leading 0x indicates FLOW_KEY_HASH is hexadecimal, otherwise decimal is assumed. If this option is not present, yafMeta2Pcap assumes IPFIX will be present on stdin.

--time FLOW_START

The start time of the flow. This option is recommended but it is not required; if not given, all packets matching the FLOW_KEY_HASH will be given, which may include multiple flows. Valid formats for FLOW_START are:

YYYY-MM-DDThh:ss:ss[.sss...]: an ISO8601 format, where the decimal point and fractional seconds are optional, e.g, 2023-07-18T12:23:34.56789
YYYY-MM-DD hh:ss:ss[.sss...]: similar to the previous, except a single space separates the date and time, e.g, 2023-07-18 12:23:34.56789
ssssssssss: the number milliseconds since the UNIX epoch as an integer, e.g., 1689683014567

getFlowKeyHash can calculate the milliseconds since epoch given the date and time.

--etime FLOW_END

The end time of the flow. This value is exported by getFlowKeyHash if present and accepted by yafMeta2Pcap. This option is suggested since it is likely to speed up search times, but it is not required. This argument overrides any end time information collected via IPFIX from getFlowKeyHash. The valid formats are the same as --time.

--packets NUM_PACKETS

NUM_PACKETS is the total number of packets for the bi-flow. This optional argument may speed up processing because it tells yafMeta2Pcap when it has found the entire flow. This can also be used if you only want the first NUM_PACKETS of the flow instead of the entire flow. This argument overrides any packet count information collected via IPFIX from getFlowKeyHash.

--yaf-program PATH_TO_YAF

PATH_TO_YAF is the location of the yaf program to use to create the PCAP file if OUTPUT_SPECIFIER is present and the pcap-meta-file has the format:

 hash | stime | path_to_pcap

If not specified, yafMeta2Pcap assumes there is program yaf on your $PATH.

--verbose

If present, print debug information to stdout.

Examples

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

The following example shows how to run yaf, getFlowKeyHash, and yafMeta2Pcap to create a PCAP for a given flow.

 $ yaf --in /data/some_large_file.pcap --out /data/ipfix.yaf \
      --pcap-meta-file /tmp/pcap-info

 $ getFlowKeyHash --sip4 10.10.25.11 --dip4 10.10.26.12 --sport 1586 \
                --dport 80 --protocol 6 --vlan 3  --date 2011-03-21 \
                --time 16:15:54.256

             sIP|            dIP|sPort|dPort|pro| vlan|      hash|                  ms
     10.10.25.11|    10.10.26.12| 1586|   80|  6|    3| 100819735|       1300724154256

 FILE PATH: 256/100819735-2011321161554_0.pcap

 $ yafMeta2Pcap --pcap-meta-file /tmp/pcap-info20150107141533_00000.meta \
                --pcap /data/some_large_file.pcap \
               --out /data/my-new-pcap.pcap \
               --hash 100819735 --time 1300724154

The following example shows how to run getFlowKeyHash directly to yafMeta2Pcap and use environment variables.

  $ export YAF_PCAP_META_FILE=/tmp/pcap-info20150107141533_00000.meta
  $ export YAF_CAPLIST=/tmp/yafcaplist.txt
  $ getFlowKeyHash --sip4 10.10.25.11 --dip4 10.10.26.12 --sport 1586 \
                --dport 80 --protocol 6 --vlan 3  --date 2011-03-21 \
                --time 16:15:54.256 -I | yafMeta2Pcap -o /data/my-new-pcap -n 10

The following example demonstrates how to use SiLK tools with getFlowKeyHash and yafMeta2Pcap. This assumes the appropriate environment variables are set.

  $ rwfilter --application=80 --pass-dest=stdout --max-pass-records=2 file.rw | \
       rwsilk2ipfix | getFlowKeyHash -I | yafMeta2Pcap -o /data/mynewpcap.pcap

The following example shows how to configure yaf to create rolling PCAP files and use yafMeta2Pcap to create a PCAP for a specific flow.

 $ yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 \
       --pcap-meta-file /tmp/pcap-info --pcap /data/pcap \
       --live pcap --index-pcap

 $ getFlowKeyHash --sip4 10.10.25.11 --dip4 10.10.26.12 --sport 1586 \
                  --dport 80 --protocol 6 --vlan 3  --date 2011-03-21 \
                  --time 16:15:54.256

            sIP|            dIP|sPort|dPort|pro| vlan|      hash|                  ms
    10.10.25.11|    10.10.26.12| 1586|   80|  6|    3| 100819735|       1300724154256

 FILE PATH: 256/100819735-2011321161554_0.pcap

 $ yafMeta2Pcap --pcap-meta-file /tmp/pcap-info20150107141533_00000.meta \
                --out /data/my-new-pcap.pcap \
                --hash 100819735 --time 1300724154

The following example shows how to run yaf creating rolling PCAP files and use yafMeta2Pcap to create a PCAP file for a particular flow.

 $ yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 \
       --pcap-meta-file /tmp/pcap-info --pcap /data/pcap \
       --live pcap

 $ yafMeta2Pcap --pcap-meta-file="/tmp/pcap-info*"\
                -o /data/new.pcap \
                --hash 100819735 --time 1300724154

The following example shows how to run yaf over a list of pcap files and use yafMeta2Pcap to create a PCAP for a specific flow

 $ yaf --in /tmp/caplist.txt --out /tmp/yaffile.yaf \
       --pcap-meta-file /tmp/pcap-info --caplist

 $ yafMeta2Pcap --caplist /tmp/caplist.txt -o flow.pcap \
                --hash 100819735 --time 1300724154 \
                --pcap-meta-file "/tmp/pcap-info*" -v

 Found 1 packets that match criteria.

Known Issues

If the pcap-meta-file was generated in a way that it has the form:

 hash | stime | path_to_pcap

and an OUTPUT_SPECIFIER is provided to --out, yafMeta2Pcap creates a temporary file with the PCAP file names that contain the flow it is searching for. This temporary file is placed in /tmp by default, but the location can be configured through the use of the TMPDIR environment variable. The temporary file is deleted upon termination of yafMeta2Pcap.

Bug reports may be sent directly to the Network Situational Awareness team at <netsa-help@cert.org>.

AUTHORS

Emily Sarneso and the CERT/CC Engineering Team, http://www.cert.org/netsa.

YAF

YAF