The configurations described below are examples to illustrate the various aspects of Pipeline. They are not CERT recommended analyses. The thresholds in the examples are made up, and not based on any analysis. Within each configuration example, the threshold is commented out to drive this point home, and to prevent direct copying and pasting into a live pipeline configuration file. Each example can be downloaded as an individual file. A tarball of all of the files can be downloaded here:
Additional examples can be found included with the distribution. There are configuration files in analysis-pipeline/src/tests2/configs. DO NOT EDIT these files as it will break the unit tests for the pipeline installation.
|Filter Examples||Various example filters illuminating features.|
|IP Watchlist||A full watchlist example.|
|DNS Watchlist||A DNS watchlist example.|
|Alert on all flows||Alerts on every flow seen by pipeline|
|Simple Count||A simple record count|
|Web Server||An example of web server detection. The thresholds are made up for the example, not to be used in production.|
|Proportion of traffic||Alert if the percentage of traffic that is TCP gets too high|
|IPv6 Tunnel Detection||Alert on all traffic identified as IPv6 tunnel traffic|
|Sensor outage||Alert if a sensor stops sending flows to pipeline|
|Passive FTP Detection||Alert on traffic thought to be passive protocol traffic|
|Beacon||Detect finite state beaconing|
The specificDport filter looks for destination port of 11111
FILTER specificDport DPORT == 11111 END FILTER
The ftpFilter looks for incoming FTP traffic (TCP port 21)
FILTER ftpFilter DPORT ==21 PROTOCOL ==6 END FILTER
The HighPorts filter looks for TCP traffic with both ports greater than 50000.
FILTER HighPorts SPORT >= 50000 DPORT >= 50000 PROTOCOL == 6 END FILTER
The writeDownHighPortSips internal filter checks to see if the flow meets the requirements of HighPorts, and puts the SIP into a list called highPortSIPs. Entries in the list are kept for 1 day.
INTERNAL FILTER writeDownHighPorts FILTER HighPorts SIP highPortSIPs 1 DAY END INTERNAL FILTER
The sipInHighPortSIPs filter looks for SIPs identified by the above internal filter writeDownHighPorts by checking if they are in the list highPortsSIPs
FILTER sipInHighPortsSIPs SIP IN LIST highPortsSIPs END FILTER
The simplest configuration file sends an alert with every flow that is seen by Pipeline. The filter allows all flows through, and the evaluation is EVERYTHING PASSES with no alert regulation.
FILTER all #No comparisons needed for a filter that lets everything through END FILTER
EVALUATION alertAll FILTER all SEVERITY 1 ALERT ALWAYS ALERT EVERYTHING CHECK EVERYTHING PASSES END CHECK END EVALUATION
This evaluation uses ALERT ALWAYS to send alerts anytime there is something available. ALERT EVERYTHING includes all outputs anytime pipeline is allowed to alert.
A standard use of the Analysis Pipeline is to perform watchlisting on a network. For Pipeline, watchlists are stored as IPSets.
Watchlisting means looking for traffic to and/or from a set of known bad IP addresses. There are two ways to look for watchlist traffic:
FILTER eitherIPInWatchlist ANY IP IN LIST "myWatchlist.set" END FILTER EVALUATION watchlistEitherDirection FILTER eitherIPInWatchlist SEVERITY 1 ALERT ALWAYS ALERT EVERYTHING CHECK EVERYTHING PASSES END CHECK END EVALUATION
FILTER SIPInWatchlist SIP IN LIST "myWatchlist.set" END FILTER EVALUATION watchlistSIP FILTER SIPInWatchlist SEVERITY 1 ALERT ALWAYS ALERT EVERYTHING CHECK EVERYTHING PASSES END CHECK END EVALUATION FILTER DIPInWatchlist DIP IN LIST "myWatchlist.set" END FILTER EVALUATION watchlistDIP FILTER DIPInWatchlist SEVERITY 1 ALERT ALWAYS ALERT EVERYTHING CHECK EVERYTHING PASSES END CHECK END EVALUATION
The second option allows for more detailed alerts as the name of the evaluation can include the directionality of the watchlist entry
A DNS watchlist checks for domain names queried by users in the network that are in a list of known malicious domains. These is different than other watchlists for three reasons. The first is the format of the watchlist which tells Pipeline that the values are to be interpretted as DNS names versus regular strings. The second difference is that the domain names are likely to come from a YAF data source, meaning there could be multiple dnsQNames in each record. The last difference is that if the record is from a YAF data source, the dnsQName is not in the main part of the record, so the evaluation needs to use EXTRA ALERT FIELD to ensure that the dnsQName makes it into the alert.
To tell Pipeline to interpret both the watchlist file's values, and the values from the records as domain names, the first line of the DNS watchlist file must be "##format:dns". Because of this first line, Pipeline will say that "www.cert.org" matches if "cert.org" is in the watchlist. If the field value matches all of the value in the watchlist, even if not all of the value is used, the comparison will return true.
Example file: dnsNames.txt
dnsQName IN LIST "dnsNames.txt"
SEVERITY 1 ALERT ALWAYS ALERT EVERYTHING CHECK EVERYTHING PASSES END CHECK EXTRA ALERT FIELD dnsQName
While it may not be useful, the simplest example of an arithmetic evaluation is to alert if more then 10000 flows have been seen by pipeline in an hour. Use the RECORD COUNT primitive and the filter all from above to allow all flows through to the eval
EVALUATION simpleCount FILTER all SEVERITY 1 ALERT ALWAYS ALERT EVERYTHING CLEAR ALWAYS CHECK THRESHOLD RECORD COUNT >10000 TIME WINDOW 1 HOUR END CHECK END EVALUATION
The CLEAR ALWAYS line causes Pipeline to reset the record count everytime is gets over 10000
This example uses a pair of evaluations to identify the webservers on our network, and the destination ip addresses that are talking to those web servers most often.
The first evaluation finds web servers, then puts those IPs into a list, that is used to filter records into a second evaluation that analyzes the recipients of packets from these webservers to find who'slistening.
A web server in this example is defined as a source IP (foreach SIP) that sends data to more than 500 distinct destination IP addresses and sends a total of 1MB (1000000 bytes) of data, all in a 10 minute window. These numbers are completely made up for the sake of this example, they are not an official CERT analysis.
The findWebServers evaluation sends the full list of web server IP addresses (set up using ALERT EVERYTHING) at most once an hour. It keeps a web server in the output list for up to 1 day (set using OUTPUT_LIST SIP ). It places each web server IP address it finds into the list called webServerList. An IP address will be removed from this list, if it is removed from the output list, which happens if the IP doesn't assert itself as a web server during that one day period (86400 seconds).
It uses the filter ourNetwork which is setup to only let through traffic coming from out network.
FILTER ourNetwork SIP == 18.104.22.168/8 #arbitrarily chosen network SPORT IN LIST [80, 8080] #Yes, there are more ports, but 80 and 8080 keep it simple END FILTER
EVALUATION findWebServers SEVERITY 2 OUTPUT_TIMEOUT 1 DAY OUTPUT_LIST SIP webServerList CLEAR ALWAYS ALERT EVERYTHING ALERT 1 TIMES 1 HOUR FILTER ourNetwork FOREACH SIP CHECK THRESHOLD DISTINCT DIP > 250 TIME_WINDOW 1 HOUR END CHECK CHECK THRESHOLD SUM BYTES > 1000000 TIME_WINDOW 10 MINUTES END CHECK END EVALUATION
The findWebServers evaluation groups state by SIP, which is why is uses FOREACH SIP.Web servers are kept in a list by using OUTPUT LIST SIP webServerList, and that the entries are kept in the list for 1 DAY is specified by OUTPUT_TIMEOUT 1 DAY. This evaluation only sends alerts once per hour. When it is permitted to alert based on the time constraints, it include all web servers that are still valid (ones that have acted like a web server within the last day, otherwise they'd time out), because ALERT EVERYTHING was used.
We only want to do this secondary analysis on traffic coming from our previously established web servers. As a result, the filter uses the output list created in the above evaluation. It checks to see if the SIP in the flow record is also in the webServerList. If so, the flow passes.
FILTER webServers SIP IN LIST webServerList END FILTER
This evaluation is taking in all traffic sent out by our established web servers, this bins the records by destination IP address, and checks to see if any of these IP addresses have received more than 10MB of data in the last 15 minutes.
EVALUATION LIS SEVERITY 2 ALERT SINCE LAST TIME ALERT 1 TIMES 60 SECONDS CLEAR ALWAYS FILTER webServers FOREACH DIP CHECK THRESHOLD SUM BYTES > 10000000 TIME_WINDOW 15 MINUTES END CHECK END EVALUATION
We know the SIP for all of this traffic is from a web server, so we group state by DIP, as we are identifying interesting IPs getting traffic from our web servers. This evaluation will send an alert at most once per minute, if there is something to include in an alert. When it does alert, it sends out all DIPs that hit the threshold since the last time alerts were sent because ALERT SINCE LAST TIME is used.
In another contrived example, we want pipeline to send an alert if the proportion of all traffic that is TCP goes over 75% within any 5 minute window. We use the all filter defined above because we want all traffic to go to this evaluation.
EVALUATION tcpTraffic SEVERITY 5 ALERT JUST_NEW_THIS_TIME ALERT 1 TIMES 0 SECONDS CLEAR NEVER FILTER all CHECK THRESHOLD PROPORTION PROTOCOL 6 >= 75 PERCENT TIME_WINDOW 5 MINUTES END CHECK END EVALUATION
There are three types of IPv6 automatic tunneling: Teredo, 6to4, and ISATAP. The details of each tunneling are slightly different, but their general format is the same. To detect all of them, there will be three distinct sets of filters, internal filters, and evaluations.
IPv6 tunneling happens in 2 stages:
The detection of IPv6 tunnels takes three stages:
Filter looking for initial public server connection<:/p>
FILTER teredoInitialConnections DIP IN_LIST [22.214.171.124, 126.96.36.199] #Insert public ipv6 server END FILTER
Take the flows that connecto to the ipv6 server, and store their SIPs in the list teredoSIPS, which have a timeout value of 1 hour
INTERNAL FILTER buildTeredoList FILTER teredoInitialConnections SIP teredoSIPS 1 HOUR END INTERNAL FILTER
Look for traffic with SIP in the list, protocol UDP, DPORT 3544
FILTER teredoTraffic SIP IN LIST teredoSIPS DPORT == 3544 PROTOCOL == 17 END FILTER
The comparisons are in that order based on the liklihood of the comparison to fail upon hitting, thus stopping the processing
Lastly, we need a basic evaluation that passes all traffic found by the teredoTraffic filter onto the alerting stage of pipeline.
EVALUATION teredo CLEAR ALWAYS SEVERITY 4 ALERT EVERYTHING ALERT 1 TIMES 0 SECONDS FILTER teredoTraffic CHECK EVERYTHING PASSES END CHECK END EVALUATION
6 to 4 is identified by protocol 41 after contacting the public server. The second filter in the 6to4 case is:
FILTER 6to4Traffic SIP IN LIST 6to4SIPS #created similarly to the first example, using a filter looking for public servers and an internal filter PROTOCOL == 41 END FILTER
ISATAP is identified the same way as 6to4, using protocol 41, but the list of public servers to initiate connection is dynamic, unlike 6to4, and teredo. As a result, the list of IPs cannot be directly typed into the filter, so an IPSet should be used, which is updated outside of Pipeline.
The SENSOR OUTAGE file evaluation is used to send an alert if a sensor does not send flows to Pipeline within a given time period. It can monitor a list of specified sensor names, or monitor all sensors included in the SiLK sensor.conf file. This evaluation type does not look at the contents of the files (flows), so it does not need to have a filter associated with it.
The examples below send alerts if a monitored sensor does not send traffic for 2 hours.
FILE_EVALUATION sensorOutage CHECK FILE_OUTAGE SENSOR_LIST ALL SENSORS TIME_WINDOW 2 HOURS END CHECK END FILE_EVALUATION
FILE_EVALUATION sensorOutage CHECK FILE_OUTAGE SENSOR_LIST [S1, S2, S3] TIME_WINDOW 2 HOURS END CHECK END FILE_EVALUATION
Protocols such as FTP can use additional connections to do their data transfers. They usually use ephemeral ports that are deteremined at the time of the original standard port connection. These passive connections may be of interest. The problem is that the flows of the transfer finish first (thus they hit pipeline first), before the original port 21 connection. As a result, to detect IP addresses engaged in passive connections, we record the IP addresses from all high port to high port connections, then compare that list to flows using port 21. First we need to collect high port flows, then put the IP address in a list using an INTERNAL FILTER, then use the HIGH PORT CHECK primitive to alert on the IP addresses.
SPORT >= 50000
DPORT >= 50000
PROTOCOL == 6 END FILTER
Put the IPs from this HighPorts filter into a specific HIGH PORT LIST using an internal filter.
INTERNAL_FILTER highPortsInternal FILTER HighPorts HIGH_PORT_LIST highPortIPs 15 MINUTES END INTERNAL_FILTER
Now look for the original port 21 (in the case of FTP) connection
FILTER port21 DPORT == 21 PROTOCOL == 6 END FILTER
Use the high port primitive to compare port 21 connection IPs to the HIGH PORT LIST contents created from high port connections
EVALUATION passiveFTP SEVERITY 2 OUTPUT_TIMEOUT 60 SECONDS ALERT SINCE_LAST_TIME ALERT 1 TIMES 60 SECONDS CLEAR ALWAYS INTERNAL_FILTER highPortsInternal FILTER port21 FOREACH SIP DIP SPORT DPORT PROTOCOL CHECK HIGH_PORT_CHECK LIST highPortIPs END CHECK END EVALUATION
EVALUATION beacon CLEAR NEVER SEVERITY 3 ALERT JUST_NEW_THIS_TIME ALERT 1 TIMES 60 SECONDS FILTER all CHECK BEACON COUNT 5 CHECK_TOLERANCE 5 PERCENT TIME_WINDOW 300 SECONDS END CHECK END EVALUATION