CERT
Software Assurance Secure Systems Organizational Security Coordinating Response Training
Skip to end of metadata
Go to start of metadata

FTP traffic consists of two types of sessions: control sessions and data transfers. The control session will consist of a client TCP connection (ephemeral port to port 21) and its return traffic. The data transfer itself will occur either in active or passive mode:

  • Active mode: Server connects from port 20 to a (client-specified) ephemeral port on the client
  • Passive mode: Client connects from an ephemeral port to a (server-specified) ephemeral port on the server

Identifying FTP flows using the --tuple option

The bash script below uses SiLK commands to create a list of FTP client/server pairs (in the files served-sipdip.txt and client-sipdip.txt), then using those lists with the --tuple-file option, which is supported in SiLK 0.11.10 and above. This is the preferred method for identifying FTP traffic, since each sip/dip pair has been verified to have an active FTP session between them during the time period being studied.

It may still be possible for those hosts to have non-FTP sessions between them which would also be identified as FTP using this methodology, but that situation would likely be rare.

The output files include

  • served-sipdip.txt: Sources are internal FTP servers, destinations are external clients
  • served-out.rw: Outbound data from internal servers to external clients
  • served-in.rw: Inbound data from external clients to internal servers
  • client-sipdip.txt: Sources are internal FTP clients, destinations are external servers
  • client-out.rw: Outbound data from internal clients to external servers
  • client-in.rw: Inbound data from external servers to internal clients
findftp.sh
#! /bin/bash

# Usage: findftp.sh YYYY/MM/DD:HH YYYY/MM/DD:HH
# where the dates specify the start/end dates
# as used in the rwfilter --start-date/--end-date commands

# For use with SiLK versions 0.11.10. and greater 

function usage() {
  echo "Usage: $0 STARTDATE ENDDATE"
  echo -n " where STARTDATE and ENDDATE are in the format" 
  echo " YYYY/MM/DD[:HH]"
}

if [ $# -lt 2 ]; then usage; exit; fi


START=$1
END=$2

IAPCLASS="--class=iap"   # The appropriate class name may differ in your environment, or
                         # you may not need a class specification

#  don't use flags, because this might be the middle of
#  a timed-out flow.  Every data transfer should be
#  preceded by a short command on port 21.

#
#  Make a list of all the sip/dip combinations for internally
#  hosted FTP servers.  Always start with outbound traffic to
#  minimize the effects of scanning.
#
echo "Creating tuple file of FTP hosts"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --sport=21 --protocol=6 --packets=2- \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    | rwuniq --fields=sip,dip  \
    | cut -f 1,2 -d "|" \
    > served-sipdip.txt
#
#  Now get all the outbound FTP traffic: use the sip/dip combination
#  for ephemeral <-> ephemeral in additon to 20<->ephemeral
#  and 21<->ephemeral, which should be all FTP traffic.
#
echo "Served out"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --tuple-file=served-sipdip.txt --tuple-direction=forward \
    --sport=20,21,1024- --dport=1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > served-out.rw
#
#  Similarly, pull the associated inbound traffic, changing the
#  tuple direction.
#
echo "Served in"
rwfilter \
    $IAPCLASS --type=in \
    --start-date=$START --end-date=$END \
    --tuple-file=served-sipdip.txt --tuple-direction=reverse \
    --dport=20,21,1024- --sport=1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > served-in.rw



#
#  Follow the same workflow for ftp traffic consumed by my network,
#  just carefully change sources for addresses and ports.
#
echo "Creating tuple file of FTP hosts"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --dport=21 --protocol=6 --packets=2- \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    | rwuniq --fields=sip,dip \
    | cut -f 1,2 -d "|" \
    > client-sipdip.txt
echo "Client out"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --tuple-file=client-sipdip.txt --tuple-direction=forward \
    --sport=1024- --dport=20,21,1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > client-out.rw
echo "Client in"
rwfilter \
    $IAPCLASS --type=in \
    --start-date=$START --end-date=$END \
    --tuple-file=client-sipdip.txt --tuple-direction=reverse \
    --dport=1024- --sport=20,21,1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > client-in.rw

Ignoring FTP flows using the --tuple option

If the goal is to eliminate FTP traffic from a particular analysis workflow, then the procedure would be to produce the served-sipdip.txt and client-sipdip.txt files in the same way as above, then remove their associated traffic from the analysis pipeline as shown below. Note the use of the --fail option to remove the traffic that matches the filter.

# Outbound traffic
 rwfilter --type=out ... \
| rwfilter \
    --tuple-file=served-sipdip.txt --tuple-direction=forward \
    --sport=20,21,1024- --dport=1024- --protocol=6 \
    --fail-destination=stdout \  # Outbound data served from internal servers
| rwfilter \
    --tuple-file=client-sipdip.txt --tuple-direction=forward \
    --sport=1024- --dport=20,21,1024- --protocol=6 \
    --fail-destination=stdout \  # Outbound client requests
| ...

# Inbound traffic
 rwfilter --type=in ... \
| rwfilter \
    --tuple-file=served-sipdip.txt --tuple-direction=reverse \
    --dport=20,21,1024- --sport=1024- --protocol=6 \
    --fail-destination=stdout  \ # Inbound requests to internal servers
| rwfilter \
    --tuple-file=client-sipdip.txt --tuple-direction=reverse \
    --dport=1024- --sport=20,21,1024- --protocol=6 \
    --fail-destination=stdout \  # Inbound data served from external servers
| ...

Identifying FTP using ipsets (SiLK versions < 0.11.10)

The method using sets is inferior to the option above, but may be faster. The set method is inferior because there may be cases where server A has an FTP session with host B, and server C has an FTP session with host D, but ephemeral-to-ephemeral traffic between server A and host D is also extracted as FTP data without further evaluation, when A and D may not have an FTP session between them.

This may be a rare case in practice, however. A cursory test showed less than 1% additional flows captured by the set method vs. the tuple method. While this difference could be acceptable for gross traffic statistics, if using the remaining flows for security purposes, the analyst should probably be more cautious and use the tuple method instead.

The following script uses sets to find FTP traffic, and produces the following output files:

  • ftpintservers.set: Internal FTP servers
  • ftpintclients.set: Internal FTP clients
  • ftpextservers.set: External FTP servers
  • ftpextclients.set: External FTP clients
  • served-out.rw: Outbound data from internal servers to external clients
  • served-in.rw: Inbound data from external clients to internal servers
  • client-out.rw: Outbound data from internal clients to external servers
  • client-in.rw: Inbound data from external servers to internal clients
findftpprev1.sh
#! /bin/bash

# Usage: findftpprev1.sh YYYY/MM/DD:HH YYYY/MM/DD:HH
# where the dates specify the start/end dates
# as used in the rwfilter --start-date/--end-date commands
 
# For use with SiLK versions before 0.11.10

function usage() {
  echo "Usage: $0 STARTDATE ENDDATE"
  echo -n " where STARTDATE and ENDDATE are in the format" 
  echo " YYYY/MM/DD[:HH]"
}

if [ $# -lt 2 ]; then usage; exit; fi


START=$1
END=$2

IAPCLASS="--class=iap"  # The appropriate class name may differ in your environment,
                        # or you may not need a class specification.

#  don't use flags, because this might be the middle of
#  a timed-out flow.  Every data transfer should be
#  preceded by a short command on port 21.

#
#  Make a list of all the sip/dip combinations for internally
#  hosted FTP servers.  Always start with outbound traffic to
#  minimize the effects of scanning.
#
echo "Creating sets of FTP hosts"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --sport=21 --protocol=6 --packets=2- \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    | rwset --sip-file=ftpintservers.set --dip-file=ftpextclients.set

#
#  Now get all the outbound FTP traffic: use the sip/dip combination
#  for ephemeral <-> ephemeral in additon to 20<->ephemeral
#  and 21<->ephemeral, which should be all FTP traffic.
#
echo "Served out"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --sipset=ftpintservers.set --dipset=ftpextclients.set \
    --sport=20,21,1024- --dport=1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > served-out.rw
#
#  Similarly, pull the associated inbound traffic, changing the
#  tuple direction.
#
echo "Served in"
rwfilter \
    $IAPCLASS --type=in \
    --start-date=$START --end-date=$END \
    --sipset=ftpextclients.set --dipset=ftpintservers.set \
    --dport=20,21,1024- --sport=1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > served-in.rw



#
#  Follow the same workflow for ftp traffic consumed by my network,
#  just carefully change sources for addresses and ports.
#
echo "Creating sets of FTP hosts"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --dport=21 --protocol=6 --packets=2- \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    | rwset --sip-file=ftpintclients.set --dip-file=ftpextservers.set
echo "Client out"
rwfilter \
    $IAPCLASS --type=out \
    --start-date=$START --end-date=$END \
    --sipset=ftpintclients.set --dipset=ftpextservers.set \
    --sport=1024- --dport=20,21,1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > client-out.rw
echo "Client in"
rwfilter \
    $IAPCLASS --type=in \
    --start-date=$START --end-date=$END \
    --sipset=ftpextservers.set --dipset=ftpintclients.set \
    --dport=1024- --sport=20,21,1024- --protocol=6 \
    --pass-destination=stdout --print-filenames --print-volume-statistics \
    > client-in.rw

In order to remove all FTP flows from an analysis pipeline, perform the following operations:

# Outbound traffic
 rwfilter --type=out ... \
| rwfilter \
    --sipset=ftpintservers.set --dipset=ftpextclients.set \
    --sport=20,21,1024- --dport=1024- --protocol=6 \
    --fail-destination=stdout \  # Outbound data served from internal servers
| rwfilter \
    --sipset=ftpintclients.set --dipset=ftpextservers.set \
    --sport=1024- --dport=20,21,1024- --protocol=6 \
    --fail-destination=stdout \  # Outbound client requests
| ...

# Inbound traffic
 rwfilter --type=in ... \
| rwfilter \
    --sipset=ftpextclients.set --dipset=ftpintservers.set \
    --dport=20,21,1024- --sport=1024- --protocol=6 \
    --fail-destination=stdout  \ # Inbound requests to internal servers
| rwfilter \
    --sipset=ftpextservers.set --dipset=ftpintclients.set \
    --dport=1024- --sport=20,21,1024- --protocol=6 \
    --fail-destination=stdout \  # Inbound data served from external servers
| ...
  • No labels