Monitoring for Large-Scale Networks

SiLK

SiLK on an Box — APT/DEB Edition

This page describes how to install and configure SiLK and YAF (and their prerequisite libfixbuf) on a single Debian machine (Ubuntu is similar) for standalone flow collection and analysis. (See the instructions for a CentOS, RedHat, or Fedora machine if that applies to you).

Building the tools

Install Prerequisites

You should have a fully functional APT/DEB-based system; these instructions use Debian 9.8.0. Run the following command to install the basic development tools (e.g., a C compiler) required to build SiLK and YAF.

apt install build-essential

(All commands in this section are run as the root user.)

Run the following command to install the GLib-2, LZO, zlib, GnuTLS, PCAP, and Python development libraries. (For Python and GnuTLS, choose the version numbers that match those currently on your system.)

apt install libglib2.0-dev liblzo2-dev zlib1g-dev libgnutls28-dev libpcap-dev python3.8-dev

To use the MaxMind country-code mapping capability of SiLK, you must install the libmaxminddb-dev package.

apt install libmaxminddb-dev

Download Software

Download the libfixbuf, SiLK, and YAF source code from the tools.netsa.cert.org web site.

cd /tmp
wget https://tools.netsa.cert.org/releases/silk-3.19.1.tar.gz
wget https://tools.netsa.cert.org/releases/libfixbuf-2.4.1.tar.gz
wget https://tools.netsa.cert.org/releases/yaf-2.12.2.tar.gz

Install libfixbuf

Unpack, configure, and install libfixbuf into the /usr/local directory. (This section assumes you downloaded the source code to /tmp.)

cd /tmp
tar -zxf /tmp/libfixbuf-2.4.1.tar.gz
cd libfixbuf-2.4.1
./configure               \
    --prefix=/usr/local   \
    --enable-silent-rules
make
make install

Install SiLK

Unpack and configure the SiLK source code. The switches to the configure command do the following:

set the default location for SiLK's hourly repository of flow files to /var/silk/data (The default location is /data.)
enable support for IPv6 addresses and flow records (The default is to support only IPv4 flow records.)
enable creating IPset files in the most compact format (The format may be changed at run-time by providing the --record-version switch to an IPset tool; see also the SILK_IPSET_RECORD_VERSION environment variable.)
enable automatic (de-)compression of binary SiLK files when reading and writing (The default is no compression. The compression may be changed at run-time by providing the --compression-method switch when invoking a tool; see also the SILK_COMPRESSION_METHOD environment variable.)
enable the SiLK Python extension (PySiLK)
enable installation of the PySiLK files under the SiLK installation tree (/usr/local) instead of with the other Python packages

cd /tmp
tar -zxf /tmp/silk-3.19.1.tar.gz
cd silk-3.19.1
./configure                              \
    --prefix=/usr/local                  \
    --enable-silent-rules                \
    --enable-data-rootdir=/var/silk/data \
    --enable-ipv6                        \
    --enable-ipset-compatibility=3.14.0  \
    --enable-output-compression          \
    --with-python                        \
    --with-python-prefix

Now that the code is configured, build and install SiLK.

make
make install

Install YAF

Unpack, configure, and install YAF into the /usr/local directory, enabling YAF's application labeling and deep packet inspection (DPI) capabilities. If you are building yaf-3.0.0 or later, add --enable-dpi to the configure options.

cd /tmp
tar -zxf /tmp/yaf-2.12.2.tar.gz
cd yaf-2.12.2
./configure                 \
    --prefix=/usr/local     \
    --enable-silent-rules   \
    --enable-applabel       \
    --enable-metadata       \
    --enable-plugins
make
make install

Manually copy the YAF start-up script into place.

cp /tmp/yaf-2.12.2/etc/init.d/yaf /etc/init.d/yaf
chmod a+x /etc/init.d/yaf

Update Dynamic Linker

Finally, you should update the cache of the dynamic linker. If you do not, it may be necessary to set the LD_LIBRARY_PATH environment variable to /usr/local/lib when you use SiLK or YAF.

Typically an entry for /usr/local/lib will already exist in the /etc/ld.so.conf.d/ directory. To confirm:

$ grep local /etc/ld.so.conf.d/*
/etc/ld.so.conf.d/libc.conf:/usr/local/lib

To update the cache in this case, run the ldconfig program.

However, if your machine does not have such an entry (or you installed to a location other than /usr/local), you should create. Create a file named silk.conf containing the following line that specifies the library directory for SiLK and YAF:

/usr/local/lib

Now copy the file into the /etc/ld.so.conf.d directory and run ldconfig.

mv silk.conf /etc/ld.so.conf.d/.
ldconfig

Configuring the tools

Configure SiLK

The first step to configuring SiLK is to create the data repository directory (/var/silk/data) and add the silk.conf file, which defines how your data is stored (see the manual page for details). Use the default silk.conf file for the twoway site, which is installed at /usr/local/share/silk/twoway-silk.conf. You may edit the sensor descriptions if desired. The default settings cause the SiLK analysis program rwfilter to consider only incoming data unless the user provides the --type or --flowtypes switch. If you want rwfilter to look at both incoming and outgoing data by default, modify the default-types line to include in inweb out outweb. If desired, also add int2int ext2ext. See the twoway packing logic manual page for the meanings of these types. Run these commands as root:

mkdir -p /var/silk/data
chmod go+rx /var/silk /var/silk/data
cp /usr/local/share/silk/twoway-silk.conf /var/silk/data/silk.conf

Configure rwflowpack

Next create the sensors.conf file that is used by rwflowpack for collecting data from yaf. Details on the file's syntax are in its manual page. Use your text editor to create a file that contains the following lines.

IMPORTANT: Make sure the ipblocks in the my-network section match your "internal" network blocks according to the ethernet interface (e.g., enp0s3, eth0) where you will configure yaf to listen below.

probe S0 ipfix
 listen-on-port 18001
 protocol tcp
 listen-as-host 127.0.0.1
end probe

group my-network
 ipblocks 192.168.1.0/24  # address of ethernet interface. CHANGE THIS.
 ipblocks 10.0.0.0/8      # other blocks considered internal. OPTIONAL.
end group

sensor S0
 ipfix-probes S0
 internal-ipblocks @my-network
 external-ipblocks remainder
end sensor

(Once the entire system is running, if you discover that all your records show up as type ext2ext, it means you did not configure your internal netblocks correctly.)

(All commands in this section are run as the root user.)

Move this file into place.

mv sensors.conf /var/silk/sensors.conf

SiLK comes with traditional "init script" start-up files, with two files per daemon: One file is invoked by the system to start the daemon, and the other contains configuration settings used by the first. Copy rwflowpack's start-up script to /etc/init.d/rwflowpack, and copy the configuration settings script to /usr/local/etc/rwflowpack.conf:

cd /usr/local
cp share/silk/etc/init.d/rwflowpack /etc/init.d/rwflowpack
cp share/silk/etc/rwflowpack.conf /usr/local/etc/rwflowpack.conf

Open the /usr/local/etc/rwflowpack.conf file in an editor and change these variables to the values shown here:

ENABLED=1
statedirectory=/var/silk
SENSOR_CONFIG=/var/silk/sensors.conf
ARCHIVE_DIR=  # empty
LOG_TYPE=legacy
LOG_DIR=/var/log
PID_DIR=/var/run

Start rwflowpack (a message about "contains no runlevels, aborting" is non-fatal):

systemctl enable rwflowpack
systemctl start rwflowpack.service

If you're in a container, which usually won't be running systemd, then start rwflowpack with its init script instead:

/etc/init.d/rwflowpack start

Look at the rwflowpack log file, /var/log/rwflowpack-YYYYYMMDD.log to ensure that it is running.

Start YAF

Like SiLK, YAF has two "init script" start-up files. Edit the YAF start-up configuration file, /usr/local/etc/yaf.conf, setting these values:

IMPORTANT: Make sure the interface (enp0s3) matches the interface on which you want to capture.

ENABLED=1
YAF_CAP_IF=enp0s3      # Ensure this is correct for your machine
YAF_IPFIX_PORT=18001   # Must match value in sensors.conf
YAF_EXTRAFLAGS="--silk --applabel --max-payload=512"

As the root user, start yaf (a message about "contains no runlevels, aborting" is non-fatal):

systemctl enable yaf
systemctl start yaf.service

If you're in a container, which usually won't be running systemd, then start yaf with its init script instead:

/etc/init.d/yaf start

Look at the yaf log file, /var/log/yaf.log to ensure that it is running.

Check the rwflowpack log file again to ensure that it received a connection from yaf. There should be a message similar to the following (the port number will likely differ):

'S0': accepted connection from 127.0.0.1:36734

If nothing is happening, it could be that yaf is waiting for network traffic. Follow the steps in the next section to generate some.

Generate traffic

Generate some traffic and wait records to be flushed. Run the following ping command, or use your web browser or the wget command to access web sites.

ping -c 4 8.8.8.8

You may have to wait 5 or 10 minutes for the first records to be flushed, but you should see messages in the rwflowpack log file that it is creating files. You may check the status of YAF and rwflowpack with the following commands:

systemctl status yaf.service
systemctl status rwflowpack.service

(You may need to include the --no-pager and -l switches to get useful information from systemctl.)

Run a test query

Run the following query to get data for the current day:

/usr/local/bin/rwfilter --sensor=S0 --type=all --all=stdout \
| rwcut --tail-recs=10

You now have a standalone flow collection and analysis machine.

See SiLK's documentation page for more information on the SiLK tool suite. In particular,

the SiLK Installation Handbook provides additional information on installing SiLK
the Analysts' Handbook: Using SiLK for Network Traffic Analysis contains more examples of the use of SiLK for analysis
the manual pages provide details for each individual SiLK tool