This page describes how to install and configure SiLK and YAF (and their prerequisite libfixbuf) on a single Debian machine (Ubuntu is similar) for standalone flow collection and analysis. (See the instructions for a CentOS, RedHat, or Fedora machine if that applies to you).
You should have a fully functional APT/DEB-based system; these instructions use Debian 9.8.0. Run the following command to install the basic development tools (e.g., a C compiler) required to build SiLK and YAF.
apt install build-essential
(All commands in this section are run as the root user.)
Run the following command to install the GLib-2, LZO, zlib, GnuTLS, PCAP, and Python development libraries. (For Python and GnuTLS, choose the version numbers that match those currently on your system.)
apt install libglib2.0-dev liblzo2-dev zlib1g-dev libgnutls28-dev libpcap-dev python3.8-dev
To use the MaxMind country-code mapping capability of SiLK, you must install the libmaxminddb-dev package.
apt install libmaxminddb-dev
Download the libfixbuf, SiLK, and YAF source code from the tools.netsa.cert.org web site.
cd /tmp wget https://tools.netsa.cert.org/releases/silk-3.19.1.tar.gz wget https://tools.netsa.cert.org/releases/libfixbuf-2.4.1.tar.gz wget https://tools.netsa.cert.org/releases/yaf-2.12.2.tar.gz
Unpack, configure, and install libfixbuf into the
/usr/local
directory. (This section assumes you
downloaded the source code to /tmp
.)
cd /tmp tar -zxf /tmp/libfixbuf-2.4.1.tar.gz cd libfixbuf-2.4.1 ./configure \ --prefix=/usr/local \ --enable-silent-rules make make install
Unpack and configure the SiLK source code. The switches to the configure command do the following:
/data
.)
/usr/local
) instead of with the other Python
packages
cd /tmp tar -zxf /tmp/silk-3.19.1.tar.gz cd silk-3.19.1 ./configure \ --prefix=/usr/local \ --enable-silent-rules \ --enable-data-rootdir=/var/silk/data \ --enable-ipv6 \ --enable-ipset-compatibility=3.14.0 \ --enable-output-compression \ --with-python \ --with-python-prefix
Now that the code is configured, build and install SiLK.
make make install
Unpack, configure, and install YAF into the /usr/local
directory, enabling YAF's application
labeling and deep packet inspection
(DPI) capabilities. If you are building yaf-3.0.0 or later, add
--enable-dpi
to the configure options.
cd /tmp tar -zxf /tmp/yaf-2.12.2.tar.gz cd yaf-2.12.2 ./configure \ --prefix=/usr/local \ --enable-silent-rules \ --enable-applabel \ --enable-metadata \ --enable-plugins make make install
Manually copy the YAF start-up script into place.
cp /tmp/yaf-2.12.2/etc/init.d/yaf /etc/init.d/yaf chmod a+x /etc/init.d/yaf
Finally, you should update the cache of the dynamic linker. If you do
not, it may be necessary to set the LD_LIBRARY_PATH environment
variable to /usr/local/lib
when you use SiLK or YAF.
Typically an entry for /usr/local/lib
will already exist
in the /etc/ld.so.conf.d/
directory. To confirm:
$ grep local /etc/ld.so.conf.d/* /etc/ld.so.conf.d/libc.conf:/usr/local/lib
To update the cache in this case, run the ldconfig program.
However, if your machine does not have such an entry (or you installed
to a location other than /usr/local
), you should
create. Create a file named silk.conf
containing the
following line that specifies the library directory for SiLK and YAF:
/usr/local/lib
Now copy the file into the /etc/ld.so.conf.d
directory
and run ldconfig.
mv silk.conf /etc/ld.so.conf.d/. ldconfig
The first step to configuring SiLK is to create the data repository
directory (/var/silk/data
) and add the
silk.conf
file, which defines how your data is stored
(see the manual page for details). Use
the default silk.conf
file for the
twoway site, which is installed at
/usr/local/share/silk/twoway-silk.conf
. You may edit the
sensor descriptions if desired. The default settings cause the SiLK
analysis program rwfilter
to consider only incoming data unless the user provides the
--type or --flowtypes switch. If
you want rwfilter to look at both incoming and
outgoing data by default, modify the default-types
line
to include in inweb out outweb. If desired,
also add int2int ext2ext. See the twoway packing logic manual page for
the meanings of these types. Run these commands as root:
mkdir -p /var/silk/data chmod go+rx /var/silk /var/silk/data cp /usr/local/share/silk/twoway-silk.conf /var/silk/data/silk.conf
Next create the sensors.conf
file that is used by rwflowpack for collecting
data from yaf. Details on the file's syntax are in
its manual page. Use your text editor
to create a file that contains the following lines.
IMPORTANT: Make sure the ipblocks in the my-network
section match your "internal" network blocks according to the ethernet
interface (e.g., enp0s3
, eth0
) where you
will configure yaf to listen below.
probe S0 ipfix listen-on-port 18001 protocol tcp listen-as-host 127.0.0.1 end probe group my-network ipblocks 192.168.1.0/24 # address of ethernet interface. CHANGE THIS. ipblocks 10.0.0.0/8 # other blocks considered internal. OPTIONAL. end group sensor S0 ipfix-probes S0 internal-ipblocks @my-network external-ipblocks remainder end sensor
(Once the entire system is running, if you discover that all your records show up as type ext2ext, it means you did not configure your internal netblocks correctly.)
(All commands in this section are run as the root user.)
Move this file into place.
mv sensors.conf /var/silk/sensors.conf
SiLK comes with traditional "init script" start-up files, with two
files per daemon: One file is invoked by the system to start the
daemon, and the other contains configuration settings used by the
first. Copy rwflowpack's start-up script to
/etc/init.d/rwflowpack
, and copy the configuration
settings script to /usr/local/etc/rwflowpack.conf
:
cd /usr/local cp share/silk/etc/init.d/rwflowpack /etc/init.d/rwflowpack cp share/silk/etc/rwflowpack.conf /usr/local/etc/rwflowpack.conf
Open the /usr/local/etc/rwflowpack.conf
file in an editor
and change these variables to the values shown here:
ENABLED=1 statedirectory=/var/silk SENSOR_CONFIG=/var/silk/sensors.conf ARCHIVE_DIR= # empty LOG_TYPE=legacy LOG_DIR=/var/log PID_DIR=/var/run
Start rwflowpack (a message about "contains no runlevels, aborting" is non-fatal):
systemctl enable rwflowpack systemctl start rwflowpack.service
If you're in a container, which usually won't be running systemd, then start rwflowpack with its init script instead:
/etc/init.d/rwflowpack start
Look at the rwflowpack log file,
/var/log/rwflowpack-YYYYYMMDD.log
to ensure that
it is running.
Like SiLK, YAF has two "init script" start-up files. Edit the YAF
start-up configuration file, /usr/local/etc/yaf.conf
,
setting these values:
IMPORTANT: Make sure the interface (enp0s3
) matches
the interface on which you want to capture.
ENABLED=1 YAF_CAP_IF=enp0s3 # Ensure this is correct for your machine YAF_IPFIX_PORT=18001 # Must match value in sensors.conf YAF_EXTRAFLAGS="--silk --applabel --max-payload=512"
As the root user, start yaf (a message about "contains no runlevels, aborting" is non-fatal):
systemctl enable yaf systemctl start yaf.service
If you're in a container, which usually won't be running systemd, then start yaf with its init script instead:
/etc/init.d/yaf start
Look at the yaf log file,
/var/log/yaf.log
to ensure that it is running.
Check the rwflowpack log file again to ensure that it received a connection from yaf. There should be a message similar to the following (the port number will likely differ):
'S0': accepted connection from 127.0.0.1:36734
If nothing is happening, it could be that yaf is waiting for network traffic. Follow the steps in the next section to generate some.
Generate some traffic and wait records to be flushed. Run the following ping command, or use your web browser or the wget command to access web sites.
ping -c 4 8.8.8.8
You may have to wait 5 or 10 minutes for the first records to be flushed, but you should see messages in the rwflowpack log file that it is creating files. You may check the status of YAF and rwflowpack with the following commands:
systemctl status yaf.service systemctl status rwflowpack.service
(You may need to include the --no-pager and -l switches to get useful information from systemctl.)
Run the following query to get data for the current day:
/usr/local/bin/rwfilter --sensor=S0 --type=all --all=stdout \ | rwcut --tail-recs=10
You now have a standalone flow collection and analysis machine.
See SiLK's documentation page for more information on the SiLK tool suite. In particular,