This page gives an overview of the portal installation process.
Prerequisites
The NetSA Portal core depends on the following software:
The NetSA Portal core modules depend on the following software:
- SiLK verion 0.11.7 or later
- RAVE version 1.9.7 or later
- Python Imaging Library 1.1.6 or later
- NumPy 1.0.2 or later
- SciPy 0.5.2 or later
- Matplotlib 0.90.0 or later
- PostgreSQL database server, version 8.2.4 or later
- Psycopg 2 database driver.
Although most of the requirements are available in the newer versions of most Linux and *BSD systems (and Mac OS X, via MacPorts), they can be difficult to build and configure by hand. install.py will run tests before installing to ensure a proper system configuration.
Quick Start
It's still a good idea to read Installation, but this should get most installations started.
The NetSA Portal is distributed as a TAR archive. To install the system on a single machine configured to run both the webserver and raved, extract the archive, change into the newly-created directory, and run:
python ./install.py install --prefix=$PREFIX
In place of $PREFIX in the above example, use the path to a directory which will contain portal configuration files and modules.
The installation script will run a series of tests to ensure that you have installed the requirements correctly, and will install the files in the archive to the path specified by $PREFIX.
When installation is finished:
- Copy $PREFIX/etc/portal.conf.sample to /etc/portal.conf and customize it for your site.
- Make sure Apache sees the configuration information in $PREFIX/etc/portal-httpd.conf. (See Apache configuration for possible necessary customizations.)
- Ensure that raved.init and/or start-raved are customized for your site. (See RAVE Configuration for more information.)
- Finally, ensure you have the watchlists and other external data enumerated in module-specific configuration information.
Installation
install.py
install.py is the portal installation script. In addition to installing the software, it will run a series of diagnostics to ensure that the environment is set up correctly. It's syntax is similar to the Python distutils' setup.py:
python install.py <command> <options>
Type python install.py help at the command line for a complete list of options. For a more concrete example, see the quick start guide above.
Split and Unified Installation
The portal can be installed in two ways, based on where you expect to run the raved analysis server. (See the RAVE page for more details.) The simplest thing to do is to install the portal on the same machine as the one running raved. However, this means that user request processing and analysis processing will both occur on the same machine. This is the so-called "unified installation."
It is more scalable to install the web components of the portal to one machine, and install the RAVE-related components on a different machine, in a "split installation". However, this is more complicated to maintain.
The optimal choice between these installation types will depend on your environment. The unified installation is a better choice if you are unsure or if you are installing the portal for the first time, and is the default supported installation.
portal.conf
After installing, copy the file portal.conf.sample in the etc directory of the portal root to /etc/portal.conf. This file contains global portal configuration options (such as the site name), and information on the SiLK sensors and their access control. You should customize this file for your site; information on each option is available inline in the document as comments.
Apache configuration
To serve the portal pages, Apache must be configured to run Python Server Pages, and must be aware of the location of the portal content.
install.py will generate an Apache configuration file named portal-httpd.conf that will look something like this:
# Note: This file is for a standard combined installation of the SiLK
# Analysis Portal. It is meant to be installed on the server host
# which is used for both analysis and web services. If you are using
# a split install, use portal-httpd-split-analysis.conf and
# portal-httpd-split-web.conf instead.
PythonInterpreter /usr/local/bin/python
PythonPath "sys.path + ['/usr/local/portal/lib/python']"
AddHandler mod_python .psp
PythonHandler mod_python.psp
# The portal web pages
Alias "/portal/" "/usr/local/portal/htdocs/"
<Location "/portal/">
Order deny,allow
Allow from all
Options Indexes FollowSymLinks
# You probably want to restrict to logged-in users. See Apache
# documentation for how to configure authentication.
# require valid-user
</Location>
# Proxy configuration
<Location "/rave/">
SetHandler mod_python
PythonHandler portal.proxy
PythonOption portal.proxy.rave-service http://127.0.0.1:8888/
PythonOption portal.proxy.timeout 120
PythonDebug on
Order allow,deny
Allow from all
</Location>
# RAVE's area to serve up content to the proxy
Alias "/rave-data/" "/usr/local/portal/cache/"
<Location "/rave-data/">
Order allow,deny
Allow from localhost
</Location>
How to use this in your Apache configuration depends on how Apache is set upon your machine. Red Hat and other Linux distributions often provide a directly in which users can place custom Apache configuration files. (e.g., /etc/httpd/conf.d) In other situations, you may have to add a line to the end of your httpd.conf file such as:
Include /usr/local/portal/etc/portal-httpd.conf
If you are running a split installation, you will need to change the portal.proxy.rave-service option in the <Location "/rave"> block to point to the location of the listening raved analysis server.
The final <Location> is used internally by the RAVE proxy to fetch content from RAVE. Unlike the other two, it will only ever be used by local connections from the RAVE proxy. If you run a split installation, this directive is only necessary on the machine running raved.
RAVE Configuration
The portal relies on RAVE for the bulk of its analysis results and graphics. In order for RAVE to be aware of the portal analyses and to simplify RAVE operation, the portal provides three files in the etc directory of the portal root:
- raved.init
- A script to use in /etc/init.d to start and stop raved easily (possibly at system startup) with portal-specific namespace configuration. This file assumes SysV-style initialization, although it could be adapted easily to a BSD-style init sytem using the rc.d framework.
- start-raved
- Similar to raved.init, but where the raved.init detaches raved from the terminal and runs as a daemon, start-raved runs n the foreground. This is useful for debugging.
- raved-log.conf
- This file configures raved logging. raved uses the Python logging facility; the syntax of this configuration file can be found here.
raved.init and start-raved are very similar, and use the same variables to customize their work. You may wish to set some of these variables, especially if you are running a split installation. In particular:
- OUTPUT_DIR
- If you wish to store the RAVE cache somewhere other than the default location, change this variable to point to the new location.
- URL_BASE
- If you are running a split installation, this should be the URL from which the RAVE proxy can retrieve analysis results.
Module-Specific Configuration
Individual analysis modules have specific configuration considerations:
Watchlists
The watchlist analyses depend on the existence of three SiLK ipset files. ipsets are (as the name implies) files describing sets of related IP addresses. You can generate these sets using the SiLK rwset and rwsetbuild commands.
The required ipsets are:
- internal.set
- A set of IP addresses internal to the monitored network
- darkspace.set
- A set of IP addresses that no activity should occur on. If you have a block of allocated addresses that are known not to be occupied, this would be an appropriate use. Otherwise, you might use addresses that are reserved by Internet standards. (E.g., Addresses reserved in RFC 1918 and RFC 3927)
- blacklist.set
- A set of IP addresses that none of your hosts should ever talk to. This would include addresses of known "corrupt" sites. (For example sites that are known to be used for collecting illicit information.) Potential sources of information for this informtion include spam and phishing blacklists.
Each of these files should be in the $PREFIX/modules/watchlists/etc directory, where "$PREFIX" is the path specified by the --prefix option of install.py.
Port Database
This module relies on the existance of a PostgreSQL database. The SQL to set up this database is in $PREFIX/modules/port_database/sql. More detailed information on setting up this database, including examples of setting up a dedicated database user for the portal, are in $PREFIX/modules/port_database/etc/README-sql, also available here.