Monitoring for Large-Scale Networks

SiLK

rwscanquery

NAME

rwscanquery - Query the network scan database

SYNOPSIS

rwscanquery [options]

Report Options:

--report=REPORT_TYPE       Select query and output options.  Values
                           for REPORT_TYPE are standard, volume,
                           scanset, scanflows, respflows, and export

--start-date=YYYY/MM/DD:HH Report on scans active after this date.
--end-date=YYYY/MM/DD:HH   Defaults to start-date.

--saddress=ADDR_SPEC       Show scans originating from matching hosts.
--sipset=IPSET_FILE        Show scans originating from hosts in set.

--daddress=IP_WILDCARD     Show only scans targeting matching hosts.
--dipset=IPSET_FILE        Show only scans targeting hosts in set.

--show-header              Display column titles at start of output.
--columnar                 Display more human-readable columnar view.
--output-path=PATH         Write results to the specified file.

Configuration Options:

--database=DBNAME          Query an alternate scan database

Help Options:

--help                     Display this brief help message.
--man                      Display the full documentation.
--version                  Display the version information.

DESCRIPTION

rwscanquery queries the network scan database---that is, the database that contains scans found by rwscan(1). The type of output rwscanquery creates is controlled by the --report switch as described in the "Report Options" section below. rwscanquery writes its output to the location specified by the --output-path switch or to the standard output when that switch is not provided.

rwscanquery runs a query of the scan database and then, depending on the report type, either displays the result set as text or creates a binary SiLK from the result set. The database rows that are part of the result set may be limited by using the --start-date, --end-date, --saddress, and --sipset switches. The result set is always limited to a time window, and the current day is used when no --start-date is given.

The following three report types produce textual output. The default output displays the values separated by a vertical bar (|) with no spacing. The --columnar switch causes the output to appear in columns with a space-delimiter between the columns. The output includes no title line unless the --show-header switch is specified.

The standard report contains most of the columns in the database for the rows in the result set. (The columns containing the scan model and scan probability are not included.)
The volume report groups the rows in the result set by day and shows sums the flows, packets, and bytes columns for each day.
The export report contains all the columns in the database for the rows in the result set, and the rows are displayed in a format compatible with rwscan.

The following three report types create a binary SiLK file as their result. These report types invoke other SiLK tools (namely rwfilter(1), rwset(1), rwsetbuild(1), and rwsetcat(1)) and the report types assume rwfilter has access to a SiLK data repository.

The first step in all three of these report types is for rwscanquery to get the distinct IP addresses for the rows in the result set and pass them into rwsetbuild to create a temporary IPset file containing the scanning IPs.

A scanflows report produces a file of SiLK Flow records whose source IP is a scanning IPs. rwscanquery uses the temporary IPset as an argument to rwfilter to find flow records in your data repository that originated from the scanning IPs within the time window. You may choose to limit the report to particular IPs targeted by the scanning IPs by specifying the --daddress or --dipset switches. The output from rwfilter is the output of the report. The rwfilter invocation uses the configuration values rw_in_class and rw_in_type if they are specified in the configuration file (c.f. "CONFIGURATION").
A respflows report produces a file of SiLK Flow records whose destination IP is a scanning IP. These flow records may represent responses to a scan. To create this report, rwscanquery performs steps similar to those for the scanflows report except the direction of the rwfilter command is reversed to find flow records going to the scanning IPs. You may choose to limit the report to particular IPs that responded to the scan by specifying the --daddress or --dipset switches. The output from rwfilter is the output of the report. The rwfilter invocation uses the configuration values rw_out_class and rw_out_type if they are specified in the configuration file (c.f. "CONFIGURATION").
A scanset report produces a binary IPset file.
- If neither the --daddress nor --dipset switches are specified, the output of the this report is the temporary IPset file containing the scanning IPs; that is, all the scanning IPs in the time window.
- Otherwise, rwscanquery performs the same steps it does as when creating scanflows report. Next, instead of returning the output from rwfilter, rwscanquery passes the flow records into rwset to create an IPset file containing the scanning IPs that targeted particular IP addresses.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

Report Options

--report=TYPE

Specify the query and the type of output to create. When this switch is not specified, the default is a standard report. The supported values for TYPE are:

standard: Write one textual line of output for each scan record in the scan database. By default, the output has no titles and it is not in columnar form. Specify the --show-header and/or --columnar switches to make the output more human readable.
volume: Write a daily scan activity volume summary report for each day within the time period. By default, the output has no titles and it is not in columnar form. Specify the --show-header and/or --columnar switches to make the output more human readable.
scanset: Write an IPset file containing the IP addresses which were the sources of scan activity during the selected time period. The output of this report type is binary, so you must redirect or pipe the output to a location or specify the --output-path switch.
scanflows: Write a SiLK Flow file containing all flows originating from scanning IP addresses within the specified time period. This flow data includes flows originating from any host that would be listed as a scan source by your query, from any time within the time period specified by --start-date and --end-date. Note that this may include flows that were not identified by the scan analysis as being part of a scan. The output of this report type is binary, so you must redirect or pipe the output to a location or specify the --output-path switch.
respflows: Write a SiLK Flow file containing all flows sent to scanning IP addresses within the specified time period---that is, possible responses to the scanners. The output of this report type is binary, so you must redirect or pipe the output to a location or specify the --output-path switch.
export: Write textual output consistent with the output format of the rwscan(1) tool. Specify the --show-header switch to include a title line.

--start-date=YYYY/MM/DD:HH

Display scans which were active after this hour. When this argument contains a date with no hour and no --end-date switch is specified, scans for that entire day are returned. If this switch is not specified at all, scans for the current day (based on the local time on the host machine) are returned.

--end-date=YYYY/MM/DD:HH

Display scans which were active before the end of this hour. If no end-date is given, defaults to the same as start-date. It is an error to provide an end-date without a start-date.

--saddress=ADDR_SPEC

Display scans originating from hosts described in ADDR_SPEC, where ADDR_SPEC is a list of addresses, address ranges, and CIDR blocks. Only scans originating from hosts in the list are displayed.

--sipset=IPSET_FILE

Display scans originating from hosts in IPSET_FILE, where IPSET_FILE is a standard SiLK IPset file as created by rwset(1) or rwsetbuild(1). Note that a very complex IPset may take a long time to process, or even fail to return any results.

--daddress=IP_WILDCARD

Display scans targeting hosts described in IP_WILDCARD, where IP_WILDCARD is a single IP address, a single CIDR block, or an IP Wildcard expression accepted by rwfilter(1). To match on multiple IPs or networks, use the --dipset switch. This switch is ignored for --report types other than scanset, scanflows, and respflows.

--dipset=IPSET_FILE

Display scans targeting hosts in IPSET_FILE, where IPSET_FILE is a standard SiLK IPset file. This switch is ignored for --report types other than scanset, scanflows, and respflows.

--show-header

Display a header line giving a short name (or title) for each field when printing textual output with the standard, volume, or export report types. By default, no header is displayed.

--columnar

Display output in more human-readable columnar format when printing textual output with the standard or volume report types. When this switch is not given, the output is presented as data fields delimited by the | character.

--output-path=PATH

Write results to PATH instead of to the standard output.

Configuration Options

--database=DBNAME: Select a database instance other than the default. The default is specified by the db_instance value in the configuration file as described in "CONFIGURATION" below.

Other Options

--help: Display a brief usage message and exit.
--man: Display full documentation for rwscanquery and exit.
--version: Print the version number and exit the application.

CONFIGURATION

rwscanquery reads configuration information from a file named .rwscanrc. If the RWSCANRC environment variable is set, it is used as the location of the .rwscanrc file. When RWSCANRC is not set, rwscanquery attempts to find a file name .rwscanrc in the directories specified in the "FILES" section below.

The format of the .rwscanrc file is name=value pairs, one per line. The configuration parameters currently read from .rwscanrc are:

db_driver: The type of database to connect to. rwscanquery supports oracle, postgresql, mysql, and sqlite.
db_userid: The userid to use when connecting to the scan database.
db_password: The password to use when connecting to the scan database.
db_instance: The name of the database instance to connect to if none is provided with the --database command line switch. If neither this configuration option nor the --database command line switch are specified, the hard-coded default database instance "SCAN" is used.
rw_in_class: The class for incoming flow data. The rw_in_class and rw_in_type values are used to query scan flows when the scanflows report type is requested or when the --daddress or --dipset switches are used for the scanset report type. If not specified, rwfilter's default is used.
rw_in_type: The type(s) for incoming flow data. See rw_in_class for details.
rw_out_class: The class for outgoing flow data. The rw_out_class and rw_out_type values are used to query scan flows when the respflows report type is requested. If not specified, rwfilter's default is used.
rw_out_type: The type(s) for outgoing flow data. See rw_out_class for details. (Note that rwfilter often defaults to querying incoming flows, so this parameter ought to be specified.)

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

Display information on all scans occurring during the 12:00 hour (12:00:00 to 12:59:59) of 2009/02/12.

$ rwscanquery --show-header --start-date=2009/02/12:12
scan-id|stime|etime|proto|srcaddr|flows|packets|bytes
499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240
365|2009-02-12 12:08:40|2009-02-12 12:14:54|6|10.146.88.117|256|256|10240
57|2009-02-12 12:28:51|2009-02-12 12:34:55|6|10.29.23.160|256|256|10240
627|2009-02-12 11:52:07|2009-02-12 12:41:16|17|10.253.24.230|1023|1023|30175
366|2009-02-12 12:41:50|2009-02-12 12:48:14|6|10.146.89.46|256|256|10240
182|2009-02-12 12:54:39|2009-02-12 13:01:20|6|10.79.26.176|256|256|10240
4|2009-02-12 12:41:19|2009-02-12 13:33:57|17|10.2.47.87|1023|1023|30205

Create the IPset file scan.set containing the scanners discovered during that hour.

$ rwscanquery --report=scanset --start-date=2009/02/12:12 \
       --output-path=scan.set
$ rwsetcat scan.set
10.2.47.87
10.29.23.160
10.79.26.176
10.146.88.117
10.146.89.46
10.199.151.231
10.253.24.230

Repeat the first query but limit the output to scanners coming from the CIDR block 10.199.0.0/16.

$ rwscanquery --show-header --start-date=2009/02/12:12 \
       --saddr=10.199.0.0/16
scan-id|stime|etime|proto|srcaddr|flows|packets|bytes
499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240

Expand the query for that CIDR block to include the preceding and following hours (11:00:00 to 13:59:59).

$ rwscanquery --start-date=2009/02/12:11 --end-date=2009/02/12:13 \
       --saddr=10.199.0.0/16
499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240
497|2009-02-12 13:33:57|2009-02-12 14:24:35|17|10.199.98.5|1023|1023|30079

Create the IPset file scanning-cidr.set that contains the CIDR block 10.199.0.0/16, and then search for scans coming from that IP on Feb 13, 2009.

$ cat scanning-cidr.txt
10.199.0.0/16
$ rwsetbuild scanning-cidr.txt scanning-cidr.set
$
$ rwscanquery --start-date=2009/02/13 --sipset=scanning-cidr.set
500|2009-02-13 22:42:25|2009-02-13 22:48:45|6|10.199.207.32|256|256|10240

Print the volume of data attributed to scans over a three day period.

$ rwscanquery --report=volume --show-header  \
       --start-date=2009/02/12 --end-date=2009/02/14
date|flows|packets|bytes
2009/02/12|137452|137499|17149008
2009/02/13|74727|76167|2798040
2009/02/14|76160|76160|2750531

The following limits the volume report to the IPs in the file scanning-cidr.set and displays the results in columns.

$ rwscanquery --report=volume --show-header --columnar  \
       --start-date=2009/02/12 --end-date=2009/02/14    \
       --sipset=scanning-cidr.set
date                     flows    packets          bytes
2009/02/12                1279       1279          40319
2009/02/13                 256        256          10240
2009/02/14                 256        256          10240

Get the SiLK Flow records coming from the scanners during the 12:00 hour on 2009/02/12 and store in the file scanning-flows.rw.

$ rwscanquery --report=scanflows --start-date=2009/02/12:12  \
       --output=scanning-flows.rw

Use rwuniq(1) to summarize the file scanning-flows.rw.

$ rwuniq --fields=sip --values=flows,packets,bytes  \
       --sort-output scanning-flows.rw
            sIP|   Records|        Packets|               Bytes|
     10.2.47.87|       373|            373|               11032|
   10.29.23.160|       256|            256|               10240|
   10.79.26.176|       203|            203|                8120|
  10.146.88.117|       256|            256|               10240|
   10.146.89.46|       256|            256|               10240|
 10.199.151.231|       256|            256|               10240|
  10.253.24.230|       846|            846|               24921|

Run a respflows report to verify that there were no responses to the scan.

$ rwscanquery --report=respflows --start-date=2009/02/12:12  \
       --output=scanning-response.rw
$
$ rwuniq --fields=sip --values=flows,packets,bytes  \
       --sort-output scanning-response.rw
            sIP|   Records|        Packets|               Bytes|

Create the IPset subnet-scan.set for scanners that targeted the 192.168.186.0/24 CIDR block during the 12:00 hour on 2009/02/12.

$ rwscanquery --report=scanset --start-date=2009/02/12:12        \
       --daddress=192.168.186.0/24 --output-path=subnet-scan.set

Store the corresponding flow records for those scans in the file subset-scan.rw.

$ rwscanquery --report=scanflows --start-date=2009/02/12:12        \
       --daddress=192.168.186.0/24 --output-path=subnet-scan.rw

Determine how many IPs in that subnet were targeted.

$ rwuniq --fields=sip --values=flows,distinct:dip subnet-scan.rw
            sIP|   Records|dIP-Distin|
   10.146.89.46|       256|       256|

Display the title line for an export report.

$ rwscanquery --report=export --start-date=2009/02/12:12  \
       --show-header | head -1
id|sip|proto|stime|etime|flows|packets|bytes|scan_model|scan_prob

ENVIRONMENT

RWSCANRC: This environment variable allows the user to specify the location of the .rwscanrc configuration file. The value may be a complete path or a file relative to the user's current directory. See the "FILES" section for standard locations of this file.
SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction for the report types of scanset, scanflows, and respflows.
SILK_CONFIG_FILE: This environment variable is used as the location for the site configuration file, silk.conf, for report types that use rwfilter. When this environment variable is not set, rwfilter searches for the site configuration file in the locations specified in the "FILES" section.
SILK_DATA_ROOTDIR: This environment variable specifies the root directory of data repository for report types that use rwfilter. This value overrides the compiled-in value. In addition, rwfilter may use this value when searching for the SiLK site configuration files. See the "FILES" section for details.
SILK_RWFILTER_THREADS: The number of threads rwfilter uses when reading files from the data store.
SILK_PATH: This environment variable gives the root of the install tree. When searching for the site configuration file, rwfilter may use this environment variable. See the "FILES" section for details.
PATH: This is the standard UNIX path (c.f., environ(7)). Depending on the report type, rwscanquery may invoke rwfilter(1), rwset(1), rwsetbuild(1), or rwsetcat(1) as part of its processing.
RWFILTER: Complete path to rwfilter. If not set, rwscanquery attempts to find rwfilter on your PATH.
RWSET: Complete path to rwset. If not set, rwscanquery attempts to find rwset on your PATH.
RWSETBUILD: Complete path to rwsetbuild. If not set, rwscanquery attempts to find rwsetbuild on your PATH.
RWSETCAT: Complete path to rwsetcat. If not set, rwscanquery attempts to find rwsetcat on your PATH.

FILES

${RWSCANRC}
${HOME}/.rwscanrc
/usr/share/silk/.rwscanrc: Possible locations for the rwscanquery configuration file, .rwscanrc. In addition, rwscanquery checks the parent directory of the directory containing the rwscanquery script.
${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/share/silk/silk.conf
/usr/share/silk.conf: Possible locations for the SiLK site configuration file---for report types that use rwfilter.