rwscanquery - Query the network scan database
--report=REPORT_TYPE Select query and output options. Values for REPORT_TYPE are standard, volume, scanset, scanflows, respflows, and export --start-date=YYYY/MM/DD:HH Report on scans active after this date. --end-date=YYYY/MM/DD:HH Defaults to start-date. --saddress=ADDR_SPEC Show scans originating from matching hosts. --sipset=IPSET_FILE Show scans originating from hosts in set. --daddress=IP_WILDCARD Show only scans targeting matching hosts. --dipset=IPSET_FILE Show only scans targeting hosts in set. --show-header Display column titles at start of output. --columnar Display more human-readable columnar view. --output-path=PATH Write results to the specified file.
--database=DBNAME Query an alternate scan database
--help Display this brief help message. --man Display the full documentation. --version Display the version information.
rwscanquery queries the network scan database---that is, the database that contains scans found by rwscan(1). The type of output rwscanquery creates is controlled by the --report switch as described in the "Report Options" section below. rwscanquery writes its output to the location specified by the --output-path switch or to the standard output when that switch is not provided.
rwscanquery runs a query of the scan database and then, depending on the report type, either displays the result set as text or creates a binary SiLK from the result set. The database rows that are part of the result set may be limited by using the --start-date, --end-date, --saddress, and --sipset switches. The result set is always limited to a time window, and the current day is used when no --start-date is given.
The following three report types produce textual output. The default output displays the values separated by a vertical bar (
|) with no spacing. The --columnar switch causes the output to appear in columns with a space-delimiter between the columns. The output includes no title line unless the --show-header switch is specified.
standard report contains most of the columns in the database for the rows in the result set. (The columns containing the scan model and scan probability are not included.)
volume report groups the rows in the result set by day and shows sums the flows, packets, and bytes columns for each day.
export report contains all the columns in the database for the rows in the result set, and the rows are displayed in a format compatible with rwscan.
The following three report types create a binary SiLK file as their result. These report types invoke other SiLK tools (namely rwfilter(1), rwset(1), rwsetbuild(1), and rwsetcat(1)) and the report types assume rwfilter has access to a SiLK data repository.
The first step in all three of these report types is for rwscanquery to get the distinct IP addresses for the rows in the result set and pass them into rwsetbuild to create a temporary IPset file containing the scanning IPs.
scanflows report produces a file of SiLK Flow records whose source IP is a scanning IPs. rwscanquery uses the temporary IPset as an argument to rwfilter to find flow records in your data repository that originated from the scanning IPs within the time window. You may choose to limit the report to particular IPs targeted by the scanning IPs by specifying the --daddress or --dipset switches. The output from rwfilter is the output of the report. The rwfilter invocation uses the configuration values
rw_in_type if they are specified in the configuration file (c.f. "CONFIGURATION").
respflows report produces a file of SiLK Flow records whose destination IP is a scanning IP. These flow records may represent responses to a scan. To create this report, rwscanquery performs steps similar to those for the
scanflows report except the direction of the rwfilter command is reversed to find flow records going to the scanning IPs. You may choose to limit the report to particular IPs that responded to the scan by specifying the --daddress or --dipset switches. The output from rwfilter is the output of the report. The rwfilter invocation uses the configuration values
rw_out_type if they are specified in the configuration file (c.f. "CONFIGURATION").
scanset report produces a binary IPset file.
If neither the --daddress nor --dipset switches are specified, the output of the this report is the temporary IPset file containing the scanning IPs; that is, all the scanning IPs in the time window.
Otherwise, rwscanquery performs the same steps it does as when creating
scanflows report. Next, instead of returning the output from rwfilter, rwscanquery passes the flow records into rwset to create an IPset file containing the scanning IPs that targeted particular IP addresses.
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.
Specify the query and the type of output to create. When this switch is not specified, the default is a
standard report. The supported values for TYPE are:
Write one textual line of output for each scan record in the scan database. By default, the output has no titles and it is not in columnar form. Specify the --show-header and/or --columnar switches to make the output more human readable.
Write a daily scan activity volume summary report for each day within the time period. By default, the output has no titles and it is not in columnar form. Specify the --show-header and/or --columnar switches to make the output more human readable.
Write an IPset file containing the IP addresses which were the sources of scan activity during the selected time period. The output of this report type is binary, so you must redirect or pipe the output to a location or specify the --output-path switch.
Write a SiLK Flow file containing all flows originating from scanning IP addresses within the specified time period. This flow data includes flows originating from any host that would be listed as a scan source by your query, from any time within the time period specified by --start-date and --end-date. Note that this may include flows that were not identified by the scan analysis as being part of a scan. The output of this report type is binary, so you must redirect or pipe the output to a location or specify the --output-path switch.
Write a SiLK Flow file containing all flows sent to scanning IP addresses within the specified time period---that is, possible responses to the scanners. The output of this report type is binary, so you must redirect or pipe the output to a location or specify the --output-path switch.
Write textual output consistent with the output format of the rwscan(1) tool. Specify the --show-header switch to include a title line.
Display scans which were active after this hour. When this argument contains a date with no hour and no --end-date switch is specified, scans for that entire day are returned. If this switch is not specified at all, scans for the current day (based on the local time on the host machine) are returned.
Display scans which were active before the end of this hour. If no end-date is given, defaults to the same as start-date. It is an error to provide an end-date without a start-date.
Display scans originating from hosts described in ADDR_SPEC, where ADDR_SPEC is a list of addresses, address ranges, and CIDR blocks. Only scans originating from hosts in the list are displayed.
Display scans originating from hosts in IPSET_FILE, where IPSET_FILE is a standard SiLK IPset file as created by rwset(1) or rwsetbuild(1). Note that a very complex IPset may take a long time to process, or even fail to return any results.
Display scans targeting hosts described in IP_WILDCARD, where IP_WILDCARD is a single IP address, a single CIDR block, or an IP Wildcard expression accepted by rwfilter(1). To match on multiple IPs or networks, use the --dipset switch. This switch is ignored for --report types other than
Display scans targeting hosts in IPSET_FILE, where IPSET_FILE is a standard SiLK IPset file. This switch is ignored for --report types other than
Display a header line giving a short name (or title) for each field when printing textual output with the
export report types. By default, no header is displayed.
Display output in more human-readable columnar format when printing textual output with the
volume report types. When this switch is not given, the output is presented as data fields delimited by the | character.
Write results to PATH instead of to the standard output.
Select a database instance other than the default. The default is specified by the
db_instance value in the configuration file as described in "CONFIGURATION" below.
Display a brief usage message and exit.
Display full documentation for rwscanquery and exit.
Print the version number and exit the application.
rwscanquery reads configuration information from a file named .rwscanrc. If the RWSCANRC environment variable is set, it is used as the location of the .rwscanrc file. When RWSCANRC is not set, rwscanquery attempts to find a file name .rwscanrc in the directories specified in the "FILES" section below.
The format of the .rwscanrc file is name=value pairs, one per line. The configuration parameters currently read from .rwscanrc are:
The type of database to connect to. rwscanquery supports
The userid to use when connecting to the scan database.
The password to use when connecting to the scan database.
The name of the database instance to connect to if none is provided with the --database command line switch. If neither this configuration option nor the --database command line switch are specified, the hard-coded default database instance "SCAN" is used.
The class for incoming flow data. The
rw_in_type values are used to query scan flows when the
scanflows report type is requested or when the --daddress or --dipset switches are used for the
scanset report type. If not specified, rwfilter's default is used.
The type(s) for incoming flow data. See
rw_in_class for details.
The class for outgoing flow data. The
rw_out_type values are used to query scan flows when the
respflows report type is requested. If not specified, rwfilter's default is used.
The type(s) for outgoing flow data. See
rw_out_class for details. (Note that rwfilter often defaults to querying incoming flows, so this parameter ought to be specified.)
In the following examples, the dollar sign (
$) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (
\) is used to indicate a wrapped line.
Display information on all scans occurring during the 12:00 hour (12:00:00 to 12:59:59) of 2009/02/12.
$ rwscanquery --show-header --start-date=2009/02/12:12 scan-id|stime|etime|proto|srcaddr|flows|packets|bytes 499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240 365|2009-02-12 12:08:40|2009-02-12 12:14:54|6|10.146.88.117|256|256|10240 57|2009-02-12 12:28:51|2009-02-12 12:34:55|6|10.29.23.160|256|256|10240 627|2009-02-12 11:52:07|2009-02-12 12:41:16|17|10.253.24.230|1023|1023|30175 366|2009-02-12 12:41:50|2009-02-12 12:48:14|6|10.146.89.46|256|256|10240 182|2009-02-12 12:54:39|2009-02-12 13:01:20|6|10.79.26.176|256|256|10240 4|2009-02-12 12:41:19|2009-02-12 13:33:57|17|10.2.47.87|1023|1023|30205
Create the IPset file scan.set containing the scanners discovered during that hour.
$ rwscanquery --report=scanset --start-date=2009/02/12:12 \ --output-path=scan.set $ rwsetcat scan.set 10.2.47.87 10.29.23.160 10.79.26.176 10.146.88.117 10.146.89.46 10.199.151.231 10.253.24.230
Repeat the first query but limit the output to scanners coming from the CIDR block 10.199.0.0/16.
$ rwscanquery --show-header --start-date=2009/02/12:12 \ --saddr=10.199.0.0/16 scan-id|stime|etime|proto|srcaddr|flows|packets|bytes 499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240
Expand the query for that CIDR block to include the preceding and following hours (11:00:00 to 13:59:59).
$ rwscanquery --start-date=2009/02/12:11 --end-date=2009/02/12:13 \ --saddr=10.199.0.0/16 499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240 497|2009-02-12 13:33:57|2009-02-12 14:24:35|17|10.199.98.5|1023|1023|30079
Create the IPset file scanning-cidr.set that contains the CIDR block 10.199.0.0/16, and then search for scans coming from that IP on Feb 13, 2009.
$ cat scanning-cidr.txt 10.199.0.0/16 $ rwsetbuild scanning-cidr.txt scanning-cidr.set $ $ rwscanquery --start-date=2009/02/13 --sipset=scanning-cidr.set 500|2009-02-13 22:42:25|2009-02-13 22:48:45|6|10.199.207.32|256|256|10240
Print the volume of data attributed to scans over a three day period.
$ rwscanquery --report=volume --show-header \ --start-date=2009/02/12 --end-date=2009/02/14 date|flows|packets|bytes 2009/02/12|137452|137499|17149008 2009/02/13|74727|76167|2798040 2009/02/14|76160|76160|2750531
The following limits the volume report to the IPs in the file scanning-cidr.set and displays the results in columns.
$ rwscanquery --report=volume --show-header --columnar \ --start-date=2009/02/12 --end-date=2009/02/14 \ --sipset=scanning-cidr.set date flows packets bytes 2009/02/12 1279 1279 40319 2009/02/13 256 256 10240 2009/02/14 256 256 10240
Get the SiLK Flow records coming from the scanners during the 12:00 hour on 2009/02/12 and store in the file scanning-flows.rw.
$ rwscanquery --report=scanflows --start-date=2009/02/12:12 \ --output=scanning-flows.rw
Use rwuniq(1) to summarize the file scanning-flows.rw.
$ rwuniq --fields=sip --values=flows,packets,bytes \ --sort-output scanning-flows.rw sIP| Records| Packets| Bytes| 10.2.47.87| 373| 373| 11032| 10.29.23.160| 256| 256| 10240| 10.79.26.176| 203| 203| 8120| 10.146.88.117| 256| 256| 10240| 10.146.89.46| 256| 256| 10240| 10.199.151.231| 256| 256| 10240| 10.253.24.230| 846| 846| 24921|
Run a respflows report to verify that there were no responses to the scan.
$ rwscanquery --report=respflows --start-date=2009/02/12:12 \ --output=scanning-response.rw $ $ rwuniq --fields=sip --values=flows,packets,bytes \ --sort-output scanning-response.rw sIP| Records| Packets| Bytes|
Create the IPset subnet-scan.set for scanners that targeted the 192.168.186.0/24 CIDR block during the 12:00 hour on 2009/02/12.
$ rwscanquery --report=scanset --start-date=2009/02/12:12 \ --daddress=192.168.186.0/24 --output-path=subnet-scan.set
Store the corresponding flow records for those scans in the file subset-scan.rw.
$ rwscanquery --report=scanflows --start-date=2009/02/12:12 \ --daddress=192.168.186.0/24 --output-path=subnet-scan.rw
Determine how many IPs in that subnet were targeted.
$ rwuniq --fields=sip --values=flows,distinct:dip subnet-scan.rw sIP| Records|dIP-Distin| 10.146.89.46| 256| 256|
Display the title line for an export report.
$ rwscanquery --report=export --start-date=2009/02/12:12 \ --show-header | head -1 id|sip|proto|stime|etime|flows|packets|bytes|scan_model|scan_prob
This environment variable allows the user to specify the location of the .rwscanrc configuration file. The value may be a complete path or a file relative to the user's current directory. See the "FILES" section for standard locations of this file.
The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction for the report types of
This environment variable is used as the location for the site configuration file, silk.conf, for report types that use rwfilter. When this environment variable is not set, rwfilter searches for the site configuration file in the locations specified in the "FILES" section.
This environment variable specifies the root directory of data repository for report types that use rwfilter. This value overrides the compiled-in value. In addition, rwfilter may use this value when searching for the SiLK site configuration files. See the "FILES" section for details.
The number of threads rwfilter uses when reading files from the data store.
This environment variable gives the root of the install tree. When searching for the site configuration file, rwfilter may use this environment variable. See the "FILES" section for details.
Complete path to rwfilter. If not set, rwscanquery attempts to find rwfilter on your PATH.
Complete path to rwset. If not set, rwscanquery attempts to find rwset on your PATH.
Complete path to rwsetbuild. If not set, rwscanquery attempts to find rwsetbuild on your PATH.
Complete path to rwsetcat. If not set, rwscanquery attempts to find rwsetcat on your PATH.
Possible locations for the rwscanquery configuration file, .rwscanrc. In addition, rwscanquery checks the parent directory of the directory containing the rwscanquery script.
Possible locations for the SiLK site configuration file---for report types that use rwfilter.