Changelog

• rwflowpack change
  • Modify the log messages produced by libfixbuf to follow the format of other rwflowpack log messages.
  • Modify NetFlow v9 support to require libfixbuf-1.1.0.
• flowcap change
  • Modify the log messages produced by libfixbuf to follow the format of other rwflowpack log messages.
  • Modify NetFlow v9 support to require libfixbuf-1.1.0.
• Building
  • Add new configure switch --enable-asa-zero-packet-hack to work around a bug in the NetFlow9 template used by Cisco ASA routers wherein the template is missing a packetTotalCount field, causing rwflowpack to treat these flows as having 0 packets. When the switch is specified, SiLK sets the packet count to 1 for flow records having a source IP, a byte count, but no packet count. In addition, if SiLK is compiled without IPv6 support, the hack causes rwflowpack to a use fully-expanded file format to store IPv4 flow records collected from netflow-v9 probes.

Changelog

• rwfilter enhancement
  • Better support when writing to a pipe and a file or another pipe simultaneously. Specifically, rwfilter used to exit when any pipe stopped receiving data. Now, rwfilter will finish writing the output to the file or other pipe when one pipe closes.
• rwset bug fix
  • Ignore IPv6 flow records.
• rwipfix2silk changes
  • Ignore IPFIX records that have a packet or byte count of zero.
  • Fix an issue where rwipfix2silk did not free the memory for the current input file before opening the next file.
• rwgeoip2ccmap bug fixes
  • Provide better error messages when the user provides the wrong input switch to the program.
  • When processing binary input, tell the user about any unrecognized values.
  • Verify that the resulting prefix map is valid before writing the map to the output.
• rwpmapbuild enhancement
  • Performance is hugely improved when building very large prefix maps.
• rwfileinfo bug fix
  • Fix an issue when processing a compressed file containing a corrupted compressed block that caused rwfileinfo to report fewer valid records than actually existed.
• rwflowpack bug fix
  • Fix an issue when processing IPFIX files where the file was never closed. This could cause rwflowpack to exit unexpectedly once it ran out of file handles.
  • Ignore return codes from libfixbuf that indicate it received a NetFlow v9 element it did not understand.
  • Ignore IPFIX records that have a packet or byte count of zero.
  • Ignore IPFIX records from yaf marked as "udp-uniflow".
• flowcap bug fix
  • Ignore return codes from libfixbuf that indicate it received a NetFlow v9 element it did not understand.
  • Ignore IPFIX records that have a packet or byte count of zero.
  • Ignore IPFIX records from yaf marked as "udp-uniflow".
• PySiLK change
  • Add a constant containing the maximum bag counter value.
• rwpdedupe bug fix
  • Fix fatal error.
• Building
  • Modify the expected result of some tests run with "make check" when standard input is not a terminal.
  • Fix a configuration issue when testing for Python on Ubuntu.
• rwsetunion, rwsetintersect
  • Mark these applications as deprecated. Use rwsettool instead.

Changelog

• flowcap, rwflowpack
  • Modify NetFlow v5 collection to handle changes in sequence number due to router reboots and sequence number roll-overs.
  • Change how missing NetFlow records are reported.
  • Fix potential deadlock that can occur when buffer holding packets to process fills to capacity.
• rwflowpack
  • For netflow-v5 probes, always use the 'log-flags' setting from sensor.conf regardless of source of the records
• rwbag
  • Stop processing input once memory is exhausted.
• rwfilter
  • Manual page enhancements.

Changelog

• rwcut, rwuniq, rwstats
  • Add a new --integer-tcp-flags switch that prints the TCP flag fields as an integer value instead of characters.
• rwtuc
  • Provide the --verbose and --stop-on-error switches that report why a field failed to parse.
• rwfilter, ipafilter.so
  • Fix a bug where the ipafilter.so plug-in prevented rwfilter from using multiple threads, even when ipafilter was not active.
• rwp2yaf2silk
  • Add a --version switch.
• Manual page updates in several tools.
• Configuring/Building
  • Have "make" create Perl and Python scripts from .in files using the paths to Perl and Python found by configure.

Changelog

• rwfilter, ipafilter.so
  • Fix an issue where rwfilter attempted to initialize ipafilter on every invocation, even when IPA-related switches were not given. This prevented rwfilter from running when SiLK was built with IPA support but run-time support for IPA was not configured.
• flowcap
  • Fix bug in start-up script that passed wrong value to the --clock-time switch.
  • Internal changes.

Changelog

• PySiLK
  • Fix bug in IPv4Addr.mask_prefix() when prefix was 32.
• rwpmapcat
  • Fix bug where final newline would not be printed for some types of output.
• rwfileinfo
  • Fix floating point exception when attempting to print record counts for very old SiLK files.
• rwpollexec
  • Fix unexpected exit that would occur when rwpollexec was run without an archive-directory.
  • Modify rwpollexec so it no longer exits when there is a problem archiving a file or moving it to the error-directory.
• Configuring/Building
  • Fix issue that prevented building with static packing logic.

Changelog

• rwuniq
  • Change the hashing function used internally. This may affect the order in which bins are printed for unsorted output.
  • Fix a bug in --sorted-output that caused the sort order to always consider start-time and plug-in fields after all other fields, regardless of order they appeared in the --fields switch.
  • Fix a bug when --fields contained start-time, end-time, and duration that caused the output to appear to contain two identically keyed bins.
  • Remove limit on number of temporary files rwuniq may open
  • WARNING: These changes will affect the output of rwuniq.
• rwstats
  • Remove limit on number of temporary files rwstats may open
  • Fix a bug when --fields contained start-time, end-time, and duration that caused the output to appear to contain two identically keyed bins.
• rwsort
  • Remove limit on number of temporary files rwsort may open.
• rwtuc
  • Fix a bug in the time parsing code that would be triggered when the time was given as seconds since the UNIX epoch and the field included trailing whitespace.
• rwcut
  • Fix a bug where --all-fields would fail unless rwcut found the mapping files for country code and address type.
• rwipaexport, rwipaimport
  • Modified to require libipa-0.5.0.
• ipafilter.so
  • New plug-in for rwfilter that supports partitioning flow records based on IPA data.
• PySiLK
  • Minor bug fixes in site initialization
  • Other minor enhancements
• rwpollexec
  • New daemon that monitors a directory for files and invokes a user-supplied command on each file.
• rwflowpack
  • Add new input mode, respool, that takes SiLK Flow files as input and puts the records into a data repository, maintaining the sensor and class/type values on the original records.
  • Fix a bug when using multiple IPFIX directory-based probes.
• flowcap
  • Add the --clock-time switch that allows flowcap to expire files at predictable wall-clock times.
• rwsender, rwreceiver
  • Add a check to determine if the timestamps on the TLS certificates provided on the command line are valid. Have the application write a log message and exit if the certificates are not valid.
  • Explicitly check the timestamps of the TLS certificates received from the other side of the connection in order to provide a better log message when expired certificates are received.
  • Fix issues in handling of simultaneous connections that could cause the process to hang.
  • Fix issues during shutdown that could cause the process to hang.
• libsilk
  • Fix an issue when reading compressed files that caused the uncompress function to be called more times than was necessary.
• Configuring/Building/C API
  • Add an SK_ prefix to all CPP macros generated by autoconf to avoid conflicts with other packages that use autoconf.
  • Change some features previously supported by plug-ins to be part of libsilk instead. This affects country code (ccfilter), address type (addrtype), and prefix map (pmapfilter) support.

Changelog

• rwflowpack
  • Fix a bug in expiring flows from the stream cache that could cause rwflowpack to deadlock or to exit unexpectedly.
• rwscan
  • Ignore additional flags when checking status of SYN flag.
• rwsort
  • Fix a bug in the handling presorted input that would occur if rwsort ran out of file descriptors.
• rwstats
  • Fix a bug in the handling presorted input that would occur if rwstats ran out of file descriptors.
  • Fix a bug in merging temporary files when no distinct fields had been specified.
• rwuniq
  • Fix a bug in the handling presorted input that would occur if rwuniq ran out of file descriptors.
  • Fix a bug in merging temporary files when no distinct fields had been specified.

Changelog

• rwfileinfo
  • Determining the number of records in a file is much faster.
• silk.conf
  • The sensor command now allows an optional textual description. To use this feature, the file's version must be set to 2.
• mapsid
  • New --print-descriptions switch prints the description of the sensors (as set in the silk.conf file).
• rwflowpack
  • Added a --flat-archive switch that prevents rwflowpack from creating subdirectories below the --archive-directory.
  • The default size of the --file-cache, which determines the number of open files to use for writing, is now 128. The file cache will now close files and remove them from the cache after a period of inactivity. Improve performance of the file cache when attempting to write to more files than will fit into the cache.
  • Improve performance when many poll-directory probes are defined, and fix an issue that could lead to too many open files.
  • In the sensor.conf file, multiple probes may specified within a single sensor block.
  • Modified NetFlow v9 support to require libfixbuf-1.0.0 (as yet unreleased).
• rwflowappend
  • Added a --flat-archive switch that prevents rwflowappend from creating subdirectories below the --archive-directory.
• rwreceiver
  • Fix a bug in handling the --post-command.

Changelog

• PySiLK
  • Country code support is now available from PySiLK. Use init_country_code() to initialize, and the IPAddr.country_code() method to return the country code.
  • IPAddr() is replaced by IPv4Addr() and IPv6Addr() which both inherit from an IPAddr() object that will never be instantiated. IPAddr() is now a constructor for IPv4Addr() and IPv6Addr().
  • INCOMPATIBILITY: The IPAddr() constructor no longer accepts an integer as an argument. You must use IPv4Addr() or IPv6Addr().
  • INCOMPATIBILITY: The IPAddr.to_ipv4() and IPAddr.to_ipv6() methods now return new IPv4Addr and IPv6Addr objects, respectively, and no longer modify the exiting IPAddr.
  • New methods on IPAddr objects: mask(), mask_prefix(), octets()
  • New method IPAddr.is_ipv6() should be used in place of IPAddr.isipv6() which is now deprecated.
  • INCOMPATIBILITY: The deprecated IPAddr.ipv6() method has been removed. Use IPAddr.is_ipv6() instead.
  • Bug fixes in plug-in support.
• rwfilter
  • New --scidr, --not-scidr, --dcidr, etc switches support partitioning by comma separated list of IPs and/or CIDR blocks.
• rwreceiver
  • Add new --duplicate-destination switch that copies received files to an additional directory. The switch may be repeated.
  • Uses finer grained locking which should improve throughput when receiving from multiple rwsenders.
• rwsender
  • Add new --local-directory switch that copies incoming files to a local directory (i.e., "send" to the local host). The switch may be repeated.
• rwflowpack
  • Add new --post-archive-command that specifies a command to run on an input file once rwflowpack has processed the file and moved the file to the archive directory.
• rwflowappend
  • Add new --reject-hours-past and --reject-hours-future switches that allow rwflowappend to reject files whose records are outside a time window (based on the current time).
• flowcap
  • Modify log message to report number of missing records when closing a file.
  • Additional log messages added when using --log-level=debug
• rwstats
  • Fix a crash that would occur when using --presorted-input on files that contained no records.
• rwresolve
  • Fix a bug that would occur if the fields were not monotonically increasing.
• num2dot
  • Fix a bug that resulted in a newline not being printed when converting the final column and there was no final delimiter.
• Prefix Maps
  • Fix a bug on big-endian 64bit machines where a prefix map file would appear to have no entries.
• rwscanquery
  • Fix a long-standing bug were rwscanquery used an old name for the rwsetcat command
• Configuring/Building/C API
  • Provide simplified APIs in C for creating new fields for rwcut, rwgroup, rwsort, rwstats, and rwuniq via plug-ins. These APIs are similar to those available in PySiLK.
  • New silk_config program can be used to determine the headers and libraries needed to link a program against the SiLK libraries.
  • The hashlib_* functions are now part of libsilk and are no longer in a separate library (libhash).

Changelog

• PySiLK
  • New register_switch() function allows user to create a command line switch that can be used when PySiLK is running as a plug-in.
  • Provide new functions to simplify the registering of fields when working with common data types.
• rwflowpack
  • In the sensor.conf file, provide a way for the administrator to give a name to a list of IPs or interface values.
  • When reading IPFIX, allow the VLAN identifiers to be stored in the SiLK Flow records in place of the SNMP interface numbers.
  • Allow rwflowpack to discard records (as opposed to packing them) when the records have an IP address or an interface value that matches a list specified by the administrator.
• rwipfix2silk
  • When reading IPFIX, allow the VLAN identifiers to be stored in the SiLK Flow records in place of the SNMP interface numbers.
• flowcap
  • When reading IPFIX, allow the VLAN identifiers to be stored in the SiLK Flow records in place of the SNMP interface numbers.
• rwfilter
  • Add --max-fail switch for consistency with --max-pass.
  • New 'app-mismatch' plug-in will pass flows when the application determined by the flow generator does not match either the source or destination port.
  • Fix a bug when running with multiple threads.
• rwcut
  • Fix bug where --end-rec-num was being ignored when no other limiting switches were present.
• rwreceiver
  • Fix a potential deadlock that could occur when an rwsender suddenly becomes unavailable.
• Building/Installation
  • Fix an issue where we attempted to install rwp2yaf2silk twice.

Changelog

• Prefix Maps
  • Add a map-name keyword to rwpmapbuild which allows a MAPNAME to be specified in the prefix map file.
  • Modify --pmap-file switch to allow an optional MAPNAME: to appear before the file name.
  • Allow rwfilter to use multiple prefix maps in a single invocation: For each MAPNAME, switches --pmap-src-MAPNAME, --pmap-dst-MAPNAME, and --pmap-any-MAPNAME are generated to partition the SiLK Flow records.
  • Allow rwcut, rwgroup, rwsort, rwstats, and rwuniq to use multiple prefix maps in a single invocation: For each MAPNAME, new src-MAPNAME and dst-MAPNAME fields are available.
  • NOTE: The prefix map code is fully backward compatible with previous releases of SiLK.
• rwuniq
  • Add --values switch that specifies the volumes (aggregate values) that rwuniq should compute. Value columns will be printed in the order they appear in this list.
  • Allow the user to define new aggregate fields by loading plug-ins written in PySiLK or C.
  • Fix issue where IPv4 addresses were being printed as IPv6 by default.
  • Fix a possible bug when sorted output is requested.
  • WARNING: There is a slight difference in the names of the columns that contain the counts of distinct IPs.
• rwstats
  • Add support for an arbitrary key. The --fields switch specifies the fields that rwstats should as the key. It supports the same fields as rwuniq.
  • Add support for computing multiple volumes. The --values switch specifies the volumes (aggregate values) that rwstats should compute. Value columns will be printed in the order they appear in this list. The first value column will be used as the basis for computing the top-N or bottom-N.
  • Add support for country codes, for generating fields from prefix maps, and for defining fields by loading PySiLK or C plug-ins.
  • Add support for defining new aggregate fields by loading plug-ins written in PySiLK or C.
  • Add support for IPv6 (when enabled at compile time).
  • Add numerous switches to specify the form of the output (--epoch-time, --integer-senors, etc).
  • NOTE: rwstats continues to support the same switches, but many switches are now deprecated.
  • WARNING: There are some differences in the headers and column titles that rwstats generates, and columns may have different widths.
• rwgroup
  • Add support for the same fields as rwcut.
  • Add support for country codes, for generating fields from prefix maps, and for defining fields by loading PySiLK or C plug-ins.
  • Add support for IPv6 (when enabled at compile time).
  • Allow the user to specify the initial ID to write into the next hop IP field via the --group-offset switch.
  • Add support for the --output-path and --copy-input switches.
  • WARNING: When the --delta-field refers to the source or destination IP address, the --delta-value switch is now taken to be the number of least significant bits to mask off prior to comparing the records.
• rwcompare
  • New tool to determine whether two SiLK Flow files contain the same records in the same order (in the spirit of UNIX cmp).
• PySiLK
  • Add Bag support (creating, reading, and writing).
  • Add Prefix map support (read-only).
  • Add operators to IPAddr objects for converting to IPv4 or IPv6.
  • Add operator to IPAddr objects for returning a string that is fully expanded and padded with 0's.
  • Modified the API for creating fields when PySiLK is used as a plug-in: register_field() replaces register_plugin_field(). The previous API is supported but deprecated.
  • Provide new register_filter() function for rwfilter plug-ins.
  • WARNING: The str() method on TCPFlags objects no longer pads the value with spaces. Use the new padded() method on TCPFlags objects to get the old string presentation.
• General Changes
  • Make the field names case insensitive in rwcut, rwdedupe, rwgroup, rwsort, rwstats, rwtuc, and rwuniq.
  • Provide a new --plugin switch for loading C plug-ins. The --dynamic-library switch on rwcut, rwfilter, rwflowpack, rwgroup, rwptoflow, rwsort, rwstats, and rwuniq is available but deprecated.
  • Allow the SILK_COUNTRY_CODES environment variable to name the location of the country code (ccfilter.so) mapping file to use.
  • Allow the SILK_ADDRESS_TYPES environment variable to name the location of the address types (addrtype.so) mapping file to use.
  • Treat protocol 58 as ICMPv6 when SiLK is compiled with IPv6 support. The rwfilter --icmp-type and --icmp-code will match ICMPv6, and the icmpTypeCode field (rwcut, rwgroup, rwsort, rwstats, rwuniq) will decode the ICMPv6 type and code.
  • Add annotation support (the --note-add family of switches) to rwgroup, rwipaexport, rwipfix2silk, rwnetmask, rwptoflow, rwsort, rwsplit, rwswapbytes, and rwtuc.
  • Allow specification of the compression method to use for the output files created by rwgroup, rwipaexport, rwnetmask and rwsplit.
• rwnetmask
  • Add support for IPv6 via the --6sip-prefix-length, --6dip-prefix-length, and --6nhip-prefix-length switches.
  • Add new --4sip-prefix-length, --4dip-prefix-length, and --4nhip-prefix-length switches for consistency. For backward compatibility, alias the existing --sip-prefix-length, etc, switches to these IPv4 names.
  • Add support for the --ipv6-policy switch.
• rwbagcat
  • Enhance the --network-structure switch to allow arbitrary CIDR blocks. You can now print information about any CIDR block size.
  • Add --mask-set switch to print the intersection of the Bag and the IPset. With --zero-counts, prints a counter value for every IP in the IPset.
• rwbagtool
  • Modify the --subtract operator to no longer treat negative counters as an error; instead the key is not included in the result.
  • Modify the --divide operator to no longer treat values less than 1 as an error; instead the key is not included in the result.
  • Add the --scalar-multiply operator which takes a positive scalar argument and multiplies every counter in the Bag by that value.
  • Add the --minimize operator which creates a Bag that contains, for each key in the input Bags, the smallest counter. A missing key is treated as if its counter is 0.
  • Add the --maximize operator which creates a Bag that contains, for each key in the input Bags, the largest counter.
  • Add the --compare operator to compares the contents of two Bags.
  • WARNING: Remove the deprecated --output-file switch. Use --output-path instead.
• rwpmapcat
  • Add --left-justify-labels switch that causes the labels to be left justified instead of right justified.
  • Allow the map to read to simply be specified on the command line; that is, no longer require the use of the --map-file switch.
• rwsetbuild
  • No longer require the file name arguments. rwsetbuild writes to the standard output when only one file is specified; additionally, it reads from the standard input when no files are specified.
• rwsetcat
  • Enhance the --network-structure switch to allow arbitrary CIDR blocks. You can now print information about any CIDR block size.
• rwsplit
  • Allow the user to specify value used to initialize the pseudo-random number generator via the new --seed switch.
• rwcat
  • Add --byte-order switch to allows the user to set the byte order of the output file.
  • Add --ipv4-output switch to allows the user to force the output to be SiLK's default IPv4 format.
• rwfilter
  • Modify the --flags-all, --flags-initial, and --flags-session switches to allow a comma separated list of HIGH/MASK flag pairs.
  • Modify the --attributes switch to allow a comma separated list of HIGH/MASK attribute pairs.
• rwscan
  • Fix a bug in the Bayesian Logistic Regression (BLR) method that may have caused it to miss some scans.
• rwflowpack
  • The "flowcap" input mode (which allowed rwflowpack to connect to a flowcap running in server-mode) has been removed. Use rwsender/rwreceiver to transfer files instead.
  • Add support for processing files created by yaf.
  • Add support for processing SiLK flow files.
  • Add the --verify-sensor-config switch which causes rwflowpack to exit after checking the syntax of the sensor.conf file.
• flowcap
  • The "server-mode" (which allowed rwflowpack to contact flowcap) has been removed. Use rwsender/rwreceiver to transfer files.
  • Add the --verify-sensor-config switch which causes flowcap to exit after checking the syntax of the sensor.conf file.
• rwsender, receiver
  • Fix bugs related to using TLS.
  • Make daemons more robust with respect to sudden loss of connectivity to their peer(s).
  • Better handle duplicate files and partially transferred files.
  • Fix a race condition in rwsender when attempting to transfer a file to multiple rwreceivers.
  • Add --error-directory switch to rwsender. rwsender will move to this directory any files that failed to transfer. The --error-directory switch is required.
• C-Code changes
  • Header files have been moved from src/include to src/include/silk and files should use #include <silk/foo.h> for file foo.h.
  • Rewrite the API to plug-ins. The old API is still supported, but it is deprecated and will be removed in a future release.
  • Provide a new API to IPsets.
  • Many additional changes.

Changelog

• rwflowpack
  • Modify internal buffering of unprocessed records. This should greatly reduce the memory usage.
• flowcap
  • Allow the compression method to be set at run-time, but continue to default to the "best" compression method available.
  • Modify internal buffering of unprocessed records. This should greatly reduce the memory usage.
• rwdedupe
  • Fix a bug that caused one record not to be written to the output.
  • Modify the sort key so that the --delta-fields have the lowest priority.
• rwrandomizeip
  • Fix a bug and potential crash on Solaris.
• rwpackchecker
  • Fix minor bugs in output and exit status.
• rwsender/rwreceiver
  • Fix a crash that could occur due to a thread synchronization error.
• silk.spec
  • Fix an issue where the daemon control scripts would look in the wrong location for their conf files.
  • Fix a bug in the pre-uninstall section
• daemon control scripts
  • Address potential shell quoting issues.

Changelog

• rwflowpack
  • Allow the packing logic to use the ingressInterface and egressInterface values in data from IPFIX probes.
  • Fix crash that would occur when multiple probes were configured to listen on the same port.
• rwsender/rwreceiver
  • Add a feature to close connections that have been completely silent for two keep-alive cycles.
  • Fix a bug that prevented keep-alive messages from being sent.
• rwipfix2silk
  • Fix a bug that prevented rwipfix2silk from handling multiple input files.

Changelog

• rwresolve
  • Add support for the ADNS library to speed IP to host mappping
  • Add support for a name cache to avoid querying DNS repeatedly for the same IP.
• rwfilter
  • Any support for --pmap-any-address and --pmap-any-port-proto
  • Fix a bug that caused threaded rwfilter to always exit with a non-zero status.
• rwsort
  • Add a --reverse switch that causes the records to be sorted from largest key to smallest
• flowrate.so
  • Add switches to the flowate.so plug-in to estimate the payload bytes of a flow and the payload bytes per second.
• cutmatch.so
  • Fix a bug that prevented the cutmatch.so plug-in from registering its field name with rwcut.
• Cygwin compatibility
  • Fix issues that prevented compilation of some packing tools
  • Fix a bug in use of getaddrinfo

Changelog

• PySiLK
  • Add support for a new SILK_PYTHON_TRACEBACK environment variable. When set, errors in the Python code will be reported to the user.
  • FUNCTIONAL CHANGE: IPSet.add(), IPSet.discard(), and IPSet.remove() now accept only a single IPAddr or a single IP Address string. (They used to support IPWildcards).
  • Fix bugs in RWRec when handling certain dates and time-ranges from Python.
  • In the register_plugin_field() function, change the name of the 'field_len' parameter to 'text_len', but allow 'field_len' for backwards compatibility.
  • The silk.plugin module is now available outside of PySiLK plug-ins, allowing the use of register_plugin_field() from library code.
  • Deprecate IPAddr.ipv6() in favor of IPAddr.isipv6().
  • Update and expand documentation.
• rwcount
  • Fix fatal error in --load-scheme=0 when used with --end-epoch
• rwip2cc
  • Fix minor bug in output when --address is specified.
• rwpmapbuild
  • Fix a bug that prevented --input-file=stdin from working.
• flowcap
  • Fix a bug that prevented processing of NetFlow v9 data.
• rwflowpack
  • When any write error occurs, force rwflowpack to shutdown.
  • Change how shutdown is initiated and the order in which structures are destroyed to avoid fatal memory errors.
• rwflowappend
  • Do not exit when attempting to open an invalid incremental file; instead, move the invalid file to the error-directory, log an error, and continue to run.
  • Fix a bug where the --hour-file-command was not being invoked.
  • Fix an issue on Mac OS X where rwflowappend would not respond to signals once the --post-command/--hour-file-command had run
• rwreceiver
  • Fix an issue on Mac OS X where rwreceiver would not respond to signals once the --post-command had run
• Support systems that do not provide getaddrinfo

Changelog

• rwsender
  • Greatly reduce the memory requirement of rwsender by memory-mapping the files as they are sent. Previously, the files were read into RAM, causing the rwsender process to have a large memory footprint.
• rwfilter
  • Fix a bug in parsing the user's times when SiLK was configured with --enable-localtime and Daylight Savings Time is active
• flowrate.so
  • Provide a manual page for the flowrate plug-in
• rwuniq
  • Fix a bug that prevented the use of PySiLK from within rwuniq
• rwreceiver
  • Fix a memory error when using the --post-command
• rwflowappend
  • Fix a memory error when using the --post-command
• Plug-in support
  • Force SiLK plug-ins to have an ".so" suffix, to better support systems that use a different suffix for shared objects.
• Minor fixes to manual pages and --help output

Changelog

• rwuniq
  • New --sort-output switch causes rwuniq to present its output in sorted order, where the sort-key is the --fields value.
• rwflowpack
  • SiLK now supports collection of NetFlow v9 when linked with libfixbuf-0.8.0.
    (As of SiLK-2.3.0, NetFlow-v9 support requires libfixbuf-1.0.0.)
  • Writing of records by rwflowpack has been greatly enhanced by using finer-grained locking and using pthread read/write locks on systems that support them.
  • New --error-directory switch allows rwflowpack to continue processing files in spite of an invalid input flow file. Previously, an invalid input file would cause rwflowpack to exit. rwflowpack will still exit unless the --error-directory is set.
  • New --file-cache-size switch allows the user to control the number of output files that rwflowpack has open simultaneously.
  • Fixed some issues that occurred when reading data from IPFIX and NetFlow-v9 probes when the UDP protocol was used.
  • rwflowpack now provides more logging messages, especially when run with --log-level=debug
  • Fixed a potential crash during shutdown when using PDU Directory polling.
• rwreceiver
  • New --post-command switch causes rwreceiver to invoke the specified command on each file once the file has been received.
• rwpmapcat
  • Enhanced the output to print Protocol/Port Prefix Maps as ranges of protocol/port pairs.
• rwpmapbuild
  • Fix a bug in reading Protocol/Port Prefix Maps where the file was not marked as containing protocol/port pairs.
• pmapfilter.so
  • Fix a crash on some operating systems caused by failure to allocate enough memory for the Prefix Map.

Changelog

• rwip2cc
  • POTENTIAL INCOMPATIBILITY. When reading the IP input from a file, the default output is now two columns: the IP and the country code. The output for a single IP is unchanged.
  • Use the new --print-ips switch to force whether IPs should be printed.
  • Additional switches have been added to control the format of the columns.
• rwfilter
  • New --flowtype switch allows selection of data from multiple class/type pair, including data from different classes.
  • Support for the --tuple-* switch is now compiled into rwfilter instead of being supported by a plug-in.
  • Fix a crash that would occur when a class in silk.conf listed no default types.
• flowrate.so
  • A new plugin exists to filter, display, sort, and bin by packets-per-second, bytes-per-second, and bytes-per-packet. The flowrate.so plugin must be explicitly loaded in the application.
• rwflowpack
  • Add capability to watch directories for NetFlow v5 files, where each directory is associated with a probe.
  • Fix a bug in parsing the sensors from sensor.conf that causes later *-probes statements to overwrite the previous probes.
• flowcap
  • Fix crash that occurred when flowcap was called with no arguments.
• rwpmapbuild
  • Fix a bug that prevented parsing of protocol/port pairs.
• rwbagtool, rwsettool
  • Fix a bug where the --note-strip switch was configured to require an argument.
• rwflowappend
  • Fix a bug where the default data format was network byte order instead of native byte order.
• PySiLK Configuration
  • Fix an issue that prevented the PySiLK code from being relocated during installation
  • Add a library whose absence prevented configuration on OpenBSD.
  • Fix a bug in the Makefile that preventing compilation when BSD make was used
• Several tools
  • Fix a crash that would occur when attempting to read SiLK flows from a non-flow file.

Changelog

• PySiLK
  • Changed the default install location for PySiLK so it is now installed with other Python modules. Use the configure script's --with-python-prefix switch to change the install location.
  • Extended the PySiLK capability to support user-defined fields in rwsort and rwuniq.
  • Changed the way the user defines PySiLK fields for rwcut. POTENTIAL INCOMPATIBILITY.
  • Fixed a reference counting bug that led to a memory leak.
  • Improved the checks for Python that occur during configuration.
• rwuniq
  • Enhanced to handle the issue of fast memory being exhausted. rwuniq will use temporary files to allow it to process more bins than will fit in memory.
  • Fixed an issue where rwuniq would not correctly process multiple input files when the --presorted-input switch was given.
• rwtotal
  • Added the ability to print only bins that meet minimum and/or maximum thresholds for bytes, packets, and/or record counts.
• rwaddrcount
  • Provided new --min-* and --max-* switches as aliases to the existing --rec-min, --rec-max, etc switches.
• rwsort
  • Modified so that the maximum buffer size is approached gradually, making its memory usage more closely reflect only what it needs.
  • Added the --print-filenames switch for consistency with other tools.
• rwdedupe
  • Modified so that the maximum buffer size is approached gradually, making its memory usage more closely reflect only what it needs.
• rwfilter
  • Fixed a bug that caused the --sensors switch not to accept a numeric range of sensors.
• rwsender
  • Improved performance by reducing the amount of memory that must be copied when reading files.
  • Improved the logging. The log now includes the time it took to send a file.
• rwreceiver
  • Improved the logging.
• rwflowappend
  • Fixed issue where rwflowappend would exit if the final record in an incremental file was invalid.
• rwflowpack
  • Enhanced the sensor.conf syntax and packing logic so that all flow records collected from a particular probe can be labeled as traveling between two networks. This allows all flows seen by that probe to be labeled as incoming, for example.
  • In sensor.conf, renamed the netflow probe to netflow-v5, but allow netflow as an alias for netflow-v5 for compatibility.
• flowcap
  • In sensor.conf, renamed the netflow probe to netflow-v5, but allow netflow as an alias for netflow-v5 for compatibility.

Changelog

• rwcut: Extended the PySiLK capability to support user-defined columns in rwcut
• rwmatch: Enhancements to allow both sides of the conversation to be included in the output.
• cutmatch.so: A new plug-in to print the values that rwmatch writes into the next-hop IP field.
• rwbagtool: Allow "--output" to be an abbreviation for "--output-path".
• rwsender: Allow the block size used when sending files to rwreceiver to be specified on the command line.
• rwuniq: Fix bug that prevented the upper bound of the --bytes, --packets, --flows, etc switches from being parsed.
• rwptoflow: Fix bug that would result in the bytes value being incorrect (the value was not being byte-swapped)
• Fix a fatal bug in the start-up of daemons that occurred when logging was set to "syslog" or "none".
• Additional minor bug fixes

Changelog

• rwfilter can support filtering using expressions written in Python, and it is possible to manipulate SiLK Flow records from within Python. This feature requires Python 2.4 or later, and you must specify --with-python when you run configure. See the "PySiLK: SiLK in Python" language reference documentation, and the --python-expr and --python-file switches on rwfilter.
• Preliminary support for IPv6 addresses can be included. Use the --enable-ipv6 switch on the configure script to include IPv6 support in SiLK. When IPv6 is present, rwfilter provides a --ip-version switch to filter on IPv4 and/or IPv6 addresses, and the tools rwuniq and rwcut provide a --ipv6-policy switch (and SILK_IPV6_POLICY environment variable) that controls the display of IPv6 addresses.
• rwfilter now supports threads. Performance is greatly improved for queries that look at many files but return few records. Use the --threads switch on rwfilter or the SILK_RWFILTER_THREADS environment variable to control the number of threads. By default, rwfilter will use a single thread. Our testing has found that performance peaks around four threads per CPU, but performance will vary depending on the type of query and the number of records returned.
• There are new binary SiLK file formats, and the format of every SiLK file has changed. SiLK-1.0.0 can read files created by earlier versions of SiLK; however, releases prior to SiLK-1.0.0 will not be able to read SiLK-1.0.0 files. Binary SiLK files now contain additional information in their headers, including the version of SiLK that produced the file.
• Delimited textual output has changed in almost all tools. Note this is a POTENTIAL INCOMPATIBILITY and may break scripts. A new --no-final-delimiter switch prevents printing of the final delimiter in the textual output of rwaddrcount, rwbagcat, rwcount, rwcut, rwpmapcat, rwsetcat, rwstats, rwtotal, rwuniq. In addition, the --delimited switch now enables --no-final-delimiter, making it easier for the output to be parsed by other tools. If you need to maintain compatibility with earlier versions of SiLK, replace --delimited=X with --no-columns --column-sep=X.
• Arbitrary notes (annotations) can be added to the headers of some SiLK files. Use the --note-add=TEXT to add a note, or --note-file-add=FILE to add text from a file. The rwfileinfo tool will view the notes. Notes are supported by rwbag, rwbagbuild, rwbagtool, rwcat, rwfilter, rwset, rwsetbuild, rwsettool.
• Site information is completely determined at run-time. The rules that rwflowpack uses to categorize flows are now controlled by a run-time plug-in that rwflowpack loads. The name of the plug-in must be passed to rwflowpack via the --packing-logic switch, or set in the silk.conf file.
• The sensor.conf file used by rwflowpack and flowcap has a completely different syntax. See the Installation Handbook and the rwflowpack(8) and sensor.conf(5) manual pages. The update-sensor-conf script converts the old syntax to the new.
• A new rwidsquery tool is provided. rwidsquery takes a Snort alert log or rule file and invokes rwfilter with the appropriate arguments to find the SiLK flow records that match the input file.
• Bugs have been fixed in processing times on Solaris when the machine's timezone was not UTC.
• Configuring SiLK to use legacy timestamps by default is no longer supported. The --legacy-timestamps switch is still supported on the applications.
• When looking for support files (such as country_codes.pmap), tools will look in $SILK_PATH/share/silk/ and $SILK_PATH/share/, but they no longer look in $SILK_PATH/.
• buildset, readset, setintersect, rwset-union:
  • These symbolic links to rwsetbuild, rwsetcat, rwsetintersect, and rwsetunion are no longer created.
• rwaddrcount:
  • See discussion of --no-final-delimiter above
• rwbag:
  • See discussion of --note-add above
• rwbagbuild:
  • The --output switch has been renamed to --output-path.
  • See discussion of --note-add above
• rwbagcat:
  • The --output switch has been renamed to --output-path.
  • See discussion of --no-final-delimiter above
• rwbagtool:
  • See discussion of --note-add above
  • The --output-file switch is deprecated. Use --output-path instead.
• rwcat:
  • See discussion of --note-add above
• rwcount:
  • Enhancement to support millisecond-sized bins. Specify a fractional value to the --bin-size switch: --bin-size=0.500
  • As a side effect of this millisecond capability, the output from the default load scheme (--load-scheme=4, splitting a flow by its active time) will now divide flows across each millisecond that the flow is active. This results in slightly different output.
  • New --end-epoch switch allows user to control the final bin to print.
  • The --delimiter switch has been removed. Use the --column-separator switch instead. (Note that the --delimited switch still exists).
  • See also discussion of --no-final-delimiter above
• rwcut:
  • A new --all-fields switch causes all possible fields to be printed.
  • New --ipv6-policy switch controls how IPv6 flows are handled
  • See also discussion of --no-final-delimiter above
• rwdedupe:
  • The --identical-fields switch has been renamed to --ignore-fields, and --sort-buffer-size has been renamed to --buffer-size.
• rwfileinfo:
  • Output has changed to reflect new SiLK binary file headers.
• rwfilter:
  • New --python-expr and --python-file switches
  • New --threads switch
  • See discussion of --note-add above
  • New tuple.so plug-in filters flow records based on any subset of the five-tuple {source-ip, destination-ip, source-port, destination-port, protocol}. The --ipport-any and --ippair-any switches are deprecated.
  • The --ippair-any and --ipport-any switches no longer work for files that use only TAB characters between the two columns of input. Change the TAB characters to spaces.
  • New --ip-version switch when IPv6 support is enabled.
  • Fix an issue where an error writing to the file system was not being correctly reported.
  • Fix a bug that caused the --site-config-file switch to be ignored
• rwmatch:
  • New --unmatched switch allows unmatched records to be written to the output.
  • New --symmetric-delta switch allows either input file to contain the initiating flow
• rwpmapbuild:
  • See discussion of --note-add above
  • rwpmapbuild has been rewritten as a C application.
• rwpmapcat:
  • See discussion of --no-final-delimiter above
• rwnetmask changes:
  • Enhancement so that it takes file names from the command line and produces a file as output.
  • Renamed switches to be more consistent with other tools but leave the old names for compatibility.
• rwscan:
  • Existing output files are no longer overwritten.
  • Printing of each filename processed, thread creation, etc. is now only done when the user specifies --verbose-progress on the command line.
  • New --verbose-results prints information about each IP.
  • New switches allow setting the parameters used by the TWR algorithm
  • New --integer-ips switch to print IPs as integers
  • In the printed output, headers and output records now end with a delimiter by default. This can be turned off with --no-final-delimiter.
  • The --scandb switch enables --no-final-delimiter.
  • The --output-file switch has been renamed to --output-path.
  • Improved manual page.
• rwset: POTENTIAL INCOMPATIBILITY.
  • Running rwset with no arguments will no longer produce an IPset. The IPset(s) to create MUST now be specified with the --sip, --dip, and/or --nhip switches.
  • See discussion of --note-add above
• rwsetbuild:
  • See discussion of --note-add above
• rwsetcat:
  • See discussion of --no-final-delimiter above
• rwsettool:
  • See discussion of --note-add above
• rwstats:
  • See discussion of --no-final-delimiter above
• rwtotal:
  • See discussion of --no-final-delimiter above
• rwuniq: POTENTIAL INCOMPATIBILITY.
  • The --threshold switch is no longer supported. Use the --flows switch instead.
  • The output from rwuniq may appear in a different order due from previous releases due to changes in the internal hash table.
  • The --sip-distinct and --dip-distinct switches are handled more efficiently for sparse IPs.
  • New --ipv6-policy switch controls how IPv6 flows are handled
  • See discussion of --no-final-delimiter above
• Summary of changes that may break old scripts or usage patterns:
  • See the discussion of --no-final-delimiter above
  • rwbagbuild: The --output switch has been renamed to --output-path. Since --output is a legal abbreviation of --output-path, no end-user effects should be seen.
  • rwbagtool: The --output switch has been renamed to --output-path. Since --output is a legal abbreviation of --output-path, no end-user effects should be seen.
  • rwcount: The --delimiter switch has been removed. Use the --column-separator switch instead. (Note that the --delimited switch still exists).
  • rwdedupe: The --identical-fields switch has been renamed to --ignore-fields, and --sort-buffer-size has been renamed to --buffer-size.
  • rwtotal: The --delimiter switch has been removed. Use the --column-separator switch instead. (Note that the --delimited switch still exists).
  • rwuniq: The --threshold switch is no longer supported. Use the --flows switch instead.
• For programmers:
  • The IP address is now an abstract object.
  • All access to the fields of an rwRec should occur through the rwRec* wrappers.
  • Time is now represented as an sktime_t (a signed 64bit integer), representing milliseconds since the UNIX epoch.
  • There have been many changes to the library functions.
• The following incompatible changes exist in the packing tools:
  • The sensor.conf syntax is completely different.
  • rwflowpack: When processing PDU-files as input, you need to use --input-mode=pdufile instead of --input-mode=file.
  • rwflowpack: The --fc-address and --fc-port switches have been removed; use --flowcap-address and --flowcap-port instead.
  • flowcap: The --sensors switch has been removed. The --probes switch offers similar functionality, but takes the names of probes, not sensors.

Changelog

• rwdedupe: New tool
  • Tool that removes duplicate SiLK Flow records from a file.
• rwsort: Enhancement
  • New --presorted-input switch allows rwsort to process previously sorted files (rwsort will merge-sort the files).
• rwsetbuild: Enhancement
  • Now supports input having an IP range on each line when the --ip-ranges switch is specified.
• rwsettool: Enhancement
  • Added a new --mask operation so a user can see which IP blocks contain an IP address.
• rwfilter: Enhancements
  • Provide new libippair.so plug-in that allows partitioning of SiLK Flow records based on the source and destination IPs as a pair.
  • Provide a mechanism to log statistics about the commands that were run and the number of files and records involved.
• flowcap, rwflowpack: Bug fix
  • Fix occassional crashes when collecting flows from IPFIX sensors. To collect flows from an IPFIX sensor, libfixbuf 0.7.2 or greater is now required.
• rwstats: Bug fix
  • Fix a bug in the output generated by the --overall-stats switch where the maximum would not be displayed correctly when the input consisted of a single flow.
• rwsender, rwreceiver: Bug fix
  • Fix a bug that was causing frequent retries and disconnects between rwsender and rwreceiver.
• rwaddrcount, rwcount, rwcut, rwtotal: Bug fix
  • Fix a bug where --output-path=/dev/null would send the textual output to stdout.
• rwtuc: Change in behavior
  • Do not create the "bad-input-lines" file when all of the input is successfully processed.

Changelog

• rwsender, rwreceiver: Enhancement
  • rwsender and rwrecevier can encrypt their communication if the GnuTLS library was found when SiLK was configured.
• rwsender: Bug fixes
  • Ensure that files are closed after reading. This fixes a bug where rwsender would eventually run out of file descriptors.
  • Fix a bug that causes rwsender to crash when it loses the connection to an rwreceiver during the transfer of a file.
• rwflowpack: Bug fix
  • Fix a bug in reading flowcap files on 64bit platforms that caused the records in the file to be ignored.
• rwscanquery: Change in behavior
  • The location of the output file must now be specified with the --output-path switch.
• rwcut and rwuniq: Bug fix
  • Fix several issues in rwcut and rwuniq when dealing with prefix map (pmap) files that had dictionary items longer than 63 characters. A new --pmap-column-width switch is available to limit the number of characters that are printed.
• rwfilter: Bug fix
  • Fix a bug where the --icmp-type and --icmp-code were not filtering out non-ICMP traffic.
• rwscan: Bug fix
  • Close the output after all worker threads have joined. This fixes the problems of missing output and double free() errors.
• rwcut: Bug fixes
  • The --copy-input switch wasn't copying its input.
  • When displaying the end-time and the milliseconds value was larger than 1000, rwcut was not properly incrementing the seconds value.
• rwnetmask: Enhancement
  • Always write the SiLK headers to the output file so that files with no data are still valid SiLK files.
• rwrandomizeip: Enhancement
  • Always write the SiLK headers to the output file so that files with no data are still valid SiLK files.
• rwswapbytes: Enhancement
  • Always write the SiLK headers to the output file so that files with no data are still valid SiLK files.
• rwset: Documentation fix
  • Fix misplaced text in the rwset man page.

Changelog

• Bug fixes: rwfilter
  • Fix a bug that occurred during parsing of the --sensors switch when only numeric sensors where specified.
  • Fix a double close() of the --print-statistics stream.
• Bug fix: rwbagcat: Recognize when the user explicitly sets 'minkey' to 0.
• Enhancement: rwsetcat: New switch --ip-ranges presents the IPset as a list of IP-ranges.
• Enhancement: rwsort: New switch --sort-buffer-size sets the amount of RAM rwsort initially tries to allocate for the buffer used to hold the SiLK Flow records prior to sorting.
• Enhancements: rwfglob
  • New switch --no-file-names suppresses printing of file names.
  • New switch --no-summary suppresses printing of number of files found.
• Enhancements: rwscanquery
  • Make the queries more efficient.
  • Make the --start-date switch more closely match the behavior of rwfilter.
• Bug fix: Add the 'pmap-example.txt' file that was missing from the SiLK-0.11.1 release.
• Bug fix: rwgeoip2ccmap: Append the string '-input' to the names of the options to match the manual page.
• Build fix: src/libskipfix src/rwipa: Make certain the CFLAGS found/set during configuration are passed to CC when building.

Changelog

This release has many changes from the previous SiLK-0.10.5 Release.

End user features, enhancements, and bug fixes:

• New scan detection system: rwscan and rwscanquery
  • rwscan reads SiLK Flow data and uses a hybrid of Threshold Random Walk and Bayesian Logistic Regression to detect scanning activity. rwscan output textual records describing the scan. If these are inserted into a relational database, rwscanquery can be used to query for the scanning activity. rwscanquery can query Oracle, Postgres, or MySQL databases.
• New tools for IPFIX support
  • rwsilk2ipfix converts SiLK Flow records to an IPFIX format.
  • rwipfix2silk converts IPFIX flow records to the SiLK format.
  • These tools can be used in place of the rwp2yaf2silk script.
  • Support for these tools requires that libfixbuf-0.6.0 be installed prior to building SiLK.
• New tools for IP storage
  • rwipaexport takes IP addresses from an IP Address Association (IPA) catalog and creates a SiLK IPset, Bag, or Prefix Map (pmap).
  • rwipaimport enters the IP addresses from a SiLK IPset, Bag, or Prefix Map into an IPA catalog.
  • Support for these tools requires that libipa-0.2.0 be installed prior to building SiLK.
• Additional new tools
  • rwsplit divides a SiLK Flow file into smaller files based on the number of flows, bytes, packets, or unique IPs. It also provides the ability to sample the input.
  • rwsettool provides the functionality of rwsetintersect and rwsetunion and additional functions such as set difference and sampling of an IPset. The rwsetintersect and rwsetunion tools are deprecated.
  • rwsetmember determines if a (textual) IP is a member of an IPset. Determinating this in previous releases of SiLK required filtering the output of rwsetcat or creating an IPset containing a single IP.
  • rwpmapcat prints the contents of a Prefix Map (pmap) file.
• rwfilter enhancements and bug fixes
  • Allow the parameter to the --flags-all, --flags-init, and --flags-session switches to be a list of HIGH/MASK pairs separated by commas, e.g., --flags-all=S/S,A/A
  • Do not print statistics or create output files when the --dry-run switch is specified.
  • Fix a file corruption issue that would occur when processing multiple files if the first input file was not successfully opened: the output file would be generated without a SiLK header.
  • Exit with a non-zero exit status if the class, type, or sensor values are invalid.
  • Fix a bug in processing the --start-date and --end-date switches when local timezone support was enabled and the local timezone was east of UTC.
• rwbag enhancements and bug fixes
  • rwbag now supports creating Bags whose key is the sensor ID, next hop IP, input interface or output interface.
  • Allow rwbag to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
  • Print errors as human readable text, not error codes
  • Fix a bug with releasing memory multiple times when rwbag ran out of memory.
• rwrandomizeip enhancement
  • Allow the user to restrict the set of IPs that are modified via two command line arguments: --dont-change-set and --only-change-set. Both switches take an IPset; the first switch prevents the IP from being changed; the second causes only the listed IPs to be changed.
• mapsid enhancement
  • The --print-classes switch will print the class(es) to which each sensor belongs.
• rwcount enhancement and changes
  • Implemented the --output-path switch which directs rwcount to write its output to the specified location.
  • Allow rwcount to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
  • The column widths have changed slightly
• rwaddrcount enhancement
  • Implemented the --output-path and --copy-input switches as described for rwcount.
• rwcut enhancement
  • Implemented the --output-path and --copy-input switches as described for rwcount.
• rwstats enhancement
  • Implemented the --output-path and --copy-input switches as described for rwcount.
• rwset enhancement
  • Implmented the --copy-input switch as described for rwcount.
• rwtotal enhancement
  • Implemented the --output-path switch as described for rwcount.
• rwuniq enhancement
  • Implemented the --output-path switch as described for rwcount.
• rwsetcat bug fix
  • Fix bug where the $PAGER was not being used.
• rwbagcat bug fixes
  • Do not print a warning message when attempting to print an empty Bag or when the min/max limits caused no entries to be printed.
  • Fix bug where the $PAGER was not being used.
  • Print errors as human readable text, not error codes
• rwbagtool bug fix
  • Print errors as human readable text, not error codes
• rwcat bug fix
  • Modify rwcat so it will always print the SiLK header to a file, even when no records are present
• rwappend enhancement and bug fix
  • New --print-statistics switch causes the number of records processed to be printed to the standard error.
  • Output change: Modified rwappend so it only prints the number of records processed when --print-statistics is given.
  • Fix a problem that occurred when SiLK was compiled with compression enabled by default and the applications were processing SiLK files produced by releases of SiLK prior to 0.10.5: the application would exit with the error message "Operation not permitted on compressed file" and no output would be generated.
• rwswapbytes bug fix
  • See compression-related bug fix for rwappend
• rwnetmask bug fix
  • See compression-related bug fix for rwappend

Administration and configuration changes:

• New "silk.conf" file removes the requirement that sensors be defined at compile-time.
  • The sensors, classes, and types are now defined at run-time through the use of a "silk.conf" text file. This file should be installed in the SILK_DATA_ROOTDIR directory.
  • The run-time configuration allows a single installation of the analysis tools to query multiple data sets; simply set the SILK_DATA_ROOTDIR environment variable to the location of the data.
  • The location of this file can also be specified by setting the SILK_CONFIG_FILE environment variable to its location, or by using the --site-config-file switch on most SiLK applications.
  • The packer (rwflowpack) still requires certain classes and types to be defined, and it cannot use new classes and types without modifying C code. This restriction will go away in a future release.
• Major changes to the build system.
  • The build system now uses all aspects of the GNU Autotools chain including 'automake' and 'libtool'.
  • The tools can now be built with shared library support, reducing the size of the binaries and allowing the kernel to use a single copy of libsilk when multiple SiLK tools are running.
  • Note that the use of shared libraries means the binaries can no longer easily be relocated; instead you should run "make install" again with the new location.
  • The SiLK headers are now copied to the install target directory
  • GNU make is no longer required to build the tools.
• New packing rules are used by default.
  • The default site has changed from "generic" to "twoway". The twoway site allows flow records to be categorized and stored as internal-to-internal (int2int) and external-to-external (ext2ext). In addition, the "out" type is no longer everything that is not "in". The files created by the generic site are forward compatible with the twoway site; however, if you wish to continue using your current packing rules, run configure with the --enable-silk-site=generic switch. See the SiLK Installation Handbook for details.
• New transfer daemons: rwsender and rwreceiver
  • These are meant to replace the direct connectivity between flowcap and rwflowpack. These daemons allow the flowcap files to be sent to multiple rwflowpack processes.
  • In addition, they allow rwflowpack to process data on one system and send small files containing SiLK Flow records (called "incremental files") to another system (where the rwflowappend daemon is running) for analysis.
• New packing tool: rwflowappend
  • rwflowappend appends SiLK Flow records contained in "incremental files" to hourly files.
• Changes to flowcap and rwflowpack
  • The flowcap and rwflowpack tools have been modified to work with the new rwsender and rwreceiver, though they can also be used in legacy mode. With the transport removed from flowcap, flowcap files can now be sent to multiple locations.
• IPFIX flow collection enhancement
  • Previous releases of SiLK (rwflowpack and flowcap) could only read IPFIX streams generated by YAF. With this release, SiLK can read flows from any IPFIX-compliant generator.
• Remove zlib requirement in rwflowpack
  • Allow rwflowpack to be built even if zlib is not available. However, rwflowpack will not be able to read files of NetFlow PDUs when zlib is not present.
• New packing tool: rwpackchecker
  • rwpackchecker performs a basic integrity check of a packed SiLK file.

Changelog

• Data file version number bump
  • Fix a forward compatibility issue in SiLK between releases prior to 0.10.0 and releases 0.10.0 through 0.10.4 when data compression is enabled (either via the --enable-output-compression switch to 'configure' or the --compression-method switch to various applications). Versions of SiLK prior to 0.10.0 did not check the value of the 'compression' byte in the header; when reading a SiLK file from 0.10.0 with compression enabled, these versions will silently attempt to read the data section without uncompressing it, leading to incorrect output.
    
    The issue is resolved in SiLK 0.10.5 by incrementing the version number of every SiLK file format that supports compression of the data section of the file (IPsets, Bags, and the output from rwfilter, rwcat, rwsort, rwflowpack, and rwptoflow).
    
    We recommend using the "silk-version-bump-0-10-5" script included with the distribution to increment the version number of files created with releases of SiLK prior to 0.10.5 that have compression enabled. The script will only modify SiLK files that have compression enabled; it will not modify non-SiLK files nor SiLK files that do no have compression enabled.
• rwcount change
  • IMPORTANT. The default binning mode (load-scheme) has changed. The former scheme put each flow's entire volume into the first second of the flow. The new scheme evenly divides the volume across each second of the flow's duration, which should help reduce "spikiness" in the data. Any scripts that rely on the former method should have "--load-scheme=1" added explicitly to rwcount's invocation.
• rwuniq enhancement and bug fix
  • New flag "--presorted-input" makes rwuniq assume that the data has been sorted with rwsort using the same set of "--fields". This reduces rwuniq's memory requirement and allows it to work like it's UNIX counterpart 'uniq'.
  • Fix a memory fault that could occur when using the --sip-distinct and/or --dip-distinct switches on large data sets.
• rwfilter changes
  • rwfilter will continue to process even if there is a problem with an input file.
  • rwfilter will now process multiple RWFILTER input files, though it prints a warning that file history is being lost.
  • rwfilter supports time filtering (via the --stime and --etime switches) to the millisecond
• New script rwp2yaf2silk:
  • rwp2yaf2silk converts a file of pcap data to SiLK Flow data; the script requires that the SiLK tool 'rwtuc' is installed and that the tools 'yaf' and 'yafscii' (http://tools.netsa.cert.org/yaf/) are installed.
• rwbagcat bug fix
  • Make certain the --bin-ips=linear switch properly handles Bag entries where the count is greater than 4294967295. These entries are now attributed to the maximum key unless the --maxcount value is used to filter out those entries.
  • When printing the output from --bin-ips=decimal, properly print the key when its value is greater than 4294967295
  • Set the output column width to 20 to maintain the columnar output when the value is very large.
  • Support values larger than 4294967295 in the --mincount and --maxcount switches
• rwbagtool bug fix
  • Fix a bug in the --invert switch which resulted in incorrect results in the output. This would occur when the value was larger than the current key.
  • Make certain the --invert switch properly handles Bag entries where the count is greater than 4294967295. These entries are now attributed to the maximum key unless the --maxcount value is used to filter out those entries.
  • Allow the --invert switch to support multiple Bag files by adding the Bags (making the switch consistent with the --coverset, --intersect, and --compliment-intersect switches). This fixes an assertion that would cause the program to abort.
  • Support values larger than 4294967295 in the --mincount and --maxcount switches
• rwflowpack input check
  • When processing NetFlow data from a file, rwflowpack now checks that the input data is in NetFlow v5 format. Previously, the version check was not made and the file would be processed as if it contained NetFlow v5 data.
• rwpmatch enhancement and bug fix
  • Provide --ports-compare and --msec-compare switches to have rwpmatch compare port data and compare times down to the millisecond.
  • Fix a bug that caused rwpmatch to assume every packet would have a corresponding flow.
  • Be more diligent about testing the length and type of packets we read.
• rwtuc change
  • Always print the SiLK header to the output, even when records were read from the input.
• flowcap fix
  • Fix a bug in flowcap that caused it to process data from only the final sensor listed in the sensor-configuration file.
  • Fix bugs in the flowcap control script.
• File relocation
  • The man page sensorconf.5 has been renamed sensor.conf.5.
  • The source POD for man pages has moved from src/APP/doc/APP.pod to src/APP/APP.pod

Changelog

• Fix a major bug in rwbagbuild that caused rwbagbuild to ignore every other line of its input.
• Fix a bug in the prefixmap (pmap) support that caused rwsort to crash when attempting to sort using fields defined in a pmap.
• Fix syntax errors in the rwfpd script that runs rwflowpack. These errors were invoked when the compression was not set or when the name of the script included a sensor-name suffix.
• Add a --no-file-locking switch to rwflowpack. With this switch, rwflowpack will not attempt to get a write lock when writing flows to data files. This switch is required for rwflowpack use filesystems that do not support file locking. During normal operation multiple rwflowpacks should never attempt to write to the same file; the use of advisory locks is not strictly necessary, but it provides protection during unusual circumstances.
• Modify rwflowpack so that when it encounters a disk error (unable to open file, obtain a lock, write the flow, etc) when trying to write a flow, it stops processing flows for that probe. If all probes encounter disk errors, rwflowpack will exit.
• Fix a communication issue between flowcap and rwflowpack: on slow and noisy networks, the ACK which rwflowpack sends to flowcap indicating that it has received a file could be lost. Since flowcap never received the ACK, it would resend the same file to rwflowpack thinking the first attempt had failed. rwflowpack would store both files, resulting in duplicate flows in the packed data. rwflowpack now stores the name of the most recent file it received. If it receives a file with the same name, the second file is ignored.
• Fix a bug related to the sensor.conf file; the growth factor for an array was too small which caused rwflowpack to abort.
• Fix a bug in parsing time ranges when fractional seconds were present.
• Ensure that compressing flows with the LZO compressor always produces the same binary output by clearing the temporary buffer that is passed into LZO.

Changelog

• There is a new Analysts' Handbook: Using SiLK for Network Traffic Analysis. This document provides a tutorial on learning the SiLK tools and describes doing analysis with the tools. The manual pages that used to be in that document have been moved into a separate document: The SiLK Reference Guide.
• The SiLK packing tools now support reading IPFIX records generated by the YAF Flow Sensor (http://tools.netsa.cert.org/yaf/). YAF must be installed prior to configuring SiLK.
• When used with YAF, SiLK supports additional fields for dealing with TCP data: The flags on the first packet on the flow are stored separately from the flags on the other packets in the flow. In addition, when a TCP session is broken into multiple flows, the flows are specially marked.
• SiLK now supports using an external compression library to further compress the "data" section of files, while leaving the "header" of the file uncompressed. This compression is available on SiLK Flow files, as well as IPsets and Bags. The supported compression methods are "none", "zlib", and "lzo1x", subject to library availability. Most tools allow one to specify the compression. The default compression is set when the 'configure' script is run (--enable-output-compression).
• The logging library has been rewritten, and now supports syslog(3). Logging messages can also be written to the standard error. "Legacy" logging is still supported (SiLK can still write its log files in a directory and rotate the files), but note that the format of log messages has changed. Also, rwflowpack will no longer automatically include the value passed to --sensor-name switch as part of the log file name and PID file name. (The rwfpd init script works around this; see the SiLK Installation Handbook.)
• For people upgrading from previous releases, note that the list of sensors has been moved from silk_site_generic.h to generic_sensors.h. Also note that the macros around the sensor list have changed; please edit carefully. See the SiLK Installation Handbook.
• A new library, libsksetbag, contains the functions to manipulate IPsets and Bags. libiptree has been removed; use libsksetbag instead.
• Additional manual pages have been added.
• Additional changes:
  • rwptoflow: does a better job of checking the validity of its input; has plug-in support; new switches allow it to produce "pass" and "fail" streams of pcap data and/or print statistics
  • rwsort: when it receives no input, it now produces a SiLK Flow file with no readers (only a header). Previously it would produce a completely empty file
  • rwfileinfo: output changed to include new compression method
  • flowcap: added a switch to manually set the ack timeout, which is useful on slow networks.

Changelog

• Critical bug fix
  • Fix a byte-swapping bug in FT_RWWWW V3 records. When converting an rwRec from or to this format and where the conversion included a byte-swap, the record would be corrupted. As long as all SiLK data was handled in the machine's native byte order, the bug would not manifest itself (the initial read of the NetFlow data was/is handled correctly, so data on little endian (not network byte order) machines is correct so long as it has always remained on little endian machines).
    
    The bug corrupted data, resulting in any of these behaviors: the source and destination ports could be swapped, the service (web-side) port could be incorrect, the TCP flags could be incorrect, the packet and byte counts could be high (64 times higher than they should be), and the millisecond times could be wrong.
• Potential Incompatibilities
  • When using SiLK flow records in contexts that do not use the millisecond field, truncate the millisecond value instead of rounding.
  • rwbagcat, rwbagtool, rwcat: When file names are listed on the command line, do not attempt to read data from the standard input unless the user explicitly uses "stdin" as the name of an input file. This change is required to allow the tools to work with cron(1).
  • rwflowpack (sensor.conf): Allow a comma to occur between the IP addresses in an ipblock list. This means that a comma cannot occur within the wildcard IP address, but it is believed few people were using this functionality.
  • rwflowpack: minor log message changes; changed the log rotation hour to 00:00; modified the umask() of log files
• New feature: Address Type Plug-in (libaddrtype.so)
  • Support for partioning by or displaying the address type requires libaddrtype.so to exist in the $SILK_PATH/lib directory and the "address_types.pmap" file to exist in the $SILK_PATH/share/silk or $SILK_PATH/share directory.
  • To create this binary "address_types.pmap" file, first list CIDR blocks in a text file (my-ips.txt) and label each as "non-routable", "internal" or "external" (any address that is not listed in the file is considered "external"), and then run the commands:
    
    rwpmapbuild -i my-ips.txt -o address_types.pmap
    
    For the best results with the pmap code, the CIDR blocks should be as large as possible. One one to convert a list of IPs (ips.txt) into a list of large CIDR blocks (cidr.txt) is to run:
    
    rwsetbuild ips.txt stdout | rwsetcat --cidr > cidr.txt
  • For more information, see the rwpmapbuild man page and the man pages of rwfilter, rwcut, rwsort, and rwuniq.
• New feature: Prefix Map Plug-in (libpmapfilter.so)
  • Experimental creation and use of the user's own prefix maps (pmaps) for partitioning (rwfilter), sorting (rwsort), counting (rwuniq), and display (rwcut, rwuniq) is provided. The interface is still considered experimental and is subject to change.
  • The rwpmapbuild tool reads a text file and builds a pmap file that can be used by the tools. This file can relate IPs or Port/Protocol pairs to some attribute (this is how the country code and addrtype pmaps work).
  • For details, see the rwpmapbuild and libpmapfilter man pages.
• New feature: Record Partitioning via IP-Port Pairs (libipport.so)
  • The --ipport-any switch to rwfilter (provided by the libipport.so plug-in) will pass a record if its source IP and port or its destination IP and port are listed in the named text file.
  • To use this plug-in, one creates a text file where each line contains a single IP address (either in dotted-decimal notation or as an integer), whitespace, and a list of ports of interest for that IP. The port list can be a single number (80), a range of numbers ("6000-6100"), or comma-separated list of numbers and ranges ("6000-6100,80"). The file may also contain blank lines and comments; comments begin with the "#" character and continue to the end of the line.
  • Support in rwfilter for partitioning records by IP-port pairs requires libipport.so to exist in the $SILK_PATH/lib directory.
• Improved sorting
  • rwsort now supports getting fields from run-time plug-ins, like rwcut and rwuniq.
  • When merging multiple temp-files, rwsort now attempts to open them all and merge them in one step, considerably reducing the I/O overhead of the merge sort.
• Better support for ICMP data
  • rwfilter: new switches allow for filtering by the ICMP type and code (--icmp-type, --icmp-code)
  • rwcut, rwsort, rwuniq: A new "icmpTypeCode" value to the --fields switch is allowed. When this value is present, the ICMP type and code will be used as part of the key when sorting (rwsort) and counting (rwuniq), and it will be displayed (by rwcut and rwuniq) in separate columns labeled 'iType' and 'iCode' (which in columnar output will shorted to 'iTy' and 'iCo'). The --icmp-type-and-code switch on rwcut is still maintained for backwards compatibility, but its use is deprecated.
  • rwstats: Supports using the ICMP type and code as a key with the --icmp switch.
• Configuration and Build System Changes
  • In preparation of using the GNU AutoTools, we've made major changes to build and configure system that bring us more in-line with the AutoTools. Note that the 'release', 'debug', and 'profile' targets have gone away. Use the --enable-debugging and --disable-optimization switches to configure for a fully debuggable binary. See configure --help to see the full list of new options.
• Miscellaneous Improvements
  • rwcount: Add a new value to the --load-scheme switch that will weigh the values assigned to each bin by the number of seconds the flow spent in the bin.
  • rwfilter: new switch to filter on a negative next-hop IP (--not-next-hop-id)
  • rwfilter: Filtering by IPsets is now supported directly in the application itself. Previously, this was handled by a plug-in.
  • flowcap: There is a new version of the flowcap file format, 5. Version 5 is identical to version 3, save for the fact that the input and output interface fields have been expanded to 16 bits.
  • rwcut, rwsort, rwuniq: Provide numerical identifiers for fields (--fields switch) that hadn't had any previously.
• Bug fixes
  • rwgroup: Fix several bugs, the majority of which have to do with the interaction between summarization and other actions.
  • rwflowpack: Use fseeko() to fix an issue when writing large files on Solaris
  • rwfilter: Fix a crash that would occur when using a combination of the switches --dynamic-library --pass for certain dynamic libraries
  • rwmatch: Several bug fixes.
  • rwstats: Fix a bug that would cause rwstats to crash when attempting to compute the top-N when no records were read as input.
  • rwtuc: Fix a bug that occurred when the user provided the --fields switch and a title line was present
  • rwuniq: Fix a display bug by using the width of the value (versus the title) for setting width of columns that we get from plug-ins.
  • rwuniq: Zero out the record prior to output to avoid getting random data values in the millisecond fields. These random values were affecting the values in the time fields.
  • libflowsource: Fix a bug that prevented it from building when used with certain parser generators.

Changelog

• New packing support: flowcap
  • The flowcap daemon allows the collection of flow data and the packing and storage of this flow data to occur on separate machines.
  • To use flowcap, the LZO real-time data compression library must be installed. If configure does not find the LZO library, flowcap will not be built.
  • Compilation and use of flowcap is optional.
• Improvements and significant changes to rwflowpack:
  • Splitting by IP address: Instead of using your router's SNMP interfaces to split traffic into inbound and outbound, rwflowpack can now split data by CIDR block.
  • rwflowpack now requires configuration via a separate sensor.conf file.
  • Many of rwflowpack's arguments have changed.
  • rwflowpack's control script, rwfpd, has been split into two parts.
• New local timezone support: Pass the --enable-localtime switch to the configure script to use the local timezone in time input and output. Without this switch, the tools will use UTC. (Data files continue to be stored in UTC.)
• Format of printed timestamps has changed, the new format is 2006/05/08T15:36:53.123. To enable the previous format by default, pass the --enable-legacy-timestamp switch to configure. The printed timestamp format can be set per invocation via the --legacy-timestamps switch.
• The tools that handle IPset files have been renamed. The old names are still supported for this release.
  • rwsetbuild replaces buildset
  • rwsetcat replaces readset
  • rwsetintersect replaces setintersect
  • rwsetunion replaces rwset-union
• New tool rwtuc: the text utility converter does the reverse of rwcut---it reads textual input and generates binary SiLK flow data from it.
• Manual pages are now included. Additional improvements to the documentation.
• Improvements to rwuniq:
  • Supports computing counts of unique source or destination IPs for small input sets; the memory requirements to support these counts can grow quickly.
  • Can be used with run-time plug-ins.
• Improvements to rwbagtool: Less memory is used during merging of multiple Bag files, and some recursive routines have been rewritten to reduce memory and increase speed.
• Changes to rwsetcat and rwbagcat: The output of the --network-structure switch has changed.
• For tools that produce textual output, columnar output and column separator can be controlled separately. These tools all support the --delimited switch; the former --delimiter switch which some tools supported is deprecated.
• Improvements to rwappend: Now supports "appending" to a nonexistent file. Restrictions on the types of files that rwappend supported have been removed.
• Configuration for multiple sites is easier, though the choice of which site to build for must still be made when you run the configure script.
• Significant rearrangement of the source code tree.

Changelog

• Fix bug where the pthreads library was not being linked into rwflowpack
• Note: Options to configure script have changed. configure now does a better job (hopefully) of testing for libraries
• Most tools will now invoke a pager to page the output. Use the SILK_PAGER environment variable to override PAGER, or the --pager switch to override SILK_PAGER. Setting SILK_PAGER to the empty string will disable paging.
• Duplicate packet detection removed from rwptoflow; use rwpdedupe to remove duplicate packets.
• Bug fixes in rwptoflow.
• Bug fixes in rwbagcat.
• Bug fixes in statistics output of readset
• Some column headers have changed; test any supporting scripts you may have.
• rwset can now build multiple sets in a single pass. Use the --sip-file, --dip-file, and --nhip-file switches to create the IP set files.
• rwsort now supports the same fields as rwcut and rwuniq
• rwuniq can now bin the start-time and end-time with the --bin-time switch
• rwstats largely rewritten. New switches (though legacy switches are still supported); added support to rwstats for computing top-N lists based on packet counts or byte counts.
• readset will now read a binary IP set from stdin
• Fix compilation problems on RedHat64

Changelog

• Bug Fix: Allow tools so write output to /dev/null.

Changelog

• New packet-support tools
  • rwptoflow: Create a single-packet SiLK flow record for every record in a tcpdump file.
  • rwpmatch: Use a SiLK Flow file to filter the contents of a tcpdump file
  • rwpcut: Output a tcpdump dump file as ASCII
• New tool rwgroup: Groups multiple records together with a common tag
• New tool rwmatch: Matches records from two files together into a common stream
• New pipe-lining tool rwnetmask: Masks off lower bits of the source and/or destination addresses allowing one to aggregate output by CIDR block
• Support for 16bit SNMP interfaces: Packing and file output formats support the full 16bits of SNMP interface values as exported in NetFlow v5
• Support for 65535 sensors: Sensor ID is now processed and stored in a 16 bit integer
• Millisecond time support: Millisecond precision for start time, end time, and duration in the file output formats. Limited application support to access this field.
• New country-code support: Allow filtering and cutting by an IP's physical location
• Enhancements to rwfilter
  • New --print-volume-statistic switch gives bytes, packet, and flow counts for the passed and failed streams
  • New --any-address and --any-ipset switches allows matching source or destination IP addresses
  • New --nhip-set switch allows matching next-hop IP address
  • New --active-time switch allows printing flows that were active at a particular time
  • New --flags-all switch to allow (yet) another way to specify TCP flags
  • Allow filtering over class and type when reading a file generated by a previous run of rwfilter
• Enhancements to rwsort
  • Remove the previous 50 million record limit by using temporary disk files when RAM is exceeded
  • Enable sorting based on elapsed time
• Enhancements to rwuniq
  • In addition to flow counts, optionally keep totals of bytes and packets, as well as the time range over which the key was active.
  • On out-of-memory, print the bins as counted so far.
• Enhancements to rwcount
  • When --start-epoch is given, use that time as the edge of a bin. This lets you view traffic in 24 hour bins that runs from noon to noon, for example.
  • Be more memory stingy by not creating bins for records that occur before the --start-epoch
  • Accepting flows in any time order (previously assumed flows were close to time-sorted order)
  • Allow --start-epoch switch to take a time string like rwfilter accepts
  • Print file names when --print-files is given
  • Add final delimiter to each line of output
• Enhancements to rwaddrcount: Allow sorting of output records by IP address
• Enhancements to rwcat: New --xargs switch to allowing reading a list of file names; this allows rwcat to accept output from the UNIX find command
• Enhancements to readset: Added switches to print details about the structure of the IPs in the IP-set

Changelog

• Critical Update. This version fixes a bug that prevents one from querying data for the new year. Any data you collected is correct; it's just that the tools prevented you querying this data.

Changelog

• New binary file format (Bag) that maps IP address to a count of bytes, packets, or flows.
• Tools are included for manipulating these files: rwbag*
• Course filtering (fglob) support removed from all tools except rwfilter.
• New rwflowpack options; previous rwfpd scripts are incompatible with the rwflowpack from this release.
• Additional documentation in analysis handbook and the installation handbook.

Changelog

• Added support to rwflowpack for accepting incoming flows from multiple interfaces.
• Fixed bugs in rwswapbytes and rwrandomizeip utilities

Changelog

• Critical Update. Public releases of the SiLK Tool Suite prior to this release (SiLK-0.3 and earlier) contained a bug that affected the packing of web records. This bug caused the source and destination ports for web records to be swapped, e.g., web connections from your network to sourceforge.net would show the sourceforge.net web service on a high port and have your client machine on port 80.
• This SiLK-0.4 release fixes that bug, and we've provided a Perl script, rwpatchwww.pl, that will repair files you've packed with previous versions. The rwpatchwww.pl script will also migrate your all of your packed files to Version 2 of the SiLK file format. Release SiLK-0.4 of the SiLK Tools will read files packed either in Version 1 or Version 2 format.

Changelog

• Added the rwfpd script that was accidentally omitted from the SiLK-0.2 release.
• Other minor fixes.

Changelog

• Critical Update. This version fixes major bugs in the initial release of rwflowpack, including a problem that cause the system to produce corrupted packed data files.

Changelog

• Initial public "preview" of the SiLK Analysis Suite and Packing System.