SiLK Release 3.8.3, 2014-Jul-31

Downloads

(MD5=d5ce485569186c6313981091436878f7)

(SHA1=60d8044d388299093b7d96580eac1e0dfe01d1e7)

Notes

  • rwstats and rwuniq
    • Fix a bug when --fields contained "dPort" followed by "icmpTypeCode" that caused the "dPort" field to display as 0.
  • Additional changes and bug fixes
Full Release Notes

SiLK Release 3.8.2, 2014-Apr-24

Downloads

(MD5=398073fb78b072dfcfccb84029ff88f7)

(SHA1=090972127175272c1cc82d6e419a65c8ecfbd275)

Notes

  • Add multiple thread support to rwflowappend.
  • Support logging of IPFIX and NetFlow v9 templates received by rwflowpack and flowcap.
Full Release Notes

SiLK Release 3.8.1, 2014-Jan-30

Downloads

(MD5=a64f7505ed89a88c7391485c35ceec29)

(SHA1=85787cd0bf8c7ce9b11013fff2ac3eea1ff5bd13)

Notes

  • Add new ICMP fields iType and iCode.
  • Add small features and fix bugs.
Full Release Notes

SiLK Release 3.8.0, 2013-Nov-21

Downloads

(MD5=5bdd091d41e30bb4c1599c93412f1698)

(SHA1=5f3887fcc7147cc5a86cdd6ffa73854b93eb24d0)

Notes

  • Allow rwpmaplookup to print the range that contains the key
  • Improve handling of records from some devices that export NetFlow v9
  • Add support for libfixbuf-1.4.0 and remove support for releases prior to libfixbuf-1.2.0
Full Release Notes

SiLK Release 3.7.2, 2013-Aug-15

Downloads

(MD5=01474e6996892d0ab67752bee858da6d)

(SHA1=32e73edf764d4c9ce54b9bcf048c13dfd40ec36b)

Notes

  • Small bug fixes.
Full Release Notes

SiLK Release 3.7.1, 2013-Jun-20

Downloads

(MD5=08c8dae123b9ef5f8fddc617a6eea999)

(SHA1=c009c47e3c9cb11e3cd8bc7101b2fa280a809680)

Notes

  • First public release of SiLK 3. Changes since SiLK 2.5.0:
  • IPsets support IPv6 addresses
    • In a mixed IPv4 and IPv6 environment, IPv4 addresses are stored in the ::ffff:0:0/96 prefix.
    • The IPset tools support a --record-version switch and a SILK_IPSET_RECORD_VERSION environment variable that determine how IPsets are written to disk. Using the numeric value of 0 as the version indicates to use the default behavior.
    • Sets containing only IPv4 addresses are stored in files identical to those in SiLK-2 (record-version=2). By default, IPsets containing IPv6 addresses are stored in a record-version=3 file format; these files can be very large. Another file format, record-version=4, is available, which is smaller than the other formats.
    • INCOMPATIBILITY: The legacy IPset tools rwsetunion and rwsetintersect have been removed. Use rwsettool instead.
  • Bags support IPv6 addresses
    • In a mixed IPv4 and IPv6 environment, IPv4 addresses are stored in the ::ffff:0:0/96 prefix.
    • Bag files now record the type of the key and counter when the Bag is created.
    • Bag files that contain only IPv4 addresses are readable by SiLK-2.x.
    • INCOMPATIBILITY: The maximum counter value supported by bag files has been reduced by 1, to 2^64 - 2.
  • Prefix maps and Country Codes support IPv6 addresses
    • In a mixed IPv4 and IPv6 environment, IPv4 addresses are stored in the ::ffff:0:0/96 prefix.
    • Prefix map files that contain only IPv4 addresses are readable by SiLK-2.x.
    • Add new --v6-csv-input switch to rwgeoip2ccmap to support building a country code prefix map that contains IPv6 addresses.
  • rwsiteinfo added
    • New tool which extends the capability of mapsid.
    • rwsiteinfo prints information about the sensors, classes, and types specified in the silk.conf site configure file.
  • int-ext-fields.so added
    • New plug-in for rwcut, rwgroup, rwsort, rwstats, and rwuniq.
    • The int-ext-fields plug-in defines fields (int-ip, ext-ip, int-port, ext-port) which can be used to print, sort by, or group by the internal or external IP or port. This plug-in is useful when a single flow file contains flows in multiple directions.
  • PySiLK enhancements
    • Support for Python 3.x has been added.
    • Support for Python 2.4 and 2.5 is now considered frozen.
    • There are numerous changes to support the IPv6 capability in IPsets and bags.
    • The silk.site class has many changes, especially regarding the reading of the silk.conf file.
    • Three new methods in silk.site class---repository_iter(), repository_silkfile_iter(), and repository_full_iter()---are provided to iterate over files in the data repository. Their use is recommended over the FGlob class.
    • INCOMPATIBILITY: The key_type attribute on Bag objects has been removed.
    • INCOMPATIBILITY: When creating PySiLK plug-ins, any filters must be registered by calling register_filter(). Previously, PySiLK would automatically register a filter.
    • INCOMPATIBILITY: When creating PySiLK plug-ins, fields must be registered with register_field(). The legacy method register_plugin_field() has been removed.
    • Add methods RWRec.is_icmp(), RWRec.is_ipv6(), RWRec.to_ipv4(), RWRec.to_ipv6().
    • Recognize an RWRec containing an IPv6 address and having a protocol of 58 as ICMPv6.
    • Limit the range of times that RWRec.stime, RWRec.duration, and RWRec.etime support.
    • Fix a bug when setting RWRec.timeout_killed, RWRec.timeout_started, and RWRec.uniform_packets.
    • Fix a bug that allowed setting arbitrary attributes on an RWRec.
    • Reflect changes to RWRec.initial_flags or RWRec.session_flags in RWRec.tcpflags and vice versa.
    • Clear RWRec.initial_flags and RWRec.session_flags when changing RWRec.protocol to a value other than TCP.
    • Throw an error when attempting to set RWRec.initial_flags or RWRec.session_flags on an RWRec where the protocol is not TCP.
    • Do not treat True and False as numbers.
    • Throw ValueError instead of OverflowError in many methods.
    • Throw an exception when an attempt is make to pickle or unpickle silk objects. Previously, attempts to pickle or unpickle these objects could cause the application to crash.
    • Add a constant containing the maximum bag counter value.
  • Changes across many analysis tools
    • Almost all analysis tools now accept the --xargs switch to read the list of names of files to process from the standard input or a file.
    • A new switch --ip-format determines how IP addresses are printed. The --integer-ips and --zero-pad-ips switches are deprecated.
    • A new switch --timestamp-format provides more control over how timestamps are printed. The --epoch-time and --legacy-timestamp switches are now deprecated.
    • A new value for 'attribute' is available. The attribute 'S' is set when the flow generator notices that all packets in the flow are the same size.
    • The key field 'dur' has been renamed to 'duration'. Most uses will be unaffected by this change, but there is a POTENTIAL INCOMPATIBILITY if a user-defined plug-in defines a key field named 'duration' or with a similar prefix.
    • Binary output to a pipe is no longer compressed by default, which provides increased throughput when piping data between SiLK applications.
    • Almost all tools support the SILK_CLOBBER environment variable. When this variable is set, SiLK allows new output files to overwrite existing files.
    • When parsing silk.conf, ensure that the names of classes, types, and sensors are of legal length and do not contain invalid characters.
    • Fix a bug during parsing the silk.conf file that prevented putting a sensorgroup into another sensorgroup.
    • When reading SiLK Flow records, ensure that the initialFlags and sessionFlags values are 0 for records that are not TCP, ignoring any value stored in the file.
    • When writing SiLK Flow records, always store the attributes value (if the file format supports it). Previously, some very old file formats only stored attributes when the protocol was TCP.
    • Modify processing of some ICMP SiLK Flow records. ICMP type and code are normally encoded in the destination port. Due to a bug when processing IPFIX bi-flow ICMP records, the type and code were sometimes stored in the source port. SiLK-3.4.0 attempts to fix this bad encoding. However, this change removes a previous work-around designed to fix issues with SiLK Flow records collected prior to SiLK-0.8.0 that originated as NetFlow v5 PDUs from some types of Cisco routers.
    • POTENTIAL INCOMPATIBILITY. Exit with an error when the path-format in silk.conf contains any unknown '%'-conversions.
    • INCOMPATIBILITY: When the --no-columns switch is specified, fields containing TCP flags and attributes are printed without embedded spaces.
    • INCOMPATIBILITY: The --dynamic-library switch has been removed, as well as support for the old dynlib API. Applications must use the --plugin switch and skplugin API.
    • Tools affected by one or more of these changes are rwaddrcount, rwbag, rwbagbuild, rwbagcat, rwbagtool, rwcat, rwcount, rwcut, rwdedupe, rwfilter, rwgroup, rwip2cc, rwipaexport, rwipfix2silk, rwmatch, rwnetmask, rwpackchecker, rwptoflow, rwrandomizeip, rwscan, rwset, rwsetbuild, rwsetcat, rwsettool, rwsilk2ipfix, rwsort, rwsplit, rwstats, rwtotal, rwtuc, rwuniq
  • rwfilter enhancements
    • Add switch --any-index to match either the SNMP input or SNMP output interface.
    • Add switch --any-cc to match either the country code of the source address or of the destination address.
    • rwfilter now exits with status 0 if there is a problem opening or reading an input flow file. Previously, its exit status was 1.
    • Better support when writing to a pipe and a file or another pipe simultaneously. Specifically, rwfilter used to exit when any pipe stopped receiving data. Now, rwfilter will finish writing the output to the file or other pipe when one pipe closes.
    • INCOMPATIBILITY: The deprecated --ipport-* and --ippair-* switches have been removed. Use the --tuple-* family of switches instead.
    • INCOMPATIBILITY: The argument to --xargs is now optional. As a result, any use of "--xargs <file>" must be changed to --xargs=<file>.
  • rwcut enhancement
    • Add new --tail-recs switch to print some number of records at the end of a file or an input stream.
  • rwcount enhancements
    • New --load-scheme=5 computes the maximum possible values for each bin, as if the entire flow record had occurred in that bin.
    • New --load-scheme=6 computes the minimum possible values for each bin, as if the entire flow record had occurred in some other bin, for records that span multiple bins.
  • rwstats and rwuniq enhancements
    • Support added for counting the number of distinct values of almost any field.
    • The value 'flows' is now provided as an alias for 'records'.
    • Fix a bug where the --bin-time value was not being applied to the eTime field unless the key fields included sTime and dur as well.
    • Fix a crash that could occur when using more than eight fields and some the fields were defined by plug-ins.
    • Fix a display issue where prefix map labels longer than 128 characters were being truncated.
    • INCOMPATIBILITY: Deprecated switches (e.g., --sip-topn) have been removed from rwstats.
  • rwset enhancement
    • The newly added --any-set switch creates a IPset containing both the source and destination IP addresses on the SiLK Flow records.
  • rwsetcat enhancements
    • The --cidr-blocks switch now accepts an optional argument.
    • Print the IPset file name when either --print-statistics or --count-ips is used and multiple files appear on command line.
    • Add new switch --print-filenames which can explicitly enable or suppress printing of the IPset file name.
    • When the --network-structure switch includes 'T' in its argument, print the total even when the IPset is empty.
    • Modify --print-statistics so output is generated when the IPset is empty. Previously, no output was generated.
    • Add a new --ip-format switch to rwsetcat that determines how IP addresses are printed. The --integer-ips and --zero-pad-ips switches are deprecated.
  • rwsetmember change
    • Process all files specified on the command line despite being unable to read some files.
  • rwsettool change
    • Modify rwsettool to exit with status 1 when it is unable to read any IPset file. Previously, rwsettool when only exit with status 1 when it was unable to read the first IPset file.
  • rwbag changes
    • rwbag no longer stops processing records when an overflow occurs for a particular key. Instead, rwbag sets that key's counter to the maximum value and continues.
    • INCOMPATIBILITY: Deprecated switches (e.g., --sf-file) have been removed, as well as the --legacy-help switch.
  • rwbagbuild enhancements
    • The new --key-type and --counter-type switches allow the type of the key and counter to be specified when the bag is created.
  • rwbagcat enhancements
    • When the --network-structure switch includes 'T' in its argument, print the total even when the Bag is empty.
    • Add a new --key-format switch to rwbagcat that determines how keys are printed. The --integer-keys and --zero-pad-ips switches are deprecated.
    • INCOMPATIBILITY: The --stats and --tree-stats switches have been removed and replaced by the new switch --print-statistics.
  • rwpmapbuild enhancements
    • There is better support for prefix maps that use numbers as labels.
    • Performance is hugely improved when building very large prefix maps.
    • The new --mode switch allows setting the type of input on the command line.
    • The new --dry-run switch checks the syntax of the file without building the prefix map.
    • The new --ignore-errors switch causes rwpmapbuild to write the output despite errors in the input.
  • rwpmaplookup added
    • New tool which extends the capability of rwip2cc.
    • rwpmaplookup finds information about specific IP address(es) or protocol/port pair(s) in a binary prefix map file and prints the result as text.
  • rwresolve enhancements
    • Support has been added for IPv6 addresses.
    • rwresolve defaults to using the C-ares asynchronous resolving library, if that library was found during compilation.
  • rwcat enhancements
    • Support for putting annotations in the header of the output file has been added (i.e., support for --note-add, --note-file-add).
  • rwmatch enhancement
    • Support for putting annotations in the header of the output file has been added (i.e., support for --note-add, --note-file-add).
    • Support for setting the compression of the output file.
  • rwfileinfo enhancements
    • New --field=16 prints information about an IPset file.
    • New --field=17 prints information about a bag file.
    • Fix an issue when processing a compressed file containing a corrupted compressed block that caused rwfileinfo to report fewer valid records than actually existed.
  • rwtuc changes
    • Add support for parsing ICMP type and ICMP code fields.
    • Add a --no-titles switch to parse the first line of input as record values when --fields is specified.
    • Fix a bug that effectively ignored the --attributes switch.
    • Allow attributes to be set regardless of presence of initialFlags and sessionFlags.
    • Ensure initialFlags and sessionFlags are 0 for non-TCP records.
    • Have initialFlags and sessionFlags take precedence over flags.
    • Describe restrictions on field combination in the manual page.
  • rwfglob changes
    • Fix a bug where the --no-file-names switch did nothing.
    • Add a --no-block-check switch to suppress checking the block count of the files.
  • rwrecgenerator added
    • New tool
    • Use a pseudo-random number generator to create SiLK Flow records that can be used to test SiLK applications.
  • rwdedupe enhancements
    • Add support for writing annotations into the header of the output file (i.e., support for --note-add, --note-file-add).
    • Copy annotations and invocation history from the headers of the input files to the header of the output file.
  • rwscan changes
    • Provide new --trw-internal-set switch that implements the behavior of the --trw-sip-set switch, and mark the latter as deprecated.
    • Generate an error when multiple internal IPsets are specified.
  • rwscanquery enhancements
    • Add support for querying a SQLite database.
    • Allow use of both --daddress and --dipset for the scanset report.
    • Perform more robust parsing of IP addresses specified by the --saddress switch.
    • Write verbose output to standard error instead of to standard out.
    • Fix a bug in date parsing where the value specified by --end-date was sometimes ignored.
    • Fix a bug where rwscanquery would attempt to create a file whose name started and ended with a quote character.
  • rwpdu2silk added
    • New tool
    • rwpdu2silk reads files containing NetFlow v5 records and writes a stream of SiLK Flow records.
  • rwipfix2silk and rwsilk2ipfix changes
    • These tools are only built when the configure script finds libfixbuf-1.0.0 or later. (As of SiLK-3.8.0, these tools require libfixbuf-1.2.0 or later.)
    • Fix a bug when processing IPFIX bi-flow ICMP records that caused the Type and Code to be recorded incorrectly.
    • Ignore IPFIX records that have a packet or byte count of zero.
  • silk.magic
    • Include a sample "magic" file for use with the UNIX file utility. The file is installed in share/silk/silk.magic.
  • Deprecated tools
    • rwip2cc is deprecated. Use rwpmaplookup instead.
    • mapsid is deprecated. Use rwsiteinfo instead.
  • Changes across all daemons
    • Add the --no-chdir switch which prevents the daemon from changing directory to / on start-up.
    • Add the --log-post-rotate switch to control post-processing of the previous day's log file.
    • Fix an issue that prevented shutdown on some BSD OSes.
    • POTENTIAL INCOMPATIBILITY. Exit with an error when any unknown '%'-conversions are specified in the argument to the --command, --post-command or --hour-file-command switches.
  • rwflowpack and flowcap enhancements
    • rwflowpack and flowcap only support IPFIX when libfixbuf-1.0.0 or later is available. (As of SiLK-3.8.0, IPFIX support requires libfixbuf-1.2.0.)
    • rwflowpack and flowcap only support NetFlow v9 when libfixbuf-1.1.0 or later is available. (As of SiLK-3.8.0, NetFlow v9 support requires libfixbuf-1.2.0.)
    • A single IPFIX probe listening on a TCP port will accept connections from multiple IPFIX clients.
    • A single NetFlow v9 or IPFIX probe listening on a UDP port will accept connections from multiple IPFIX clients. (Requires libfixbuf-1.2.0 or later.)
    • Add support for logging number of missing NetFlow v9 packets. (Requires libfixbuf-1.3.0 or later.)
    • In sensor.conf, the accept-from-host clause is now supported for IPFIX and NetFlow v9 probes.
    • rwflowpack and flowcap support listening for flow records on IPv6 addresses.
    • Add support for reading TCP flags in a subTemplateMultiList as exported in IPFIX records created by yaf-2.0.
    • Collection statistics generated by yaf are now recorded in the log file.
    • Write messages to the log file for IPFIX/NetFlow v9 records that are ignored--for example, records that represent firewall events.
    • Accept NF_F_FWD_FLOW_DELTA_BYTES and NF_F_REV_FLOW_DELTA_BYTES as volume values for NetFlow v9 flow records.
    • Accept initiatorOctets, initiatorPackets, responderOctets, and responderPackets as volume values for IPFIX flow records.
    • When reading IPFIX data, ignore initialTCPFlags and unionTCPFlags when the protocol is not TCP.
    • Ignore IPFIX records that have a packet or byte count of zero.
    • Fix a bug where future-dated records could be created when the NetFlow v9 sysUptime rolled over.
    • Fix a bug when processing IPFIX bi-flow ICMP records that caused the Type and Code to be recorded incorrectly.
    • The sensor.conf file accepts a host name as a valid address to listen as or to accept a connection from.
    • INCOMPATIBILITY: Parsing of the sensor.conf file is more strict. Some statements that previous versions of SiLK used to ignore will now cause errors.
  • rwflowpack changes
    • Add a new output-mode "incremental-files" which is similar to the "sending" output-mode except it leaves the incremental files in the incremental-directory. Deprecate the "sending" output-mode, but continue to accept it for backwards compatibility.
    • POTENTIAL INCOMPATIBILITY: If incremental files from a previous releases of SiLK are in the incremental-directory when this release of rwflowpack is started in the sending output-mode, the files will not be moved to the sender-directory.
    • A manual page now exists for each packing-logic plug-in.
    • Allow for categorizing flow records solely based on VLAN tags.
    • Fix an issue when processing IPFIX files where the file was never closed. This could cause rwflowpack to exit unexpectedly once it ran out of file handles.
    • Fix a potential race condition between opening a file and getting the write lock on the file.
    • Write log messages about missing NetFlow v5 records when reading PDUs from a file.
  • flowcap enhancements
    • Print a log message that specifies the number of packets received when closing an empty file. Previously, the message was only printed when closing a file that contained records.
  • rwflowappend enhancements
    • Use advisory write locks on files in the repository to avoid conflicts when multiple rwflowappend processes attempt to write to the same hourly file.
    • The new switch --no-file-locking can be used to disable these advisory locks.
    • Truncate the repository file to the size it had when it was opened if there is a write error while appending an incremental file to the repository file.
    • POTENTIAL INCOMPATIBILITY: Modify rwflowappend to use the packed file information in a file's header instead of relying on the name of the file. The file name will only be checked if the file does not contain the necessary header.
  • rwsender and rwreceiver enhancements
    • rwsender and rwreceiver support communicating over IPv6 addresses.
    • When running in server mode, rwsender and rwreceiver support binding to a particular IP address on multi-homed machines.
    • Add support for GnuTLS 3.x.
    • Slow how quickly a client reconnects to a server when the server rejects the client due to unrecognized identifier.
  • rwsender enhancement
    • Add the --unique-local-copies switch to create a complete copy of files (as opposed to using hard links) when creating duplicates of the incoming files via the --local-directory switch.
  • rwreceiver enhancements
    • Add ability to monitor disk usage when either --freespace-minimum or --space-maximum-percent is specified.
    • Add the --unique-duplicates switch to create a complete copy of files (as opposed to using hard links) when creating duplicates via the --duplicate-destination switch.
    • Handle the case where rwreceiver attempts to receive files with the same name from multiple rwsender processes simultaneously.
    • Fix a potential deadlock when disk space is exhausted.
  • rwpollexec changes
    • On systems where "/bin/sh -c" does not use exec, attempt to find a shell that does use exec.
    • Fix an issue where the stdout and stderr from the command would not appear in the log file after the log file had been rotated.
  • Numerous changes to C functions in libraries.
    • Support for dynlib API has been removed. Use the skplugin API instead.
    • The iochecks.h header has been deleted.
    • Functions for skipaddr_t have been moved into skipaddr.h, which you may need to include.
    • The bag API has been largely rewritten, and the old API is deprecated.
    • Many deprecated functions have been removed.
  • Building and configuration.
    • Modify the expected result of some tests run with "make check" when standard input is not a terminal.
    • Require automake 1.12 or later to rebuild the Makefile.in files.
Full Release Notes

SiLK Release 2.5.0, 2012-Jun-28

Downloads

(MD5=abd24b6164759171c95a2ffcf467a503)

(SHA1=7ce02198742da6475c47b017bfd43c438429ff58)

Notes

  • Provide a new configure switch to work-around issues when reading NetFlow v9 flow records from a Cisco ASA router.
Full Release Notes