Packages

  • package root

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

    Please see the documentation for the individual packages for more details on their use.

    Scala Packages

    These packages are useful in Scala code without involving Spark:

    org.cert.netsa.data

    This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

    org.cert.netsa.io.ipfix

    The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

    org.cert.netsa.io.silk

    To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

    org.cert.netsa.util

    The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

    Spark Packages

    These packages require the use of Apache Spark:

    org.cert.netsa.mothra.datasources

    Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

    The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

    org.cert.netsa.mothra.analysis

    A grab-bag of analysis helper functions and example analyses.

    org.cert.netsa.mothra.functions

    This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package cert
    Definition Classes
    org
  • package netsa
    Definition Classes
    cert
  • package io
    Definition Classes
    netsa
  • package ipfix

    The ipfix package provides classes and objects for reading and writing IPFIX data.

    The ipfix package provides classes and objects for reading and writing IPFIX data.

    Class / Object Overview

    (For a quick overview of the IPFIX format, see the end of this description.)

    The Message trait describes the attributes of an IPFIX message, and the CollectedMessage class and object are implementations of that trait when reading data. (Record export does not create specific Message instance.)

    The IpfixSet abstract class and object hold the attributes of a Set. The TemplateSet class may represent a Template Set or an Options Template Set.

    The Template class and object are used to represent a Template Record or an Options Template Record.

    The IEFieldSpecifier class and object represent a Field Specifier within an existing Template. To search for a field within a Template, the user of the ipfix package creates a FieldSpec (the companion object) and attempts to find it within a Template.

    The Field Specifier uses the numeric Identifier to identify an Information Element, and an Element is represented by the InfoElement class and object. The InfoModel class and object represent the Information Model.

    To describe the attributes of an InfoElement, several support classes are defined: DataTypes is an enumeration that describes the type of data that the element contains, and DataType is a class that extracts a Field Value with that DataType. IESemantics describes the data semantics of an Information Element (e.g., a counter, an identifier, a set of flags), and IEUnits describes its units.

    The Data Set is represented by the RecordSet class and object.

    A Data Record is represented by the Record abstract class. This class has three subclasses:

    1. The CollectedRecord class and object are its implementation when reading data. Its members are always referenced by numeric position.
    2. The ArrayRecord (I do not like this name) and object may be used to build a Record from Scala objects; its fields are also referenced by numeric position.
    3. ExportRecord is an abstract class that also supports building a Record from Scala objects. The user extends the class and uses the IPFIXExtract annotation to mark the members of the subclass that are to be used when writing the Record.

    A user-defined class that extends Fillable trait may use the Record's fill() method to copy fields from a Record to the user's class. It also uses the IPFIXExtract annotation.

    A Structured Data Field Value in a Data Record is represented by the ListElement abstract class. That abstract class has three abstract subclasses, and each of those has two concrete subclasses (one for reading and one for writing):

    1. The BasicList abstract class (object) has subclasses CollectedBasicList and ExportBasicList.
    2. The SubTemplateList abstract class (object) has subclasses CollectedSubTemplateList and ExportSubTemplateList.
    3. The SubTemplateMultiList abstract class (object) has subclasses CollectedSubTemplateMultiList and ExportSubTemplateMultiList.

    Reading data

    When reading data, a Record instance is returned by a RecordReader. The RecordReader uses a class that extends the MessageReader trait. The ipfix package includes two: ByteBufferMessageReader and StreamMessageReader.

    A Session value represent an IPFIX session, which is part of a SessionGroup.

    Writing data

    For writing data, an instance of an ExportStream must be created using a Session and the destination FileChannel. The user adds Records or Templates to the ExportStream and they are written to the FileChannel.

    Overview of IPFIX

    An IPFIX stream is composed of Messages. Each Message has a 16-byte Message Header followed by one or more Sets. There are three types of Sets: A Data Set, a Template Set, and an Options Template Set.

    Each Set has a 4-byte set header followed by one or more Records. A Data Set contains Data Records and a Template Set contains Template Records.

    A Template Record describes the shape of the data that appears in a Data Record. A Template Record contains a 4-byte header followed by zero or more Field Specifiers. Each Field Specifier is either a 4-byte or an 8-byte value that describes a field in the Data Record.

    A Field Specifier has two parts. The first is the numeric Information Element Identifier that is defined in an Information Model. The second is the number of octets the field occupies in the Data Record.

    A Data Set contains one or more Data Records of the same type, where the type is determined by the Template Record that the Data Set Header refers to. Each Data Record contains one or more Field Values, where the order and length of the Field Values is given by the Template.

    A Field Value in a Data Record may be a Structured Data. There are three types of Structured Data:

    1. A Basic List contains one or more instances of a Single Information Element.
    2. A SubTemplateList references a single Template ID, and it contains one or more Records that match that Template.
    3. The SubTemplateMultiList contains a series of Template IDs and Records that match that Template ID.

    An IPFIX stream exists in a Transport Session, where a Transport Session is part of a Session Group. All Sessions in a Session Group use the same Transport Protocol, and only differ in the numeric Observation Domain that is part of the Message Header.

    Definition Classes
    io
  • package silk

    SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

    SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

    RWRec is the type of SiLK flow records.

    You can use RWRecReader to read SiLK files from Scala, including compressed files if Hadoop native libraries are available. For example:

    import org.cert.netsa.io.silk.RWRecReader
import java.io.FileInputStream

val inputFile = new FileInputStream("path/to/silk/rw/file")

for ( rec <- RWRecReader.ofInputStream(inputFile) ) {
  println(rec.sIP)
}
    Definition Classes
    io
    See also

    org.cert.netsa.mothra.datasources.silk.flow for working with SiLK data in Spark using the Mothra SiLK datasource.

  • package config
  • package io
  • BagDataType
  • BagReader
  • BagResult
  • BagWriter
  • CompressionMethod
  • FileFormat
  • FlowType
  • Header
  • HeaderEntry
  • IPSetReader
  • IPSetWriter
  • PrefixMapProtocolPortPair
  • PrefixMapReader
  • PrefixMapResult
  • RWRec
  • RWRecReader
  • RWRecWriter
  • Sensor
  • SilkConfig
  • SilkDataFormatException
  • SilkVersion
  • TCPState

package silk

SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

RWRec is the type of SiLK flow records.

You can use RWRecReader to read SiLK files from Scala, including compressed files if Hadoop native libraries are available. For example:

import org.cert.netsa.io.silk.RWRecReader
import java.io.FileInputStream

val inputFile = new FileInputStream("path/to/silk/rw/file")

for ( rec <- RWRecReader.ofInputStream(inputFile) ) {
  println(rec.sIP)
}
See also

org.cert.netsa.mothra.datasources.silk.flow for working with SiLK data in Spark using the Mothra SiLK datasource.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. silk
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Package Members

  1. package config
  2. package io

Type Members

  1. sealed abstract class BagDataType extends ShortEnumEntry

    A SiLK bag data type.

    A SiLK bag data type. May be converted to and from Short values.

    See also

    the companion object for more details

    BagReader

  2. abstract class BagReader[T] extends Iterator[(T, Long)]

    A reader of binary SiLK Bag files.

    A reader of binary SiLK Bag files. This is usable as an Iterator over a pair representing the key and counter in the Bag. Depending on the contents of the Bag file, the key is either an IPAddress or an Int. The counter is always a Long.

    Since a Bag file may contain different key types, the BagReader factory methods return a BagResult wrapper over BagReader.

    The keyType and counterType methods return a BagDataType that specifies the type of the key and the counter that were specified in the Bag file's header.

    Example:

    1. This example uses the single argument form of the companion object's ofInputStream() method to read the Bag file "example.bag". This code may be used outside of Hadoop when it is known that the Bag's key is an IPAddress.

      val stream = new java.io.FileInputStream("example.bag")
val bagresult = BagReader.ofInputStream(stream)
val bag = bagresult match {
  case BagResult.IPAddressBag(iter) => iter
  case _ => null
}
bag.hasNext
val (ipaddr, counter) = bag.next()
    Note

    While the SiLK command lines tools display an unspecified key type as an IPv4 address, the BagReader class treats an unspecified key type as an integer.

    See also

    the companion object for more details.

  3. sealed abstract class BagResult extends AnyRef

    Return type for the ofInputStream() method of the companion object to the BagReader class.

  4. class BagWriter extends AnyRef

    A writer of binary SiLK Bag files.

    A writer of binary SiLK Bag files.

    To include a header in the Bag file that specifies the type of the and counter, run setKeyType() and/or setCounterType() prior to writing the Bag.

    Example:

    1. This example reads the contents of "example.bag" and writes it to "copy.bag", where the keys are IP addresses:

      val in = new java.io.FileInputStream("example.bag")
val out = new java.io.FileOutputStream("copy.bag")
val bagresult = BagReader.ofInputStream(in)
val bag = bagresult match {
  case BagResult.IPAddressBag(iter) => iter
  case _ => null
}
val writer = BagWriter.toOutputStream(out)
if ( None != bag.keyType ) {
  writer.setKeyType(bag.keyType)
}
if ( None != bag.counterType ) {
  writer.setCounterType(bag.counterType)
}
writer.appendIPAddresses(bag)
writer.close()
    See also

    the companion object for more details

  5. sealed abstract class CompressionMethod extends ByteEnumEntry

    A SiLK compression method.

    A SiLK compression method. May be converted to and from Byte values.

    See the companion object for more details.

  6. sealed abstract class FileFormat extends ByteEnumEntry

    A SiLK file format.

    A SiLK file format. May be converted to and from Byte values. Equipped with a partial function mapping Short file versions to Unpackers.

    See the companion object for more details.

  7. final case class FlowType(toByte: Byte) extends AnyVal with Product with Serializable

    A SiLK flow type (type and class), as represented by a Byte value.

    A SiLK flow type (type and class), as represented by a Byte value.

    toByte

    The byte value representing this flow type.

  8. case class Header(fileFlags: Byte, fileFormat: FileFormat, fileVersion: Byte, compressionMethod: CompressionMethod, silkVersion: SilkVersion, recordSize: Short, recordVersion: Short, headerEntries: IndexedSeq[HeaderEntry]) extends Product with Serializable

    A SiLK file header, including contained header entries.

    A SiLK file header, including contained header entries. Supports only "new-style" header format (SiLK versions 1.0+).

    fileFlags

    The bits encoding file flags. Currently only whether the file is big-endian.

    fileFormat

    The SiLK file format contained within this file.

    fileVersion

    The SiLK file version--specifically the version of the header format.

    compressionMethod

    The compression method used by data in this file.

    silkVersion

    The version of SiLK used to create this file.

    recordSize

    The size of individual (uncompressed) records in this file.

    recordVersion

    The record version of the file format.

    headerEntries

    Sequence of additional extensible header records of various types.

    See also

    Header.isBigEndian

  9. sealed abstract class HeaderEntry extends AnyRef

    One of a variety of SiLK extended header entries.

    One of a variety of SiLK extended header entries.

    See the companion object for more details.

  10. abstract class IPSetReader extends Iterator[IPBlock]

    A reader of binary SiLK IPset files.

    A reader of binary SiLK IPset files. This is usable as an Iterator over IPBlock objects.

    Example:

    1. This example uses the single argument form of the companion object's ofInputStream() method to read the IPset file "example.set"; the code may be used outside of Hadoop.

      val stream = new java.io.FileInputStream("example.set")
val ipset = IPSetReader.ofInputStream(stream)
ipset.hasNext
val ipblock = ipset.next()
println(ipblock.min + "/" + ipblock.prefixLength)
    See also

    the companion object for more details

  11. class IPSetWriter extends AnyRef

    A writer of binary SiLK IPset files.

    A writer of binary SiLK IPset files.

    Example:

    1. This example reads the contents of "example.set" and writes it to "copy.set":

      val in = new java.io.FileInputStream("example.set")
val out = new java.io.FileOutputStream("copy.set")
val ipset = IPSetReader.ofInputStream(in)
val writer = IPSetWriter.toOutputStream(out)
writer.append(ipset)
writer.close()
    See also

    the companion object for more details

  12. final case class PrefixMapProtocolPortPair(startProtocol: Protocol, startPort: Port, endProtocol: Protocol, endPort: Port) extends Product with Serializable

    A type representing a range of (org.cert.netsa.data.net.Protocol, org.cert.netsa.data.net.Port) Pairs.

    A type representing a range of (org.cert.netsa.data.net.Protocol, org.cert.netsa.data.net.Port) Pairs.

    This is the type of the key when using an Iterator over the contents of a PrefixMap file that contains (Protocol, Port) Pairs.

    See also

    PrefixMapReader

  13. abstract class PrefixMapReader[T] extends Iterator[(T, String)]

    A reader of binary SiLK PrefixMap files.

    A reader of binary SiLK PrefixMap files. This is usable as an Iterator over a pair representing a key and value. Depending on the contents of the PrefixMap file, the key is either an IPBlock or a PrefixMapProtocolPortPair. The value is always a String.

    Because the PrefixMap may contain different key types, the PrefixMapReader factory methods return a PrefixMapResult wrapper over PrefixMapReader.

    The mapName method returns the map name that was specified in the PrefixMap file's header, if any.

    Example:

    1. This example uses the single argument form of the companion object's ofInputStream() method to read the PrefixMap file "example.pmap". The code may be used outside of Hadoop when it is known that the key is an IPBlock.

      val stream = new java.io.FileInputStream("example.pmap")
val pmapresult = PrefixMapReader.ofInputStream(stream)
val pmap = pmapresult match {
  case PrefixMapResult.IPBlockPrefixMap(iter) => iter
  case _ => null
}
pmap.hasNext
val (ipblock, name) = pmap.next()
    See also

    the companion object for more details

  14. sealed abstract class PrefixMapResult extends AnyRef

    Return type for the ofInputStream() method of the companion object to the PrefixMapReader class.

  15. case class RWRec(startTime: Instant, elapsed: Duration, sPort: Port, dPort: Port, protocol: Protocol, flowType: FlowType, sensor: Sensor, flags: TCPFlags, initFlags: TCPFlags, restFlags: TCPFlags, tcpState: TCPState, application: Port, memo: Short, input: SNMPInterface, output: SNMPInterface, packets: Long, bytes: Long, sIP: IPAddress, dIP: IPAddress, nhIP: IPAddress) extends Product with Serializable

    A SiLK flow record.

    A SiLK flow record.

    Note that in addition to the fields of the case class, some derived fields are also provided. (See below.)

    startTime

    The instant that the first packet in this flow was observed.

    elapsed

    The duration between the instants the first and last packets in this flow were observed.

    sPort

    The source port of this flow, or zero if this flow is neither a TCP nor a UDP flow.

    dPort

    The destination port of this flow, or zero if this flow is neither a TCP nor a UDP flow.

    protocol

    The IP protocol of this flow.

    flowType

    The SiLK flow type (class and type) of this flow, or FlowType(0) if unknown.

    sensor

    The SiLK sensor that observed this flow, or Sensor(0) if unknown.

    flags

    The union of all TCP flags observed in this flow, or TCPFlags(0) if this flow is not a TCP flow.

    initFlags

    The flags observed in the initial packet of this TCP flow, or TCPFlags(0) if this flow is not a TCP flow or if extended flags are not available.

    restFlags

    The union of all TCP flags observed after the initial packet of this flow, or TCPFlags(0) if this flow is not a TCP flow or if extended flags are not available.

    tcpState

    Flags relating to the observed status of this flow, including whether extended TCP flags are available. See TCPState for more details.

    application

    The detected application of this flow, expressed as the common port number for that application, or Port(0) if no application was detected.

    memo

    A Short value stored as a memo on this flow, or zero if no such memo has been set.

    input

    The input SNMP routing interface for this flow, or SNMPInterface(0) if routing information is not available.

    output

    The output SNMP routing interface for this flow, or SNMPInterface(0) if routing information is not available.

    packets

    The number of IP packets observed in this flow.

    bytes

    The number of bytes in packets observed in this flow.

    sIP

    The source IP address of packets in this flow.

    dIP

    The destination IP address of packets in this flow.

    nhIP

    The next-hop IP address of packets in this flow, or IPAddress("0.0.0.0") or IPAddress("::") if routing information is not available.

  16. class RWRecReader extends Iterator[RWRec]

    A reader of SiLK flow records.

    A reader of SiLK flow records. This is usable as an Iterator over RWRec records.

    See the companion object for more details.

  17. class RWRecWriter extends AnyRef

    A writer of binary SiLK RWRec files that are readable by SiLK.

    A writer of binary SiLK RWRec files that are readable by SiLK.

    Example:

    1. This example reads the contents of "example.rw" and writes it to "copy.rw":

      val in = new java.io.FileInputStream("example.rw")
val out = new java.io.FileOutputStream("copy.rw")
val reader = RWRecReader.ofInputStream(in)
val writer = RWRecWriter.toOutputStream(out)
writer.append(reader)
writer.close()
    See also

    the companion object for more details

  18. final case class Sensor(toShort: Short) extends AnyVal with Product with Serializable

    A SiLK sensor, as represented by a Short value.

    A SiLK sensor, as represented by a Short value.

    toShort

    The short value representing this sensor.

  19. case class SilkConfig(version: Option[Int], defaultClassName: Option[String], packingLogicPath: Option[String], pathFormat: String, groups: Map[String, GroupConfig], sensors: SensorMap, classes: Map[String, ClassConfig]) extends Product with Serializable

    SiLK data spool configuration.

    SiLK data spool configuration.

    version

    The version of the config file format used.

    defaultClassName

    The default class to be examined if none is specified.

    packingLogicPath

    The path to the plugin to be loaded by the packer for determining where to pack flows.

    pathFormat

    The format used for filenames in the data spool.

    groups

    The sensor groups defined in this configuration.

    sensors

    The sensors defined in this configuration, usable as a value of type Map[Sensor, SensorConfig].

    classes

    The classes defined in this configuration.

  20. class SilkDataFormatException extends IOException

    Signals that a SiLK input file has been found to be malformed.

  21. final case class SilkVersion(toInt: Int) extends AnyVal with Product with Serializable

    A SiLK version number, as represented by an Int value encoding the major version, minor version, and patch numbers.

    A SiLK version number, as represented by an Int value encoding the major version, minor version, and patch numbers.

    toInt

    The int value representing this SiLK version.

  22. final case class TCPState(toByte: Byte) extends AnyVal with Product with Serializable

    A SiLK TCP state flag vector, encoding various properties of TCP packets in a TCP flow, as encoded in a Byte value.

    A SiLK TCP state flag vector, encoding various properties of TCP packets in a TCP flow, as encoded in a Byte value.

    toByte

    The byte value representing this state vector.

Value Members

  1. case object BagDataType extends ShortEnum[BagDataType] with Product with Serializable
  2. object BagReader

    The BagReader object provides support for creating a BagReader.

  3. object BagResult

    The object returned by the ofInputStream() method of the companion object to the BagReader class.

  4. object BagWriter

    The BagWriter companion object provides support for creating an BagWriter.

  5. case object CompressionMethod extends ByteEnum[CompressionMethod] with Product with Serializable
  6. object FileFormat extends ByteEnum[FileFormat]
  7. object FlowType extends Serializable
  8. object Header extends Serializable
  9. object HeaderEntry
  10. object IPSetReader

    The IPSetReader object provides support for creating an IPSetReader.

  11. object IPSetWriter

    The IPSetWriter companion object provides support for creating an IPSetWriter.

  12. object PrefixMapProtocolPortPair extends Serializable

    The PrefixMapProtocolPortPair object provides support for creating a PrefixMapProtocolPortPair.

  13. object PrefixMapReader

    The PrefixMapReader object provides support for creating an PrefixMapReader.

  14. object PrefixMapResult

    The object returned by the ofInputStream() method of the companion object to the PrefixMapReader class.

  15. object RWRecReader
  16. object RWRecWriter

    The RWRecWriter companion object provides support for creating an RWRecWriter.

  17. object Sensor extends Serializable
  18. object SilkConfig extends Serializable
  19. object SilkVersion extends Serializable
  20. object TCPState extends Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped