Packages

  • package root

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

    Please see the documentation for the individual packages for more details on their use.

    Scala Packages

    These packages are useful in Scala code without involving Spark:

    org.cert.netsa.data

    This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

    org.cert.netsa.io.ipfix

    The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

    org.cert.netsa.io.silk

    To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

    org.cert.netsa.util

    The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

    Spark Packages

    These packages require the use of Apache Spark:

    org.cert.netsa.mothra.datasources

    Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

    The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

    org.cert.netsa.mothra.analysis

    A grab-bag of analysis helper functions and example analyses.

    org.cert.netsa.mothra.functions

    This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package cert
    Definition Classes
    org
  • package netsa
    Definition Classes
    cert
  • package io
    Definition Classes
    netsa
  • package silk

    SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

    SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

    RWRec is the type of SiLK flow records.

    You can use RWRecReader to read SiLK files from Scala, including compressed files if Hadoop native libraries are available. For example:

    import org.cert.netsa.io.silk.RWRecReader
import java.io.FileInputStream

val inputFile = new FileInputStream("path/to/silk/rw/file")

for ( rec <- RWRecReader.ofInputStream(inputFile) ) {
  println(rec.sIP)
}
    Definition Classes
    io
    See also

    org.cert.netsa.mothra.datasources.silk.flow for working with SiLK data in Spark using the Mothra SiLK datasource.

  • package config
    Definition Classes
    silk
  • package io
    Definition Classes
    silk
  • BagDataType
  • BagReader
  • BagResult
  • BagWriter
  • CompressionMethod
  • FileFormat
  • FlowType
  • Header
  • HeaderEntry
  • IPSetReader
  • IPSetWriter
  • PrefixMapProtocolPortPair
  • PrefixMapReader
  • PrefixMapResult
  • RWRec
  • RWRecReader
  • RWRecWriter
  • Sensor
  • SilkConfig
  • SilkDataFormatException
  • SilkVersion
  • TCPState

case class RWRec(startTime: Instant, elapsed: Duration, sPort: Port, dPort: Port, protocol: Protocol, flowType: FlowType, sensor: Sensor, flags: TCPFlags, initFlags: TCPFlags, restFlags: TCPFlags, tcpState: TCPState, application: Port, memo: Short, input: SNMPInterface, output: SNMPInterface, packets: Long, bytes: Long, sIP: IPAddress, dIP: IPAddress, nhIP: IPAddress) extends Product with Serializable

A SiLK flow record.

Note that in addition to the fields of the case class, some derived fields are also provided. (See below.)

startTime

The instant that the first packet in this flow was observed.

elapsed

The duration between the instants the first and last packets in this flow were observed.

sPort

The source port of this flow, or zero if this flow is neither a TCP nor a UDP flow.

dPort

The destination port of this flow, or zero if this flow is neither a TCP nor a UDP flow.

protocol

The IP protocol of this flow.

flowType

The SiLK flow type (class and type) of this flow, or FlowType(0) if unknown.

sensor

The SiLK sensor that observed this flow, or Sensor(0) if unknown.

flags

The union of all TCP flags observed in this flow, or TCPFlags(0) if this flow is not a TCP flow.

initFlags

The flags observed in the initial packet of this TCP flow, or TCPFlags(0) if this flow is not a TCP flow or if extended flags are not available.

restFlags

The union of all TCP flags observed after the initial packet of this flow, or TCPFlags(0) if this flow is not a TCP flow or if extended flags are not available.

tcpState

Flags relating to the observed status of this flow, including whether extended TCP flags are available. See TCPState for more details.

application

The detected application of this flow, expressed as the common port number for that application, or Port(0) if no application was detected.

memo

A Short value stored as a memo on this flow, or zero if no such memo has been set.

input

The input SNMP routing interface for this flow, or SNMPInterface(0) if routing information is not available.

output

The output SNMP routing interface for this flow, or SNMPInterface(0) if routing information is not available.

packets

The number of IP packets observed in this flow.

bytes

The number of bytes in packets observed in this flow.

sIP

The source IP address of packets in this flow.

dIP

The destination IP address of packets in this flow.

nhIP

The next-hop IP address of packets in this flow, or IPAddress("0.0.0.0") or IPAddress("::") if routing information is not available.

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. RWRec
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new RWRec(startTime: Instant, elapsed: Duration, sPort: Port, dPort: Port, protocol: Protocol, flowType: FlowType, sensor: Sensor, flags: TCPFlags, initFlags: TCPFlags, restFlags: TCPFlags, tcpState: TCPState, application: Port, memo: Short, input: SNMPInterface, output: SNMPInterface, packets: Long, bytes: Long, sIP: IPAddress, dIP: IPAddress, nhIP: IPAddress)

    startTime

    The instant that the first packet in this flow was observed.

    elapsed

    The duration between the instants the first and last packets in this flow were observed.

    sPort

    The source port of this flow, or zero if this flow is neither a TCP nor a UDP flow.

    dPort

    The destination port of this flow, or zero if this flow is neither a TCP nor a UDP flow.

    protocol

    The IP protocol of this flow.

    flowType

    The SiLK flow type (class and type) of this flow, or FlowType(0) if unknown.

    sensor

    The SiLK sensor that observed this flow, or Sensor(0) if unknown.

    flags

    The union of all TCP flags observed in this flow, or TCPFlags(0) if this flow is not a TCP flow.

    initFlags

    The flags observed in the initial packet of this TCP flow, or TCPFlags(0) if this flow is not a TCP flow or if extended flags are not available.

    restFlags

    The union of all TCP flags observed after the initial packet of this flow, or TCPFlags(0) if this flow is not a TCP flow or if extended flags are not available.

    tcpState

    Flags relating to the observed status of this flow, including whether extended TCP flags are available. See TCPState for more details.

    application

    The detected application of this flow, expressed as the common port number for that application, or Port(0) if no application was detected.

    memo

    A Short value stored as a memo on this flow, or zero if no such memo has been set.

    input

    The input SNMP routing interface for this flow, or SNMPInterface(0) if routing information is not available.

    output

    The output SNMP routing interface for this flow, or SNMPInterface(0) if routing information is not available.

    packets

    The number of IP packets observed in this flow.

    bytes

    The number of bytes in packets observed in this flow.

    sIP

    The source IP address of packets in this flow.

    dIP

    The destination IP address of packets in this flow.

    nhIP

    The next-hop IP address of packets in this flow, or IPAddress("0.0.0.0") or IPAddress("::") if routing information is not available.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val application: Port
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. val bytes: Long
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  8. val dIP: IPAddress
  9. val dPort: Port
  10. val elapsed: Duration
  11. def endTime: Instant

    The instant that the last packet in this flow was observed.

  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  14. val flags: TCPFlags
  15. val flowType: FlowType
  16. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def icmpCode: ICMPCode

    The code of this ICMP flow, or garbage if this is a TCP or UDP flow, or ICMPType(0) if this is not an ICMP, TCP, or UDP flow.

  18. def icmpType: ICMPType

    The type of this ICMP flow, or garbage if this is a TCP or UDP flow, or ICMPType(0) if this is not an ICMP, TCP, or UDP flow.

  19. val initFlags: TCPFlags
  20. val input: SNMPInterface
  21. def isIPv6: Boolean

    True if this flow's addresses are IPv6 addresses.

  22. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  23. val memo: Short
  24. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. val nhIP: IPAddress
  26. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  28. val output: SNMPInterface
  29. val packets: Long
  30. def productElementNames: Iterator[String]
    Definition Classes
    Product
  31. val protocol: Protocol
  32. val restFlags: TCPFlags
  33. val sIP: IPAddress
  34. val sPort: Port
  35. val sensor: Sensor
  36. val startTime: Instant
  37. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  38. val tcpState: TCPState
  39. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  40. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  41. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Derived Members

Ungrouped