Packages

package root

This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

Please see the documentation for the individual packages for more details on their use.

Scala Packages

These packages are useful in Scala code without involving Spark:

org.cert.netsa.data

This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

org.cert.netsa.data.net - types for working with network data
org.cert.netsa.data.time - types for working with time data
org.cert.netsa.data.unsigned - types for working with unsigned integral values

org.cert.netsa.io.ipfix

The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

org.cert.netsa.io.silk

To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

org.cert.netsa.util

The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

Spark Packages

These packages require the use of Apache Spark:

org.cert.netsa.mothra.datasources

Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

org.cert.netsa.mothra.analysis

A grab-bag of analysis helper functions and example analyses.

org.cert.netsa.mothra.functions

This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

Definition Classes: root

package org

Definition Classes: root

package cert

Definition Classes: org

package netsa

Definition Classes: cert

package mothra

Definition Classes: netsa

package datasources

This package contains the Mothra datasources, along with mechanisms for working with those datasources.

This package contains the Mothra datasources, along with mechanisms for working with those datasources. The primary novel feature of these datasources is the fields mechanism.

To use the IPFIX or SiLK data sources, you can use the following methods added by the implicit CERTDataFrameReader on DataFrameReader after importing from this package:

import org.cert.netsa.mothra.datasources._
val silkDF = spark.read.silkFlow()                                    // to read from the default SiLK repository
val silkRepoDF = spark.read.silkFlow(repository="...")                // to read from an alternate SiLK repository
val silkFilesDF = spark.read.silkFlow("/path/to/silk/files")          // to read from loose SiLK files
val ipfixDF = spark.read.ipfix(repository="/path/to/mothra/data/dir") // for packed Mothra IPFIX data
val ipfixS3DF = spark.read.ipfix(s3Repository="bucket-name")          // for packed Mothra IPFIX data from an S3 bucket
val ipfixFilesDF = spark.read.ipfix("/path/to/ipfix/files")           // for loose IPFIX files

(The additional methods are defined on the implicit class CERTDataFrameReader.)

Using the fields method allows you to configure which SiLK or IPFIX fields you wish to retrieve. (This is particularly important for IPFIX data, as IPFIX files may contains many many possible fields organized in various ways.)

import org.cert.netsa.mothra.datasources._
val silkDF = spark.read.fields("sIP", "dIP").silkFlow(...)
val ipfixDF = spark.read.fields("sourceIPAddress", "destinationIPAddress").ipfix(...)

Both of these dataframes will contain only the source and destination IP addresses from the specified data sources. You may also provide column names different from the source field names:

val silkDF = spark.read.fields("server" -> "sIP", "client" -> "dIP").silkFlow(...)
val ipfixDF = spark.read.fields("server" -> "sourceIPAddress", "client" -> "destinationIPAddress").ipfix(...)

You may also mix the mapped and the default names in one call:

val df = spark.read.fields("sIP", "dIP", "s" -> "sensor").silkFlow(...)

Definition Classes: mothra
See also: IPFIX datasource
SiLK flow datasource

package ipfix

A data source as defined by the Spark Data Source API for reading IPFIX records from Mothra data spools and from loose files.

You can use this by importing org.cert.netsa.mothra.datasources._ like this:

import org.cert.netsa.mothra.datasources._
val df1 = spark.read.ipfix("path/to/mothra/data/dir") // for packed Mothra IPFIX data
val df2 = spark.read.ipfix("path/to/ipfix/files")     // for loose IPFIX files

The IPFIX datasource uses the fields mechanism from org.cert.netsa.mothra.datasources. You can make use of this mechanism like these examples:

import org.cert.netsa.mothra.datasources._
val df1 = spark.read.fields(
  "startTime", "endTime", "sourceIPAddress", "destinationIPAddress"
).ipfix(...)

val df2 = spark.read.fields(
  "startTime", "endTime", "TOS" -> "ipClassOfService"
).ipfix(...)

with arbitrary sets of fields and field name mappings.

Default Fields

The default set of fields (defined in IPFIXFields.default) is:

"startTime" -> "func:startTime"
"endTime" -> "func:endTime"
"sourceIPAddress" -> "func:sourceIPAddress"
"sourcePort" -> "func:sourcePort"
"destinationIPAddress" -> "func:destinationIPAddress"
"destinationPort" -> "func:destinationPort"
"protocolIdentifier"
"observationDomainId"
"vlanId"
"reverseVlanId"
"silkAppLabel"
"packetCount" -> "packetTotalCount|packetDeltaCount"
"reversePacketCount" -> "reversePacketTotalCount|reversePacketDeltaCount"
"octetCount" -> "octetTotalCount|octetDeltaDcount"
"reverseOctetCount" -> "reverseOctetTotalCount|reverseOctetDeltaCount"
"initialTCPFlags"
"reverseInitialTCPFlags"
"unionTCPFlags"
"reverseUnionTCPFlags"

Some of these defaults are defined simply as IPFIX Information Elements. For example, "protocolIdentifier" and "vlanId" are exactly the Information Elements that are named. No "right-hand-side" is given for these definitions, because the name of the field is the same as the name of the Information Element.

Others have simple expressions. For example, packetCount is defined as "packetTotalCount|packetDeltaCount". This expressions means that the value should be found from the packetTotalCount IE, or if that is not set from the packetDeltaCount IE. This allows this field to be used regardless of which Information Element contains the data.

Some others are derived in more complex ways from basic IPFIX fields. For example, the startTime field is produced using "func:startTime", which runs the "gauntlet of time" to determine the start time for a flow by whatever means possible. Other time fields are similarly defined.

Some of the "func:..." fields are actually quite simple. For example, "func:sourceIPAddress", practically speaking, is the same as "sourceIPv4Address|sourceIPv6Address". However, these fields are defined using the func: extension mechanism so that partitioning on them is possible. (This restriction may be lifted in a future Mothra version.)

Field Types

The mappings between IPFIX types and Spark types are:

octetArray → Array[Byte]
unsigned8 → Short
unsigned16 → Int
unsigned32 → Long
unsigned64 → Long
signed8 → Byte
signed16 → Short
signed32 → Int
signed64 → Long
float32 → Float
float64 → Double
boolean → Boolean
macAddress → String
string → String
dateTimeSeconds → Timestamp
dateTimeMilliseconds → Timestamp
dateTimeMicroseconds → Timestamp
dateTimeNanoseconds → Timestamp
ipv4Address → String
ipv6Address → String

IPFIX's basicList, subTemplateList, and subTemplateMultiList data types are handled differently.

Field Expressions

As noted above, field expressions may contain simple IPFIX Information Element names, or collections of names separated by pipe characters to indicate taking the first matching choice. This language has a number of other capabilities which are documented for now in the IPFIX field parser object.

Functional Fields

A number of pre-defined "functional fields" are available. Some of these combine other information elements in ways that the expression language cannot (applying the so-called "gauntlet of time", for example). Others provide support for the Mothra repository partitioning system. And finally, a few are for debugging purposes and provide high-level overviews of IPFIX records or point to file locations on disk.

Function fields are all defined and described in the org.cert.netsa.mothra.datasources.ipfix.fields.func package.

Definition Classes: datasources

package fields

Most of these classes and traits relate to the definition of IPFIX fields as IPFIX record processing objects.

The IPFIXFieldParsing object defines the parser used for IPFIX field expressions, and includes the documentation for that language.

Other mechanisms, including implementations of the IPFIXField trait, provide the ability to define new "function" fields and register them into the Func registry. This is an experimental capabilty and is likely to be deprecated and then removed from public access in the future.

Definition Classes: ipfix
Note: This is an experimental interface and is likely to be removed or made private in a future version.

SlicingRelationSparkVerImpl

org.cert.netsa.mothra.datasources.ipfix

IPFIXFields

object IPFIXFields

Collections of fields relevant to IPFIX data sources.

These are primarily the fields produced by YAF version 3 and up, although the same fields are produced by other tools.

These collections and the individual fields within them may be passed to spark.read.fields directly and all of the fields in the collection will be included in the output.

If using a REPL with tab-completion, many of the provided symbols may be accessed using tab completion. In addition, most of the fields and field collections provide useful information if converted to strings. For example:

scala> import org.cert.netsa.mothra.datasources.ipfix.IPFIXFields
import org.cert.netsa.mothra.datasources.ipfix.IPFIXFields

scala> println(IPFIXFields.default)
// Most common fields for flow-based traffic analysis
FieldsSpec( // #default
  // Core flow fields, including time and the 5-tuple
  FieldsSpec( // #core
    // Timestamp of the first packet of this Flow
    "startTime" -> "func:startTime",
    // Timestamp of the final packet of this Flow
    "endTime" -> "func:endTime",
    // IPv4 or IPv6 source for incoming packets in this Flow
    "sourceIPAddress" -> "func:sourceIPAddress",
    // Any source port identifier from the transport header
    "sourcePort" -> "func:sourcePort",
    // IPv4 or IPv6 destination for incoming packets in this Flow
    "destinationIPAddress" -> "func:destinationIPAddress",
    // Any destination port identifier from the transport header
    "destinationPort" -> "func:destinationPort",
    // Protocol number from the IP packet header
    "protocolIdentifier"
  ),
  ...
)

This is printing out all of the fields in the "default" collection. The fields each include their definition as an IPFIX field expression and a short description. Some fields are grouped into larger collections which are also given names and descriptions.

Each of the field names like "startTime" may be used as an index to pull out a specific field from the collection:

scala> println(IPFIXFields.default("startTime"))
FieldsSpec(
  // Timestamp of the first packet of this Flow
  "startTime" -> "func:startTime"
)

And the collection names such as "#core" may also be used to pull out sub-collections:

scala> println(IPFIXFields.default("#tcpflags"))
// Fields for TCP protocol flag information
FieldsSpec( // #tcpflags
  // TCP flags of first incoming packet of a TCP Flow
  "initialTCPFlags",
  // TCP flags of first outgoing packet of a TCP Flow
  "reverseInitialTCPFlags",
  // Union of TCP flags of all incoming packets after the first
  "unionTCPFlags",
  // Union of TCP flags of all outgoing packets after the first
  "reverseUnionTCPFlags"
)

Linear Supertypes

AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

IPFIXFields
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Value Members

final def !=(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def ##: Int
Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0
Definition Classes
Any
def clone(): AnyRef
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.CloneNotSupportedException]) @native()
val core: FieldsSpec
Core IPFIX flow fields, including time and the 5-tuple.
Core IPFIX flow fields, including time and the 5-tuple. startTime, endTime, sourceIPAddress, sourcePort, destinationIPAddress, destinationPort, and protocolIdentifier.
val counts: FieldsSpec
Fields for basic volume statistics.
Fields for basic volume statistics. packetCount, reversePacketCount, octetCount, and reverseOctetCount.
val counts_uniflow: FieldsSpec
Fields for basic volume statistics for uniflow.
Fields for basic volume statistics for uniflow. packetCount and octetCount.
val default: FieldsSpec
Default flow fields, including core fields, label fields, counts, and TCP flags, all as described above.
val default_uniflow: FieldsSpec
Default uniflow fields, including core fields, and uniflow versions of label fields, counts, and TCP flags, all as described above.
final def eq(arg0: AnyRef): Boolean
Definition Classes
AnyRef
def equals(arg0: AnyRef): Boolean
Definition Classes
AnyRef → Any
val everything: FieldsSpec
Accessible collection of all fields useful for working with YAF3 in particular.
Accessible collection of all fields useful for working with YAF3 in particular. You should not use all of these at once, but select individual fields everything("fieldName") or collections everything("#collectionName") that you wish to use. You can select multiple items at once like everything("field1", "#coll1", "field2", ...). You can use the facilities described at org.cert.netsa.mothra.datasources to add your own fields. And finally, note that if the same field occurs more than once only the first definition will be kept.
def finalize(): Unit
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.Throwable])
final def getClass(): Class[_ <: AnyRef]
Definition Classes
AnyRef → Any
Annotations
@native()
def hashCode(): Int
Definition Classes
AnyRef → Any
Annotations
@native()
final def isInstanceOf[T0]: Boolean
Definition Classes
Any
val label: FieldsSpec
Fields used for flow labeling and partitioning.
Fields used for flow labeling and partitioning. observationDomainId, vlanId, reverseVlanId, and silkAppLabel.
val label_uniflow: FieldsSpec
Fields used for flow labeling and partitioning for uniflow.
Fields used for flow labeling and partitioning for uniflow. observationDomainId, vlanId, silkAppLabel.
val legacy: FieldsSpec
Older default definitions, which are no longer recommanded.
Older default definitions, which are no longer recommanded. These names are closer to what users of SiLK might expect, but this hides some of the differences in the IPFIX format and the bidirectional flow data commonly used in IPFIX.
final def ne(arg0: AnyRef): Boolean
Definition Classes
AnyRef
final def notify(): Unit
Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit
Definition Classes
AnyRef
Annotations
@native()
final def synchronized[T0](arg0: => T0): T0
Definition Classes
AnyRef
val tcpflags: FieldsSpec
Fields for TCP flag information.
Fields for TCP flag information. initialTCPFlags, reverseInitialTCPFlags, unionTCPFlags, and reverseUnionTCPFlags.
val tcpflags_uniflow: FieldsSpec
Fields for TCP flag information for uniflow.
Fields for TCP flag information for uniflow. initialTCPFlags and unionTCPFlags.
def toString(): String
Definition Classes
AnyRef → Any
final def wait(): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long, arg1: Int): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException]) @native()
object cert_tool extends Multiple
Miscellaneous internal templates for CERT NetSA tools.
Miscellaneous internal templates for CERT NetSA tools.
Note: These are *not* included in everything.
object dpi extends Multiple
Flow fields for deep packet inspection data, broken down by protocol.
object plugins extends Multiple
Flow fields for fields provided by YAF plugins, broken down by plugin.

Packages

Scala Packages

org.cert.netsa.data

org.cert.netsa.io.ipfix

org.cert.netsa.io.silk

org.cert.netsa.util

Spark Packages

org.cert.netsa.mothra.datasources

org.cert.netsa.mothra.analysis

org.cert.netsa.mothra.functions

Default Fields

Field Types

Field Expressions

Functional Fields

IPFIXFields

object IPFIXFields

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

Scala Packages

org.cert.netsa.data

org.cert.netsa.io.ipfix

org.cert.netsa.io.silk

org.cert.netsa.util

Spark Packages

org.cert.netsa.mothra.datasources

org.cert.netsa.mothra.analysis

org.cert.netsa.mothra.functions

Default Fields

Field Types

Field Expressions

Functional Fields

IPFIXFields

object IPFIXFields

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

IPFIXFields