Packages

package root

This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

Please see the documentation for the individual packages for more details on their use.

Scala Packages

These packages are useful in Scala code without involving Spark:

org.cert.netsa.data

This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

org.cert.netsa.data.net - types for working with network data
org.cert.netsa.data.time - types for working with time data
org.cert.netsa.data.unsigned - types for working with unsigned integral values

org.cert.netsa.io.ipfix

The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

org.cert.netsa.io.silk

To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

org.cert.netsa.util

The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

Spark Packages

These packages require the use of Apache Spark:

org.cert.netsa.mothra.datasources

Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

org.cert.netsa.mothra.analysis

A grab-bag of analysis helper functions and example analyses.

org.cert.netsa.mothra.functions

This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

Definition Classes: root

package org

Definition Classes: root

package cert

Definition Classes: org

package netsa

Definition Classes: cert

package io

Definition Classes: netsa

package ipfix

The ipfix package provides classes and objects for reading and writing IPFIX data.

Class / Object Overview

(For a quick overview of the IPFIX format, see the end of this description.)

The Message trait describes the attributes of an IPFIX message, and the CollectedMessage class and object are implementations of that trait when reading data. (Record export does not create specific Message instance.)

The IpfixSet abstract class and object hold the attributes of a Set. The TemplateSet class may represent a Template Set or an Options Template Set.

The Template class and object are used to represent a Template Record or an Options Template Record.

The IEFieldSpecifier class and object represent a Field Specifier within an existing Template. To search for a field within a Template, the user of the ipfix package creates a FieldSpec (the companion object) and attempts to find it within a Template.

The Field Specifier uses the numeric Identifier to identify an Information Element, and an Element is represented by the InfoElement class and object. The InfoModel class and object represent the Information Model.

To describe the attributes of an InfoElement, several support classes are defined: DataTypes is an enumeration that describes the type of data that the element contains, and DataType is a class that extracts a Field Value with that DataType. IESemantics describes the data semantics of an Information Element (e.g., a counter, an identifier, a set of flags), and IEUnits describes its units.

The Data Set is represented by the RecordSet class and object.

A Data Record is represented by the Record abstract class. This class has three subclasses:

The CollectedRecord class and object are its implementation when reading data. Its members are always referenced by numeric position.
The ArrayRecord (I do not like this name) and object may be used to build a Record from Scala objects; its fields are also referenced by numeric position.
ExportRecord is an abstract class that also supports building a Record from Scala objects. The user extends the class and uses the IPFIXExtract annotation to mark the members of the subclass that are to be used when writing the Record.

A user-defined class that extends Fillable trait may use the Record's fill() method to copy fields from a Record to the user's class. It also uses the IPFIXExtract annotation.

A Structured Data Field Value in a Data Record is represented by the ListElement abstract class. That abstract class has three abstract subclasses, and each of those has two concrete subclasses (one for reading and one for writing):

The BasicList abstract class (object) has subclasses CollectedBasicList and ExportBasicList.
The SubTemplateList abstract class (object) has subclasses CollectedSubTemplateList and ExportSubTemplateList.
The SubTemplateMultiList abstract class (object) has subclasses CollectedSubTemplateMultiList and ExportSubTemplateMultiList.

Reading data

When reading data, a Record instance is returned by a RecordReader. The RecordReader uses a class that extends the MessageReader trait. The ipfix package includes two: ByteBufferMessageReader and StreamMessageReader.

A Session value represent an IPFIX session, which is part of a SessionGroup.

Writing data

For writing data, an instance of an ExportStream must be created using a Session and the destination FileChannel. The user adds Records or Templates to the ExportStream and they are written to the FileChannel.

Overview of IPFIX

An IPFIX stream is composed of Messages. Each Message has a 16-byte Message Header followed by one or more Sets. There are three types of Sets: A Data Set, a Template Set, and an Options Template Set.

Each Set has a 4-byte set header followed by one or more Records. A Data Set contains Data Records and a Template Set contains Template Records.

A Template Record describes the shape of the data that appears in a Data Record. A Template Record contains a 4-byte header followed by zero or more Field Specifiers. Each Field Specifier is either a 4-byte or an 8-byte value that describes a field in the Data Record.

A Field Specifier has two parts. The first is the numeric Information Element Identifier that is defined in an Information Model. The second is the number of octets the field occupies in the Data Record.

A Data Set contains one or more Data Records of the same type, where the type is determined by the Template Record that the Data Set Header refers to. Each Data Record contains one or more Field Values, where the order and length of the Field Values is given by the Template.

A Field Value in a Data Record may be a Structured Data. There are three types of Structured Data:

A Basic List contains one or more instances of a Single Information Element.
A SubTemplateList references a single Template ID, and it contains one or more Records that match that Template.
The SubTemplateMultiList contains a series of Template IDs and Records that match that Template ID.

An IPFIX stream exists in a Transport Session, where a Transport Session is part of a Session Group. All Sessions in a Session Group use the same Transport Protocol, and only differ in the numeric Observation Domain that is part of the Message Header.

Definition Classes: io

package datatype

Definition Classes: ipfix

package testing

Definition Classes: ipfix

package util

Definition Classes: ipfix

ArrayRecord

BasicList

BasicListFieldExtractor

ByteBufferMessageReader

CollectedBasicList

CollectedMessage

CollectedRecord

CollectedSubTemplateList

CollectedSubTemplateMultiList

ExportStreamTemplatesFirst

ExportSubTemplateList

ExportSubTemplateMultiList

IPFIXRuntimeException

Identifier

IllegalBasicListException

IllegalFieldSpecifierException

IllegalInfoElementAttributeException

IllegalListException

IllegalSetException

IllegalSubTemplateListException

IllegalSubTemplateMultiListException

IllegalTemplateRecordException

InvalidIPFIXMessageException

InvalidInfoElementException

MessageTooLongException

NoTemplateException

OutOfTemplateIdsException

SessionMismatchException

TruncatedReadException

org.cert.netsa.io.ipfix

Session

Companion object Session

sealed class Session extends StrictLogging

A session contains information about a single IPFIX session, from a single observation Domain. Specifically, a session takes care of handling the template library that maps template IDs to templates, and it keeps track of message sequence numbers.

The Session class's contructor is not accessible. Instead these classes must be used: The StreamSession is used when reading data from a file or from a TCP stream. The DatagramSession is used when reading data over UDP.

See also: The companion object for additional details.

Linear Supertypes

StrictLogging, AnyRef, Any

Known Subclasses

DatagramSession, StreamSession

Ordering

Alphabetic
By Inheritance

Inherited

Session
StrictLogging
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Instance Constructors

new Session(group: SessionGroup, id: Int)
Auxiliary constructor
Auxiliary constructor
Creates a new session from the given session group and observation domain ID. The new session creates a new information model that inherits from the SessionGroup's information model.
group
the session group that owns the session
id
the observation domain ID
Attributes
protected

Value Members

final def !=(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def ##: Int
Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean
Definition Classes
AnyRef → Any
def add(template: Template, tid: Int): Option[Template]
Adds a Template to this Session using tid as the templateId and replacing the Template that previously used that ID, if any.
Adds a Template to this Session using tid as the templateId and replacing the Template that previously used that ID, if any. If a Template is replaced, it is returned.
To add template only if it is not already present, use getOrAdd.
If template.size is 0, withdraws and returns the Template whose ID is tid. The Template if not withdrawn if this is a DatagramSession.
Exceptions thrown
Session.TemplateInterferenceException when a template is replaced that may not be replaced, by policy.
java.lang.RuntimeException when the Template's session is not this session.
def addTemplateMetadata(metadata: TemplateMetadata): Unit
Updates the metadata for a particular template in this Session.
Updates the metadata for a particular template in this Session. Ignores metadata if its template ID is invalid.
final def apply(template: Template): Int
The ID used by template within this Session.
The ID used by template within this Session.
Since
1.3.1
Exceptions thrown
NoTemplateException if the Session does not know about that Template.
See also
getId
final def apply(tid: Int): Template
The Template from the template library with this template ID.
The Template from the template library with this template ID.
Since
1.3.1
Exceptions thrown
NoTemplateException if the Session does not have a Template with that ID.
See also
getTemplate
final def asInstanceOf[T0]: T0
Definition Classes
Any
def clone(): AnyRef
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.CloneNotSupportedException]) @native()
final def eq(arg0: AnyRef): Boolean
Definition Classes
AnyRef
def equals(arg0: AnyRef): Boolean
Definition Classes
AnyRef → Any
var expectedSequence: Long
The sequence number that is expected for the next message in this session.
The sequence number that is expected for the next message in this session.
It is value a value between 0 and 2^32-1, or -1 if no message has yet been observed in this session.
def finalize(): Unit
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.Throwable])
final def findUnusedId(): Option[Int]
Find a template ID not currently being used by this session.
Find a template ID not currently being used by this session. Returns None when the Session has no available IDs. (Not necessarily deterministic.)
returns
a currently unused template ID
final def getClass(): Class[_ <: AnyRef]
Definition Classes
AnyRef → Any
Annotations
@native()
final def getId(template: Template): Option[Int]
Gets the ID for the Template within this session as an Option.
def getOrAdd(template: Template, tid: Option[Int] = None): Int
Returns the ID for template in this Session, adding template to this Session if it does not exist.
Returns the ID for template in this Session, adding template to this Session if it does not exist.
If template must be added to the Session, an arbitrary ID is used when tid is None. If tid is not None, that ID is used if possible; otherwise an arbitrary ID is used. Use the add method if template must use a particular ID.
Takes no action and returns 0 if template.size is 0.
template
The Template to get or add
tid
An optional ID for template if it does not exist in the Session
returns
The ID of template in this Session
def getPersistentSession(): Session
Returns a read-only copy of this session that will not change.
Returns a read-only copy of this session that will not change.
returns
the persistent session
final def getTemplate(tid: Int): Option[Template]
Gets a Template from the template library based on the template ID as an Option.
Gets a Template from the template library based on the template ID as an Option.
tid
the template ID
returns
The template associated with the template ID, or None
def getTemplateMetadata(tid: Int): Option[TemplateMetadata]
Gets the metadata for the Template whose ID is tid as an Option.
def hashCode(): Int
Definition Classes
AnyRef → Any
Annotations
@native()
final var idGetTmpl: TrieMap[Int, Template]
Maps a templateId to a Template.
Maps a templateId to a Template. "protected" since ReadOnlySession needs access
Attributes
protected
final val infoModel: InfoModel
final def isInstanceOf[T0]: Boolean
Definition Classes
Any
final def iterator: Iterator[Template]
Returns an Iterator over the Templates in the Session.
val logger: Logger
Attributes
protected
Definition Classes
StrictLogging
final def ne(arg0: AnyRef): Boolean
Definition Classes
AnyRef
final def noteMessage(m: Message): Unit
Check for missing or out of sequence records.
Check for missing or out of sequence records. If found, invoke any SequenceCallback objects that have been registered and log a message. A log message is not written if no SequenceCallback objects have been registered.
final def notify(): Unit
Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit
Definition Classes
AnyRef
Annotations
@native()
final val observationDomain: Int
def register(callback: SequenceCallback): Unit
Registers a new sequence callback with this session.
Registers a new sequence callback with this session. A session can have any number of callbacks registered to it. Each registered callback will be called whenever a message arrives with an unexpected sequence number.
callback
the sequence number callback
def register(callback: Iterable[TemplateCallback]): Unit
Registers multiple templates callbacks with this session.
def register(callback: TemplateCallback): Unit
Registers a new template callback with this session.
Registers a new template callback with this session. A session can have any number of callbacks registered to it. Each registered callback will be called whenever a template is added or removed from the session template library.
callback
the template callback
def remove(template: Template): Option[Template]
Removes a template from this session.
Removes a template from this session.
template
The template to remove
returns
the template that was removed, or None if none
def remove(tid: Int): Option[Template]
Removes a template from this session.
Removes a template from this session.
tid
the template ID of the template to remove
returns
the template that was removed, or None if none
final def synchronized[T0](arg0: => T0): T0
Definition Classes
AnyRef
final var templateMetadata: TrieMap[Int, TemplateMetadata]
Maps a templateId to its metadata
Maps a templateId to its metadata
Attributes
protected
final var tmplGetId: TrieMap[Template, Int]
Maps a Template to its templateId in this Session.
Maps a Template to its templateId in this Session.
Attributes
protected
def toString(): String
Definition Classes
AnyRef → Any
final val transport: AnyRef
final def wait(): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long, arg1: Int): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException]) @native()
def withdraw(t: Template, tid: Int): Option[Template]
Internal function that removes a template when a withdrawal template is received on the input stream.
Internal function that removes a template when a withdrawal template is received on the input stream. That is, when add() is called with a withdrawal template.
Attributes
protected

Packages

Scala Packages

org.cert.netsa.data

org.cert.netsa.io.ipfix

org.cert.netsa.io.silk

org.cert.netsa.util

Spark Packages

org.cert.netsa.mothra.datasources

org.cert.netsa.mothra.analysis

org.cert.netsa.mothra.functions

Class / Object Overview

Reading data

Writing data

Overview of IPFIX

Session

Companion object Session

sealed class Session extends StrictLogging

Instance Constructors

Value Members

Inherited from StrictLogging

Inherited from AnyRef

Inherited from Any

Ungrouped