Packages

  • package root

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

    Please see the documentation for the individual packages for more details on their use.

    Scala Packages

    These packages are useful in Scala code without involving Spark:

    org.cert.netsa.data

    This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

    org.cert.netsa.io.ipfix

    The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

    org.cert.netsa.io.silk

    To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

    org.cert.netsa.util

    The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

    Spark Packages

    These packages require the use of Apache Spark:

    org.cert.netsa.mothra.datasources

    Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

    The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

    org.cert.netsa.mothra.analysis

    A grab-bag of analysis helper functions and example analyses.

    org.cert.netsa.mothra.functions

    This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package cert
    Definition Classes
    org
  • package netsa
    Definition Classes
    cert
  • package io
    Definition Classes
    netsa
  • package ipfix

    The ipfix package provides classes and objects for reading and writing IPFIX data.

    The ipfix package provides classes and objects for reading and writing IPFIX data.

    Class / Object Overview

    (For a quick overview of the IPFIX format, see the end of this description.)

    The Message trait describes the attributes of an IPFIX message, and the CollectedMessage class and object are implementations of that trait when reading data. (Record export does not create specific Message instance.)

    The IpfixSet abstract class and object hold the attributes of a Set. The TemplateSet class may represent a Template Set or an Options Template Set.

    The Template class and object are used to represent a Template Record or an Options Template Record.

    The IEFieldSpecifier class and object represent a Field Specifier within an existing Template. To search for a field within a Template, the user of the ipfix package creates a FieldSpec (the companion object) and attempts to find it within a Template.

    The Field Specifier uses the numeric Identifier to identify an Information Element, and an Element is represented by the InfoElement class and object. The InfoModel class and object represent the Information Model.

    To describe the attributes of an InfoElement, several support classes are defined: DataTypes is an enumeration that describes the type of data that the element contains, and DataType is a class that extracts a Field Value with that DataType. IESemantics describes the data semantics of an Information Element (e.g., a counter, an identifier, a set of flags), and IEUnits describes its units.

    The Data Set is represented by the RecordSet class and object.

    A Data Record is represented by the Record abstract class. This class has three subclasses:

    1. The CollectedRecord class and object are its implementation when reading data. Its members are always referenced by numeric position.
    2. The ArrayRecord (I do not like this name) and object may be used to build a Record from Scala objects; its fields are also referenced by numeric position.
    3. ExportRecord is an abstract class that also supports building a Record from Scala objects. The user extends the class and uses the IPFIXExtract annotation to mark the members of the subclass that are to be used when writing the Record.

    A user-defined class that extends Fillable trait may use the Record's fill() method to copy fields from a Record to the user's class. It also uses the IPFIXExtract annotation.

    A Structured Data Field Value in a Data Record is represented by the ListElement abstract class. That abstract class has three abstract subclasses, and each of those has two concrete subclasses (one for reading and one for writing):

    1. The BasicList abstract class (object) has subclasses CollectedBasicList and ExportBasicList.
    2. The SubTemplateList abstract class (object) has subclasses CollectedSubTemplateList and ExportSubTemplateList.
    3. The SubTemplateMultiList abstract class (object) has subclasses CollectedSubTemplateMultiList and ExportSubTemplateMultiList.

    Reading data

    When reading data, a Record instance is returned by a RecordReader. The RecordReader uses a class that extends the MessageReader trait. The ipfix package includes two: ByteBufferMessageReader and StreamMessageReader.

    A Session value represent an IPFIX session, which is part of a SessionGroup.

    Writing data

    For writing data, an instance of an ExportStream must be created using a Session and the destination FileChannel. The user adds Records or Templates to the ExportStream and they are written to the FileChannel.

    Overview of IPFIX

    An IPFIX stream is composed of Messages. Each Message has a 16-byte Message Header followed by one or more Sets. There are three types of Sets: A Data Set, a Template Set, and an Options Template Set.

    Each Set has a 4-byte set header followed by one or more Records. A Data Set contains Data Records and a Template Set contains Template Records.

    A Template Record describes the shape of the data that appears in a Data Record. A Template Record contains a 4-byte header followed by zero or more Field Specifiers. Each Field Specifier is either a 4-byte or an 8-byte value that describes a field in the Data Record.

    A Field Specifier has two parts. The first is the numeric Information Element Identifier that is defined in an Information Model. The second is the number of octets the field occupies in the Data Record.

    A Data Set contains one or more Data Records of the same type, where the type is determined by the Template Record that the Data Set Header refers to. Each Data Record contains one or more Field Values, where the order and length of the Field Values is given by the Template.

    A Field Value in a Data Record may be a Structured Data. There are three types of Structured Data:

    1. A Basic List contains one or more instances of a Single Information Element.
    2. A SubTemplateList references a single Template ID, and it contains one or more Records that match that Template.
    3. The SubTemplateMultiList contains a series of Template IDs and Records that match that Template ID.

    An IPFIX stream exists in a Transport Session, where a Transport Session is part of a Session Group. All Sessions in a Session Group use the same Transport Protocol, and only differ in the numeric Observation Domain that is part of the Message Header.

    Definition Classes
    io
  • package util
    Definition Classes
    ipfix
  • BufferIterator
  • ListView
c

org.cert.netsa.io.ipfix.util

BufferIterator

abstract class BufferIterator[T] extends Iterator[T]

An abstract class to assist when Iterating over the contents of a ByteBuffer.

This class provides the hasNext and next() methods. Subclasses must implment the fromBuffer() method to process the data and return an object. The fromBuffer() method is only called when the ByteBuffer has at least some minimum number of bytes available.

Linear Supertypes
Iterator[T], IterableOnceOps[T, Iterator, Iterator[T]], IterableOnce[T], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BufferIterator
  2. Iterator
  3. IterableOnceOps
  4. IterableOnce
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new BufferIterator(buffer: ByteBuffer, minimum: Int)

    buffer

    The buffer to copy data out of.

    minimum

    The minimum number of bytes buffer must contain in order for the fromBuffer() method to be called.

Type Members

  1. class GroupedIterator[B >: A] extends AbstractIterator[Seq[B]]
    Definition Classes
    Iterator

Abstract Value Members

  1. abstract def fromBuffer(b: ByteBuffer): T

    Gets the next object from the buffer.

    Gets the next object from the buffer.

    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ++[B >: T](xs: => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator
    Annotations
    @inline()
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def addString(b: StringBuilder): b.type
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  6. final def addString(b: StringBuilder, sep: String): b.type
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  7. def addString(b: StringBuilder, start: String, sep: String, end: String): b.type
    Definition Classes
    IterableOnceOps
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. val buffer: ByteBuffer
  10. def buffered: BufferedIterator[T]
    Definition Classes
    Iterator
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  12. def collect[B](pf: PartialFunction[T, B]): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  13. def collectFirst[B](pf: PartialFunction[T, B]): Option[B]
    Definition Classes
    IterableOnceOps
  14. def concat[B >: T](xs: => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator
  15. def contains(elem: Any): Boolean
    Definition Classes
    Iterator
  16. def copyToArray[B >: T](xs: Array[B], start: Int, len: Int): Int
    Definition Classes
    IterableOnceOps
  17. def copyToArray[B >: T](xs: Array[B], start: Int): Int
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecatedOverriding()
  18. def copyToArray[B >: T](xs: Array[B]): Int
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecatedOverriding()
  19. def corresponds[B](that: IterableOnce[B])(p: (T, B) => Boolean): Boolean
    Definition Classes
    IterableOnceOps
  20. def count(p: (T) => Boolean): Int
    Definition Classes
    IterableOnceOps
  21. def distinct: Iterator[T]
    Definition Classes
    Iterator
  22. def distinctBy[B](f: (T) => B): Iterator[T]
    Definition Classes
    Iterator
  23. def drop(n: Int): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  24. def dropWhile(p: (T) => Boolean): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  25. def duplicate: (Iterator[T], Iterator[T])
    Definition Classes
    Iterator
  26. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  27. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  28. def exists(p: (T) => Boolean): Boolean
    Definition Classes
    IterableOnceOps
  29. def filter(p: (T) => Boolean): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  30. def filterNot(p: (T) => Boolean): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  31. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  32. def find(p: (T) => Boolean): Option[T]
    Definition Classes
    IterableOnceOps
  33. def flatMap[B](f: (T) => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  34. def flatten[B](implicit ev: (T) => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  35. def fold[A1 >: T](z: A1)(op: (A1, A1) => A1): A1
    Definition Classes
    IterableOnceOps
  36. def foldLeft[B](z: B)(op: (B, T) => B): B
    Definition Classes
    IterableOnceOps
  37. def foldRight[B](z: B)(op: (T, B) => B): B
    Definition Classes
    IterableOnceOps
  38. def forall(p: (T) => Boolean): Boolean
    Definition Classes
    IterableOnceOps
  39. def foreach[U](f: (T) => U): Unit
    Definition Classes
    IterableOnceOps
  40. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  41. def grouped[B >: T](size: Int): GroupedIterator[B]
    Definition Classes
    Iterator
  42. final def hasNext: Boolean

    Tests whether this iterator can provide another element.

    Tests whether this iterator can provide another element.

    returns

    true if a subsequent call to next() will yield an element, false otherwise.

    Definition Classes
    BufferIterator → Iterator
  43. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  44. def indexOf[B >: T](elem: B, from: Int): Int
    Definition Classes
    Iterator
  45. def indexOf[B >: T](elem: B): Int
    Definition Classes
    Iterator
  46. def indexWhere(p: (T) => Boolean, from: Int): Int
    Definition Classes
    Iterator
  47. def isEmpty: Boolean
    Definition Classes
    Iterator → IterableOnceOps
    Annotations
    @deprecatedOverriding()
  48. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  49. def isTraversableAgain: Boolean
    Definition Classes
    IterableOnceOps
  50. final def iterator: Iterator[T]
    Definition Classes
    Iterator → IterableOnce
    Annotations
    @inline()
  51. def knownSize: Int
    Definition Classes
    IterableOnce
  52. final def length: Int
    Definition Classes
    Iterator
    Annotations
    @inline()
  53. def map[B](f: (T) => B): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  54. def max[B >: T](implicit ord: Ordering[B]): T
    Definition Classes
    IterableOnceOps
  55. def maxBy[B](f: (T) => B)(implicit ord: Ordering[B]): T
    Definition Classes
    IterableOnceOps
  56. def maxByOption[B](f: (T) => B)(implicit ord: Ordering[B]): Option[T]
    Definition Classes
    IterableOnceOps
  57. def maxOption[B >: T](implicit ord: Ordering[B]): Option[T]
    Definition Classes
    IterableOnceOps
  58. def min[B >: T](implicit ord: Ordering[B]): T
    Definition Classes
    IterableOnceOps
  59. def minBy[B](f: (T) => B)(implicit ord: Ordering[B]): T
    Definition Classes
    IterableOnceOps
  60. def minByOption[B](f: (T) => B)(implicit ord: Ordering[B]): Option[T]
    Definition Classes
    IterableOnceOps
  61. def minOption[B >: T](implicit ord: Ordering[B]): Option[T]
    Definition Classes
    IterableOnceOps
  62. val minimum: Int
  63. final def mkString: String
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  64. final def mkString(sep: String): String
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  65. final def mkString(start: String, sep: String, end: String): String
    Definition Classes
    IterableOnceOps
  66. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  67. final def next(): T

    The next element for this iterator.

    The next element for this iterator.

    Definition Classes
    BufferIterator → Iterator
    Exceptions thrown

    java.util.NoSuchElementException when the iterator is depleted.

  68. def nextOption(): Option[T]
    Definition Classes
    Iterator
  69. def nonEmpty: Boolean
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecatedOverriding()
  70. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  71. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  72. def padTo[B >: T](len: Int, elem: B): Iterator[B]
    Definition Classes
    Iterator
  73. def partition(p: (T) => Boolean): (Iterator[T], Iterator[T])
    Definition Classes
    Iterator
  74. def patch[B >: T](from: Int, patchElems: Iterator[B], replaced: Int): Iterator[B]
    Definition Classes
    Iterator
  75. def product[B >: T](implicit num: Numeric[B]): B
    Definition Classes
    IterableOnceOps
  76. def reduce[B >: T](op: (B, B) => B): B
    Definition Classes
    IterableOnceOps
  77. def reduceLeft[B >: T](op: (B, T) => B): B
    Definition Classes
    IterableOnceOps
  78. def reduceLeftOption[B >: T](op: (B, T) => B): Option[B]
    Definition Classes
    IterableOnceOps
  79. def reduceOption[B >: T](op: (B, B) => B): Option[B]
    Definition Classes
    IterableOnceOps
  80. def reduceRight[B >: T](op: (T, B) => B): B
    Definition Classes
    IterableOnceOps
  81. def reduceRightOption[B >: T](op: (T, B) => B): Option[B]
    Definition Classes
    IterableOnceOps
  82. def reversed: Iterable[T]
    Attributes
    protected
    Definition Classes
    IterableOnceOps
  83. def sameElements[B >: T](that: IterableOnce[B]): Boolean
    Definition Classes
    Iterator
  84. def scanLeft[B](z: B)(op: (B, T) => B): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  85. def size: Int
    Definition Classes
    IterableOnceOps
  86. def slice(from: Int, until: Int): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  87. def sliceIterator(from: Int, until: Int): Iterator[T]
    Attributes
    protected
    Definition Classes
    Iterator
  88. def sliding[B >: T](size: Int, step: Int): GroupedIterator[B]
    Definition Classes
    Iterator
  89. def span(p: (T) => Boolean): (Iterator[T], Iterator[T])
    Definition Classes
    Iterator → IterableOnceOps
  90. def splitAt(n: Int): (Iterator[T], Iterator[T])
    Definition Classes
    IterableOnceOps
  91. def stepper[S <: Stepper[_]](implicit shape: StepperShape[T, S]): S
    Definition Classes
    IterableOnce
  92. def sum[B >: T](implicit num: Numeric[B]): B
    Definition Classes
    IterableOnceOps
  93. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  94. def take(n: Int): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  95. def takeWhile(p: (T) => Boolean): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  96. def tapEach[U](f: (T) => U): Iterator[T]
    Definition Classes
    Iterator → IterableOnceOps
  97. def to[C1](factory: Factory[T, C1]): C1
    Definition Classes
    IterableOnceOps
  98. def toArray[B >: T](implicit arg0: ClassTag[B]): Array[B]
    Definition Classes
    IterableOnceOps
  99. final def toBuffer[B >: T]: Buffer[B]
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  100. def toIndexedSeq: IndexedSeq[T]
    Definition Classes
    IterableOnceOps
  101. def toList: List[T]
    Definition Classes
    IterableOnceOps
  102. def toMap[K, V](implicit ev: <:<[T, (K, V)]): Map[K, V]
    Definition Classes
    IterableOnceOps
  103. def toSeq: Seq[T]
    Definition Classes
    IterableOnceOps
  104. def toSet[B >: T]: Set[B]
    Definition Classes
    IterableOnceOps
  105. def toString(): String
    Definition Classes
    Iterator → AnyRef → Any
  106. def toVector: Vector[T]
    Definition Classes
    IterableOnceOps
  107. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  108. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  109. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  110. def withFilter(p: (T) => Boolean): Iterator[T]
    Definition Classes
    Iterator
  111. def zip[B](that: IterableOnce[B]): Iterator[(T, B)]
    Definition Classes
    Iterator
  112. def zipAll[A1 >: T, B](that: IterableOnce[B], thisElem: A1, thatElem: B): Iterator[(A1, B)]
    Definition Classes
    Iterator
  113. def zipWithIndex: Iterator[(T, Int)]
    Definition Classes
    Iterator → IterableOnceOps

Deprecated Value Members

  1. final def /:[B](z: B)(op: (B, T) => B): B
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use foldLeft instead of /:

  2. final def :\[B](z: B)(op: (T, B) => B): B
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use foldRight instead of :\

  3. def aggregate[B](z: => B)(seqop: (B, T) => B, combop: (B, B) => B): B
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) For sequential collections, prefer foldLeft(z)(seqop). For parallel collections, use ParIterableLike#aggregate.

  4. final def copyToBuffer[B >: T](dest: Buffer[B]): Unit
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use dest ++= coll instead

  5. final def hasDefiniteSize: Boolean
    Definition Classes
    Iterator → IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) hasDefiniteSize on Iterator is the same as isEmpty

  6. def scanRight[B](z: B)(op: (T, B) => B): Iterator[B]
    Definition Classes
    Iterator
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) Call scanRight on an Iterable instead.

  7. def seq: BufferIterator.this.type
    Definition Classes
    Iterator
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) Iterator.seq always returns the iterator itself

  8. final def toIterator: Iterator[T]
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use .iterator instead of .toIterator

  9. final def toStream: Stream[T]
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use .to(LazyList) instead of .toStream

Inherited from Iterator[T]

Inherited from IterableOnceOps[T, Iterator, Iterator[T]]

Inherited from IterableOnce[T]

Inherited from AnyRef

Inherited from Any

Ungrouped