Packages

  • package root

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

    This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

    Please see the documentation for the individual packages for more details on their use.

    Scala Packages

    These packages are useful in Scala code without involving Spark:

    org.cert.netsa.data

    This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

    org.cert.netsa.io.ipfix

    The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

    org.cert.netsa.io.silk

    To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

    org.cert.netsa.util

    The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

    Spark Packages

    These packages require the use of Apache Spark:

    org.cert.netsa.mothra.datasources

    Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

    The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

    org.cert.netsa.mothra.analysis

    A grab-bag of analysis helper functions and example analyses.

    org.cert.netsa.mothra.functions

    This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package cert
    Definition Classes
    org
  • package netsa
    Definition Classes
    cert
  • package io
    Definition Classes
    netsa
  • package silk

    SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

    SiLK file formats, data types, and methods to read them, including support for reading them from Spark.

    RWRec is the type of SiLK flow records.

    You can use RWRecReader to read SiLK files from Scala, including compressed files if Hadoop native libraries are available. For example:

    import org.cert.netsa.io.silk.RWRecReader
import java.io.FileInputStream

val inputFile = new FileInputStream("path/to/silk/rw/file")

for ( rec <- RWRecReader.ofInputStream(inputFile) ) {
  println(rec.sIP)
}
    Definition Classes
    io
    See also

    org.cert.netsa.mothra.datasources.silk.flow for working with SiLK data in Spark using the Mothra SiLK datasource.

  • package config
    Definition Classes
    silk
  • package io
    Definition Classes
    silk
  • BagDataType
  • BagReader
  • BagResult
  • BagWriter
  • CompressionMethod
  • FileFormat
  • FlowType
  • Header
  • HeaderEntry
  • IPSetReader
  • IPSetWriter
  • PrefixMapProtocolPortPair
  • PrefixMapReader
  • PrefixMapResult
  • RWRec
  • RWRecReader
  • RWRecWriter
  • Sensor
  • SilkConfig
  • SilkDataFormatException
  • SilkVersion
  • TCPState

abstract class BagReader[T] extends Iterator[(T, Long)]

A reader of binary SiLK Bag files. This is usable as an Iterator over a pair representing the key and counter in the Bag. Depending on the contents of the Bag file, the key is either an IPAddress or an Int. The counter is always a Long.

Since a Bag file may contain different key types, the BagReader factory methods return a BagResult wrapper over BagReader.

The keyType and counterType methods return a BagDataType that specifies the type of the key and the counter that were specified in the Bag file's header.

Example:

  1. This example uses the single argument form of the companion object's ofInputStream() method to read the Bag file "example.bag". This code may be used outside of Hadoop when it is known that the Bag's key is an IPAddress.

    val stream = new java.io.FileInputStream("example.bag")
val bagresult = BagReader.ofInputStream(stream)
val bag = bagresult match {
  case BagResult.IPAddressBag(iter) => iter
  case _ => null
}
bag.hasNext
val (ipaddr, counter) = bag.next()
Note

While the SiLK command lines tools display an unspecified key type as an IPv4 address, the BagReader class treats an unspecified key type as an integer.

See also

the companion object for more details.

Linear Supertypes
Iterator[(T, Long)], IterableOnceOps[(T, Long), Iterator, Iterator[(T, Long)]], IterableOnce[(T, Long)], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BagReader
  2. Iterator
  3. IterableOnceOps
  4. IterableOnce
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new BagReader(reader: BufferReader, header: Header)
    Attributes
    protected

Type Members

  1. class GroupedIterator[B >: A] extends AbstractIterator[Seq[B]]
    Definition Classes
    Iterator

Abstract Value Members

  1. abstract def next(): (T, Long)

    Produces the next (key, counter) pair of this iterator.

    Produces the next (key, counter) pair of this iterator.

    returns

    the next pair of this iterator if hasNext is true

    Definition Classes
    BagReader → Iterator
    Exceptions thrown

    java.util.NoSuchElementException when the iterator is depleted

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ++[B >: (T, Long)](xs: => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator
    Annotations
    @inline()
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def addString(b: StringBuilder): StringBuilder
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  6. final def addString(b: StringBuilder, sep: String): StringBuilder
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  7. def addString(b: StringBuilder, start: String, sep: String, end: String): StringBuilder
    Definition Classes
    IterableOnceOps
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. var bufOffset: Int

    Current position (octet offset) in the current buffer.

    Current position (octet offset) in the current buffer.

    Attributes
    protected[this]
  10. var buffer: Array[Byte]

    Current buffer of bytes being processed

    Current buffer of bytes being processed

    Attributes
    protected[this]
  11. def buffered: BufferedIterator[(T, Long)]
    Definition Classes
    Iterator
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  13. def collect[B](pf: PartialFunction[(T, Long), B]): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  14. def collectFirst[B](pf: PartialFunction[(T, Long), B]): Option[B]
    Definition Classes
    IterableOnceOps
  15. def concat[B >: (T, Long)](xs: => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator
  16. def contains(elem: Any): Boolean
    Definition Classes
    Iterator
  17. def copyToArray[B >: (T, Long)](xs: Array[B], start: Int, len: Int): Int
    Definition Classes
    IterableOnceOps
  18. def copyToArray[B >: (T, Long)](xs: Array[B], start: Int): Int
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecatedOverriding()
  19. def copyToArray[B >: (T, Long)](xs: Array[B]): Int
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecatedOverriding()
  20. def corresponds[B](that: IterableOnce[B])(p: ((T, Long), B) => Boolean): Boolean
    Definition Classes
    IterableOnceOps
  21. def count(p: ((T, Long)) => Boolean): Int
    Definition Classes
    IterableOnceOps
  22. def counterType: Option[BagDataType]

    Provides the data type of the counter that was specified in the Bag file's header, if any.

    Provides the data type of the counter that was specified in the Bag file's header, if any.

    returns

    the type of data in the Counter field of the Bag or None if no header was present

  23. def distinct: Iterator[(T, Long)]
    Definition Classes
    Iterator
  24. def distinctBy[B](f: ((T, Long)) => B): Iterator[(T, Long)]
    Definition Classes
    Iterator
  25. def drop(n: Int): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  26. def dropWhile(p: ((T, Long)) => Boolean): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  27. def duplicate: (Iterator[(T, Long)], Iterator[(T, Long)])
    Definition Classes
    Iterator
  28. var endOfFile: Boolean

    Whether the end of the input stream has been reached.

    Whether the end of the input stream has been reached.

    Attributes
    protected[this]
  29. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  30. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  31. def exists(p: ((T, Long)) => Boolean): Boolean
    Definition Classes
    IterableOnceOps
  32. def fillBuffer(): Unit

    Read a new buffer into memory if needed.

    Read a new buffer into memory if needed. Does no work if the current buffer still has unread records, or if the reader is at end of file. Sets endOfFile to true if the EOF is reached.

    Attributes
    protected[this]
  33. def filter(p: ((T, Long)) => Boolean): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  34. def filterNot(p: ((T, Long)) => Boolean): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  35. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  36. def find(p: ((T, Long)) => Boolean): Option[(T, Long)]
    Definition Classes
    IterableOnceOps
  37. def flatMap[B](f: ((T, Long)) => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  38. def flatten[B](implicit ev: ((T, Long)) => IterableOnce[B]): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  39. def fold[A1 >: (T, Long)](z: A1)(op: (A1, A1) => A1): A1
    Definition Classes
    IterableOnceOps
  40. def foldLeft[B](z: B)(op: (B, (T, Long)) => B): B
    Definition Classes
    IterableOnceOps
  41. def foldRight[B](z: B)(op: ((T, Long), B) => B): B
    Definition Classes
    IterableOnceOps
  42. def forall(p: ((T, Long)) => Boolean): Boolean
    Definition Classes
    IterableOnceOps
  43. def foreach[U](f: ((T, Long)) => U): Unit
    Definition Classes
    IterableOnceOps
  44. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  45. def grouped[B >: (T, Long)](size: Int): GroupedIterator[B]
    Definition Classes
    Iterator
  46. def hasNext: Boolean

    Tests whether this iterator can provide another (key, counter) pair.

    Tests whether this iterator can provide another (key, counter) pair.

    returns

    true if a subsequent call to next() will yield an element, false otherwise.

    Definition Classes
    BagReader → Iterator
  47. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  48. val header: Header
    Attributes
    protected[this]
  49. def indexOf[B >: (T, Long)](elem: B, from: Int): Int
    Definition Classes
    Iterator
  50. def indexOf[B >: (T, Long)](elem: B): Int
    Definition Classes
    Iterator
  51. def indexWhere(p: ((T, Long)) => Boolean, from: Int): Int
    Definition Classes
    Iterator
  52. def isEmpty: Boolean
    Definition Classes
    Iterator → IterableOnceOps
    Annotations
    @deprecatedOverriding()
  53. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  54. def isTraversableAgain: Boolean
    Definition Classes
    IterableOnceOps
  55. final def iterator: Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnce
    Annotations
    @inline()
  56. def keyType: Option[BagDataType]

    Provides the data type of the key that was specified in the Bag file's header, if any.

    Provides the data type of the key that was specified in the Bag file's header, if any.

    returns

    the type of data in the Key field of the Bag or None if no header was present

  57. def knownSize: Int
    Definition Classes
    IterableOnce
  58. final def length: Int
    Definition Classes
    Iterator
    Annotations
    @inline()
  59. def map[B](f: ((T, Long)) => B): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  60. def max[B >: (T, Long)](implicit ord: Ordering[B]): (T, Long)
    Definition Classes
    IterableOnceOps
  61. def maxBy[B](f: ((T, Long)) => B)(implicit cmp: Ordering[B]): (T, Long)
    Definition Classes
    IterableOnceOps
  62. def maxByOption[B](f: ((T, Long)) => B)(implicit cmp: Ordering[B]): Option[(T, Long)]
    Definition Classes
    IterableOnceOps
  63. def maxOption[B >: (T, Long)](implicit ord: Ordering[B]): Option[(T, Long)]
    Definition Classes
    IterableOnceOps
  64. def min[B >: (T, Long)](implicit ord: Ordering[B]): (T, Long)
    Definition Classes
    IterableOnceOps
  65. def minBy[B](f: ((T, Long)) => B)(implicit cmp: Ordering[B]): (T, Long)
    Definition Classes
    IterableOnceOps
  66. def minByOption[B](f: ((T, Long)) => B)(implicit cmp: Ordering[B]): Option[(T, Long)]
    Definition Classes
    IterableOnceOps
  67. def minOption[B >: (T, Long)](implicit ord: Ordering[B]): Option[(T, Long)]
    Definition Classes
    IterableOnceOps
  68. final def mkString: String
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  69. final def mkString(sep: String): String
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  70. final def mkString(start: String, sep: String, end: String): String
    Definition Classes
    IterableOnceOps
  71. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  72. def nextOption(): Option[(T, Long)]
    Definition Classes
    Iterator
  73. def nonEmpty: Boolean
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecatedOverriding()
  74. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  75. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  76. def padTo[B >: (T, Long)](len: Int, elem: B): Iterator[B]
    Definition Classes
    Iterator
  77. def partition(p: ((T, Long)) => Boolean): (Iterator[(T, Long)], Iterator[(T, Long)])
    Definition Classes
    Iterator
  78. def patch[B >: (T, Long)](from: Int, patchElems: Iterator[B], replaced: Int): Iterator[B]
    Definition Classes
    Iterator
  79. def product[B >: (T, Long)](implicit num: Numeric[B]): B
    Definition Classes
    IterableOnceOps
  80. val reader: BufferReader
    Attributes
    protected[this]
  81. val recordLength: Short

    Record length

    Record length

    Attributes
    protected[this]
  82. def reduce[B >: (T, Long)](op: (B, B) => B): B
    Definition Classes
    IterableOnceOps
  83. def reduceLeft[B >: (T, Long)](op: (B, (T, Long)) => B): B
    Definition Classes
    IterableOnceOps
  84. def reduceLeftOption[B >: (T, Long)](op: (B, (T, Long)) => B): Option[B]
    Definition Classes
    IterableOnceOps
  85. def reduceOption[B >: (T, Long)](op: (B, B) => B): Option[B]
    Definition Classes
    IterableOnceOps
  86. def reduceRight[B >: (T, Long)](op: ((T, Long), B) => B): B
    Definition Classes
    IterableOnceOps
  87. def reduceRightOption[B >: (T, Long)](op: ((T, Long), B) => B): Option[B]
    Definition Classes
    IterableOnceOps
  88. def reversed: Iterable[(T, Long)]
    Attributes
    protected
    Definition Classes
    IterableOnceOps
  89. def sameElements[B >: (T, Long)](that: IterableOnce[B]): Boolean
    Definition Classes
    Iterator
  90. def scanLeft[B](z: B)(op: (B, (T, Long)) => B): Iterator[B]
    Definition Classes
    Iterator → IterableOnceOps
  91. def size: Int
    Definition Classes
    IterableOnceOps
  92. def slice(from: Int, until: Int): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  93. def sliceIterator(from: Int, until: Int): Iterator[(T, Long)]
    Attributes
    protected
    Definition Classes
    Iterator
  94. def sliding[B >: (T, Long)](size: Int, step: Int): GroupedIterator[B]
    Definition Classes
    Iterator
  95. def span(p: ((T, Long)) => Boolean): (Iterator[(T, Long)], Iterator[(T, Long)])
    Definition Classes
    Iterator → IterableOnceOps
  96. def splitAt(n: Int): (Iterator[(T, Long)], Iterator[(T, Long)])
    Definition Classes
    IterableOnceOps
  97. def stepper[S <: Stepper[_]](implicit shape: StepperShape[(T, Long), S]): S
    Definition Classes
    IterableOnce
  98. def sum[B >: (T, Long)](implicit num: Numeric[B]): B
    Definition Classes
    IterableOnceOps
  99. val swap: Boolean

    Whether to byte-swap the bytes as they are read from the buffer

    Whether to byte-swap the bytes as they are read from the buffer

    Attributes
    protected[this]
  100. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  101. def take(n: Int): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  102. def takeWhile(p: ((T, Long)) => Boolean): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  103. def tapEach[U](f: ((T, Long)) => U): Iterator[(T, Long)]
    Definition Classes
    Iterator → IterableOnceOps
  104. def to[C1](factory: Factory[(T, Long), C1]): C1
    Definition Classes
    IterableOnceOps
  105. def toArray[B >: (T, Long)](implicit arg0: ClassTag[B]): Array[B]
    Definition Classes
    IterableOnceOps
  106. final def toBuffer[B >: (T, Long)]: Buffer[B]
    Definition Classes
    IterableOnceOps
    Annotations
    @inline()
  107. def toIndexedSeq: IndexedSeq[(T, Long)]
    Definition Classes
    IterableOnceOps
  108. def toList: List[(T, Long)]
    Definition Classes
    IterableOnceOps
  109. def toMap[K, V](implicit ev: <:<[(T, Long), (K, V)]): Map[K, V]
    Definition Classes
    IterableOnceOps
  110. def toSeq: Seq[(T, Long)]
    Definition Classes
    IterableOnceOps
  111. def toSet[B >: (T, Long)]: Set[B]
    Definition Classes
    IterableOnceOps
  112. def toString(): String
    Definition Classes
    Iterator → AnyRef → Any
  113. def toVector: Vector[(T, Long)]
    Definition Classes
    IterableOnceOps
  114. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  115. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  116. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  117. def withFilter(p: ((T, Long)) => Boolean): Iterator[(T, Long)]
    Definition Classes
    Iterator
  118. def zip[B](that: IterableOnce[B]): Iterator[((T, Long), B)]
    Definition Classes
    Iterator
  119. def zipAll[A1 >: (T, Long), B](that: IterableOnce[B], thisElem: A1, thatElem: B): Iterator[(A1, B)]
    Definition Classes
    Iterator
  120. def zipWithIndex: Iterator[((T, Long), Int)]
    Definition Classes
    Iterator → IterableOnceOps

Deprecated Value Members

  1. final def /:[B](z: B)(op: (B, (T, Long)) => B): B
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use foldLeft instead of /:

  2. final def :\[B](z: B)(op: ((T, Long), B) => B): B
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use foldRight instead of :\

  3. def aggregate[B](z: => B)(seqop: (B, (T, Long)) => B, combop: (B, B) => B): B
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) aggregate is not relevant for sequential collections. Use foldLeft(z)(seqop) instead.

  4. final def copyToBuffer[B >: (T, Long)](dest: Buffer[B]): Unit
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use dest ++= coll instead

  5. final def hasDefiniteSize: Boolean
    Definition Classes
    Iterator → IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) hasDefiniteSize on Iterator is the same as isEmpty

  6. def scanRight[B](z: B)(op: ((T, Long), B) => B): Iterator[B]
    Definition Classes
    Iterator
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) Call scanRight on an Iterable instead.

  7. def seq: BagReader.this.type
    Definition Classes
    Iterator
    Annotations
    @deprecated
    Deprecated

    (Since version 2.13.0) Iterator.seq always returns the iterator itself

  8. final def toIterator: Iterator[(T, Long)]
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use .iterator instead of .toIterator

  9. final def toStream: Stream[(T, Long)]
    Definition Classes
    IterableOnceOps
    Annotations
    @deprecated @inline()
    Deprecated

    (Since version 2.13.0) Use .to(LazyList) instead of .toStream

Inherited from Iterator[(T, Long)]

Inherited from IterableOnceOps[(T, Long), Iterator, Iterator[(T, Long)]]

Inherited from IterableOnce[(T, Long)]

Inherited from AnyRef

Inherited from Any

Ungrouped