Packages

•  package root

  This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data.

  This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.

  Please see the documentation for the individual packages for more details on their use.

  Scala Packages

  These packages are useful in Scala code without involving Spark:

  org.cert.netsa.data

  This package, which is collected as the netsa-data library, provides types for working with various kinds of information:

  • org.cert.netsa.data.net - types for working with network data
  • org.cert.netsa.data.time - types for working with time data
  • org.cert.netsa.data.unsigned - types for working with unsigned integral values

  org.cert.netsa.io.ipfix

  The netsa-io-ipfix library provides tools for reading and writing IETF IPFIX data from various connections and files.

  org.cert.netsa.io.silk

  To read and write CERT NetSA SiLK file formats and configuration files, use the netsa-io-silk library.

  org.cert.netsa.util

  The "junk drawer" of netsa-util so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.

  Spark Packages

  These packages require the use of Apache Spark:

  org.cert.netsa.mothra.datasources

  Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple spark.read... calls.

  The mothra-datasources library contains both IPFIX and SiLK functionality, while mothra-datasources-ipfix and mothra-datasources-silk contain only what's needed for the named datasource.

  org.cert.netsa.mothra.analysis

  A grab-bag of analysis helper functions and example analyses.

  org.cert.netsa.mothra.functions

  This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the mothra-functions library.

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package cert

  Definition Classes

  org

•  package netsa

  Definition Classes

  cert

•  package data

  The org.cert.netsa.data.net package is for working with network-related data.

  The org.cert.netsa.data.net package is for working with network-related data. This includes types for IP addresses, port numbers, protocol numbers, and the like. Many of these types have namespaces managed by IANA, and the types provide mechanisms for looking up names from numbers and vice-versa based on embedded copies of IANA's tables.

  In org.cert.netsa.data.time you can find an Ordering for Java LocalDate objects, and a type LocalDateSet for working with sets of those dates.

  Finally, org.cert.netsa.data.unsigned contains types for working with unsigned integer values.

  Definition Classes

  netsa

•  package net

  Data types for working with network data.

  Data types for working with network data. This currently includes IP addresses and CIDR blocks (both v4 and v6), and a variety of ID numbers, many of which are given names by IANA.

  Overview

  IP addresses and CIDR blocks are represented by IPAddress and IPBlock types, like so:

  scala> val addr4 = IPAddress("1.2.3.4")
  addr4: org.cert.netsa.data.net.IPAddress = 1.2.3.4
  
  scala> val addr6 = IPAddress("ffef::a:b:c:d:1.2.3.4")
  addr6: org.cert.netsa.data.net.IPAddress = ffef:0:a:b:c:d:102:304
  
  scala> val cidr4 = IPBlock("1.2.0.0/16")
  cidr4: org.cert.netsa.data.net.IPBlock = 1.2.0.0/16
  
  scala> val cidr6 = IPBlock("feff::/16")
  cidr6: org.cert.netsa.data.net.IPBlock = feff:0:0:0:0:0:0:0/16
  
  scala> val a = cidr4.contains(addr4)
  a: Boolean = true
  
  scala> val b = cidr6.contains(addr6)
  b: Boolean = false
  
  scala> val c = cidr4.overlaps(cidr6)
  c: Boolean = false
  
  scala> val d = cidr4.overlaps(IPBlock("1.0.0.0/8"))
  d: Boolean = true

  Types like Port and Protocol are used for IANA-registered service port numbers and information about those service mappings:

  scala> val port1 = Port(80)
  port1: org.cert.netsa.data.net.Port = Port(80)
  
  scala> val port2 = Port("https")
  port2: org.cert.netsa.data.net.Port = Port(443)
  
  scala> val port3 = Port(65535)
  port3: org.cert.netsa.data.net.Port = Port(65535)
  
  scala> for ( p <- Seq(port1, port2, port3) )
       |   println(f"${p.toString}%15s ${p.toShort}%6d ${p.serviceName}%15s")
         Port(80)     80      Some(http)
        Port(443)    443     Some(https)
      Port(65535)     -1            None

  In general, these types use the smallest available (signed) integer type as their bitwise representation. They provide a mechanism for getting the name given by IANA ("serviceName" for port numbers). Some also provide constants for easy access to the most common values:

  scala> Protocol.TCP
  res0: org.cert.netsa.data.net.Protocol = Protocol(6)

  Others have additional methods to provide appropraite facilities for breaking the values down further, or provide nothing more than what is required to distinguish these IDs from integers.

  See the individual types in this package for more details.

  Definition Classes

  data

•  package time

  Definition Classes

  data

•  package unsigned

  A variety of unsigned integral types, and new methods on the built-in integral types for working with them.

  A variety of unsigned integral types, and new methods on the built-in integral types for working with them.

  Import the implicit conversions from this package to add toUInt methods and the like to standard Scala types.

  Features

  The overall pattern for each integral type (UByte, UShort, UInt, ULong) is the following (using UByte as the example):

  Unsigned alues can be constructed from signed Byte and Int values using UByte(b: Byte) and UByte(i: Int).

  x.toUByte, x.toUShort, etc. and x.toByte, x.toShort, etc. methods are included.

  All of the expected comparison, arithmetic, and bitwise operations are present. In addition, UByte extends Comparable, and equipped with an Ordering and membership in the Integral type class.

  UByte.MinValue and UByte.MaxValue are defined.

  If you import implicits.ByteUnsignedConversions, then x.toUByte, etc. methods will be available by implicit conversion on Byte values.

  Definition Classes

  data

  Note

  If you are concerned with efficiency, do not create arrays of unsigned values, as the will be boxed into objects. Instead, create arrays of normal signed values and then convert to and from unsigned when getting and setting the values.

• UByte
• UInt
• ULong
• UShort
• implicits

org.cert.netsa.data

unsigned