package func
The objects in this package represent "function fields" usable in the
IPFIX data source. These fields may each be accessed by the field
expression "func:<fieldName>"
, for example "func:recordInfo"
in a
field expression will produce a human-readable string record as
described below.
- See also
org.cert.netsa.mothra.datasources.ipfix for examples of field expressions
IPFIXFieldParsing for details about the field path grammar
- Alphabetic
- By Inheritance
- func
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- case object TCPFlags extends IPFIXField[Int] with Product with Serializable
The numeric bit-wise union of all TCP flags from the CERT IPFIX
initialTCPFlags
(only first packet) andunionTCPFlags
(all packets after first packet) Information Elements.The numeric bit-wise union of all TCP flags from the CERT IPFIX
initialTCPFlags
(only first packet) andunionTCPFlags
(all packets after first packet) Information Elements.- See also
unionTCPFlags Information Element
initialTCPFlags Information Element
- case object appLabel extends IPFIXField[String] with Product with Serializable
The name of the service associated with the CERT IPFIX
silkAppLabel
Information Element, if a name is defined.The name of the service associated with the CERT IPFIX
silkAppLabel
Information Element, if a name is defined. For most values, this is the same as the IANA service name for the appLabel as a port number. For some, such as "dns", they differ from the official ("domain" in this case).- See also
silkAppLabel Information Element
- case object destinationIPAddress extends SimpleField[String] with Product with Serializable
Either the IPv4 or the IPv6 destination address of this record.
Either the IPv4 or the IPv6 destination address of this record. This is equivalent to the path expression
"destinationIPv4Address | destinationIPv6Address"
, except that when filtering on this function, partitions on either field are used to improve efficiency in Mothra IPFIX repositories.- See also
IANA registry for these Information Elements
- case object destinationPort extends SimpleField[Int] with Product with Serializable
The destination transport port, stored in any of the common Information Elements.
The destination transport port, stored in any of the common Information Elements. This is equivalent to the path expression
"destinationTransportPort | tcpDestinationPort | udpDestinationPort"
, except that when filtering on this function, partitions on either field are used to improve efficiency in Mothra IPFIX repositories.- See also
IANA registry for these Information Elements
- case object durationMicroseconds extends IPFIXField[Long] with Product with Serializable
The duration in microseconds of this record, computed using TimeGauntlet.computeDurationNanos.
- case object durationMilliseconds extends IPFIXField[Long] with Product with Serializable
The duration in milliseconds of this record, computed using TimeGauntlet.computeDurationNanos.
- case object endTime extends IPFIXField[Timestamp] with Product with Serializable
The flow end time of this record, extracted or computed using TimeGauntlet.computeEndTime.
The flow end time of this record, extracted or computed using TimeGauntlet.computeEndTime. When filtering on this field, time-based partitioning is used to improve efficiency in Mothra IPFIX repositories.
- case object filename extends IPFIXField[String] with LazyLogging with Product with Serializable
The filename of the file or the key of the S3 object which stores this record.
The filename of the file or the key of the S3 object which stores this record. Mostly useful for debugging.
- case object icmpCode extends IPFIXField[Int] with Product with Serializable
The ICMP code, stored in any of the common Information Elements.
The ICMP code, stored in any of the common Information Elements. When
protocolIdentifier
is not 1 (ICMP) or 46 (IPv6-ICMP), this is always null. Otherwise, it checksicmpTypeCodeIPv4
,icmpTypeCodeIPv6
,icmpCodeIPv4
,icmpCodeIPv6
, anddestinationTransportPort
for data, and extracts the code from those fields if necessary.When filtering on this field, all of the candidate fields are checked for partitions and used to select partitions.
- See also
IANA registry for these Information Elements
- case object icmpType extends IPFIXField[Int] with Product with Serializable
The ICMP type, stored in any of the common Information Elements.
The ICMP type, stored in any of the common Information Elements. When
protocolIdentifier
is not 1 (ICMP) or 46 (IPv6-ICMP), this is always null. Otherwise, it checksicmpTypeCodeIPv4
,icmpTypeCodeIPv6
,icmpCTypeIPv4
,icmpTypeIPv6
, anddestinationTransportPort
for data, and extracts the code from those fields if necessary.When filtering on this field, all of the candidate fields are checked for partitions and used to select partitions.
- See also
IANA registry for these Information Elements
- case object initialTCPFlags extends SimpleField[Int] with Product with Serializable
Finds initial TCP flags from deeper within the record to support legacy YAF versions.
- case object isIPv6 extends IPFIXField[Boolean] with Product with Serializable
True if the record contains a
sourceIPv6Address
ordestinationIPv6Address
Information Element. - case object nextHopIPAddress extends SimpleField[String] with Product with Serializable
For Either IPv4 or IPv6, the BGP or IP next-hop address of this record.
For Either IPv4 or IPv6, the BGP or IP next-hop address of this record. This is equivalent to the path expression
"ipNextHopIPv4Address | ipNextHopIPv6Address | bgpNextHopIPv4Address | bgpNextHopIPv6Address"
, except that when filtering on this function, partitions on either field are used to improve efficiency in Mothra IPFIX repositories.- See also
IANA registry for these Information Elements
- case object octetCount extends SimpleField[Long] with Product with Serializable
The number of octets in this record, from whatever source possible.
The number of octets in this record, from whatever source possible. This is equivalent to the path expression
"octetDeltaCount | octetTotalCount | initiatorOctets | postOctetDeltaCount | postOctetTotalCount"
.- See also
IANA registry for these Information Elements
- case object packetCount extends SimpleField[Long] with Product with Serializable
The number of packets in this record, from whatever source possible.
The number of packets in this record, from whatever source possible. This is equivalent to the path expression
"packetDeltaCount | packetTotalCount | initiatorPackets | postPacketDeltaCount | postPacketTotalCount"
.- See also
IANA registry for these Information Elements
- case object recordInfo extends IPFIXField[String] with Product with Serializable
A human-readable text representation of this record (also works in sub-records).
A human-readable text representation of this record (also works in sub-records). The format is similar to JSON, but is not parsable as JSON. Most useful for debugging or exploring data.
- case object reverseOctetCount extends SimpleField[Long] with Product with Serializable
The number of reverse-direction octets in this record, from whatever source possible.
The number of reverse-direction octets in this record, from whatever source possible. This is equivalent to the path expression
"reverseOctetDeltaCount | reverseOctetTotalCount | responderOctets | reversePostOctetDeltaCount | reversePostOctetTotalCount"
.- See also
IANA registry for these Information Elements
- case object reversePacketCount extends SimpleField[Long] with Product with Serializable
The number of reverse-direction packets in this record, from whatever source possible.
The number of reverse-direction packets in this record, from whatever source possible. This is equivalent to the path expression
"reversePacketDeltaCount | reversePacketTotalCount | responderPackets | reversePostPacketDeltaCount | reversePostPacketTotalCount"
.- See also
IANA registry for these Information Elements
- case object sourceIPAddress extends SimpleField[String] with Product with Serializable
Either the IPv4 or the IPv6 source address of this record.
Either the IPv4 or the IPv6 source address of this record. This is equivalent to the path expression
"sourceIPv4Address | sourceIPv6Address"
, except that when filtering on this function, partitions on either field are used to improve efficiency in Mothra IPFIX repositories.- See also
IANA registry for these Information Elements
- case object sourcePort extends SimpleField[Int] with Product with Serializable
The source transport port, stored in any of the common Information Elements.
The source transport port, stored in any of the common Information Elements. This is equivalent to the path expression
"sourceTransportPort | tcpSourcePort | udpSourcePort"
, except that when filtering on this function, partitions on either field are used to improve efficiency in Mothra IPFIX repositories.- See also
IANA registry for these Information Elements
- case object startTime extends IPFIXField[Timestamp] with Product with Serializable
The flow start time of this record, extracted or computed using TimeGauntlet.computeStartTime.
The flow start time of this record, extracted or computed using TimeGauntlet.computeStartTime. When filtering on this field, time-based partitioning is used to improve efficiency in Mothra IPFIX repositories.
- case object unionTCPFlags extends SimpleField[Int] with Product with Serializable
Finds union TCP flags from deeper within the record to support legacy YAF versions.
This is documentation for Mothra, a collection of Scala and Spark library functions for working with Internet-related data. Some modules contain APIs of general use to Scala programmers. Some modules make those tools more useful on Spark data-processing systems.
Please see the documentation for the individual packages for more details on their use.
Scala Packages
These packages are useful in Scala code without involving Spark:
org.cert.netsa.data
This package, which is collected as the
netsa-data
library, provides types for working with various kinds of information:org.cert.netsa.data.net
- types for working with network dataorg.cert.netsa.data.time
- types for working with time dataorg.cert.netsa.data.unsigned
- types for working with unsigned integral valuesorg.cert.netsa.io.ipfix
The
netsa-io-ipfix
library provides tools for reading and writing IETF IPFIX data from various connections and files.org.cert.netsa.io.silk
To read and write CERT NetSA SiLK file formats and configuration files, use the
netsa-io-silk
library.org.cert.netsa.util
The "junk drawer" of
netsa-util
so far provides only two features: First, a method for equipping Scala scala.collection.Iterators with exception handling. And second, a way to query the versions of NetSA libraries present in a JVM at runtime.Spark Packages
These packages require the use of Apache Spark:
org.cert.netsa.mothra.datasources
Spark datasources for CERT file types. This package contains utility features which add methods to Apache Spark DataFrameReader objects, allowing IPFIX and SiLK flows to be opened using simple
spark.read...
calls.The
mothra-datasources
library contains both IPFIX and SiLK functionality, whilemothra-datasources-ipfix
andmothra-datasources-silk
contain only what's needed for the named datasource.org.cert.netsa.mothra.analysis
A grab-bag of analysis helper functions and example analyses.
org.cert.netsa.mothra.functions
This single Scala object provides Spark SQL functions for working with network data. It is the entirety of the
mothra-functions
library.