netsa_silk — NetSA Python/PySiLK Support¶
The netsa_silk module contains a shared API for working with common Internet data in both netsa-python and PySiLK. If netsa-python is installed but PySiLK is not, the less efficient but more portable pure-Python version of this functionality that is included in netsa-python is used. If PySiLK is installed, then the high-performance C version of this functionality that is included in PySiLK is used.
This document describes version 1.0 of the netsa_silk module API.
IPv6 Support¶
Depending on which version of the netsa_silk functionality is in use, IPv6 support may not be present or may be limited. The following functions allow determining what variety of IPv6 support is available.
- netsa_silk.has_IPv6Addr() → bool¶
Returns True if the most basic form of IPv6 support—support for IPv6 addresses—is available. If it is not available, then this function returns False, IPAddr will raise ValueError when given an IPv6 address, and any call to IPv6Addr will raise NotImplementedError.
See also ip_set.supports_ipv6.
IP Addresses¶
An IP address is represented by either an IPv4Addr or an IPv6Addr. Both of these are subclasses of the generic IPAddr class.
- class netsa_silk.IPAddr(address : str or IPAddr) → IPv4Addr or IPv6Addr¶
Converts the input into an IP address, either IPv4 or IPv6. Returns either an IPv4Addr or IPv6Addr object, depending on whether the given input is parsed as an IPv4 or an IPv6 address.
If IPv6 address support is not available (has_IPv6Addr returns False), then attempting to parse an IPv6 address will raise ValueError.
Examples:
>>> addr1 = IPAddr('192.160.1.1') >>> addr2 = IPAddr('2001:db8::1428:57ab') >>> addr3 = IPAddr('::ffff:12.34.56.78') >>> addr4 = IPAddr(addr1) >>> addr5 = IPAddr(addr2)
- class netsa_silk.IPv4Addr(address : int or str or IPAddr)¶
Converts the input into a new IPv4 address. If the integer input is too large a value, if the string input is unparseable as an IPv4 address, or if the IPAddr input is not convertible to an IPv4 address, raises a ValueError.
IPv4Addr is a subclass of IPAddr.
Examples:
>>> addr1 = IPv4Addr('192.160.1.1') >>> addr2 = IPv4Addr(IPAddr('::ffff.12.34.56.78')) >>> addr3 = IPv4Addr(addr1) >>> addr4 = IPv4Addr(0x10000000)
- class netsa_silk.IPv6Addr(address : int or str or IPAddr)¶
Converts the input into a new IPv6 address. If the integer input is too large a value, or if the string input is unparseable as an IPv6 address, raises a ValueError. If the input is an IPv4Addr, the address is converted to IPv6 via IPv4-mapped address embedding. (“1.2.3.4” becomes ”::ffff:1.2.3.4”).
If IPv6 address support is not available (has_IPv6Addr returns False), then calling IPv6Addr will raise NotImplementedError.
IPv6Addr is a subclass of IPAddr.
Examples:
>>> addr1 = IPv6Addr('2001:db8::1428:57ab') >>> addr2 = IPv6Addr(IPAddr('192.168.1.1')) >>> addr3 = IPv6Addr(addr1) >>> addr4 = IPv6Addr(0x100000000000000000000000)
Comparisons¶
Whenever an IPv4 address is compared to an IPv6 address, the IPv4 address is converted to IPv6 using IPv4-mapped address embedding. This means that IPAddr('0.0.0.0') equals IPAddr('::ffff:0.0.0.0'). You can distinguish IPv4 addresses from IPv6 address by using the is_ipv6() method.
| Operation | Result |
|---|---|
| a == b | if a is equal to b, then True, else False |
| a != b | if a is equal to b, then False, else True |
| a < b | if a‘s integer representation is less than b‘s, then True, else False |
| a <= b | if a‘s integer representation is less than or equal to b‘s, then True, else False |
| a >= b | if a‘s integer representation is greater than or equal to b‘s, then True, else False |
| a > b | if a‘s integer representation is greater than b‘s, then True, else False |
Conversions¶
The following operations and methods may be used to convert between IPv4 and IPv6 addresses and between IP addresses and other types.
| Operation | Result | Notes |
|---|---|---|
| addr.is_ipv6() | if addr is an IPv6 address, then True, else False | |
| addr.to_ipv4() | the IPv4 equivalent of addr, or None if no such equivalent exists | (1) |
| addr.to_ipv6() | the IPv6 equivalent of addr | (2) |
| int(addr) | the integer representation of addr | (3) |
| str(addr) | the human-readable string representation of addr | (4) |
| addr.padded() | a zero-padded human-readable string representation of addr | (5) |
| addr.octets() | a tuple containing each octet of addr in network byte order as an unsigned integer |
Notes:
- If the address is already an IPv4 address, does nothing. If the address is an IPv6 address using IPv4-mapped address embedding (e.g. ”::ffff:1.2.3.4”), returns the equivalent IPv4 address. Otherwise, returns None.
- If the address is already an IPv6 address, does nothing. If the address is an IPv4 address, returns the address converted to an IPv6 address using IPv4-mapped address embedding.
- If the address is an IPv4 address, returns an unsigned 32-bit integer value. If the address is an IPv6 address, returns an unsigned 128-bit integer value.
- The address is returned in its canonical form.
- If the address is an IPv4 address, returns a string of the form “xxx.xxx.xxx.xxx”, where each field is one octet of the address as a zero-padded base-10 integer. If the address is an IPv6 address, returns a string of the form “xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx”, where each field is two octets of the address as a zero-padded base-16 integer.
Masking¶
Masking operations return a copy of the address with all non-masked bits set to zero.
| Operation | Result | Notes |
|---|---|---|
| addr.mask(mask) | the address addr masked by the bits of the address mask | (1) |
| addr.mask_prefix(len) | the address addr masked to a length of len prefix bits |
Notes:
- If addr is an IPv6 address but mask is an IPv4 address, mask is converted to IPv6 and then the mask is applied. If addr was not an IPv4-mapped embedded IPv6 address, the result may not be what was expected. Prefer addr.mask_prefix(len) for IPv6 addresses when possible.
IP Sets¶
While there are multiple different IP set implementations with different qualities, netsa_silk provides a standard API for these sets, and a standard mechanism for acquiring a value of some set when you don’t need a specific implementation.
- class netsa_silk.ip_set([iterable])¶
Returns a new IP set object from an unspecified implementation that matches the following API. The elements of the set must be IP addresses. The values in iterable may be IPAddr objects, strings parsable by IPAddr, IPWildcard objects, or strings parsable by IPWildcard.
In all of the following descriptions, s, s1, and s2 must be ip_set obejcts, addr may be an IPAddr object or a string that is parsable as an IPAddr. iterable may be any iterable containing IPAddr objects, IPWildcard objects, and strings, as described above.
Cardinality¶
Since IP sets can grow very large, an additional method for querying the cardinality which supports large values is available.
| Operation | Result | Notes |
|---|---|---|
| s.cardinality() | the cardinality of s | |
| len(s) | the cardinality of s | (1) |
Notes:
- If the cardinality of s is exceptionally large, this may raise OverflowError due to limitations in Python. Using s.cardinality() is highly preferred for IP sets.
Membership¶
| Operation | Result |
|---|---|
| addr in s | if addr is a member of s, then``True``, else False |
| addr not in s | if addr is a member of s, then False, else True |
Comparison¶
| Operation | Result |
|---|---|
| s1 == s2 | if s has exactly the same elements as s2, then True, else False |
| s1 != s2 | if s1 does not have exactly the same elements as s2, then True, else False |
| s.isdisjoint(iterable) | if s has no elements in common with iterable, then True, else False |
| s.issubset(iterable) | if every element of s is also in iterable, then True, else False |
| s1 <= s2 | if every element of s1 is also in s2, then True, else False |
| s1 < s2 | if s1 < s2 and s1 != s2, then True, else False |
| s.issuperset(iterable) | if every element of iterable is also in s, then True, else False |
| s1 >= s2 | if every element of s2 is also in s1, then True, else False |
| s1 > s2 | if s1 > s2 and s1 != s2, then True, else False |
Manipulation¶
The following operations return a new IP set with the desired changes, while leaving the original IP set unmodified.
| Operation | Result |
|---|---|
| s.union(iterable, ...) | a set with all elements that are in s or any iterable |
| s1 | s2 | a set with all elements that are in s1 or s2 |
| s.intersection(iterable, ...) | a set with only elements that are in s and every iterable |
| s1 & s2 | a set with only elements that are in both s1 and s2 |
| s.difference(iterable, ...) | a set with all elements that are in s but in no iterable |
| s1 - s2 | a set with all elements that are in s1 but not in s2 |
| s.symmetric_difference(iterable) | a set with all elements that are in s or iterable but not both |
| s1 ^ s2 | a set with all elements that are in s1 or s2 but not both |
| s.copy() | a shallow copy of s |
Modification¶
The following operations modify the target IP set in place.
| Operation | Result | Notes |
|---|---|---|
| s.update(iterable, ...) | updates s by adding all elements from each iterable | |
| s1 |= s2 | updates s1 by adding all elements from s2 | |
| s.intersection_update(iterable, ...) | updates s by removing all elements that do not appear in every iterable | |
| s1 &= s2 | updates s1 by removing all elements that do not appear in s2 | |
| s.difference_update(iterable, ...) | updates s by removing all elements that appear in any iterable | |
| s1 -= s2 | updates s1 by removing all elements that appear in s2 | |
| s.symmetric_difference_update(iterable) | updates s, keeping only elements found in s or iterable but not both | |
| s1 ^= s2 | updates s1, keeping only elements found in s1 or s2 but not both | |
| s.add(addr) | adds addr to s | |
| s.remove(addr) | removes addr from s | (1) |
| s.discard(addr) | removes addr from s | |
| s.pop() | removes and returns an arbitrary element from s | (2) |
| s.clear() | removes all elements from s |
Notes:
CIDR Block Iteration¶
- ip_set.cidr_iter() → (IPAddr, int) iter¶
Returns an iterator over the CIDR blocks covered by this IP set. Each value in the iterator is a pair (addr, prefix_len) where addr is the first IP address in the block, and prefix_len is the prefix length of the block.
IPv6 Support¶
Some ip_set implementations do not provide IPv6 support. Such an implementation will raise an exception.TypeError on any attempt to add an IPv6Addr to the set. The following class method can be used to determine if a given implementation has IPv6 support:
- classmethod ip_set.supports_ipv6() → bool¶
Returns True if this IP set implementation provides support for IPv6 addresses, or False otherwise.
IP Wildcards¶
An IPWildcard object represents the specification of a set of IP addresses using SiLK IP wildcard syntax. Not all sets of IP addresses can be represented by a single IP wildcard.
- class netsa_silk.IPWildcard(wildcard : str or IPWildcard)¶
Returns a new IPWildcard object constructed from wildcard. The string wildcard may contain an IP address, an IP address with a CIDR prefix designation, an integer, an integer with a CIDR prefix designation, or a SiLK wildcard expression. In SiLK wildcard notation, a wildcard is represented as an IP address in canonical form with each octet (for IPv4 addresses) or octet pair (IPv6) holding a single value, a range of values, a comma-separated list of values and ranges, or the character ‘x’ to accept all values.
Examples:
>>> wild1 = IPWildcard('1.2.3.0/24') >>> wild2 = IPWildcard('ff80::/16') >>> wild3 = IPWildcard('1.2.3.4') >>> wild4 = IPWildcard('::FFFF:0102:0304') >>> wild5 = IPWildcard('16909056') >>> wild6 = IPWildcard('16909056/24') >>> wild7 = IPWildcard('1.2.3.x') >>> wild8 = IPWildcard('1:2:3:4:5:6:7:x') >>> wild9 = IPWildcard('1.2,3.4,5.6,7') >>> wild10 = IPWildcard('1.2.3.0-255') >>> wild11 = IPWildcard('::2-4') >>> wild12 = IPWildcard('1-2:3-4:5-6:7-8:9-a:b-c:d-e:0-ffff')
Membership¶
The primary operation on IPWildcard objects is testing whether an address is contained in the set covered by the wildcard. Both IPAddr and str values may be tested for membership.
| Operation | Results | Notes |
|---|---|---|
| addr in wildcard | addr matches wildcard, then True, else False | (1) |
| addr not in wildcard | addr matches wildcard, then False, else True | (1) |
Notes:
- addr may be an IPAddr or a string. Strings are automatically converted as with IPAddr(addr)
Other¶
The following additional operations are available on IPWildcard objects:
| Operation | Result |
|---|---|
| str(wildcard) | the string that was used to construct wildcard |
| wildcard.is_ipv6() | if wildcard contains IPv6 addresses then True, else False |
TCP Flags¶
A TCPFlags object represents the eight bits of flags from a TCP session.
- class netsa_silk.TCPFlags(value : int or str or TCPFlags)¶
Returns a new TCPFlags object with the given flags set. If value is an integer, it is interpreted as the bitwise integer representation of the flags. If value is a string, it is interpreted as a case-insensitive sequence of letters indicating individual flags, and optional white space. The mapping is described below.
Each supported flag has an assigned letter in string representations, is available as an attribute on TCPFlags values, and is available as a TCPFlags constant in netsa_silk:
| Flag | Meaning | Letter | TCPFlags attribute | netsa_silk constant |
|---|---|---|---|---|
| FIN | No more data from sender | F | flags.fin | TCP_FIN |
| SYN | Synchronize sequence numbers | S | flags.syn | TCP_SYN |
| RST | Reset the connection | R | flags.rst | TCP_RST |
| PSH | Push Function | P | flags.psh | TCP_PSH |
| ACK | Acknowledgment field significant | A | flags.ack | TCP_ACK |
| URG | Urgent Pointer field significant | U | flags.urg | TCP_URG |
| ECE | ECN-echo (RFC 3168) | E | flags.ece | TCP_ECE |
| CWR | Congestion window reduced (RFC 3168) | C | flags.cwr | TCP_CWR |
Bit-Manipulation¶
The following bit-manipulation operations are available on TCPFlags objects:
| Operation | Result |
|---|---|
| ~flags | the bitwise inversion (not) of flags |
| flags1 & flags2 | the bitwise intersection (and) of flags1 and flags2 |
| flags1 | flags2 | the bitwise union (or) of flags1 and flags2 |
| flags1 ^ flags2 | the bitwise exclusive disjunction (xor) of flags1 and flags2 |
Conversions¶
The following operations and methods may be used to convert TCPFlags objects into other types.
| Operation | Result |
|---|---|
| int(flags) | the integer value of the flags set in flags |
| str(flags) | a string representation of the flags set in flags |
| flags.padded() | a space-padded column aligned string representation of the flags set in flags |
| bool(flags) | if any flag is set in flags, then True, else False |
Matching¶
- TCPFlags.matches(flagmask : str) → bool¶
The TCPFlags.matches method may be used to determine if a TCPFlags value matches a given flag/mask specification. The specification is given as a string containing a set of flags that must be set, optionally followed by a slash and a set of flags that must be checked. (i.e. if “A” is not in the flag list but is in the mask, it must be false. If “U” is not in either, it may have any value.) For example, flags.matches('S/SA') would return True if SYN was set and ACK was not set in flags.
Examples:
>>> flags = TCPFlags('SAU') >>> flags.matches('S') True >>> flags.matches('SA/SA') True >>> flags.matches('S/SP') True >>> flags.matches('S/SA') False >>> flags.matches('SP/SP') False >>> flags.matches('A/SA') False
Support for SiLK versions before 3.0¶
Although netsa_silk is only fully supported by SiLK as of version 3.0, some legacy support for older versions of PySiLK is available. Some specific things to watch for if you need to work with older SiLK versions:
- For TCPFlags, the lower-case flag name attributes (e.g. flags.syn) on the object are not available. To work around this, use the TCPFlags.matches method, or perform bitwise operations on the constants in the module. (For example, instead of if flags.fin: stuff..., use if flags & TCP_FIN: stuff ....)
