Overview¶

The Golem Script Automation framework is a specialized extension of the NetSA Scripting Framework for constructing automated analysis scripts. Such scripts might be launched periodically from a cron job in order to produce regular sets of results in a data repository. In addition to the golem-specific extensions, netsa.script.golem offers the same functionality as netsa.script.

At its heart, Golem is a template engine combined with some synchronization logic across analytic time bins. The templates define the output paths for result data and the command line sequences and pipelines used to generate these results. Template variables are known as tags. Golem provides certain tags by default and others can be added or modified via the golem configuration functions.

Golem Automation is designed to allow developers to build scripts that easily consume the output results of other golem scripts without the need for detailed knowledge of the implementation details of the external script (e.g. how often it runs or what pathnames it uses for populating its data repository). When provided the path to the external script in its configuration, a golem script will interrogate the external script for information regarding its output results, automatically synchronize across processing windows, and make the result paths available for use in command templates as input paths.

In addition to analysis automation, Golem scripts also offer a query mode so that results in the data repository can be easily examined or pulled to local directories.

Golem offers a number of shortcuts and convenience functions specific to the SiLK Flow Analysis suite, but is not limited to using SiLK for analysis.

See the examples at the end of this chapter to learn how to write a golem script. See the description of template and tag usage for details on tags provided for use within templates. See the API reference for more thorough documentation of the features and interface. See the CLI reference for the standard command line parameters that golem enables.

Command Line Usage¶

Golem-enabled scripts offer standard command line parameters grouped into three categories: basic, repository-related, and query-related. Parameters must be enabled by the script author. Parameters can be enabled individually or by category. The add_golem_params function will enable all parameters.

Golem scripts also have the standard netsa.script command line options --help, --help-expert, and --verbose.

Metadata Functions¶

Golem scripts are an extension of netsa.script. As such, golem scripts offer the same functions as netsa.script. Script authors are encouraged to use the following metadata functions:

netsa.script.golem.set_title(script_title : str)

Set the title for this script. This should be the human-readable name of the script, and denote its purpose.

netsa.script.golem.set_description(script_description : str)

Set the description for this script. This should be a longer human-readable description of the script’s purpose, including simple details of its behavior and required inputs.

netsa.script.golem.set_version(script_version : str)

Set the version number of this script. This can take any form, but the standard major . minor (. patch ) format is recommended.

netsa.script.golem.set_package_name(script_package_name : str)

Set the package name for this script. This should be the human-readable name of a collection of scripts.

netsa.script.golem.set_contact(script_contact : str)

Set the point of contact email for support of this script, which must be a single string. The form should be suitable for treatment as an email address. The recommended form is a string containing:

Full Name <full.name@contact.email.org>

netsa.script.golem.set_authors(script_authors : str list)

Set the list of authors for this script, which must be a list of strings. It is recommended that each author be listed in the form described for set_contact.

netsa.script.golem.add_author(script_author : str)

Add another author to the list of authors for this script, which must be a single string. See set_authors for notes on the content of this string.

Please see netsa.script for additional functions, such as those used to add custom command line parameters.

Configuration Functions¶

Golem scripts are configured by calling functions from within a particular imported module. You can either import netsa.script.golem directly:

from netsa.script import golem

Or import as netsa.script.golem.script:

from netsa.script.golem import script

In either case, the imported module will provide identical functionality. All functions available within netsa.script are also available from within golem, with the addition of the golem-specific functions and classes. Please consult the netsa.script documentation for details on functions offered by that module.

The following functions configure the behavior of golem scripts.

netsa.script.golem.set_default_home(path : str[, path : str, ...])¶

Sets the default base path for this golem script in cases where the GOLEM_HOME environment variable is not set. Multiple arguments will be joined together as with os.path.join. If the provided path is relative, it is assumed to be relative to the directory in which script resides.

The actual home path will be decided by the first available source in the following order:

1. The GOLEM_HOME environment variable
2. The default home if set by this function
3. The directory in which the script resides

Subsequent path settings (e.g. set_repository) will be relative to the script home if they are not absolute paths.

netsa.script.golem.set_repository(path : str[, path : str, ...])¶

Sets the default path for the output results data repository. Multiple arguments will be joined together as with os.path.join. Output results will be stored in this directory or in a subdirectory beneath, depending on how each output template is specified. Relative paths are considered to be relative to the home path.

netsa.script.golem.set_suite_name(name : str)¶

Set the short suite name for this script, if it belongs to a suite of related scripts. This should be a simple label suitable for use as a component in paths or filenames. Defaults to None.

netsa.script.golem.set_name(name : str)¶

Set the short name for this script. The name should be a simple label suitable for use as a component in paths or filenames. This defaults to the basename of the script file itself, minus the ‘.py’ extension, if present.

netsa.script.golem.set_interval([days : int, minutes : int, hours : int, weeks : int])¶

Set how often this golem script is expected to generate results.

The interval roughly corresponds to how often the script should be run (such as from a cron job). Golem scripts will only process data for incomplete intervals over a provided date range, unless told otherwise via the --overwrite option.

netsa.script.golem.set_span([days : int, minutes : int, hours : int, weeks : int])¶

Set the span over which this golem script will expect input data for each processing interval. Defaults to the processing interval.

The span will manifest as how much data is being pulled from a SiLK repository or possibly how many outputs from another golem script are being consumed. For example, a script having a 4 week span might run once a week, pulling 4 weeks worth of data each time it runs.

See Intervals and Spans Explained for more details.

netsa.script.golem.set_skew([days : int, minutes : int, hours : int, weeks : int])¶

Shift the epoch from which all time bins and spans are anchored. For example, given a golem script with an interval of one week, skew can control which day of the week processing occurs.

netsa.script.golem.set_lag([days : int, minutes : int, hours : int, weeks : int])¶

Set the lag for this golem script relative to the current date and time. Defaults to 0.

As an example, 3 hours is a typical value for data to finish accumulating in a given hour within a SiLK repository. Setting lag to 3 hours effectively shifts the script’s concept of the current time that far into the past.

netsa.script.golem.set_realtime(enable=True)¶

Set whether or not this golem script will report output results in real time or not. Defaults to False.

Normally a golem script will wait until a processing interval has completely passed before performing any processing or reporting any output results. With set_realtime enabled, the golem script will consider the current processing bin to be the most recent, even if it extends to a future date.

Enabling realtime has a side effect of setting lag to zero.

netsa.script.golem.set_tty_safe(enable=True)¶

Set whether or not query results are safe to send to the terminal. Defaults to False.

netsa.script.golem.set_passive_mode(enable=False)¶

Controls whether or not an output option is required before running the main loop of the script. Defaults to False. Normally at least one repository-related or query-related option must be present or the script aborts. This is useful for scripts that require query behavior by default or are maintaining the repository in a custom fashion. If enabled, script authors should explicitely check whether or not repository updates were requested via the command line prior to updating the repository.

netsa.script.golem.add_tag(name : str, value : str or func)¶

Set a command template tag with key name. The provided value can be callable, in which case it is resolved once the main function is invoked. Tags can reference other tags. See template and tag usage for more information.

netsa.script.golem.add_loop(name : str, value : str list or func[, group_by : str or func, group_name : str, sep=', '])¶

Add a template tag under key name whose values cycle through those provided by values, either as an iterable or callable. In the latter case, the values are not resolved until the main function is invoked.

Optional keyword arguments:

group_by

Specifies how to group entries from values into a single loop entry. If the provided value is callable, the function should accept a single entry from values and return the group label for that entry or the original string, e.g. ‘LAB2’ might return ‘LAB’. If the value is a dictionary, it will resolve to the mapped value if present. If it is an iterable of prefix matches, they will be converted into a regular expression to be applied to the beginning of each entry. In all cases, if there is no match or result, the original entry becomes its own group label.

group_name

This is the name of the template tag under which the group label appears, defaulting to the value of name appended with ‘_group’. In the above example, if name is ‘sensor’, then %(sensor)s might resolve to ‘LAB1,LAB2,LAB3’ whereas %(sensor_group)s would merely resolve to ‘LAB’.

sep

Depending on how these loops are visited, the template value under name might contain multiple values from values. Under these circumstances, the resulting string is joined by the value of ‘sep’ (default: ‘,’)

Note that adding a loop will automatically add an additional query command line parameter named after the name for limiting which values to process within the loop. If grouping was requested, an additional parameter named after the group_name is also added. If these parameters are not desired or need to be modified, use the modify_golem_param function.

See template usage for more information on templates and loop values.

netsa.script.golem.add_sensor_loop([name='sensor' : str, sensors : str list or func, group_by : str or func, group_name : str, auto_group=False])¶

This is a convenience function for adding a template loop tag under key name whose values are based on sensors defined in a SiLK repository. Special note is taken that these loop values represent SiLK sensors, so any netsa.script.Flow_params tags that are defined (see add_flow_tag) will have their sensors parameter automatically bound (if not explicitly bound to something else) to the last sensor loop defined by this function.

Optional keyword arguments:

name

The tag name within the template. Defaults to ‘sensor’, accessible from within templates as the tag %(sensor)s

sensors

The source of sensor names, specified either as an iterable or callable. Defaults to the get_sensors function, which will interrogate the local SiLK repository once the main function is invoked.

group_by

Same as with add_loop

group_name

Same as with add_loop

auto_group

Causes group_by to be set to the get_sensor_group function, a convenience function that strips numbers, possibly preceded by an underscore, from the end of sensor names. This can provide serviceable sensor grouping for descriptive sensor names (e.g. ‘LAB0’, ‘LAB1’, ‘LAB2’) but will not be of much use if they are generically named (e.g. ‘S0’, ‘S1’, etc).

netsa.script.golem.add_flow_tag(name : str[, flow_class : str, flow_type : str, flowtypes : str, sensors : str, start_date : str, end_date : str, input_pipe : str, xargs : str, filenames : str])¶

Add a netsa.script.Flow_params object as a template tag under key name. The rest of the keyword arguments correspond to the same parameters accepted by the netsa.script.Flow_params constructor and serve to map these fields to either specific template tags in the golem script or to the literal string if it is not present in the template tags.. If not otherwise specified, the start_date and end_date attributes are bound to the values of %(golem_start_date)s and %(golem_end_date)s, respectively, for each loop iteration. Additionally, if any sensor-specific loops were specified via add_sensor_loop, the sensors parameter defaults to the tag associated with the last defined sensor loop (typically %(sensor)s). The resulting netsa.script.Flow_params object is associated with a tag entry specified by name. The values of tags associated with netsa.script.Flow_params attributes in this way are still accessible under their original tag names.

The following optional keyword arguments are available to map template tag values to their corresponding attributes in the flow params object: flow_class, flow_type, flowtypes, sensors, start_date, end_date, input_pipe, xargs, and filenames.

See template and tag usage for more information on templates.

netsa.script.golem.add_output_template(name : str, template : str[, scope : int, mime_type : str, description : str])¶

Define an output template tag key name with the provided template. The provided template can use any of the tags available for each processing iteration. Wildcards are acceptable in the template specification. Absolute paths are not allowed since outputs must reside in the data repository.

The following optional keyword arguments are available. Pass any of them a value of None to disable entirely:

scope

Defines how many intervals of this output are required to represent a complete analysis result in cases where the output from a single interval represents a partial result. For example, a golem script might have an interval of 1 day whereas a “complete” set of results is 7 days worth relative to any particular day.

mime_type

The expected MIME Content-Type of the output file, if any.

description

A long-form text description, if any, of the contents of this output file.

See template and tag usage for more information on templates.

Templates are not required to reference all distinguishing tags and can therefore be ‘lossy’ across loop iterations if such a thing is desired.

netsa.script.golem.add_input_template(name : str, template : str[, required=True, mime_type : str, descriptions : str])¶

Define an input template tag under key name. This is useful for defining inputs not produced by other golem scripts. Wildcards are allowed in the template specification.

The provided template can use the any of the set of tags available on each processing iteration. The resolved string is available to command templates under key name.

Optional keyword arguments:

required

When False will ignore missing inputs once the template is resolved. Defaults to True.

count

Specifies how many intervals (as defined by this script) over which to resolve the input template. For example, if this script has an interval of one week, a count of 4 will resolve to the last 4 days of the week in question.

offset

Specify how far backwards (in intervals) to anchor this input. If a count has been specified, the offset shifts the entire count of intervals.

cover

If True, a count will be calculated such that the template will be resolved over all intervals covered by the span of this script. For example, a script with an interval of one day and span of seven, the input template will be resolved for all seven days in the span.

source_name

Specify what produced this input For informational purposes when listing script inputs.

mime_type

The expected MIME Content-Type of the input.

description

A long-form text description of the expected contents of this input.

See template and tag usage for more information on templates.

netsa.script.golem.add_golem_input(golem_script : str, name : str [, output_name : str, count : int, cover=False, offset : int, span : timedelta, join_on : str or str list, join : dict, required=True)¶

Specify a tag, under key name, that represents the path (or paths) to an output of an external golem script. Only a single output can be assosciated at at time – if the external script has multiple outputs defined, then additional calls to this function are necessary for each output of interest.

For each output template, efforts are made to synchronize across time intervals and loop tags as appropriate. By default, this is the output of the most recent interval of the other golem script that corresponds to the interval currently under consideration within the local script. Covering multiple intervals of the input is controlled by cover, count, or span. If, for example, you’re pulling inputs with an hourly interval into a script with a daily interval, the cover parameter should probably be used, otherwise only a single hour would be pulled in.

Matching loop tags are automatically joined unless the join parameter is provided, in which case the only joins that happen are the ones provided.

Optional keyword arguments:

output_name

The tag name used for this output in the external script if it differs from the value of name used locally for this input. By default the names are assumed to be identical.

count

Specifies how many intervals (as defined by the external script) of output data are to be used as input. By default, the most recent interval of the other golem script that corresponds to the local interval currently under consideration is provided.

For example, if the other golem script has an interval of one week, a count of 4 will provide the last 4 weeks of output from that script.

offset

Specify how far back (in units of the other script’s interval) to reference for this input. Defaults to the most recent corresponding interval. If a count has been specified, the offset shifts the entire count of intervals.

For example, if the other script’s interval is one week, an offset of -1 will reference the output from the week prior to the most recent week. Negative and positive offsets are equivalent for these purposes, they always reach backwards through time.

cover

If True, a count is calculated that will fully cover the local interval under consideration. This option cannot be used with the count or offset options. Defaults to False.

For example, if the local interval is one week, and the other interval is one day, then this is equivalent to specifying a count of 7 (days).

span

A datetime object that represents a span of time that covers the intervals of interest. Based on the other script’s interval, a count is calculated that will cover the provided span. Cannot be used simultaneously with count, offset, or cover.

join

A dictionary or iterable of tuples that provides an equivalence mapping between template loops defined in the other golem script and locally defined loops. If no join mapping is provided, an attempt is made to join on loops sharing the same name. If the join parameter is provided, no auto-join is performed.

For example, if the other script defines a template loop on the %(my_sensor)s tag, and the local script defines a loop on %(sensor)s, a mapping from ‘my_sensor’ to ‘sensor’ will ensure that for each iteration over the values of %(sensor)s the input tag value is also sensor-specific based on its %(my_sensor)s loop. Without this association, the input tag would resolve to all outputs across all sensors, regardless of which sensor or sensor group was currently under local consideration. Iterations with no valid mapping are ignored, as opposed to when a valid association exists but the other output is missing.

required

If False or 0 and the expected output from the other golem script is missing, continue processing rather than raising an exception. If specified as a positive integer, it means at least that many of the other inputs should exist, otherwise an exception is thrown. By default, at least one input is required. It is up to the developer to handle missing inputs (i.e. empty template tags) appropriately in these cases.

netsa.script.golem.add_input_group(name : str, group : list)¶

Group the given inputs under the provided tag as a single GolemArgs object. The result can be used the same as the regular inputs.

netsa.script.golem.add_output_group(name : str, group : list)¶

Group the given outputs under the provided tag as a single GolemArgs object. The result can be used the same as the regular outputs, including as inputs to other golem scripts.

netsa.script.golem.add_query_handler(name : str, query_handler : func)¶

Define a query handler under tag key name to be processed by the callable query_handler. The output will only be generated dynamically when specifically requested via query-related parameters.

The provided callable will be passed the name of this query and a ‘tags’ dictionary for use with templates. And additonal tag %(golem_query_tgt)s will be provided in the standard dictionary that contains the output path for this query. The function is responsible for creating this output.

netsa.script.golem.add_self_input(name : str, output_name : str[, count : int, offset : int, span : timedelta])¶

Specify an input template tag, under key name, associated with this script’s own output from prior intervals. Since output tag names will necessarily collide, name is the new tag name for the input and output_name is the name of the output template.

Optional keyword arguments:

count

Same as with add_golem_input

offset

Same as with add_golem_input, except defaults to -1. If self-referencing for purposes of delta-encoding, 0 should probably be specified.

span

Same as with add_golem_input

Assuming a local loop over the template tag %(sensor)s, the following example:

>>> script.add_self_input('result', 'prior_result')

...is equivalent to this:

>>> script.add_golem_input(script.get_script_path(), 'prior_result',
>>>     output_name='result',
>>>     offset=-1,
>>>     join_on=['sensor'],
>>>     required=False)

Note

If this output happens to have a scope the developer should ensure that this self-reference means what is intended for ‘most recent’ output.

For example, if this golem script has an interval of 1 day, but the output has a scope of 28 (days), the example above would capture the prior 28 days of output due to the offset of -1.

If the script is delta-encoding its current result with its own prior results, however, probably what is desired would be the prior 27 days, in which case offset should be specified as 0. The 28th day in this scope scenario is the very result currently being generated, which does not exist yet, and therefore will not appear in the template as a ‘prior’ result. After processing is complete, however, it will appear in the collected outputs if a different golem script references this interval as input.

netsa.script.golem.add_golem_source(path : str)¶

Adds a directory within which to search for other golem scripts. Directories are searched in the following order: 1) paths within the colon-separated list of directories in the GOLEM_SOURCES environment variable; 2) any paths added by this function (multiple invocations allowed); 3) this script’s own directory as reported by the get_script_dir function, and finally 4) this script’s home directory as reported by the get_home function, if different than the script directory. These directories are only searched if the external script is specified as a relative path.

netsa.script.golem.get_script_path() → str¶

Returns the normalized absolute path to this script.

netsa.script.golem.get_script_dir() → str¶

Returns the normalized absolute path to the directory in which this script resides.

netsa.script.golem.get_home() → str¶

Returns the current value of this script’s home path if it has been set. Otherwise, defaults to the contents of the GOLEM_HOME environment variable (if set), or finally, the directory in which this script resides.

netsa.script.golem.get_repository() → str¶

Returns the current path for this script’s data output repository, or None if not set.

Parameter Functions¶

The following functions are used to enable and modify the standard command-line parameters available for golem scripts. No command line parameters are enabled by default, so at least one of these should be invoked in a typical golem script:

netsa.script.golem.add_golem_params([without_params : str list])¶

Enables all golem command line parameters. Equivalent to individually invoking each of add_golem_basic_params, add_golem_repository_params, and add_golem_query_params. Optionally accepts a list of parameters to exclude.

netsa.script.golem.add_golem_basic_params([without_params : str list])¶

Enables basic golem command line parameters. Optionally accepts a list of parameters to exclude.

netsa.script.golem.add_golem_query_params([without_params : str list])¶

Enables query-related golem command line parameters. Optionally accepts a list of parameters to exclude.

netsa.script.golem.add_golem_repository_params([without_params : str list])¶

Adds repository-related golem command line parameters. Optionally accepts a list of parameters to exclude.

netsa.script.golem.add_golem_param(name : str[, alias : str])¶

Enables a particular golem command line parameter. Accepts the same optional keyword parameters as the modify_golem_param function with the exception of enable, which is implied. An example is the alias parameter, which which can be provided to change the default parameter string (for example, aliasing --last-date to --date in cases where date ranges are not desired).

netsa.script.golem.modify_golem_param(name : str[, enabled : bool, alias : str, help : str, ...])¶

Modifies the settings for the given golem script parameter. Accepts new values for the golem-specific keywords enabled and alias, along with the usual netsa.script parameter keywords (e.g. help, required, default, default_help, description, mime_type)

Processing and Status Functions¶

The following functions are intended for use within the main function during processing or examination of status.

netsa.script.golem.execute(func)¶

Executes the main function of a golem script. This should be called as the last line of any golem script, with the script’s main function (whatever it might be named) as its only argument.

Warning

It is important that most, if not all, actual work the script does is done within this function. Golem scripts (as with all NetSA Scripting Framework scripts) may be loaded in such a way that they are not executed, but merely queried for metadata instead. If the golem script performs significant work outside of the main function, metadata queries will no longer be efficient. Golem scripts must use this execute function rather than netsa.script.execute.

netsa.script.golem.process([golem_view : GolemView])¶

Returns a GolemProcess wrapper around the given golem view, which defaults to the main script view. The result behaves much like a GolemTags object. Iterating over it returns a dictionary of resolved template tags while performing system level interactions (such as checking for input existence and creating output directories and/or files) in preparation for whatever processing the developer specifies in the processing loop. Views that have already completed processing are ignored.

Also takes an optional ‘exception_handler’ keyword argument which must be a function that accepts an exception and a golem view object (advanced use only).

netsa.script.golem.loop([golem_view : GolemView])¶

Returns a GolemTags view of the given golem view, which defaults to the main script view. No system level processing happens while iterating over or interacting with this view.

netsa.script.golem.inputs([golem_view : GolemView])¶

Returns a GolemInputs view of the given golem view, which defaults to the main script view. No system level processing happens while iterating over or interacting with this view.

netsa.script.golem.outputs([golem_view : GolemView])¶

Returns a GolemOutputs view of the given golem view, which defaults to the main script view.. No system level processing happens while iterating over or interacting with this view.

netsa.script.golem.is_complete([golem_view : GolemView])¶

Examines the outputs of the optionally provided GolemView object, which defaults to the main script view, and examines the status of the outputs for each processing interval. If all appear to be complete, returns True, otherwise False.

netsa.script.golem.script_view()¶

Returns the currently defined global GolemView object.

netsa.script.golem.current_view([golem_view : GolemView])¶

Returns a version of the given GolemView object, which defaults to the main script view, for the most recent interval available.

Utility Functions¶

The following functions provide some SiLK-specific tools and other potentially useful features for script authors.

netsa.script.golem.get_sensors() → str list¶

Retrieves a list of sensors as defined in the local SiLK repository configuration.

netsa.script.golem.get_sensor_group(sensor : str) → str¶

Convenience function, such as can be passed as a value for the group_by parameter in add_sensor_loop, for extracting a sensor ‘group’ out of a sensor name. Groups are determined by extracting prefixes made of ‘word’ characters excluding ‘_’. For example, two sensors called ‘LAB0’ and ‘LAB1’ would be grouped under ‘LAB’.

netsa.script.golem.get_sensors_by_group([grouper : func, sensors : str list])¶

Convenience function that uses the callable grouper to construct a tuple of named pairs of the form (group_name, members) suitable for use in constructing a dictionary. Defaults to the get_sensor_group function.

netsa.script.golem.get_args() → GolemArgs¶

Returns a GolemArgs object containing any non-parameter arguments that were provided to the script on the command line.

Additional Functions¶

Please see the following sections in the netsa.script documentation for details regarding other functions available within netsa.script.golem:

• Adding Script Parameters
• Verbose Output
• Flow Data Parameters
• Adding Additional Output

Usage of Tags, Loops, and Templates¶

Golem is a templating engine. Based on a script’s configuration, the processing loop will iterate over time bins (processing intervals) and the values of any other loops defined for tags in the script. For each iteration, a dictionary of template tags is produced for use with such utilities as netsa.util.shell. A typical golem script looks something like this:

#!/usr/bin/env python

from netsa.script import golem
from netsa.util import shell

# golem configuration and command line parameter
# configuration up here

def main():

    # All 'real' work should happen in this function.
    # Before invoking the processing loop, perhaps do some
    # configuration and prep work

    for tags in golem.process():

        # Set up per-iteration prep work, such as perhaps
        # some temp files

        # maybe modify contents of tags
        tags['temp_file'] = ...

        # set up command templates

        cmd1 = ...
        cmd2 = ...
        ...

        # run the commands
        shell.run_parallel(cmd1, cmd2, vars=tags)

# pass main function to module for invocation and handling
golem.execute(main)

The following template tags are automatically available to each iteration over a golem processing loop:

For intervals, time bins are bounded by golem_bin_date and golem_end_date. Spans, on the other hand, are bounded by golem_start_date and golem_end_date. For details on how intervals and spans relate to one another and format precisions, see Intervals and Spans Explained.

In addition to the standard golem tags defined above, all other tags, loops, inputs, and outputs defined in the initial script configuration are available for use in templates. For example, assume a script makes the following declarations:

script.add_tag('in_types',  'in,inweb')
script.add_tag('out_types', 'out,outweb')
script.add_tag('month', "%(golem_month)02d/%(golem_year)d")
script.add_sensor_loop()
script.add_flow_tag('in_flow',  flow_type='in_types')
script.add_flow_tag('out_flow', flow_type='out_types')
script.add_output_template('juicy_set', ext='.set',
    description="Target 'juicy' set to generate.",
    mime_type='application/x-silk-ipset')

During each iteration of the processing loop, the tags dictionary will now include the following additional template entries:

in_types

"in,inweb"

out_types

"out,outweb"

month

"%(golem_month)02d/%(golem_year)d" \
    % (tags['golem_month'], tags['golem_year'])

sensor

the current iteration value of script.get_sensors()

in_flow

Flow_params(
    start_date = tags['golem_start_date'],
    end_date   = tags['golem_end_date'],
    sensors    = tags['sensor'],
    flow_type  = tags['in_types'])

out_flow

Flow_params(
    start_date = tags['golem_start_date'],
    end_date   = tags['golem_end_date'],
    sensors    = tags['sensor'],
    flow_type  = tags['out_types'])

juicy_set

"%(golem_name)s/%(sensor)s/"                             \
"%(golem_name)s.%(sensor)s.%(golem_start_date_iso)s.set" \
    % (tags['golem_name'], tags['sensor'],               \
       tags['golem_start_date_iso'])

Sometimes, depending on how loops and dependencies are arranged and how views are being manipulated, an input or output tag for the current view might contain multiple values (e.g. multiple filenames). In these cases, the resolved values are bundled into a GolemArgs object, which in turn resolves as a string of paths separated by spaces in the final command template.

Intervals and Spans Explained¶

Intervals and spans represent two different concepts.

An interval is a processing interval which represents how frequently the script is intended to produce results. This is roughly analogous to how frequently the script might be invoked via a cron job, except that golem scripts will back-fill missing results upon request and ignore intervals that appear to already have results present. An interval is always represented by the first timestamp contained within the interval.

A span, on the other hand, is a data window which represents how much input data the script is expected to consume, whether it be from a SiLK repository, results from other golem scripts, or other sources.

By default, the span of a golem script is the same as its interval. Not much surprising happens when the values are equal. They can be different, however. For example, a script might have a weekly interval yet consume 4 weeks worth of data for each of those weeks. Alternatively, a script might run every 4 weeks yet consume only a day’s worth of data, akin to a monthly snapshot.

Intervals are anchored relative to a particular epoch. Intervals are always relative to midnight of the first Monday after January 1st, 1970 which was January 5th. Weeks therefore begin with Monday and multiples of weeks are always relative to that particular Monday. If another day of the week is desired, use the set_skew configuration function.

Spans are always anchored relative to the end of the processing interval.

In the tags dictionary provided for each processing loop, the interval is represented by the golem_bin_date entry and the span is represented by golem_start_date and golem_end_date. Given a 3 week interval and a 4 week span, for example, these values are aligned like so:

interval:             bin_date             next_bin_date
                         |                       |
                 |-------|-------|-------|-------|
                 |                               |
span:       start_date                        end_date

Note that end_date is not inclusive—its actual value is the value of next_bin_date minus one millisecond. next_bin_date, on the other hand, can be handy if you want to represent your results files by the end of the processing interval (“as of” next_bin_date as opposed to “begining with” bin_date).

Each of these entries are represented by datetime.datetime objects along with an assortment of formatted string representations. If both the interval and span have a magnitude of at least a day or more, the formatted string variations look like so:

If either the interval or span is less than a day, hours are included:

In all of the examples covered in this documentation, result templates are all based on the golem_bin_date values, i.e. the processing interval. As the example diagram above illustrates, this may not be intuitive as to what data is represented in the results. It is up to script authors to decide how to name their results, but they should choose a convention, stay consistent with it, and document the decision.

Examples¶

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell

script.set_title('Daily IP Sets')
script.set_description("""
    Convert daily IP lists into rwset files.
""")
script.set_version("0.1")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('daily_set')
script.set_interval(days=1)
script.set_span(days=1)

script.add_golem_params()

script.set_repository('dat')

script.add_input_template('daily_txt',
    "/data/incoming/daily.%(golem_bin_iso)s.txt",
    description="Daily IP text files",
    mime_type='text/plain')

script.add_output_template('daily_set',
    "daily/daily.%(golem_bin_iso)s.set",
    description="Daily IP Sets",
    mime_type='application/x-silk-ipset')

def main():
    cmd = "rwsetbuild %(daily_txt)s %(daily_set)s"
    for tags in script.process():
        shell.run_parallel(cmd, vars=tags)

script.execute(main)

from netsa.script.golem import script
from netsa.util import shell

script.set_title('Daily IP Sets')
script.set_description("""
    Convert daily IP lists into rwset files.
""")
script.set_version("0.1")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('daily_set')

script.set_interval(days=1)
script.set_span(days=1)

script.add_golem_params()

script.set_repository('dat')

script.add_input_template('daily_txt',
    "/data/incoming/daily.%(golem_bin_iso)s.txt",
    description="Daily IP text files",
    mime_type='text/plain')

script.add_output_template('daily_set',
    "daily/daily.%(golem_bin_iso)s.set",
    description="Daily IP Sets",
    mime_type='application/x-silk-ipset')

def main():
    cmd = "rwsetbuild %(daily_txt)s %(daily_set)s"
    for tags in script.process():
        shell.run_parallel(cmd, vars=tags)

script.execute(main)

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell
from netsa import files

script.set_title('SiLK Daily Active Internal Hosts')
script.set_description("""
    Daily inventory of observed internal host activity.
""")
script.set_version("0.1")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('daily_set')
script.set_interval(days=1)
script.set_span(days=1)

script.add_golem_params()

script.set_repository('dat')

script.add_tag('in_types',  'in,inweb')
script.add_tag('out_types', 'out,outweb')

script.add_output_template('internal_set',
    "internal/daily/daily.%(golem_bin_iso)s.set",
    description="Daily Internal host activity",
    mime_type='application/x-silk-ipset')

def main():
    for tags in script.process():
        tags['out_fifo'] = files.get_temp_pipe_name()
        tags['in_fifo']  = files.get_temp_pipe_name()
        cmd1 = [
            "rwfilter --start-date=%(golem_start_silk)s"
                " --end-date=%(golem_end_silk)s"
                " --type=%(in_types)s"
                " --proto=0-255 --pass=stdout",
            "rwset --sip=%(out_fifo)s"]
        cmd2 = [
            "rwfilter --start-date=%(golem_start_silk)s"
                " --end-date=%(golem_end_silk)s"
                " --type=%(out_types)s"
                " --proto=0-255 --pass=stdout",
            "rwset --dip=%(in_fifo)s"]
        cmd3 = [
            "rwsettool --union --output-path=%(internal_set)s"
                " %(out_fifo)s %(in_fifo)s"]
        shell.run_parallel(cmd1, cmd2, cmd3, vars=tags)

script.execute(main)

script.add_tag('in_types',  'in,inweb')
script.add_tag('out_types', 'out,outweb')

def main():
    for tags in script.process():
        tags['out_fifo'] = files.get_temp_pipe_name()
        tags['in_fifo']  = files.get_temp_pipe_name()

cmd1 = [
    "rwfilter --start-date=%(silk_start)s"
        " --end-date=%(silk_end)s"
        " --type=%(in_types)s"
        " --proto=0-255 --pass=stdout",
    "rwset --sip=%(out_fifo)s"]
cmd2 = [
    "rwfilter --start-date=%(silk_start)s"
        " --end-date=%(silk_end)s"
        " --type=%(out_types)s"
        " --proto=0-255 --pass=stdout",
    "rwset --dip=%(in_fifo)s"]
cmd3 = [
    "rwsettool --union --output-path=%(internal_set)s"
        " %(out_fifo)s %(in_fifo)s"]
shell.run_parallel(cmd1, cmd2, cmd3, vars=tags)

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell

script.set_title('Weekly Active Internal Host Set')
script.set_description("""
    Aggregate daily internal activity sets over the last week.
""")
script.set_version("0.1")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('weekly_set')
script.set_interval(weeks=1)
script.set_span(weeks=1)

script.add_golem_params()

script.set_repository('dat')

script.add_golem_input('daily_set.py', 'daily_set', cover=True)

script.add_output_template('weekly_set',
    "weekly/weekly.%(golem_bin_iso)s.set",
    description="Aggregated weekly sets.",
    mime_type='application/x-silk-ipset')

def main():
    cmd = "rwsettool --union --output-path=%(weekly_set)s %(daily_set)s"
    for tags in script.process():
        shell.run_parallel(cmd, vars=tags)

script.execute(main)

script.set_interval(weeks=1)
script.set_span(weeks=1)

script.add_golem_input('daily_set.py', 'daily_set', cover=True)

def main():
    cmd = "rwsettool --union --output-path=%(weekly_set)s %(daily_set)s"
    for tags in script.process():
        shell.run_parallel(cmd, vars=tags)

script.execute(main)

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell
from netsa import files

script.set_title('Active Internal Hosts')
script.set_description("""
    Per-sensor inventory of observed internal host activity over a
    four week window of observation, generated every three weeks.
""")
script.set_version("0.1")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('internal')
script.set_interval(weeks=3)
script.set_span(weeks=4)

script.add_golem_params()

script.set_repository('dat')

script.add_tag('in_types',  'in,inweb')
script.add_tag('out_types', 'out,outweb')

script.add_loop('sensor', ["S0", "S1", "S2", "S3"])

script.add_output_template('internal_set',
    "internal/internal.%(sensor)s.%(golem_bin_iso)s.set",
    description="Internal host activity",
    mime_type='application/x-silk-ipset')

def main():
    for tags in script.process():
        tags['out_fifo'] = files.get_temp_pipe_name()
        tags['in_fifo']  = files.get_temp_pipe_name()
        cmd1 = [
            "rwfilter --start-date=%(golem_start_silk)s"
                " --end-date=%(golem_end_silk)s"
                " --type=%(in_types)s"
                " --sensors=%(sensor)s"
                " --proto=0-255 --pass=stdout",
            "rwset --sip=%(out_fifo)s"]
        cmd2 = [
            "rwfilter --start-date=%(golem_start_silk)s"
                " --end-date=%(golem_end_silk)s"
                " --type=%(out_types)s"
                " --sensors=%(sensor)s"
                " --proto=0-255 --pass=stdout",
            "rwset --dip=%(in_fifo)s"]
        cmd3 = [
            "rwsettool --union --output-path=%(internal_set)s"
                " %(out_fifo)s %(in_fifo)s"]
        shell.run_parallel(cmd1, cmd2, cmd3, vars=tags)

script.execute(main)

script.set_interval(weeks=3)
script.set_span(weeks=4)

script.add_loop('sensor', ["S0", "S1", "S2", "S3"])

script.add_output_template('internal_set',
    "internal/internal.%(sensor)s.%(golem_bin_iso)s.set",
    description="Internal host activity",
    mime_type='application/x-silk-ipset')

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell
from netsa import files

script.set_title('Active Internal Hosts')
script.set_description("""
    Per-sensor inventory of observed internal host activity over a
    four week window of observation, generated every three weeks.
""")
script.set_version("0.2")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('internal')
script.set_interval(weeks=3)
script.set_span(weeks=4)

script.add_golem_params()

script.set_repository('dat')

script.add_tag('in_types',  'in,inweb')
script.add_tag('out_types', 'out,outweb')

script.add_sensor_loop()

script.add_flow_tag('in_flow',  flow_type='in_types')
script.add_flow_tag('out_flow', flow_type='out_types')

script.add_output_template('internal_set',
    "internal/internal.%(sensor)s.%(golem_bin_iso)s.set",
    description="Internal host activity",
    mime_type='application/x-silk-ipset')

def main():
    for tags in script.process():
        tags['out_fifo'] = files.get_temp_pipe_name()
        tags['in_fifo']  = files.get_temp_pipe_name()
        cmd1 = [
            "rwfilter %(out_flow)s --proto=0-255 --pass=stdout",
            "rwset --sip=%(out_fifo)s"]
        cmd2 = [
            "rwfilter %(in_flow)s --proto=0-255 --pass=stdout",
            "rwset --dip=%(in_fifo)s"]
        cmd3 = [
            "rwsettool --union --output-path=%(internal_set)s"
                " %(out_fifo)s %(in_fifo)s"]
        shell.run_parallel(cmd1, cmd2, cmd3, vars=tags)

script.execute(main)

script.add_sensor_loop()

script.add_flow_tag('in_flow',  flow_type='in_types')
script.add_flow_tag('out_flow', flow_type='out_types')

cmd1 = [
    "rwfilter %(out_flow)s --proto=0-255 --pass=stdout",
    "rwset --sip=%(out_fifo)s"]
cmd2 = [
    "rwfilter %(in_flow)s --proto=0-255 --pass=stdout",
    "rwset --dip=%(in_fifo)s"]

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell
from netsa import files

script.set_title('Daily Internal Port 25 Activity')
script.set_description("""
    Daily per-sensor-group inventory of observed internal host
    activity on port 25.
""")
script.set_version("0.1")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('p25')
script.set_interval(days=1)
script.set_span(days=1)

script.add_golem_params()

script.set_repository('dat')

script.add_sensor_loop(auto_group=True)

script.add_tag('out_type', 'out,outweb')
script.add_flow_tag('out_flow', flow_type='out_type')

script.add_golem_input('internal.py', 'internal_set', join_on='sensor')

script.add_output_template('p25_set',
    "p25/p25.%(sensor_group)s.%(golem_bin_iso)s.set",
    description="Daily IP set for internal port 25 activity",
    mime_type='application/x-silk-ipset')

def main():
    for tags in script.process():
        tags['in_fifo'] = files.get_temp_pipe_name()
        cmd1 = [
            "rwsettool --union"
                " --output-path=%(in_fifo)s
                " %(internal_set)s"]
        cmd2 = [
            "rwfilter %(out_flow)s"
                " --proto=6"
                " --sport=25"
                " --packets=2-"
                " --sipset=%(in_fifo)s"
                " --pass=stdout",
            "rwset --sip-set=%(p25_set)s"]
        shell.run_parallel(cmd1, cmd2, vars=tags)

script.execute(main)

script.set_interval(days=1)
script.set_span(days=1)

script.add_sensor_loop(auto_group=True)

script.add_golem_input('internal.py', 'internal_set', join_on='sensor')

script.add_output_template('p25_set',
    "p25/p25.%(sensor_group)s.%(golem_bin_iso)s.set",
    description="Daily IP set for internal port 25 activity",
    mime_type='application/x-silk-ipset')

def main():
    for tags in script.process():
        tags['in_fifo'] = files.get_temp_pipe_name()
        cmd1 = [
            "rwsettool --union"
                " --output-path=%(in_fifo)s
                " %(internal_set)s"]
        cmd2 = [
            "rwfilter %(out_flow)s"
                " --proto=6"
                " --sport=25"
                " --packets=2-"
                " --sipset=%(in_fifo)s"
                " --pass=stdout",
            "rwset --sip-set=%(p25_set)s"]
        shell.run_parallel(cmd1, cmd2, vars=tags)

script.execute(main)

#!/usr/bin/env python

from netsa.script.golem import script
from netsa.util import shell
from netsa import files

script.set_title('Active Internal Hosts')
script.set_description("""
    Daily per-sensor inventory of observed internal host activity,
    delta-encoded using the prior four weeks of results.
""")
script.set_version("0.3")
script.set_contact("H.P. Fnord <fnord@example.com>")
script.set_authors(["H.P. Fnord <fnord@example.com>"])

script.set_name('internal')
script.set_interval(days=1)
script.set_span(days=1)

script.add_golem_params()

script.set_repository('dat')

script.add_tag('in_types',  'in,inweb')
script.add_tag('out_types', 'out,outweb')

script.add_sensor_loop()

script.add_flow_tag('in_flow',  flow_type='in_types')
script.add_flow_tag('out_flow', flow_type='out_types')

script.add_output_template('internal_set',
    "internal/internal.delta.%(sensor)s.%(golem_bin_iso)s.set",
    description="Delta set of internal host activity",
    mime_type='application/x-silk-ipset',
    scope=28)

script.add_self_input('prior_set', 'internal_set', offset=0)

def main():
    for tags in script.process():
        tags['out_fifo'] = files.get_temp_pipe_name()
        tags['in_fifo']  = files.get_temp_pipe_name()
        cmds = []
        cmds.append([
            "rwfilter %(out_flow)s --proto=0-255 --pass=stdout",
            "rwset --sip=%(out_fifo)s"])
        cmds.append([
            "rwfilter %(in_flow)s --proto=0-255 --pass=stdout",
            "rwset --dip=%(in_fifo)s"])
        cmds.append([
            "rwsettool --union --output-path=%(current_set)s"
                " %(out_fifo)s %(in_fifo)s"])
        if tags['prior_set']:
            tags['current_set'] = files.get_temp_pipe_name()
            cmds.append([
              "rwsettool --difference"
                  " --output-path=%(internal_set)s"
                  " %(current_out)s %(prior_set)s"])
        else:
            tags['current_set'] = tags['internal_set']

        shell.run_parallel(vars=tags, *cmds)

script.execute(main)

script.set_interval(days=1)
script.set_span(days=1)

script.add_output_template('internal_set',
    "internal/internal.delta.%(sensor)s.%(golem_bin_iso)s.set",
    description="Delta set of internal host activity",
    mime_type='application/x-silk-ipset',
    scope=28)

script.add_self_input('prior_set', 'internal_set', offset=0)

def main():
    for tags in script.process():
        tags['out_fifo'] = files.get_temp_pipe_name()
        tags['in_fifo']  = files.get_temp_pipe_name()
        cmds = []
        cmds.append([
            "rwfilter %(out_flow)s --proto=0-255 --pass=stdout",
            "rwset --sip=%(out_fifo)s"])
        cmds.append([
            "rwfilter %(in_flow)s --proto=0-255 --pass=stdout",
            "rwset --dip=%(in_fifo)s"])
        cmds.append([
            "rwsettool --union --output-path=%(current_set)s"
                " %(out_fifo)s %(in_fifo)s"])
        if tags['prior_set']:
            tags['current_set'] = files.get_temp_pipe_name()
            cmds.append([
              "rwsettool --difference"
                  " --output-path=%(internal_set)s"
                  " %(current_out)s %(prior_set)s"])
        else:
            tags['current_set'] = tags['internal_set']

        shell.run_parallel(vars=tags, *cmds)

script.execute(main)

Classes¶