Monitoring for Large-Scale Networks

fixbuf

libfixbuf: Lists in IPFIX

How-To Handle BasicLists, SubTemplateLists, & SubTemplateMultiLists

General Information

Background on the list types is in "Export of Structured Data in IPFIX" (RFC 6313). Each of the list structures uses a nested list of data. The basic list nests a single information element, while the others use a nested template. The template used for nesting is part of the listed templates sent to the collector when the connection is made, or when the data transfer begins. There is no way to mark a template from this list as one that will be nested, or one that will be used as the highest level template. Each of the templates in the list are treated as equals.

The collector does not learn which template or information element is nested until the data arrives. This requires flexibility in the collectors to handle each of the possible results.

Internal Templates for Sub Templates

The setting of the internal template has not changed with the addition of the list structures. The internal template is still used to perform the initial decoding of the data that arrives at the collector.

Basic lists are not transcoded in the same way as templates because they contain just one information element, thus having no order, so the data can just be parsed and copied to a buffer.

The question with decoding sub templates becomes, what do we use as an internal template for any sub templates that arrive? The answer is a new structure in fixbuf that pairs external and internal template IDs for use in decoding sub templates. The pairs are added to the session that is used for the connection, using fbSessionAddTemplatePair().

Because the external template IDs are only unique for that session, the collector must know the IDs of the templates that are collected in order to pair an internal template with the external template. As a result, callback functionality may be enabled (via fbSessionAddNewTemplateCallback()) to alert the user when a new external template has arrived. The callback functions are stored in the session structure, which manages the templates. The callback function, fbNewTemplateCallback_fn, receives the session pointer, the template, the template ID, a context pointer for the application to use, and the location in which to store the template's context variable. The callback also gives the user another callback that can be used to free the context variable upon template deletion. This information is sufficient for the application to successfully add template pairs to the session for sub template decoding.

If the application does not use the callback or does not add any template pairs to the session, then fixbuf will transcode each of the sub templates as if the external and internal template were same. This causes all of the fields sent over the wire to be transcoded into the data buffer on the collecting side. The details of that template are passed up to the collector upon receipt of data so it knows how the data is structured in the buffer.

If the application adds any template pair to the list, then the list will be referenced for each transcode. Any external template the application wishes to process MUST have an entry in the list. There are 3 cases for entries in the list:

There is no entry for the given external template ID, so the entire sub template is ignored by the transcoder. The collector will be given a sub template list (or multi list entry) struct with the number of elements in the list set to 0, and the data pointer set to NULL.
The listing exists, and the external and internal template IDs are set to the same value. When decoding, the list of internal templates is queried to see if a template exists with the same ID as the external template. If not, the transcoder decodes each of the information elements, in the same order, into the buffer. This is a change as setting them equal to each other used to force a full decode. This change highlights the need for careful template ID management.
The listing exists, and the external and internal template IDs are different. This will transcode in the standard way external templates have been transcoded into internal templates, selecting the desired elements (listed in the internal template) from the data that arrived in the external template.

Iterating Over the Lists

There are four scenarios in which the user needs to iterate through the elements in a list, whether to fill in, or process the data:

Iterating over the repeated information element data in a basic list
Iterating over the decoded data elements in a sub template list
Iterating over the entries that make up a sub template multi list
Iterating over the decoded data elements in an entry of a sub template multi list

The two iterating mechanisms are the same in each case: Each of the function names start with the structure being iterated over, e.g., fbBasicList, or fbSubTemplateMultiListEntry

Indexing (by position)

The function used here is (structName)GetIndexed(dataPtr or entry)() It takes a pointer to the struct, and the index to be retrieved. Example usage:
```
for (i = 0; myStructPtr = ...GetIndexedDataPtr(listPtr, i); ++i) {
    process the data that myStructPtr points to.
}
```
The loop will end when the function returns NULL because i is beyond the end of the list.
Incrementing

The function used here is (structName)GetNext(dataPtr or entry)() It takes a pointer to the struct, and a pointer to an element in the list. Pass in NULL at the beginning to get the first element back.

Example usage:
```
myStructPtr = NULL;
while (myStructPtr = ...GetNextPtr(listPtr, myStructPtr)) {
    process the data that myStructPtr points to.
}
```
The loop will end when the function returns NULL because it is at the end of the list. A key part here is initializing myStructPtr to NULL at the beginning.

When working with a basicList, the user must be prepared to handle elements that use reduced length encoding.

Previous: Connection-less Collector | Next: RFC 5610