IPFIX File Collectors

How-To Read IPFIX Files

Using fixbuf to read from IPFIX Files as a Collecting Process is very much like the Export case.

Create an fbInfoModel_t using fbInfoModelAlloc(). If needed, add any vendor-specific information elements using fbInfoModelAddElement(), fbInfoModelAddElementArray(), fbInfoModelReadXMLFile(), or fbInfoModelReadXMLData().

Create an fbSession_t state container using fbSessionAlloc().

For each type of record you expect to read, create an internal template to describe it: Use fbTemplateAlloc() to allocate the template and fbTemplateAppendSpecArray(), fbTemplateAppendSpec(), fbTemplateAppend(), fbTemplateAppendArraySpecId(), or fbTemplateAppendSpecId(), to specify its elements. Unlike when exporting data, external templates do not need to be added for collection as they are defined by the templates in the file.

Use fbSessionAddTemplate() to add each Template to the Session.

Create the fbCollector_t to encapsulate the file, using the fbCollectorAllocFP() or fbCollectorAllocFile() calls. (For network collection, see Network Collectors).

With an fbSession_t and an fbCollector_t available, create a Buffer (fBuf_t) for reading via fBufAllocForCollection().

To read all records using a single template, set the internal template with fBufSetInternalTemplate(). Call fBufNext() to read records from the IPFIX Messages in the input stream until it returns FALSE.

To vary the internal template depending on the external template, the main reading loop is slightly different. First call fBufNextCollectionTemplate() to examine the template of the next record, then use fBufSetInternalTemplate() and fBufNext() to get that record.

By default, fBufNext() will consume an IPFIX Message from the input stream when the end of the message buffer is reached on read. The fBufSetAutomaticNextMessage() call can be used to modify this behavior, causing fBufNext() to return FB_ERROR_EOM when at end of message. Use this if your application requires manual control of message collection. In this case, fBufNextMessage() will consume a Message from the input stream.

Complete sample_collector.c program:

#include <fixbuf/public.h>
#define FATAL(e)                                \
    { fprintf(stderr, "Failed at %s:%d: %s\n",  \
              __FILE__, __LINE__, e->message);  \
        exit(1); }
 
int main()
{
    fbInfoElementSpec_t collectTemplate[] = {
        {"flowStartMilliseconds",               8, 0 },
        {"flowEndMilliseconds",                 8, 0 },
        {"sourceIPv4Address",                   4, 0 },
        {"destinationIPv4Address",              4, 0 },
        {"sourceTransportPort",                 2, 0 },
        {"destinationTransportPort",            2, 0 },
        {"protocolIdentifier",                  1, 0 },
        {"paddingOctets",                       3, 0 },
        {"packetTotalCount",                    8, 0 },
        {"octetTotalCount",                     8, 0 },
        {"ipPayloadPacketSection",              0, 0 },
        FB_IESPEC_NULL
    };
    struct collectRecord_st {
        uint64_t      flowStartMilliseconds;
        uint64_t      flowEndMilliseconds;
        uint32_t      sourceIPv4Address;
        uint32_t      destinationIPv4Address;
        uint16_t      sourceTransportPort;
        uint16_t      destinationTransportPort;
        uint8_t       protocolIdentifier;
        uint8_t       padding[3];
        uint64_t      packetTotalCount;
        uint64_t      octetTotalCount;
        fbVarfield_t  payload;
    } collectRecord;
 
    fbInfoModel_t   *model;
    fbSession_t     *session;
    fbCollector_t   *collector;
    fbTemplate_t    *tmpl;
    fBuf_t          *fbuf;
    uint16_t         tid;
    size_t           reclen;
    GError          *err = NULL;
 
    memset(&collectRecord, 0, sizeof(collectRecord));
 
    model = fbInfoModelAlloc();
    //  Use if needed to define elements used by YAF.
    //if (!fbInfoModelReadXMLFile(model, "cert_ipfix.xml", &err))
    //    FATAL(err);
 
    session = fbSessionAlloc(model);
 
    tmpl = fbTemplateAlloc(model);
    if (!fbTemplateAppendSpecArray(tmpl, collectTemplate, ~0, &err))
        FATAL(err);
    if (!(tid = fbSessionAddTemplate(
              session, TRUE, FB_TID_AUTO, tmpl, NULL, &err)))
        FATAL(err);
 
    collector = fbCollectorAllocFP(NULL, stdin);
    fbuf = fBufAllocForCollection(session, collector);
 
    if (!fBufSetInternalTemplate(fbuf, tid, &err))
        FATAL(err);
 
    reclen = sizeof(collectRecord);
    while (fBufNext(fbuf, (uint8_t *)&collectRecord, &reclen, &err)) {
        //  Print or process the collectRecord
    }
    if (!g_error_matches(err, FB_ERROR_DOMAIN, FB_ERROR_EOF))
        FATAL(err);
    g_clear_error(&err);
 
    //  This frees the Buffer, Session, Templates, and Collector.
    fBufFree(fbuf);
    fbInfoModelFree(model);
 
    return 0;
}
// EndOfExample

Previous: Exporter Usage | Next: Template Definition