Monitoring for Large-Scale Networks

SiLK

rwsender

NAME

rwsender - Watch directory for files and transfer them to rwreceiver(s)

SYNOPSIS

To listen for incoming connections:

  rwsender --mode=server --server-port=[HOST:]PORT
        --client-ident=IDENT [--client-ident=IDENT ...]
        ...

To make outgoing connections:

  rwsender --mode=client --server-address=IDENT:HOST:PORT
        [--server-address=IDENT:HOST:PORT ...]
        ...

  rwsender  SERVER_MODE_OR_CLIENT_MODE_SWITCHES
        --identifier=IDENT --incoming-directory=DIR_PATH
        --processing-directory=DIR_PATH --error-directory=DIR_PATH
        [ --tls-ca=TRUST_FILE
          { { --tls-cert=CERTIFICATE_FILE --tls-key=KEY_FILE }
            | --tls-pkcs12=PKCS12_FILE }
          [--tls-priority=TLS_PRIORITY] [--tls-security=TLS_SECURITY]
          [--tls-crl=CRL_FILE] [--tls-debug-level=DB_LEVEL] ]
        [--local-directory=[[IDENT]:]DIR_PATH
          [--local-directory=[[IDENT]:]DIR_PATH ...]]
        [--unique-local-copies]
        [--filter=IDENT:REGEXP] [--priority=NUM:REGEXP]
        [--polling-interval=NUM]
        [--send-attempts=NUM] [--block-size=NUM]
        { --log-destination=DESTINATION
          | --log-pathname=FILE_PATH
          | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]
            [--log-post-rotate=COMMAND] }
        [--log-level=LEVEL] [--log-sysfacility=NUMBER]
        [--pidfile=FILE_PATH] [--no-chdir] [--no-daemon]

  rwsender --help

  rwsender --version

DESCRIPTION

rwsender is a daemon which watches an incoming directory for files. As files are added to the incoming directory, they are moved into a processing directory and then transferred over the network to one or more rwreceiver(8) processes. Files in the incoming directory may also be "transferred" to one or more local directories.

As rwsender scans the incoming directory, it ignores a file if its size is 0 bytes or if its name begins with a dot (.). On each scan, if rwsender detects a file name that was not present in the previous scan, it records the name and size of the file. If the file has a different size on the next scan, the new size is recorded. Once the file has the same size on two consecutive scans, rwsender moves the file to the processing directory and queues it for transfer.

Interaction with rwreceiver

Either rwsender or rwreceiver may act as the server with the other acting as the client. That is, an rwsender server may listen for connections from rwreceiver clients, or an rwsender client may attempt to connect to one or more rwreceiver servers.

In addition, each rwsender and rwreceiver is configured with an identifier of its own and the identifier(s) of the rwreceiver(s) or rwsender(s) that may connect to it. The connection is closed if the identifier provided by other process is not recognized.

Every rwsender that communicates with the same rwreceiver must have a unique identifier; likewise, every rwreceiver that communicates with the same rwsender must have a unique identifier. Ideally, the identifier should provide some information about where the rwsender or rwreceiver program is running and what sort of data it is transferring.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

Application-specific switches

The following set of switches are required:

--identifier=IDENT: Use the name IDENT when establishing a connection with an rwreceiver process. The identifier should contain only printable, non-whitespace characters; the following characters are illegal: colon (:), slash (/ and \), period (.), and comma (,).
--mode=MODE: Specify how the connection between rwsender and rwreceiver(s) should be established. When MODE is server, rwsender listens for connections from rwreceiver clients; when MODE is client, rwsender attempts to connect to rwreceiver servers.
--incoming-directory=DIR_PATH: Periodically scan the directory DIR_PATH for files to transfer. As rwsender scans DIR_PATH, it ignores a file if its name begins with a dot (.) or if its size is 0 bytes. When a file is first detected, its size is recorded, and the file must have the same size for two consecutive scans before rwsender will add it to sending queue. The interval between scans is set by --polling-interval. DIR_PATH must be a complete directory path.
--processing-directory=DIR_PATH: Use DIR_PATH as a location to cache files until they are successfully transferred. For each rwreceiver IDENT specified on the command line, a subdirectory is created under DIR_PATH to hold a copy of each file that is to be sent to that rwreceiver. (rwsender uses a reference (a hard link) to the file instead of a copy of the file when possible.) DIR_PATH must be a complete directory path.
--error-directory=DIR_PATH: Move a file that is rejected by an rwreceiver (for example, because it has a duplicate filename) to the subdirectory IDENT of DIR_PATH, where IDENT is the identifier of the rwreceiver that rejected the file. DIR_PATH must be a complete directory path.

Server-mode switches

When running in server mode, the following switches are required:

--server-port=[HOST:]PORT: Listen for incoming rwreceiver client connections on PORT as HOST. If HOST is omitted, rwsender listens on any address. HOST may be a name or an IP address; when HOST is an IPv6 address, it must be enclosed in square brackets.
--client-ident=IDENT: Allow connections from an rwreceiver client whose identifier is IDENT. This switch may be repeated to allow multiple rwreceiver clients to connect. rwsender closes the connection if an rwreceiver client connects and does not provide a valid identifier.

Client-mode switch

When running in client mode, the following switch is required:

--server-address=IDENT:HOST:PORT: Attempt to connect to the rwreceiver server listening to port number PORT on the machine HOST. rwsender closes the connection unless the rwreceiver identifies itself as IDENT. This switch may be repeated to connect to multiple rwreceiver servers. HOST may be a name or an IP address; when HOST is an IPv6 address, it must be enclosed in square brackets.

Transport Layer Security switches

It is possible to build SiLK with support for the GnuTLS Transport Layer Security library (https://www.gnutls.org/) which allows rwsender and rwreceiver to use an encrypted/authenticated channel for their communication. When SiLK includes GnuTLS support, the following switches are available. To enable use of GnuTLS, specify the --tls-ca switch and either the --tls-pkcs12 switch or both the --tls-cert and --tls-key switches.

--tls-ca=TRUST_FILE: Set the trusted certificate authorities to those in TRUST_FILE, where TRUST_FILE is the complete path to a file containing a PEM-encoded list of certificates. This list of authorities is used to verify the certificate sent by rwreceiver. (Store intermediate certificates in either CERTIFICATE_FILE or PKCS12_FILE.) This switch must be used in conjunction with either the --tls-pkcs12 switch or both the --tls-cert and the --tls-key switches.
--tls-cert=CERTIFICATE_FILE: Set the certificate chain (path) for rwsender's private key to the list of certificates in CERTIFICATE_FILE, where CERTIFICATE_FILE is the complete path to a file containing the PEM-encoded certificates. The certificate corresponding to KEY_FILE must be listed first; each subsequent intermediate certificate certifies the one before it. This switch may only be used in conjunction with the --tls-ca and --tls-key switches.
--tls-key=KEY_FILE: Read rwsender's private encryption key for TLS from KEY_FILE, where KEY_FILE is the complete path to a PEM-encoded file. The key must not be password protected as there is no support in rwsender for obtaining the password, but note that --tls-pkcs12 allows a password. This switch may only be used in conjunction with the --tls-ca and --tls-cert switches.
--tls-pkcs12=PKCS12_FILE: Set rwsender's encryption certificate and private key for TLS to the contents of PKCS12_FILE, where PKCS12_FILE is the complete path to a file containing the PKCS#12 contents in DER-format. If intermediate certificates are needed, they should be included in the file. This switch may only be used in conjunction with the --tls-ca switch. rwsender uses the value in the RWSENDER_TLS_PASSWORD environment variable to decrypt the PKCS#12 file. If this variable is not set, rwsender assumes the password is the empty string.
--tls-priority=TLS_PRIORITY: Set the preference order (priority) for ciphers, key exchange methods, message authentication codes, and compression methods to those in TLS_PRIORITY. This switch is optional; the default value is NORMAL. The argument is parsed by the GnuTLS library, and the available arguments depend on the version of GnuTLS linked with SiLK. Detailed information on the format of the argument is available in the GnuTLS documentation under Priority Strings (e.g., https://gnutls.org/manual/html_node/Priority-Strings.html) provides the set for the most recent version of GnuTLS; the values used at your site may be different). See also the output of running gnutls-cli(1) with the --priority-list switch. Since SiLK 3.18.0.
--tls-security=TLS_SECURITY: Set the security level to use when generating Diffie-Hellman parameters to TLS_SECURITY, where TLS_SECURITY is one of low, medium, high, or ultra. This switch is optional, and when not specified a value of medium is used. For the meaning of these values see Selecting cryptographic key sizes in the GnuTLS documentation at your site (e.g., https://gnutls.org/manual/html_node/Selecting-cryptographic-key-sizes.html). Since SiLK 3.18.0.
--tls-crl=CRL_FILE: Update the list of trusted certificates with the certificate revocation lists contained in CRL_FILE, where CRL_FILE is the complete path to a file containing PEM-encoded list of CRLs. This switch is optional. Since SiLK 3.18.0.
--tls-debug-level=DB_LEVEL: Set the debugging level used internally by the GnuTLS library to DB_LEVEL, an integer between 0 and 99 inclusive. The messages are written to the log destation at the info level. The default value of 0 disables debugging. Larger values may reveal sensitive information and should be used carefully. A value above 10 enables all debugging options. Since SiLK 3.18.0.

Required logging switches

One of the following logging switches is required:

--log-destination=DESTINATION

Specify the destination where logging messages are written. When DESTINATION begins with a slash /, it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with /, it must be one of the following strings:

none: Messages are not written anywhere.
stdout: Messages are written to the standard output.
stderr: Messages are written to the standard error.
syslog: Messages are written using the syslog(3) facility.
both: Messages are written to the syslog facility and to the standard error (this option is not available on all platforms).

--log-directory=DIR_PATH

Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form

 DIR_PATH/LOG_BASENAME-YYYYMMDD.log

where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files are rotated: At midnight local time, a new log is opened, the previous file is closed, and the command specified by --log-post-rotate is invoked on the previous day's log file. (Old log files are not removed by rwsender; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) is also written in this directory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH

Use FILE_PATH as the complete path to the log file. The log file is not rotated.

Optional application-specific switches

These are application-specific switches that are not required:

--local-directory=[[IDENT]:]DIR_PATH

Create a duplicate of each incoming file in the directory DIR_PATH. This switch may be specified multiple times to create multiple duplicates. The duplicate is made by a reference (a hard link) to the file in the processing-directory if possible, otherwise a complete copy is made. (Note that any modification-in-place to a file reference affects all references to that file; use --unique-local-copies to avoid this). If IDENT is specified, filters may be used to determine which files get copied to DIR_PATH. See --filter for filter details. When DIR_PATH contains the colon character and no IDENT is wanted, a single colon may precede DIR_PATH to designate an empty IDENT.

--unique-local-copies

Force the duplicate file created in each local-directory to be a complete copy of the file in the processing-directory instead of a reference (a hard link) to the file. Using references saves disk space and is faster than making a complete copy; however, any modification-in-place to one file affects all files. rwsender always makes a complete copy when it is unable to make a reference. This switch is ignored when the --local-directory switch is not provided.

--filter=IDENT:REGEXP

Configure rwsender to transfer files matching the regular expression REGEXP to the rwreceiver whose identifier is IDENT, or to copy files to the local directory labeled as IDENT. This switch may be repeated. When this switch is not provided, all rwreceivers and local directories get all files. When this switch is provided, any files not matching a REGEXP are left in the incoming directory and are not transferred.

The regular expression must be a POSIX 1003.2 modern or extended regular expressions, roughly akin to those used by egrep(1). Documentation might be found in the regex(7) or re_format(7) manual pages on your system.

The filter is only applied to files in the incoming-directory. Once a file has been moved into an rwreceiver-specific subdirectory of the processing-directory, restarting rwsender with a different set of --filter switches does not affect the files previously queued for each rwreceiver. To apply the filters to unsent files, you must stop the rwsender process, move all files from the subdirectories of the processing-directory to the incoming-directory, and restart the rwsender process.

--priority=NUM:REGEXP

Set the priority of files that match REGEXP to NUM. NUM must be an integer between 0 and 100 inclusive. In the current version of rwsender, priorities 0 through 50 get grouped into a single low priority bin, and priorities 51 through 100 get grouped into a single high priority bin. Files in the high priority bin are generally be sent before files in the low priority bin. The default priority of a file is 50. This switch may be repeated for multiple priorities.

--polling-interval=NUM

Configure rwsender to check the incoming directory for new files every NUM seconds. The default polling interval is 15 seconds.

--send-attempts=NUM

For each file going to an rwreceiver, make NUM attempts to open the file, map its contents, and send the contents to that rwreceiver. After NUM attempts, the file is ignored by rwsender but the file remains in the rwreceiver-specific subdirectory of the processing directory. Unless the file is manually removed from the processing directory, rwsender again attempts to send the file when rwsender is restarted. The limit may be set to a value from 1 to 65535. When NUM is 0, there is no limit. The default number of attempts is 5.

--block-size=NUM

Specify the chunk size in bytes that rwsender uses when sending files to rwreceivers. The default number of bytes is 8192; the valid range is 256 to 65535.

Optional logging and daemon switches

The following are optional switches related to logging and running as a daemon:

--log-level=LEVEL: Set the severity of messages that are logged. The levels from most severe to least are: emerg, alert, crit, err, warning, notice, info, debug. The default is info.
--log-sysfacility=NUMBER: Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to LOG_USER on the system where rwsender is running. This switch produces an error unless --log-destination=syslog is specified.
--log-basename=LOG_BASENAME: Use LOG_BASENAME in place of the application name in the name of log files in the log directory. See the description of the --log-directory switch. This switch does not affect the name of the process-ID file.
--log-post-rotate=COMMAND: Run COMMAND on the previous day's log file after log rotation. When this switch is not specified, the previous day's log file is compressed with gzip(1). When the switch is specified and COMMAND is the empty string, no action is taken on the log file. Each occurrence of the string %s in COMMAND is replaced with the full path to the log file, and each occurrence of %% is replaced with %. If any other character follows %, rwsender exits with an error. Specifying this switch without also using --log-directory is an error.
--pidfile=FILE_PATH: Set the complete path to the file in which rwsender writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwsender.pid.
--no-chdir: Do not change directory to the root directory. When rwsender becomes a daemon process, it changes its current directory to the root directory so as to avoid potentially running on a mounted file system. Specifying --no-chdir prevents this behavior, which may be useful during debugging. The application does not change its directory when --no-daemon is given.
--no-daemon: Force rwsender to run in the foreground---it does not become a daemon process. This may be useful during debugging.

Help switches

The following switches provide help:

--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

ENVIRONMENT

RWSENDER_TLS_PASSWORD: Specifies the password to use to decrypt the PKCS#12 file specified in the --tls-pkcs12 switch. When this is not provided, a NULL password is used. Set this environment variable to the empty string for an empty password.

BUGS

An attempt should be made to use a unique name for each file put into the incoming directory. When a file is added to the incoming directory that has the same name as a file in the processing directory, the file added to the incoming directory replaces the existing file in the processing directory.