Monitoring for Large-Scale Networks

SiLK

rwreceiver

NAME

rwreceiver - Accepts files transferred from rwsender(s)

SYNOPSIS

To listen for incoming connections:

  rwreceiver --mode=server --server-port=[HOST:]PORT
        --client-ident=IDENT [--client-ident=IDENT ...]
        ...

To make outgoing connections:

  rwreceiver --mode=client --server-address=IDENT:HOST:PORT
        [--server-address=IDENT:HOST:PORT ...]
        ...

  rwreceiver  SERVER_MODE_OR_CLIENT_MODE_SWITCHES
        --identifier=IDENT --destination-directory=DIR_PATH
        [ --tls-ca=TRUST_FILE
          { { --tls-cert=CERTIFICATE_FILE --tls-key=KEY_FILE }
            | --tls-pkcs12=PKCS12_FILE }
          [--tls-priority=TLS_PRIORITY] [--tls-security=TLS_SECURITY]
          [--tls-crl=CRL_FILE] [--tls-debug-level=DB_LEVEL] ]
        [--post-command=COMMAND]
        [--duplicate-destination=DIR_PATH
          [--duplicate-destination=DIR_PATH...] ]
        [--unique-duplicates]
        [--freespace-minimum=SIZE] [--space-maximum-percent=NUM]
        { --log-destination=DESTINATION
          | --log-pathname=FILE_PATH
          | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]
            [--log-post-rotate=COMMAND] }
        [--log-level=LEVEL] [--log-sysfacility=NUMBER]
        [--pidfile=FILE_PATH] [--no-chdir] [--no-daemon]

  rwreceiver --help

  rwreceiver --version

DESCRIPTION

rwreceiver is a daemon which accepts files transferred from one or more rwsender(8) processes. The received files are stored in a destination directory.

rwreceiver creates multiple copies of the files it receives when one or more --duplicate-destination switches are specified. If possible, the duplicate file is created as a reference (a hard link) to the original file. The --unique-duplicates switch tells rwreceiver not to use hard links and forces rwreceiver to create an individual copy of the file in each duplicate destination directory. Failure to create a file in any of the duplicate destination directories is noted in rwreceiver's log but it is not treated as a failure to transfer the file. Only when a file cannot be created in the destination-directory does rwreceiver consider the transfer as failed.

The --post-command switch tells rwreceiver to execute a command on each file that it successfully receives after the file has been written to the destination directory and copied to each duplicate destination directory. The command may include a placeholder which rwreceiver fills with the path to the file in the destination directory. The exit status of the command is ignored by rwreceiver. Any output on stdout or stderr from COMMAND normally appears in the log when the log messages are being written to a local log file. See also the rwpollexec(8) daemon.

Interaction with rwsender

Either rwsender or rwreceiver may act as the server with the other acting as the client. That is, an rwsender server may listen for connections from rwreceiver clients, or an rwsender client may attempt to connect to one or more rwreceiver servers.

In addition, each rwsender and rwreceiver is configured with an identifier of its own and the identifier(s) of the rwreceiver(s) or rwsender(s) that may connect to it. The connection is closed if the identifier provided by other process is not recognized.

Every rwsender that communicates with the same rwreceiver must have a unique identifier; likewise, every rwreceiver that communicates with the same rwsender must have a unique identifier. Ideally, the identifier should provide some information about where the rwsender or rwreceiver program is running and what sort of data it is transferring.

Disk Usage

By default, if the disk that rwreceiver writes to becomes full, rwreceiver prints a message to the log file and exits.

To prevent this, specify the --freespace-minimum and/or --space-maximum-percent switches, which cause rwreceiver to monitor its disk usage. These switches were added in SiLK 3.6.

If receiving a file from an rwsender process would violate the limits specified in those switches, rwreceiver closes the connection to that rwsender. This causes the connection to be reestablished, and rwsender tries to transfer the file again. If the filesystem is still full, rwreceiver closes the connection again. After a delay, the connection is reestablished. This loop is repeated until the file is successfully transferred. The delay between each retry starts at five seconds and grows in five second increments to a maximum of one minute.

When monitoring its disk usage, rwreceiver accounts for one copy of the number of bytes in the file. rwreceiver does not account for the filesystem overhead associated with creating a file, and it does not consider the space required to create multiple copies of the file (cf., --duplicate-destination).

File Creation

The following describes the process rwreceiver uses when creating a file it receives from rwsender. Administrators may find this information useful when configuring other software to work with rwreceiver.

rwsender sends the name of the file, the size of the file, and the file's permission bits to rwreceiver.
If a file with that name already exists in rwreceiver's destination directory, rwreceiver checks the file's on-disk size. If the size is 0 and no other rwreceiver thread is currently handling that file, rwreceiver assumes it is an aborted attempt to send the file, and rwreceiver removes the existing file. Otherwise, rwreceiver tells rwsender that the name represents a duplicate file, at which point rwsender moves the file to its error directory.
When neither --freespace-minimum nor --space-maximum-percent is specified, processing moves to the next step. Otherwise, rwreceiver verifies that there is space on the filesystem to hold one copy of the file. As described in the "Disk Usage" section above, rwreceiver delays processing the file until space is available.
rwreceiver creates a 0-length placeholder file having the name of the file being transferred, and rwreceiver closes this file. The permission bits on this file are all 0.
The rwreceiver process creates a second file whose name consists of a dot (.) followed by the name of the file being transferred. The permission bits on this file are those sent by rwsender.
rwreceiver writes the data it receives from rwsender into the dot file.
Once the transfer is complete, rwreceiver closes the dot file.
If any duplicate destination directories have been specified, rwreceiver copies the dot file to each of those directories (using a hard link if possible). A failure to copy the file into a duplicate destination is noted in the log file, but otherwise the error is ignored.
rwreceiver renames the dot file to replace the placeholder file.
The rwreceiver process tells the rwsender process that the transfer was successfully completed.
rwreceiver prepares the command specified by the --post-command switch, perhaps filling in the complete path to the file in the destination directory, and executes the command.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

Application-specific switches

The following set of switches are required:

--identifier=IDENT: Use the name IDENT when establishing a connection with an rwsender process. The identifier should contain only printable, non-whitespace characters; the following characters are illegal: colon (:), slash (/ and \), period (.), and comma (,).
--mode=MODE: Specify how the connection between rwsender and rwreceiver(s) should be established. When MODE is server, rwreceiver listens for connections from rwsender clients; when MODE is client, rwreceiver attempts to connect to rwsender servers.
--destination-directory=DIR_PATH: Place the transferred files into DIR_PATH. Note that rwreceiver uses this as its processing directory; see the "File Creation" section above for details.

Server-mode switches

When running in server mode, the following switches are required:

--server-port=[HOST:]PORT: Listen for incoming rwsender client connections on PORT as HOST. If HOST is omitted, rwreceiver listens on any address. HOST may be a name or an IP address; when HOST is an IPv6 address, it must be enclosed in square brackets.
--client-ident=IDENT: Allow connections from an rwsender client whose identifier is IDENT. This switch may be repeated to allow multiple rwsender clients to connect. rwreceiver closes the connection if an rwsender client connects and does not provide a valid identifier.

Client-mode switch

When running in client mode, the following switch is required:

--server-address=IDENT:HOST:PORT: Attempt to connect to the rwsender server listening to port number PORT on the machine HOST. rwreceiver closes the connection unless the rwsender identifies itself as IDENT. This switch may be repeated to connect to multiple rwsender servers. HOST may be a name or an IP address; when HOST is an IPv6 address, it must be enclosed in square brackets.

Transport Layer Security switches

It is possible to build SiLK with support for the GnuTLS Transport Layer Security library (https://www.gnutls.org/) which allows rwsender and rwreceiver to use an encrypted/authenticated channel for their communication. When SiLK includes GnuTLS support, the following switches are available. To enable use of GnuTLS, specify the --tls-ca switch and either the --tls-pkcs12 switch or both the --tls-cert and --tls-key switches.

--tls-ca=TRUST_FILE: Set the trusted certificate authorities to those in TRUST_FILE, where TRUST_FILE is the complete path to a file containing a PEM-encoded list of certificates. This list of authorities is used to verify the certificate sent by rwsender. (Store intermediate certificates in either CERTIFICATE_FILE or PKCS12_FILE.) This switch must be used in conjunction with either the --tls-pkcs12 switch or both the --tls-cert and the --tls-key switches.
--tls-cert=CERTIFICATE_FILE: Set the certificate chain (path) for rwreceiver's private key to the list of certificates in CERTIFICATE_FILE, where CERTIFICATE_FILE is the complete path to a file containing the PEM-encoded certificates. The certificate corresponding to KEY_FILE must be listed first; each subsequent intermediate certificate certifies the one before it. This switch may only be used in conjunction with the --tls-ca and --tls-key switches.
--tls-key=KEY_FILE: Read rwreceiver's private encryption key for TLS from KEY_FILE, where KEY_FILE is the complete path to a PEM-encoded file. The key must not be password protected as there is no support in rwreceiver for obtaining the password, but note that --tls-pkcs12 allows a password. This switch may only be used in conjunction with the --tls-ca and --tls-cert switches.
--tls-pkcs12=PKCS12_FILE: Set rwreceiver's encryption certificate and private key for TLS to the contents of PKCS12_FILE, where PKCS12_FILE is the complete path to a file containing the PKCS#12 contents in DER-format. If intermediate certificates are needed, they should be included in the file. This switch may only be used in conjunction with the --tls-ca switch. rwreceiver uses the value in the RWRECEIVER_TLS_PASSWORD environment variable to decrypt the PKCS#12 file. If this variable is not set, rwreceiver assumes the password is the empty string.
--tls-priority=TLS_PRIORITY: Set the preference order (priority) for ciphers, key exchange methods, message authentication codes, and compression methods to those in TLS_PRIORITY. This switch is optional; the default value is NORMAL. The argument is parsed by the GnuTLS library, and the available arguments depend on the version of GnuTLS linked with SiLK. Detailed information on the format of the argument is available in the GnuTLS documentation under Priority Strings (e.g., https://gnutls.org/manual/html_node/Priority-Strings.html) provides the set for the most recent version of GnuTLS; the values used at your site may be different). See also the output of running gnutls-cli(1) with the --priority-list switch. Since SiLK 3.18.0.
--tls-security=TLS_SECURITY: Set the security level to use when generating Diffie-Hellman parameters to TLS_SECURITY, where TLS_SECURITY is one of low, medium, high, or ultra. This switch is optional, and when not specified a value of medium is used. For the meaning of these values see Selecting cryptographic key sizes in the GnuTLS documentation at your site (e.g., https://gnutls.org/manual/html_node/Selecting-cryptographic-key-sizes.html). Since SiLK 3.18.0.
--tls-crl=CRL_FILE: Update the list of trusted certificates with the certificate revocation lists contained in CRL_FILE, where CRL_FILE is the complete path to a file containing PEM-encoded list of CRLs. This switch is optional. Since SiLK 3.18.0.
--tls-debug-level=DB_LEVEL: Set the debugging level used internally by the GnuTLS library to DB_LEVEL, an integer between 0 and 99 inclusive. The messages are written to the log destation at the info level. The default value of 0 disables debugging. Larger values may reveal sensitive information and should be used carefully. A value above 10 enables all debugging options. Since SiLK 3.18.0.

Required logging switches

One of the following logging switches is required:

--log-destination=DESTINATION

Specify the destination where logging messages are written. When DESTINATION begins with a slash /, it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with /, it must be one of the following strings:

none: Messages are not written anywhere.
stdout: Messages are written to the standard output.
stderr: Messages are written to the standard error.
syslog: Messages are written using the syslog(3) facility.
both: Messages are written to the syslog facility and to the standard error (this option is not available on all platforms).

--log-directory=DIR_PATH

Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form

 DIR_PATH/LOG_BASENAME-YYYYMMDD.log

where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files are rotated: At midnight local time, a new log is opened, the previous file is closed, and the command specified by --log-post-rotate is invoked on the previous day's log file. (Old log files are not removed by rwreceiver; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) is also written in this directory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH

Use FILE_PATH as the complete path to the log file. The log file is not rotated.

Optional application-specific switches

The following switches are optional:

--post-command=COMMAND

Run COMMAND on a file once it has been successfully received. The following %-conversions are supported in COMMAND: %s is replaced with the full path of the transferred file in the destination directory, %I is replaced with the identifier of the rwsender that sent the file, and %% is replaced with %. If any other character follows %, rwreceiver exits with an error. Note that COMMAND is only invoked on files in the destination directory; however, at the time COMMAND is invoked, rwreceiver has already copied the file into each of the duplicate destination directories, if any. See also the rwpollexec(8) daemon.

--duplicate-destination=DIR_PATH

Create a duplicate of each transferred file in the directory DIR_PATH. This option may be specified multiple times to create multiple duplicates. This duplicate is made by a hard link to the file in the destination-directory if possible, otherwise a complete copy is made (see also --unique-duplicates). If there are errors copying the file to this directory, the error is logged but the process continues as if the transfer was successful. (rwreceiver considers a transfer as failed only when the file cannot be created in the destination-directory.)

--unique-duplicates

Force the duplicate file created in each duplicate-destination directory to be a complete copy of the file in the destination-directory instead of a hard link to the file. Using hard links saves disk space and is faster than making a complete copy; however, any modification-in-place to one file affects all files. This switch is ignored when the --duplicate-destination switch is not provided.

--freespace-minimum=SIZE

Set the minimum amount free space (in bytes) to maintain on the file system where the --destination-directory is located. rwreceiver delays processing of any file that would cause it to violate this limit (see "Disk Usage" above). The default value of this switch is 0, which tells rwreceiver not to monitor its disk usage. See also --space-maximum-percent.

SIZE may be given as an ordinary integer, or as a real number followed by a suffix K, M, G, or T, which represents the numerical value multiplied by 1,024 (kilo), 1,048,576 (mega), 1,073,741,824 (giga), and 1,099,511,627,776 (tera), respectively. For example, 1.5K represents 1,536 bytes, or one and one-half kilobytes.

--space-maximum-percent=NUM

Use no more than this percentage of the file system containing the --destination-directory. The default is to use all of the file system (100%). rwreceiver delays processing of files that would cause it to violate this limit. The NUM parameter does not need to be an integer. See also --freespace-minimum and "Disk Usage".

Optional logging and daemon switches

The following are optional switches related to logging and running as a daemon:

--log-level=LEVEL: Set the severity of messages that are logged. The levels from most severe to least are: emerg, alert, crit, err, warning, notice, info, debug. The default is info.
--log-sysfacility=NUMBER: Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to LOG_USER on the system where rwreceiver is running. This switch produces an error unless --log-destination=syslog is specified.
--log-basename=LOG_BASENAME: Use LOG_BASENAME in place of the application name in the name of log files in the log directory. See the description of the --log-directory switch. This switch does not affect the name of the process-ID file.
--log-post-rotate=COMMAND: Run COMMAND on the previous day's log file after log rotation. When this switch is not specified, the previous day's log file is compressed with gzip(1). When the switch is specified and COMMAND is the empty string, no action is taken on the log file. Each occurrence of the string %s in COMMAND is replaced with the full path to the log file, and each occurrence of %% is replaced with %. If any other character follows %, rwreceiver exits with an error. Specifying this switch without also using --log-directory is an error.
--pidfile=FILE_PATH: Set the complete path to the file in which rwreceiver writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwreceiver.pid.
--no-chdir: Do not change directory to the root directory. When rwreceiver becomes a daemon process, it changes its current directory to the root directory so as to avoid potentially running on a mounted file system. Specifying --no-chdir prevents this behavior, which may be useful during debugging. The application does not change its directory when --no-daemon is given.
--no-daemon: Force rwreceiver to run in the foreground---it does not become a daemon process. This may be useful during debugging.

Help switches

The following switches provide help:

--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

ENVIRONMENT

RWRECEIVER_TLS_PASSWORD: Specifies the password to use to decrypt the PKCS#12 file specified in the --tls-pkcs12 switch. When this is not provided, a NULL password is used. Set this environment variable to the empty string for an empty password.