omhdfs: Hadoop Filesystem Output Module¶

Module Name:	omhdfs
Author:	Rainer Gerhards <rgerhards@adiscon.com>

Purpose¶

This module supports writing message into files on Hadoop’s HDFS file system.

Configuration Parameters¶

Note

Parameter names are case-insensitive; camelCase is recommended for readability.

Module Parameters¶

Parameter	Summary
omhdfsFileName	Sets the HDFS path of the file that omhdfs writes log messages to.
omhdfsHost	Selects the hostname or IP address of the HDFS system omhdfs connects to.
omhdfsPort	Specifies the TCP port number used to reach the configured HDFS host.
omhdfsDefaultTemplate	Sets the template omhdfs applies when no template is specified for an action.

Caveats/Known Bugs¶

Building omhdfs is a challenge because we could not yet find out how to integrate Java properly into the autotools build process. The issue is that HDFS is written in Java and libhdfs uses JNI to talk to it. That requires that various system-specific environment options and paths be set correctly. At this point, we leave this to the user. If someone knows how to do it better, please drop us a line!

In order to build, you need to set these environment variables BEFORE running ./configure:
- JAVA_INCLUDES - must have all include paths that are needed to build JNI C programs, including the -I options necessary for gcc. An example is # export JAVA_INCLUDES=”-I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include -I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include/linux”
- JAVA_LIBS - must have all library paths that are needed to build JNI C programs, including the -l/-L options necessary for gcc. An example is # export export JAVA_LIBS=”-L/usr/java/jdk1.6.0_21/jre/lib/amd64 -L/usr/java/jdk1.6.0_21/jre/lib/amd64/server -ljava -ljvm -lverify”
As of HDFS architecture, you must make sure that all relevant environment variables (the usual Java stuff and HADOOP’s home directory) are properly set.
As it looks, libhdfs makes Java throw exceptions to stdout. There is no known work-around for this (and it usually should not case any troubles.

Examples¶

Example 1¶

$ModLoad omhdfs
$omhdfsFileName /var/log/hdfs/system.log
*.* :omhdfs:

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project