Use this documentation with care! It describes the outdated version 7, which was actively developed around 2014 and is considered dead by the rsyslog team.

This documentation reflects the latest update of the v7-stable branch. It describes the 7.6.8 version, which was never released. As such, it contains some content that does not apply to any released version.

To obtain the doc that properly matches your installed v7 version, obtain the doc set from your distro. Each version of rsyslog contained the version that exactly matches it.

As general advise, it is strongly suggested to upgrade to the current version supported by the rsyslog project. The current version can always be found on the right-hand side info box on the rsyslog web site.

Note that there is only limited rsyslog community support available for the outdated v7 version (officially we do not support it at all, but we usually are able to answer simple questions). If you need to stick with v7, it probably is best to ask your distribution for support.

omhdfs: Hadoop Filesystem Output Module

Module Name:    omhdfs

Available since: 5.7.1

Author:Rainer Gerhards <rgerhards@adiscon.com>

Description:

This module supports writing message into files on Hadoop’s HDFS file system.

Configuration Directives:

  • $OMHDFSFileName [name]
    The name of the file to which the output data shall be written.
  • $OMHDFSHost [name]
    Name or IP address of the HDFS host to connect to.
  • $OMHDFSPort [name] Port on which to connect to the HDFS host.
  • $OMHDFSDefaultTemplate [name] Default template to be used when none is specified. This saves the work of specifying the same template ever and ever again. Of course, the default template can be overwritten via the usual method.

Caveats/Known Bugs:

Building omhdfs is a challenge because we could not yet find out how to integrate Java properly into the autotools build process. The issue is that HDFS is written in Java and libhdfs uses JNI to talk to it. That requires that various system-specific environment options and pathes be set correctly. At this point, we leave this to the user. If someone know how to do it better, please drop us a line!

  • In order to build, you need to set these environment variables BEFORE running ./configure:
    • JAVA_INCLUDES - must have all include pathes that are needed to build JNI C programms, including the -I options necessary for gcc. An example is # export JAVA_INCLUDES=”-I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include -I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include/linux”
    • JAVA_LIBS - must have all library pathes that are needed to build JNI C programms, including the -l/-L options necessary for gcc. An example is # export export JAVA_LIBS=”-L/usr/java/jdk1.6.0_21/jre/lib/amd64 -L/usr/java/jdk1.6.0_21/jre/lib/amd64/server -ljava -ljvm -lverify”
  • As of HDFS architecture, you must make sure that all relevant environment variables (the usual Java stuff and HADOOP’s home directory) are properly set.
  • As it looks, libhdfs makes Java throw exceptions to stdout. There is no known work-around for this (and it usually should not case any troubles.

Sample:

$ModLoad omhdfs $OMHDFSFileName /var/log/logfile \*.\* :omhdfs:

This documentation is part of the rsyslog project.

Copyright © 2010-2014 by Rainer Gerhards and Adiscon. Released under the GNU GPL version 3 or higher.

Previous topic

omfwd: syslog Forwarding Output Module

Next topic

omlibdbi: Generic Database Output Module

This Page