Original post: Scalable and Flexible Elasticsearch Reindexing via rsyslog by @Sematext This recipe is useful in a two scenarios: migrating data from one Elasticsearch cluster to another (e.g. when you’re upgrading from Elasticsearch 1.x to 2.x or later) reindexing data from one index to another in a cluster pre 2.3. For clusters on version 2.3 […]
Connecting with Logstash via Apache Kafka
Original post: Recipe: rsyslog + Kafka + Logstash by @Sematext This recipe is similar to the previous rsyslog + Redis + Logstash one, except that we’ll use Kafka as a central buffer and connecting point instead of Redis. You’ll have more of the same advantages: rsyslog is light and crazy-fast, including when you want it […]
Recipe: Apache Logs + rsyslog (parsing) + Elasticsearch
Original post: Recipe: Apache Logs + rsyslog (parsing) + Elasticsearch by @Sematext This recipe is about tailing Apache HTTPD logs with rsyslog, parsing them into structured JSON documents, and forwarding them to Elasticsearch (or a log analytics SaaS, like Logsene, which exposes the Elasticsearch API). Having them indexed in a structured way will allow you […]
Coupling with Logstash via Redis
Original post: Recipe: rsyslog + Redis + Logstash by @Sematext OK, so you want to hook up rsyslog with Logstash. If you don’t remember why you want that, let me give you a few hints: Logstash can do lots of things, it’s easy to set up but tends to be too heavy to put on […]
Tutorial: Sending impstats Metrics to Elasticsearch Using Rulesets and Queues
Originally posted on the Sematext blog: Monitoring rsyslog’s Performance with impstats and Elasticsearch If you’re using rsyslog for processing lots of logs (and, as we’ve shown before, rsyslog is good at processing lots of logs), you’re probably interested in monitoring it. To do that, you can use impstats, which comes from input module for process […]
rsyslog and ElasticSearch
by Micah Yoder, originally published on rackspace. Minor changes through Adiscon. There is a clear benefit to being able to aggregate logs from various servers and services into one place and be able to search them for any sort of arbitrary event. Traditional syslog can aggregate logs, but aggregating events from them sometimes involves grep and […]
Storing and forwarding remote messages
In this scenario, we want to store remote sent messages into a specific local file and forward the received messages to another syslog server. Local messages should still be locally stored. Things to think about How should this work out? Basically, we need a syslog listener for TCP and one for UDP, the local logging […]
Using the Text File Input Module
Log files should be processed by rsyslog. Here is some information on how the file monitor works. This will only describe setting up the Text File Input Module. Further configuration like processing rules or output methods will not be described. Things to think about The configuration given here should be placed on top of the […]
On the Use of English
I ventured to write this book in English because … it will be more easily read in poor English, than in good German by 90% of my intended readers. — HANS J. STETTER, Analysis of Discretization Methods for Ordinary Differential Equations (1973) There is not much I could add to Mr. Stetter’s thought, except, maybe, […]
What this book is about
This book offers a cookbook-approach to configuring rsyslog. While the official documentation focusses on concepts, components and configuration statements, this book takes a completely different approach. It will not tell you about rsyslog concepts. Instead, it will offer a wide-range of recipies for configuring rsyslog so that it performs some specific task. The individual recipies […]