How to use grok exporter to create prometheus metrics from unstructured logs

Read Time: 4 mins

Long time, No see. It’s been weeks since my last blog post. Let’s the break the ice and talk on  grok exporter . In this article, I will explain how you can use grok exporter to create prometheus metrics from unstructured logs.

Grok is popular for processing logs in ELK stack (Elastic Search, Logtash, Kibana) and thanks to Fabian Stäber for developing  grok exporter .

Here is the official documentation of grok exporter => https://github.com/fstab/grok_exporter

Step 1: Install Grok exporter

Let’s get ready made grok exporter zip file from https://github.com/fstab/grok_exporter/releases.

  1. Go to releases and click on latest version (Now it is v0.2.7).
  2. Then Download the zip file appropriate for your operating system. My OS is a 64bit Linux. So the command would be.
3.  unzip  the file and cd to the extracted directory.

4. Then run below command to start the grok exporter.

Now you can see sample metrics at http://localhost.localdomain:9144/metrics .

Step 2: Lets process some custom logs

Let’s process some sample logs with Grok exporter. Here is some randome logs made by me.

As you might have guessed, the log shows user login activity to a box. Now I need create a Prometheus metric out of this.

In the Step1, you might have noticed the path of config.xml mentioned in the  grok exporter starter command. Open the config file and replace the content with below data.

The above config is made from below skeleton.

Step 3: Tweak heart of the grok exporter

In the above config, most interesting section is metrics where we specify how log lines should be mapped to Prometheus metrics.

The syntax of the grok pattern is %{SYNTAX:SEMANTIC}   where SYNTAX is the name of the pattern that will match the log and SEMANTIC is the field name to assign the value of the matched log. Take %{HOSTNAME:instance}  as an example, HOSTNAME is grok pattern which only scrap IP part from log and that “IP part” is saved to instance(you can give any name here) that you can use later. You need to note that, each SYNTAX has their own purpose,  that means you cannot scrap IP address in the log with DATE syntax. And as the name suggests, DATE , TIME ,  HOSTNAME , USER  and GREEDYDATA  scraps date, time, hostname and ‘any message’ respectively.

 

You can use labels to decide which parameter based metric should be generated. From the above config, you can understand that metric is based on user’s name. Note that you need to use SEMANTIC of the SYNTAX for label . There should be at-least two parameters for a metric. Here we have user’s name in one axis and the count of occurrence of the user in other axis. The second parameter is decided based metric type counter . Like Counter, grok exporter has other metric types, learn about them from official doc.

 

Now run the grok exporter ./grok_exporter -config ./config.yml and open metrics on the browser. You will see metrics created with name ‘user_activity’ as we specified in the config file.

Plotting this metric as a graph in Prometheus is out of this article’s scope. But it is very straight forward, you just need to specify the metric endpoint in the prometheus configuration so that Prometheus can scrap the metrics data and plot the graph for you.

Hurray, you have completed the course. Congratulations 🙂

Thanks for the time taken to read my blog. Subscribe to this blog so that you don’t miss out anything useful  (Checkout Right Sidebar for the Subscription Form and Facebook follow button)  . Please also put your thoughts on this article as  comments .

A Site Reliability Engineer and an automation enthusiast.
Posts created 32

Begin typing your search term above and press enter to search. Press ESC to cancel.

Back To Top
x