How to use grok exporter to create prometheus metrics from unstructured logs

---

Long time, No see. It’s been weeks since my last blog post. Let’s the break the ice and talk on  grok exporter . In this article, I will explain how you can use grok exporter to create prometheus metrics from unstructured logs.

Grok is popular for processing logs in ELK stack (Elastic Search, Logtash, Kibana) and thanks to Fabian Stäber for developing  grok exporter .

Here is the official documentation of grok exporter => https://github.com/fstab/grok_exporter

Step 1: Install Grok exporter

Let’s get ready made grok exporter zip file from https://github.com/fstab/grok_exporter/releases.

1. Go to releases and click on latest version (Now it is v0.2.7).
2. Then Download the zip file appropriate for your operating system. My OS is a 64bit Linux. So the command would be.

wget https://github.com/fstab/grok_exporter/releases/download/v0.2.7/grok_exporter-0.2.7.linux-amd64.zip

3. unzip  the file and cd to the extracted directory.

4. Then run below command to start the grok exporter.

[root@localhost grok_exporter-0.2.7.linux-amd64]# ./grok_exporter -config ./config.yml
Starting server on http://localhost.localdomain:9144/metrics

Now you can see sample metrics at http://localhost.localdomain:9144/metrics .

Step 2: Lets process some custom logs

Let’s process some sample logs with Grok exporter. Here is some randome logs made by me.

30.07.2016 04:33:03 10.3.4.1 user=Nijil message="logged in"
30.07.2016 06:47:03 10.3.4.2 user=Alex message="logged failed"
30.07.2016 06:55:03 10.3.4.2 user=Alex message="logged in"
30.07.2016 07:03:03 10.3.4.3 user=Alan message="logged in"
30.07.2016 07:37:03 10.3.4.1 user=Nijil message="logged out"
30.07.2016 08:47:03 10.3.4.2 user=Alex message="logged out"
30.07.2016 14:34:03 10.3.4.3 user=Alan message="logged out"

As you might have guessed, the log shows user login activity to a box. Now I need create a Prometheus metric out of this.

In the Step1, you might have noticed the path of config.xml mentioned in the  grok exporter starter command. Open the config file and replace the content with below data.

global:
    config_version: 2
input:
    type: file
    path: ./example/nijil.log  # Specify the location of the your log
    readall: true              # This should be True if you want to read whole log and False if you want to read only new lines.
grok:
    patterns_dir: ./patterns    
metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

server:
    port: 9144

The above config is made from below skeleton.

global:
    # Config version
input:
    # How to read log lines (file or stdin).
grok:
    # Available Grok patterns.
metrics:
    # How to map Grok fields to Prometheus metrics.
server:
    # How to expose the metrics via HTTP(S).

Step 3: Tweak heart of the grok exporter

In the above config, most interesting section is metrics where we specify how log lines should be mapped to Prometheus metrics.

metrics:
    - type: counter
      name: user_activity
      help: Counter metric example with labels.
      match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\""
      labels:
          user    : '{{.user}}'

The syntax of the grok pattern is %{SYNTAX:SEMANTIC}   where SYNTAX is the name of the pattern that will match the log and SEMANTIC is the field name to assign the value of the matched log. Take %{HOSTNAME:instance}  as an example, HOSTNAME is grok pattern which only scrap IP part from log and that “IP part” is saved to instance(you can give any name here) that you can use later. You need to note that, each SYNTAX has their own purpose,  that means you cannot scrap IP address in the log with DATE syntax. And as the name suggests, DATE , TIME ,  HOSTNAME , USER  and GREEDYDATA  scraps date, time, hostname and ‘any message’ respectively.

 

You can use labels to decide which parameter based metric should be generated. From the above config, you can understand that metric is based on user’s name. Note that you need to use SEMANTIC of the SYNTAX for label . There should be at-least two parameters for a metric. Here we have user’s name in one axis and the count of occurrence of the user in other axis. The second parameter is decided based metric type counter . Like Counter, grok exporter has other metric types, learn about them from official doc.

 

Now run the grok exporter ./grok_exporter -config ./config.yml and open metrics on the browser. You will see metrics created with name ‘user_activity’ as we specified in the config file.

# TYPE user_activity counter
user_activity{user="Alan"} 2
user_activity{user="Alex"} 3
user_activity{user="Nijil"} 2

Plotting this metric as a graph in Prometheus is out of this article’s scope. But it is very straight forward, you just need to specify the metric endpoint in the prometheus configuration so that Prometheus can scrap the metrics data and plot the graph for you.

Hurray, you have completed the course. Congratulations 🙂

Thanks for the time taken to read my blog. Subscribe to this blog so that you don’t miss out anything useful  (Checkout Right Sidebar for the Subscription Form and Facebook follow button)  . Please also put your thoughts on this article as  comments .