Long time, No see. It’s been weeks since my last blog post. Let’s the break the ice and talk on grok exporter . In this article, I will explain how you can use grok exporter to create prometheus metrics from unstructured logs.
Grok is popular for processing logs in ELK stack (Elastic Search, Logtash, Kibana) and thanks to Fabian Stäber for developing grok exporter .
Here is the official documentation of grok exporter => https://github.com/fstab/grok_exporter
Step 1: Install Grok exporter
Let’s get ready made grok exporter zip file from https://github.com/fstab/grok_exporter/releases.
- Go to releases and click on latest version (Now it is v0.2.7).
- Then Download the zip file appropriate for your operating system. My OS is a 64bit Linux. So the command would be.
wget https://github.com/fstab/grok_exporter/releases/download/v0.2.7/grok_exporter-0.2.7.linux-amd64.zip
3. unzip the file and cd to the extracted directory.
4. Then run below command to start the grok exporter.
[root@localhost grok_exporter-0.2.7.linux-amd64]# ./grok_exporter -config ./config.yml Starting server on http://localhost.localdomain:9144/metrics
Now you can see sample metrics at http://localhost.localdomain:9144/metrics .
Step 2: Lets process some custom logs
Let’s process some sample logs with Grok exporter. Here is some randome logs made by me.
30.07.2016 04:33:03 10.3.4.1 user=Nijil message="logged in" 30.07.2016 06:47:03 10.3.4.2 user=Alex message="logged failed" 30.07.2016 06:55:03 10.3.4.2 user=Alex message="logged in" 30.07.2016 07:03:03 10.3.4.3 user=Alan message="logged in" 30.07.2016 07:37:03 10.3.4.1 user=Nijil message="logged out" 30.07.2016 08:47:03 10.3.4.2 user=Alex message="logged out" 30.07.2016 14:34:03 10.3.4.3 user=Alan message="logged out"
As you might have guessed, the log shows user login activity to a box. Now I need create a Prometheus metric out of this.
In the Step1, you might have noticed the path of config.xml mentioned in the grok exporter starter command. Open the config file and replace the content with below data.
global: config_version: 2 input: type: file path: ./example/nijil.log # Specify the location of the your log readall: true # This should be True if you want to read whole log and False if you want to read only new lines. grok: patterns_dir: ./patterns metrics: - type: counter name: user_activity help: Counter metric example with labels. match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\"" labels: user : '{{.user}}' server: port: 9144
The above config is made from below skeleton.
global: # Config version input: # How to read log lines (file or stdin). grok: # Available Grok patterns. metrics: # How to map Grok fields to Prometheus metrics. server: # How to expose the metrics via HTTP(S).
Step 3: Tweak heart of the grok exporter
In the above config, most interesting section is metrics where we specify how log lines should be mapped to Prometheus metrics.
metrics: - type: counter name: user_activity help: Counter metric example with labels. match: "%{DATE} %{TIME} %{HOSTNAME:instance} user=%{USER:user} message=\"%{GREEDYDATA:data}\"" labels: user : '{{.user}}'
The syntax of the grok pattern is %{SYNTAX:SEMANTIC} where SYNTAX is the name of the pattern that will match the log and SEMANTIC is the field name to assign the value of the matched log. Take %{HOSTNAME:instance} as an example, HOSTNAME is grok pattern which only scrap IP part from log and that “IP part” is saved to instance(you can give any name here) that you can use later. You need to note that, each SYNTAX has their own purpose, that means you cannot scrap IP address in the log with DATE syntax. And as the name suggests, DATE , TIME , HOSTNAME , USER and GREEDYDATA scraps date, time, hostname and ‘any message’ respectively.
You can use labels to decide which parameter based metric should be generated. From the above config, you can understand that metric is based on user’s name. Note that you need to use SEMANTIC of the SYNTAX for label . There should be at-least two parameters for a metric. Here we have user’s name in one axis and the count of occurrence of the user in other axis. The second parameter is decided based metric type counter . Like Counter, grok exporter has other metric types, learn about them from official doc.
Now run the grok exporter ./grok_exporter -config ./config.yml and open metrics on the browser. You will see metrics created with name ‘user_activity’ as we specified in the config file.
# TYPE user_activity counter user_activity{user="Alan"} 2 user_activity{user="Alex"} 3 user_activity{user="Nijil"} 2
Plotting this metric as a graph in Prometheus is out of this article’s scope. But it is very straight forward, you just need to specify the metric endpoint in the prometheus configuration so that Prometheus can scrap the metrics data and plot the graph for you.
Hurray, you have completed the course. Congratulations 🙂
Thanks for the time taken to read my blog. Subscribe to this blog so that you don’t miss out anything useful (Checkout Right Sidebar for the Subscription Form and Facebook follow button) . Please also put your thoughts on this article as comments .