Problem
We run many internal and customer applications, in our cloud infrastructure that has many servers, creating a huge amounts of log data. Many text files across many computers and docker containers make searching and debugging very difficult.
The logs can be hard to access unless you have shell access to the server and container. Providing this level of access is complex and difficult. Many project team members need access to these logs to support, monitor and debug application issues.
Solution
Gerhard Hipfinger investigated and evaluated various logging infrastructure solutions including the established ELK (Elastic Logstash Kibana) based solutions. However the Loki and Grafana prototype he presented to our DevOps team was very well received.
Its main advantages were:
oAuth Single Sign on via Keycloak and Grafana.
Familiar and simple user interface in Grafana.
Simple log processing and collection with Promtail.
Implementation
Loki stores and indexes the log data and provides a query interface.
Promtail collects, interprets and posts log file data to Loki.
Grafana queries Loki to retrieve log data for presentation in custom dashboards.
We configure our cloud infrastructure using Ansible. It allows us to configure many servers accurately and consistently. The promtails.conf Ansible template configures the Promtail service to collect system logs and Docker container logs, as well as defining the labels for the Loki indexes.
$ cat conf/promtail/docker-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: {{ loki_url }}
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
instance: {{ inventory_hostname }}
job: varlogs
__path__: /var/log/*log
- job_name: containers
static_configs:
- targets:
- localhost
labels:
instance: {{ inventory_hostname }}
job: containerlogs
__path__: /var/lib/docker/containers/*/*log
pipeline_stages:
- json:
expressions:
stream: stream
attrs: attrs
tag: attrs.tag
- labels:
tag:
stream:
In order to get a human readable application name in the Loki tag index for the Docker containers, we needed to configure the log options "tag" in the Docker daemon json to be the container ".Name".
$ cat conf/docker/docker-daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "5",
"tag": "{{.Name}}"
}
}
In Grafana we can then see the system logs
and the Docker container logs
Conclusion
We now have a logging infrastructure component that is easy to roll out to many servers and containerised applications with minimum effort. Grafana provides a flexible UI for creating custom dashboards that can be used by all team members. I am really happy with this solution. I think that the ease of maintaining and using it means that will hopefully have a better team adoption than other more complex Kibana based solutions that we have used in the past.
The simple things in life are often the best.