Kubernetes Logging with fluentd and logz.io

With the self-hosted Kubernetes Cluster in place the next step is the setup of a centralized logging system.

This example will focus on Fluentd1 as log collector and Logz.io2 as backend.

Basics

Fluentd is an open source collector for log files and a good choice when deployed into Kubernetes. We will create a DaemonSet3 and use the fluentd-kubernetes-daemonset4 docker image. A DaemonSet ensures, that the configured pods run on each node in the cluster and new notes are automatically provisioned.

You can find multiple tags of the image which provide support for different backends (e.g. Elasticsearch, Cloudwatch or Stackdriver).

As a backend we will use Logz.io which is build on the ELK stack and offers a free Community plan with 3GB data daily.

Setup

All required Kubernetes files are available in the GitHub repository5 and should be cloned first.

Namespace

To separate the logging from the existing namespaces, we create a new namespace kube-logging first.

kubectl create -f logging-namespace.yml 

Secret

Logz.io requires a Token and Type as authentication parameters. You can find the values after a sucessful registration in your settings.

Both values are encoded in base64 and added to the fluentd-secret.yml.

echo -n YOUR_TOKEN | base64
echo -n YOUR_TYPE | base64

Now we can create the secret in our cluster:

kubectl create -f fluentd-secret.yml 

Daemonset, ServiceAccount and ClusterRole

The last step is the creation of a ServiceAccount used by the fluentd pods and a matching ClusterRole.

kubectl create -f  fluentd-daemonset.yml 

Check installation

You can now verify the process of the DaemonSet deployment:

kubectl get ds -n kube-logging 

NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd   2         2         2       2            2           <none>          27s

And get the Pods:

kubectl get pods -n kube-logging 

NAME            READY   STATUS    RESTARTS   AGE
fluentd-t644b   1/1     Running   0          67m
fluentd-t6sxw   1/1     Running   0          67m

Inside the logs of fluentd you should see the parsed log-files:

kubectl logs -f -n kube-logging fluentd-5wg4c

2019-01-10 08:08:54 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-scheduler-kubernetes_kube-system_kube-scheduler-20ba3a20f492e72da577408089cde0122046483f954f5e41ee2e1698dc34633f.log
2019-01-10 08:08:54 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-flannel-ds-grnn2_kube-system_install-cni-e97eb766bc841909bdb412c7b6b580855950175d62bac1fd81e518fa0371a49e.log
2019-01-10 08:08:54 +0000 [info]: #0 fluentd worker is now running worker=0
2019-01-10 08:09:36 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 2, pod_cache_size: 4, pod_cache_watch_misses: 5, pod_cache_watch_ignored: 1, pod_cache_watch_delete_ignored: 1, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4
2019-01-10 08:10:07 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 2, pod_cache_size: 4, pod_cache_watch_misses: 5, pod_cache_watch_ignored: 1, pod_cache_watch_delete_ignored: 1, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4
...

Summary

If everything works it will take only seconds until the first logs become available in the Kibana Dashboard6.

Footnotes

Related Posts