Kubernetes Logging with fluentd and logz.io
Reading time3 Minutes
tl;dr: By using logz.io, it is relatively easy to outsource Kubernetes logfiles that do not contain sensitive data to an external service and analyze them there with Kibana.
With the self-hosted Kubernetes Cluster in place the next step is the setup of a centralized logging system.
This example will focus on Fluentd1 as log collector and Logz.io2 as backend.
Fluentd is an open source collector for log files and a good choice when deployed into Kubernetes. We will create a DaemonSet3 and use the fluentd-kubernetes-daemonset4 docker image. A DaemonSet ensures, that the configured pods run on each node in the cluster and new notes are automatically provisioned.
You can find multiple tags of the image which provide support for different backends (e.g. Elasticsearch, Cloudwatch or Stackdriver).
As a backend we will use Logz.io which is build on the ELK stack and offers a free Community plan with 3GB data daily.
All required Kubernetes files are available in the GitHub repository5 and should be cloned first.
To separate the logging from the existing namespaces, we create a new namespace
kubectl create -f logging-namespace.yml
Logz.io requires a Token and Type as authentication parameters. You can find the values after a sucessful registration in your settings.
Both values are encoded in base64 and added to the fluentd-secret.yml.
echo -n YOUR_TOKEN | base64 echo -n YOUR_TYPE | base64
Now we can create the secret in our cluster:
kubectl create -f fluentd-secret.yml
Daemonset, ServiceAccount and ClusterRole
The last step is the creation of a ServiceAccount used by the fluentd pods and a matching ClusterRole.
kubectl create -f fluentd-daemonset.yml
You can now verify the process of the DaemonSet deployment:
kubectl get ds -n kube-logging NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd 2 2 2 2 2 <none> 27s
And get the Pods:
kubectl get pods -n kube-logging NAME READY STATUS RESTARTS AGE fluentd-t644b 1/1 Running 0 67m fluentd-t6sxw 1/1 Running 0 67m
Inside the logs of fluentd you should see the parsed log-files:
kubectl logs -f -n kube-logging fluentd-5wg4c 2019-01-10 08:08:54 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-scheduler-kubernetes_kube-system_kube-scheduler-20ba3a20f492e72da577408089cde0122046483f954f5e41ee2e1698dc34633f.log 2019-01-10 08:08:54 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-flannel-ds-grnn2_kube-system_install-cni-e97eb766bc841909bdb412c7b6b580855950175d62bac1fd81e518fa0371a49e.log 2019-01-10 08:08:54 +0000 [info]: #0 fluentd worker is now running worker=0 2019-01-10 08:09:36 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 2, pod_cache_size: 4, pod_cache_watch_misses: 5, pod_cache_watch_ignored: 1, pod_cache_watch_delete_ignored: 1, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4 2019-01-10 08:10:07 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 2, pod_cache_size: 4, pod_cache_watch_misses: 5, pod_cache_watch_ignored: 1, pod_cache_watch_delete_ignored: 1, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4 ...
If everything works it will take only seconds until the first logs become available in the Kibana Dashboard6.
Spring Boot Application in OpenShift / OKDNow that we have packaged an existing Spring Boot application into a Docker Image, we can deploy it to a Kubernet cluster as well.
In this example the additional features of OpenShift/OKD are used to enable a continuous deployment of the application.
Setup a Kubernetes Cluster with AnsibleAlthough all large Cloud provider nowadays offer Managed Kubernetes Clusters, I prefer to have access to a local cluster especially during development. In this post, we will setup a Kubernetes Cluster using Ansible and Kubeadm. The cluster will include a single master node and two (or more) worker nodes. Most of the work done here is based on a tutorial by bsder1. Requirements I will use three Ubuntu 18.04 LTS (Bionic Beaver) servers, each with 4GB RAM and 2 CPUs, you should also be fine with 1GB RAM.
Autoscaling GitLab Runner Instances on Google Cloud PlatformMigrating GitLab CI jobs to Google Cloud Platform is possible with little effort due to the good support provided by GitLab and relieves the load off your hardware.
This can be worthwhile even for small projects or private GitLab instances without generating major costs.