Introduction to Fluentd on Kubernetes

Prerequisites

You will need a basic understanding of Fluentd before you attempt to run it on Kubernetes.
Fluentd and Kubernetes have a bunch of moving parts.
To understand the basics of Fluentd, I highly recommend you start with this video:

We need a Kubernetes cluster

Lets create a Kubernetes cluster to play with using kind

kind create cluster --name fluentd --image kindest/node:v1.19.1

Fluentd Manifests

I would highly recommend to use manifests from the official fluentd github repo for production usage

The manifests found here are purely for demo purpose.
The manifests in this repo are broken down and simplified for educational purpose.

In this example I will use the most common use case and we'll break it down to get an understanding of each component.

Fluentd Docker

I would recommend to start with the official fluentd docker image.
You may want to build your own image if you want to install plugins. In this demo I will be using the fluentd elasticsearch plugin
It's pretty simple to adjust fluentd to send logs to any other destination in case you are not an elasticsearch user.

Let's build our docker image in the introduction folder:

cd monitoring\logging\fluentd\introduction

#note: use your own tag!
docker build . -t aimvector/fluentd-demo

#note: use your own tag!
docker push aimvector/fluentd-demo

Fluentd Namespace

I like to run certain infrastructure components in their own namespaces.
If you are using the official manifests, they may be using the kube-system namespace instead.
You may want to carefully adjust it based on your preference
Let's create a fluentd namespace:

kubectl create ns fluentd

Fluentd Configmap

In my fluentd introduction video, I talk about how fluentd allows us to simplify our configs using the include statement.
This helps us prevent having a large complex file.

We have 3 files in our fluentd-configmap.yaml :

fluent.conf: Our main config which includes all other configurations
pods-fluent.conf: tail config that sources all pod logs on the kubernetes host
file-fluent.conf: match config to capture all logs and write it to file for testing log collection
elastic-fluent.conf: match config that captures all logs and sends it to elasticseach

Let's deploy our configmap:

kubectl apply -f .\monitoring\logging\fluentd\kubernetes\fluentd-configmap.yaml

Fluentd Daemonset

Let's deploy our daemonset:

kubectl apply -f .\monitoring\logging\fluentd\kubernetes\fluentd-rbac.yaml 
kubectl apply -f .\monitoring\logging\fluentd\kubernetes\fluentd.yaml

kubectl -n fluentd get pods

NOT message:("pattern not matched") and NOT message:("/var/log/containers/")

Demo ElasticSearch and Kibana

kubectl create ns elastic-kibana

kubectl -n elastic-kibana apply -f .\monitoring\logging\fluentd\kubernetes\elastic\elastic-demo.yaml
kubectl -n elastic-kibana apply -f .\monitoring\logging\fluentd\kubernetes\elastic\kibana-demo.yaml

Kibana

kubectl -n elastic-kibana port-forward svc/kibana 5601

3.6 KiB Raw Blame History