# Introduction to Fluentd on Kubernetes fluentd-k8s ## Prerequisites You will need a basic understanding of Fluentd before you attempt to run it on Kubernetes.
Fluentd and Kubernetes have a bunch of moving parts.
To understand the basics of Fluentd, I highly recommend you start with this video:
Fluentd The most important components to understand is the fluentd `tail` plugin.
This plugin is used to read logs from containers and pods on the file system and collect them. ## We need a Kubernetes cluster Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) ``` kind create cluster --name fluentd --image kindest/node:v1.19.1 ``` ## Fluentd Manifests I would highly recommend to use manifests from the official fluentd [github repo](https://github.com/fluent/fluentd-kubernetes-daemonset) for production usage
The manifests found here are purely for demo purpose.
The manifests in this repo are broken down and simplified for educational purpose.

In this example I will use the most common use case and we'll break it down to get an understanding of each component. ## Fluentd Docker I would recommend to start with the official [fluentd](https://hub.docker.com/r/fluent/fluentd/) docker image.
You may want to build your own image if you want to install plugins. In this demo I will be using the `fluentd` elasticsearch plugin
It's pretty simple to adjust `fluentd` to send logs to any other destination in case you are not an `elasticsearch` user.

Let's build our [docker image](https://github.com/marcel-dempers/docker-development-youtube-series/blob/master/monitoring/logging/fluentd/introduction/dockerfile) in the introduction folder: ``` cd .\monitoring\logging\fluentd\kubernetes\ #note: use your own tag! docker build . -t aimvector/fluentd-demo #note: use your own tag! docker push aimvector/fluentd-demo ``` ## Fluentd Namespace I like to run certain infrastructure components in their own namespaces.
If you are using the official manifests, they may be using the `kube-system` namespace instead.
You may want to carefully adjust it based on your preference
Let's create a `fluentd` namespace:
``` kubectl create ns fluentd ``` ## Fluentd Configmap In my [fluentd introduction video](https://youtu.be/Gp0-7oVOtPw), I talk about how `fluentd` allows us to simplify our configs using the `include` statement.
This helps us prevent having a large complex file.
We have 5 files in our `fluentd-configmap.yaml` : * fluent.conf: Our main config which includes all other configurations * pods-kind-fluent.conf: `tail` config that sources all pod logs on the `kind` cluster. Note: `kind` cluster writes its log in a different format * pods-fluent.conf: `tail` config that sources all pod logs on the `kubernetes` host in the cloud.
Note: When running K8s in the cloud, logs may go into JSON format. * file-fluent.conf: `match` config to capture all logs and write it to file for testing log collection
Note: This is great to test if collection of logs works * elastic-fluent.conf: `match` config that captures all logs and sends it to `elasticseach` Let's deploy our `configmap`: ``` kubectl apply -f .\monitoring\logging\fluentd\kubernetes\fluentd-configmap.yaml ``` ## Fluentd Daemonset Let's deploy our `daemonset`: ``` kubectl apply -f .\monitoring\logging\fluentd\kubernetes\fluentd-rbac.yaml kubectl apply -f .\monitoring\logging\fluentd\kubernetes\fluentd.yaml kubectl -n fluentd get pods ``` Let's deploy our example app that writes logs to `stdout` ``` kubectl apply -f .\monitoring\logging\fluentd\kubernetes\counter.yaml kubectl get pods ``` ## Demo ElasticSearch and Kibana ``` kubectl create ns elastic-kibana # deploy elastic search kubectl -n elastic-kibana apply -f .\monitoring\logging\fluentd\kubernetes\elastic\elastic-demo.yaml kubectl -n elastic-kibana get pods # deploy kibana kubectl -n elastic-kibana apply -f .\monitoring\logging\fluentd\kubernetes\elastic\kibana-demo.yaml kubectl -n elastic-kibana get pods ``` ## Kibana ``` kubectl -n elastic-kibana port-forward svc/kibana 5601 ```