Kubernetes Daemonsets
We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using kind
Because a Daemonset is all about running pods on every node, lets create a 3 node cluster:
cd kubernetes/daemonsets
kind create cluster --name daemonsets --image kindest/node:v1.20.2 --config kind.yaml
Test our cluster:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
daemonsets-control-plane Ready control-plane,master 65s v1.20.2
daemonsets-worker Ready <none> 31s v1.20.2
daemonsets-worker2 Ready <none> 31s v1.20.2
daemonsets-worker3 NotReady <none> 31s v1.20.2
Introduction
Kubernetes provide documentation for what a Daemonset is with examples.
Basic Daemonset
Let's deploy a daemonset that runs a pod on each node and collects the name of the node
kubectl apply -f daemonset.yaml
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-daemonset-8lcr5 1/1 Running 0 3m21s 10.244.2.4 daemonsets-worker <none> <none>
example-daemonset-9jhgx 1/1 Running 0 81s 10.244.3.4 daemonsets-worker2 <none> <none>
example-daemonset-lvvsd 1/1 Running 0 2m41s 10.244.1.4 daemonsets-worker3 <none> <none>
example-daemonset-xxcv9 1/1 Running 0 119s 10.244.0.7 daemonsets-control-plane <none> <none>
We can see the logs of any pod
kubectl logs <example-daemonset-xxxx>
Cleanup:
kubectl delete ds example-daemonset
Basic Daemonset: Exposing HTTP
Let's deploy a daemonset that runs a pod on each node and exposes an HTTP endpoint on each node.
In this demo we'll use a simple NGINX on port 80
kubectl apply -f daemonset-communication.yaml
kubectl get pods
Communicating with Daemonset Pods
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#communicating-with-daemon-pods
Let's deploy a pod that can talk to our daemonset
kubectl apply -f pod.yaml
kubectl exec -it pod -- bash
Service Type: ClusterIP or LoadBalancer
Let's deploy a service of type ClusterIP
kubectl apply -f ./services/clusterip-service.yaml
while true; do curl http://daemonset-svc-clusterip; sleep 1s; done
Hello from daemonsets-worker2
Hello from daemonsets-control-plane
Hello from daemonsets-worker2
Hello from daemonsets-worker3
Hello from daemonsets-worker
Hello from daemonsets-worker
Node IP and Node Port
We can add the nodePort field to the pods port section to expose a port on the node.
Let's expose the node port in the pod spec:
ports:
- containerPort: 80
hostPort: 80
name: "http"
This means we can contact the daemonset pod using the Node IP and port:
# get the node ips
kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP KERNEL-VERSION CONTAINER-RUNTIME
daemonsets-control-plane Ready control-plane,master 112m v1.20.2 172.18.0.4 <none>
daemonsets-worker Ready <none> 111m v1.20.2 172.18.0.3 <none>
daemonsets-worker2 Ready <none> 111m v1.20.2 172.18.0.2 <none>
daemonsets-worker3 Ready <none> 111m v1.20.2 172.18.0.6 <none>
#example:
bash-5.1# curl http://172.18.0.4:80
Hello from daemonsets-control-plane
bash-5.1# curl http://172.18.0.2:80
Hello from daemonsets-worker2
Service: Headless service
Let's deploy a headless service where clusterIP: None
kubectl apply -f ./services/headless-service.yaml
There are a few ways to discover our pods:
- Discover the DNS records via the "Headless" services
apk add --no-cache bind-tools
dig daemonset-svc-headless.default.svc.cluster.local
Notice it resolves to multiple DNS records for each pod:
;; ANSWER SECTION:
daemonset-svc-headless.default.svc.cluster.local. 30 IN A 10.244.0.5
daemonset-svc-headless.default.svc.cluster.local. 30 IN A 10.244.2.2
daemonset-svc-headless.default.svc.cluster.local. 30 IN A 10.244.3.2
daemonset-svc-headless.default.svc.cluster.local. 30 IN A 10.244.1.2
- Discover pods endpoints by retrieving the endpoints for the headless service
kubectl describe endpoints daemonset-svc-headless
Example:
Addresses: 10.244.0.5,10.244.1.2,10.244.2.2,10.244.3.2
Get A records for each pod by using the following format:
<pod-ip-address>.<my-namespace>.pod.<cluster-domain.example>.
#examples:
10-244-0-5.default.pod.cluster.local
10-244-1-2.default.pod.cluster.local
10-244-2-2.default.pod.cluster.local
10-244-3-2.default.pod.cluster.local
Communicate with the pods over DNS:
curl http://10-244-0-5.default.pod.cluster.local
Hello from daemonsets-control-plane
Real world Examples:
Monitoring Nodes: Node-Exporter Daemonset
We clone the official kube-prometheus repo to get monitoring manifests for Kubernetes.
git clone https://github.com/prometheus-operator/kube-prometheus.git
Check the compatibility matrix here
For this demo, we will use the compatible version tag 0.8
git checkout v0.8.0
Deploy Prometheus Operator and CRDs
cd .\manifests\
kubectl create -f .\setup\
Deploy remaining resources including node exporter daemonset
kubectl create -f .
# wait for pods to be up
kubectl get pods -n monitoring
#access prometheus in the browser
kubectl -n monitoring port-forward svc/prometheus-k8s 9090
See the Daemonset communications on the Prometheus targets page
Checkout my monitoring guide for kubernetes for more in depth info
Monitoring: Logging via Fluentd
Take a look at my monitoring guide for Fluentd