marko/docker-development-youtube-series
1
0
Fork 0
mirror of https://github.com/marcel-dempers/docker-development-youtube-series.git synced 2025-06-06 17:01:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

add files

Browse Source
This commit is contained in:
marcel-dempers 2020-07-07 19:24:55 +10:00
parent 45cd191f3c
commit 4b9dd849cd
8 changed files with 358 additions and 117 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
jenkins/amazon-eks/jenkins.pv.yaml Normal file
View File

@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: fs-92b853aa

11
jenkins/amazon-eks/jenkins.pvc.yaml Normal file
View File

@ -0,0 +1,11 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi

184
jenkins/amazon-eks/readme.md Normal file
View File

@ -0,0 +1,184 @@
# Jenkins on Amazon Kubernetes
## Create a cluster
Follow my Introduction to Amazon EKS for beginners guide, to create a cluster <br/>
Video [here](https://youtu.be/QThadS3Soig)
## Setup our Cloud Storage
```
# deploy EFS storage driver
kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"
# get VPC ID
aws eks describe-cluster --name getting-started-eks --query "cluster.resourcesVpcConfig.vpcId" --output text
# Get CIDR range
aws ec2 describe-vpcs --vpc-ids vpc-id --query "Vpcs[].CidrBlock" --output text
# security for our instances to access file storage
aws ec2 create-security-group --description efs-test-sg --group-name efs-sg --vpc-id VPC_ID
aws ec2 authorize-security-group-ingress --group-id sg-xxx --protocol tcp --port 2049 --cidr VPC_CIDR
# create storage
aws efs create-file-system --creation-token eks-efs
# create mount point
aws efs create-mount-target --file-system-id FileSystemId --subnet-id SubnetID --security-group GroupID
# grab our volume handle to update our PV YAML
aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text
```
More details about EKS storage [here](https://aws.amazon.com/premiumsupport/knowledge-center/eks-persistent-storage/)
### Setup a namespace
```
kubectl create ns jenkins
```
### Setup our storage for Jenkins
```
kubectl get storageclass
# create volume
kubectl apply -f ./jenkins/amazon-eks/jenkins.pv.yaml
kubectl get pv
# create volume claim
kubectl apply -n jenkins -f ./jenkins/amazon-eks/jenkins.pvc.yaml
kubectl -n jenkins get pvc
```
### Deploy Jenkins
```
# rbac
kubectl apply -n jenkins -f ./jenkins/jenkins.rbac.yaml
kubectl apply -n jenkins -f ./jenkins/jenkins.deployment.yaml
kubectl -n jenkins get pods
```
### Expose a service for agents
```
kubectl apply -n jenkins -f ./jenkins/jenkins.service.yaml
```
## Jenkins Initial Setup
```
kubectl -n jenkins exec -it <podname> cat /var/jenkins_home/secrets/initialAdminPassword
kubectl port-forward -n jenkins <podname> 8080
# setup user and recommended basic plugins
# let it continue while we move on!
```
## SSH to our node to get Docker user info
```
eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa
ssh -i ~/.ssh/id_rsa ec2-user@ec2-13-239-41-67.ap-southeast-2.compute.amazonaws.com
id -u docker
cat /etc/group
# Get user ID for docker
# Get group ID for docker
```
## Docker Jenkins Agent
Docker file is [here](../dockerfiles/dockerfile) <br/>
```
# you can build it
cd ./jenkins/dockerfiles/
docker build . -t aimvector/jenkins-slave
```
## Continue Jenkins setup
Install Kubernetes Plugin <br/>
Configure Plugin: Values I used are [here](../readme.md) <br/>
Install Kubernetes Plugin <br/>
## Try a pipeline
```
pipeline {
agent {
kubernetes{
label 'jenkins-slave'
}
}
environment{
DOCKER_USERNAME = credentials('DOCKER_USERNAME')
DOCKER_PASSWORD = credentials('DOCKER_PASSWORD')
}
stages {
stage('docker login') {
steps{
sh(script: """
docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
""", returnStdout: true)
}
}
stage('git clone') {
steps{
sh(script: """
git clone https://github.com/marcel-dempers/docker-development-youtube-series.git
""", returnStdout: true)
}
}
stage('docker build') {
steps{
sh script: '''
#!/bin/bash
cd $WORKSPACE/docker-development-youtube-series/python
docker build . --network host -t aimvector/python:${BUILD_NUMBER}
'''
}
}
stage('docker push') {
steps{
sh(script: """
docker push aimvector/python:${BUILD_NUMBER}
""")
}
}
stage('deploy') {
steps{
sh script: '''
#!/bin/bash
cd $WORKSPACE/docker-development-youtube-series/
#get kubectl for this demo
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
./kubectl apply -f ./kubernetes/configmaps/configmap.yaml
./kubectl apply -f ./kubernetes/secrets/secret.yaml
cat ./kubernetes/deployments/deployment.yaml | sed s/1.0.0/${BUILD_NUMBER}/g | ./kubectl apply -f -
./kubectl apply -f ./kubernetes/services/service.yaml
'''
}
}
}
}
```

186
jenkins/dockerfiles/jenkins-slave
View File

@ -1,94 +1,94 @@
#!/usr/bin/env sh #!/usr/bin/env sh
# The MIT License # The MIT License
# #
# Copyright (c) 2015, CloudBees, Inc. # Copyright (c) 2015, CloudBees, Inc.
# #
# Permission is hereby granted, free of charge, to any person obtaining a copy # Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal # of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights # in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is # copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions: # furnished to do so, subject to the following conditions:
# #
# The above copyright notice and this permission notice shall be included in # The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software. # all copies or substantial portions of the Software.
# #
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE. # THE SOFTWARE.
# Usage jenkins-slave.sh [options] -url http://jenkins [SECRET] [AGENT_NAME] # Usage jenkins-slave.sh [options] -url http://jenkins [SECRET] [AGENT_NAME]
# Optional environment variables : # Optional environment variables :
# * JENKINS_TUNNEL : HOST:PORT for a tunnel to route TCP traffic to jenkins host, when jenkins can't be directly accessed over network # * JENKINS_TUNNEL : HOST:PORT for a tunnel to route TCP traffic to jenkins host, when jenkins can't be directly accessed over network
# * JENKINS_URL : alternate jenkins URL # * JENKINS_URL : alternate jenkins URL
# * JENKINS_SECRET : agent secret, if not set as an argument # * JENKINS_SECRET : agent secret, if not set as an argument
# * JENKINS_AGENT_NAME : agent name, if not set as an argument # * JENKINS_AGENT_NAME : agent name, if not set as an argument
# * JENKINS_AGENT_WORKDIR : agent work directory, if not set by optional parameter -workDir # * JENKINS_AGENT_WORKDIR : agent work directory, if not set by optional parameter -workDir
if [ $# -eq 1 ]; then if [ $# -eq 1 ]; then
# if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
exec "$@" exec "$@"
else else
# if -tunnel is not provided try env vars # if -tunnel is not provided try env vars
case "$@" in case "$@" in
*"-tunnel "*) ;; *"-tunnel "*) ;;
*) *)
if [ ! -z "$JENKINS_TUNNEL" ]; then if [ ! -z "$JENKINS_TUNNEL" ]; then
TUNNEL="-tunnel $JENKINS_TUNNEL" TUNNEL="-tunnel $JENKINS_TUNNEL"
fi ;; fi ;;
esac esac
# if -workDir is not provided try env vars # if -workDir is not provided try env vars
if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
case "$@" in case "$@" in
*"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;; *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
*) *)
WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;; WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
esac esac
fi fi
if [ -n "$JENKINS_URL" ]; then if [ -n "$JENKINS_URL" ]; then
URL="-url $JENKINS_URL" URL="-url $JENKINS_URL"
fi fi
if [ -n "$JENKINS_NAME" ]; then if [ -n "$JENKINS_NAME" ]; then
JENKINS_AGENT_NAME="$JENKINS_NAME" JENKINS_AGENT_NAME="$JENKINS_NAME"
fi fi
if [ -z "$JNLP_PROTOCOL_OPTS" ]; then if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior" echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true" JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
fi fi
# If both required options are defined, do not pass the parameters # If both required options are defined, do not pass the parameters
OPT_JENKINS_SECRET="" OPT_JENKINS_SECRET=""
if [ -n "$JENKINS_SECRET" ]; then if [ -n "$JENKINS_SECRET" ]; then
case "$@" in case "$@" in
*"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;; *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
*) *)
OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;; OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
esac esac
fi fi
OPT_JENKINS_AGENT_NAME="" OPT_JENKINS_AGENT_NAME=""
if [ -n "$JENKINS_AGENT_NAME" ]; then if [ -n "$JENKINS_AGENT_NAME" ]; then
case "$@" in case "$@" in
*"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;; *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
*) *)
OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;; OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
esac esac
fi fi
#TODO: Handle the case when the command-line and Environment variable contain different values. #TODO: Handle the case when the command-line and Environment variable contain different values.
#It is fine it blows up for now since it should lead to an error anyway. #It is fine it blows up for now since it should lead to an error anyway.
exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@" exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
fi fi

30
jenkins/jenkins.deployment.yaml
View File

@ -1,48 +1,42 @@
apiVersion: extensions/v1beta1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: jenkins name: jenkins
labels: labels:
name: jenkins
app: jenkins app: jenkins
spec: spec:
replicas: 1
selector: selector:
matchLabels: matchLabels:
name: jenkins app: jenkins
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template: template:
metadata: metadata:
labels: labels:
app: jenkins app: jenkins
name: jenkins
name: jenkins
spec: spec:
serviceAccountName: jenkins serviceAccountName: jenkins
containers: containers:
- env: - name: jenkins
image: jenkins/jenkins:2.235.1-lts-alpine
imagePullPolicy: IfNotPresent
env:
- name: JAVA_OPTS - name: JAVA_OPTS
value: -Xmx2048m -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 value: -Xmx2048m -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
image: jenkins/jenkins #:lts-alpine
imagePullPolicy: IfNotPresent
name: jenkins
ports: ports:
- containerPort: 8080 - containerPort: 8080
protocol: TCP protocol: TCP
- containerPort: 50000 - containerPort: 50000
protocol: TCP protocol: TCP
# resources:
# limits:
# cpu: "1"
# memory: 1Gi
# requests:
# cpu: "1"
# memory: 1Gi
volumeMounts: volumeMounts:
- mountPath: /var/jenkins_home - mountPath: /var/jenkins_home
name: jenkins name: jenkins
restartPolicy: Always restartPolicy: Always
securityContext: securityContext:
#fsGroup: 1000
runAsUser: 0 runAsUser: 0
terminationGracePeriodSeconds: 30 terminationGracePeriodSeconds: 30
volumes: volumes:

18
jenkins/jenkins.rbac.yaml
View File

@ -4,7 +4,7 @@ kind: ServiceAccount
metadata: metadata:
name: jenkins name: jenkins
--- ---
kind: Role kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
metadata: metadata:
name: jenkins name: jenkins
@ -20,7 +20,19 @@ rules:
verbs: ["get","list","watch"] verbs: ["get","list","watch"]
- apiGroups: [""] - apiGroups: [""]
resources: ["secrets"] resources: ["secrets"]
verbs: ["get"] verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["ingresses"]
verbs: ["create","delete","get","list","patch","update"]
--- ---
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
@ -29,7 +41,7 @@ metadata:
name: jenkins name: jenkins
roleRef: roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
kind: Role kind: ClusterRole
name: jenkins name: jenkins
subjects: subjects:
- kind: ServiceAccount - kind: ServiceAccount

11
jenkins/readme.md
View File

@ -1,3 +1,13 @@
# Jenkins on Amazon Kubernetes
For running Jenkins on AMAZON, start [here](./amazon-eks/readme.md)
# Jenkins on Local (Docker Windows \ Minikube \ etc)
For running Jenkins on Local Docker for Windows or Minikube <br/>
Watch the [video](https://youtu.be/eRWIJGF3Y2g)
# Setting up Jenkins Agent # Setting up Jenkins Agent
After installing `kubernetes-plugin` for Jenkins After installing `kubernetes-plugin` for Jenkins
@ -14,6 +24,7 @@ After installing `kubernetes-plugin` for Jenkins
* Add Kubernetes Pod Template * Add Kubernetes Pod Template
* Name: jenkins-slave * Name: jenkins-slave
* Namespace: jenkins * Namespace: jenkins
* Service Account: jenkins
* Labels: jenkins-slave (you will need to use this label on all jobs) * Labels: jenkins-slave (you will need to use this label on all jobs)
* Containers | Add Template * Containers | Add Template
* Name: jnlp * Name: jnlp

20
kubernetes/cloud/amazon/getting-started.md
View File

@ -9,7 +9,7 @@ docker run -it --rm -v ${PWD}:/work -w /work --entrypoint /bin/sh amazon/aws-cli
cd ./kubernetes/cloud/amazon cd ./kubernetes/cloud/amazon
yum install jq yum install jq gzip nano tar git
``` ```
## Login to AWS ## Login to AWS
@ -116,13 +116,27 @@ aws eks create-nodegroup \
## EKS CTL example ## EKS CTL example
``` ```
eksctl create cluster --name getting-started-eks-1 \ # Install EKS CTL
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
mv /tmp/eksctl /usr/local/bin
# Create SSH key for Node access (if you need it)
yum install openssh
mkdir -p ~/.ssh/
PASSPHRASE="mysuperstrongpassword"
ssh-keygen -t rsa -b 4096 -N "${PASSPHRASE}" -C "your_email@example.com" -q -f ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_rsa*
eksctl create cluster --name getting-started-eks \
--region ap-southeast-2 \ --region ap-southeast-2 \
--version 1.16 \ --version 1.16 \
--managed \ --managed \
--node-type t2.small \ --node-type t2.small \
--nodes 1 \ --nodes 1 \
--node-volume-size 200 --ssh-access \
--ssh-public-key=~/.ssh/id_rsa.pub \
--node-volume-size 200
``` ```
## Create some sample containers ## Create some sample containers