From c4fd6e6deaf745cd1a660355ad7007929490b472 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Mon, 22 Nov 2021 14:36:29 +1100
Subject: [PATCH 01/16] add python redis code and guide

---
 .../part-5.database.redis/readme.md           |   2 +-
 .../part-5.database.redis/README.md           | 499 ++++++++++++++++++
 .../part-5.database.redis/dockerfile          |  14 +
 .../part-5.database.redis/requirements.txt    |   2 +
 .../part-5.database.redis/src/app.py          |  99 ++++
 5 files changed, 615 insertions(+), 1 deletion(-)
 create mode 100644 python/introduction/part-5.database.redis/README.md
 create mode 100644 python/introduction/part-5.database.redis/dockerfile
 create mode 100644 python/introduction/part-5.database.redis/requirements.txt
 create mode 100644 python/introduction/part-5.database.redis/src/app.py

diff --git a/golang/introduction/part-5.database.redis/readme.md b/golang/introduction/part-5.database.redis/readme.md
index b7cd39f..2cac64b 100644
--- a/golang/introduction/part-5.database.redis/readme.md
+++ b/golang/introduction/part-5.database.redis/readme.md
@@ -23,7 +23,7 @@ Code is over [here](../../../storage/redis/clustering/readme.md)
 
 ## Go Dev Environment
 
-The same as Part 1+2+3, we start with a [dockerfile](./dockerfile) where we declare our version of `go`.
+The same as Part 1+2+3+4, we start with a [dockerfile](./dockerfile) where we declare our version of `go`.
 
 The `dockerfile`:
 
diff --git a/python/introduction/part-5.database.redis/README.md b/python/introduction/part-5.database.redis/README.md
new file mode 100644
index 0000000..eefa375
--- /dev/null
+++ b/python/introduction/part-5.database.redis/README.md
@@ -0,0 +1,499 @@
+# Introduction to Python: Storing data in Redis Database
+
+So far, we've learnt Python fundamentals, worked with data, files , HTTP and more importantly, basic data structures like `csv` and `json`.
+
+Be sure to checkout: </br>
+[Part 1: Intro to Python](../README.md) </br>
+[Part 2: Files](../part-2.files/README.md) </br>
+[Part 3: JSON](../part-3.json/README.md) </br>
+
+## Start up a Redis Cluster
+
+Follow my Redis clustering Tutorial </br>
+
+<a href="https://youtube.com/playlist?list=PLHq1uqvAteVtlgFkmOlIqWro3XP26y_oW" title="Redis"><img src="https://i.ytimg.com/vi/L3zp347cWNw/hqdefault.jpg" width="50%" alt="Redis Guide" /></a>
+
+Code is over [here](../../../storage/redis/clustering/readme.md)
+
+## Python Dev Environment
+
+The same as Part 1+2+3+4, we start with a [dockerfile](./dockerfile) where we declare our version of `python`.
+
+```
+FROM python:3.9.6-alpine3.13 as dev
+
+WORKDIR /work
+```
+
+Let's build and start our container: 
+
+```
+cd python\introduction\part-5.database.redis
+
+docker build --target dev . -t python
+docker run -it -v ${PWD}:/work -p 5000:5000 --net redis python sh
+
+/work # python --version
+Python 3.9.6
+
+```
+
+## Our application
+
+We're going to use what we've learnt in part 1,2 & 3 and create
+our customer app that handles customer data </br>
+Firstly we have to import our dependencies:
+
+```
+import os.path
+import csv
+import json
+from flask import Flask
+from flask import request
+
+```
+
+Then we have a class to define what a customer looks like:
+```
+class Customer:
+  def __init__(self, c="",f="",l=""):
+    self.customerID = c
+    self.firstName  = f
+    self.lastName   = l
+  def fullName(self):
+    return self.firstName + " " + self.lastName
+```
+
+And also set a global variable for the location of our videos `json` file:
+
+```
+dataPath = "./customers.json"
+```
+
+Then we need a function which returns our customers:
+```
+def getCustomers():
+  if os.path.isfile(dataPath):
+    with open(dataPath, newline='') as customerFile:
+      data = customerFile.read()
+      customers = json.loads(data)
+      return customers
+  else: 
+    return {}
+```
+
+Here is a function to return a specific customer:
+```
+def getCustomer(customerID):
+  customers = getCustomers()
+
+  if customerID in customers:
+    return customers[customerID]
+  else:
+    return {}
+```
+And finally a function for updating our customers:
+
+```
+def updateCustomers(customers):
+  with open(dataPath, 'w', newline='') as customerFile:
+    customerJSON = json.dumps(customers)
+    customerFile.write(customerJSON)
+```
+
+In the previous episode, we've created a `json` file to hold all our customers. </br>
+We've learnt how to read and write to file and temporarily use the file for our storage. </br>
+
+Let's create a file called `customers.json` : 
+```
+{
+  "a": {
+    "customerID": "a",
+    "firstName": "James",
+    "lastName": "Baker"
+  },
+  "b": {
+    "customerID": "b",
+    "firstName": "Jonathan",
+    "lastName": "D"
+  },
+  "c": {
+    "customerID": "c",
+    "firstName": "Aleem",
+    "lastName": "Janmohamed"
+  },
+  "d": {
+    "customerID": "d",
+    "firstName": "Ivo",
+    "lastName": "Galic"
+  },
+  "e": {
+    "customerID": "e",
+    "firstName": "Joel",
+    "lastName": "Griffiths"
+  },
+  "f": {
+    "customerID": "f",
+    "firstName": "Michael",
+    "lastName": "Spinks"
+  },
+  "g": {
+    "customerID": "g",
+    "firstName": "Victor",
+    "lastName": "Savkov"
+  }
+}
+```
+
+Now that we have our customer data and functions to read and update our customer data, let's define our `Flask` application:
+
+
+```
+app = Flask(__name__)
+```
+
+We create our route to get all customers:
+
+```
+@app.route("/", methods=['GET'])
+def get_customers():
+    customers = getCustomers()
+    return json.dumps(customers)
+```
+
+A route to get one customer by ID:
+
+```
+@app.route("/get/<string:customerID>", methods=['GET'])
+def get_customer(customerID):
+    customer = getCustomer(customerID)
+
+    if customer == {}:
+      return {}, 404
+    else:
+      return customer
+```
+
+And finally a route to update or add customers called `/set` :
+
+```
+@app.route("/set", methods=['POST'])
+def add_customer():
+    jsonData = request.json
+
+    if "customerID" not in jsonData:
+      return "customerID required", 400
+    if "firstName" not in jsonData:
+      return "firstName required", 400
+    if "lastName" not in jsonData:
+      return "lastName required", 400
+    
+    customers = getCustomers()
+    customers[jsonData["customerID"]] = Customer( jsonData["customerID"], jsonData["firstName"], jsonData["lastName"]).__dict__
+    updateCustomers(customers)
+    return "success", 200
+```
+
+Before we can be done, we need to import our `Flask` dependency we covered in our Python HTTP video. </br>
+Let's create a `requirements.txt` file:
+
+```
+Flask == 2.0.2
+```
+
+We can install our dependencies using:
+
+```
+pip install -r requirements.txt
+```
+
+This gives us a web application that handles customer data and using a file as it's storage </br>
+To test it, we can start up Flask:
+
+```
+export FLASK_APP=src/app
+flask run -h 0.0.0 -p 5000
+```
+
+Now we can confirm it's working by accessing our application in the browser on `http://localhost:5000`
+
+## Redis
+
+To connect to Redis, we'll use a popular library called `redis-py` which we can grab from [here](https://github.com/redis/redis-py) </br>
+The pip install is over [here](https://pypi.org/project/redis/3.5.3/) </br>
+
+Let's add that to our `requirements.txt` dependency files.
+
+```
+redis == 3.5.3
+```
+
+We can proceed to install it using `pip install`
+
+```
+pip install -r requirements.txt
+```
+
+Now to connect to Redis in a highly available manner, we need to take a look at the 
+`Sentinel support` section of the guide </br>
+
+Let's test the library. The beauty of Python is that it's a scripting language, so we don't have to compile and keep restarting our application, we can test each line of code. </br>
+
+```
+python
+from redis.sentinel import Sentinel
+
+sentinel = Sentinel([('sentinel-0', 5000),('sentinel-1', 5000),('sentinel-2', 5000)], socket_timeout=0.1)
+
+sentinel.discover_master('mymaster')
+
+sentinel.discover_slaves('mymaster')
+
+master = sentinel.master_for('mymaster',password = "a-very-complex-password-here", socket_timeout=0.1)
+
+slave = sentinel.slave_for('mymaster',password = "a-very-complex-password-here", socket_timeout=0.1)
+
+master.set('foo', 'bar')
+slave.get('foo')
+```
+
+We can demonstrate reading and writing a key value pair. </br>
+We can also demonstrate failure, when we stop the current master, we'll get a connection error. It's important to implement retry logic. </br>
+If we wait a moment and execute commands again, we will see that it starts to work.
+
+
+```
+# stop current master
+docker rm -f redis-0
+
+master.set('foo', 'bar2')
+
+redis.exceptions.ConnectionError: Connection closed by server.
+
+# retry moments later...
+
+master.set('foo', 'bar2')
+slave.get('foo')
+
+sentinel.discover_master('mymaster')
+sentinel.discover_slaves('mymaster')
+```
+
+We can find the current master by running `docker inspect` to see who owns that IP address.
+
+Start up `redis-0` again, to simulate a recovery from failure.
+
+## Connecting our App to Redis
+
+To connect to redis, we'll want to read the connection info from environment variables. Let's set some global variables.
+
+```
+import os
+
+redis_sentinels = os.environ.get('REDIS_SENTINELS')
+redis_master_name = os.environ.get('REDIS_MASTER_NAME')
+redis_password = os.environ.get('REDIS_PASSWORD')
+
+```
+
+We will need to restart our container so we can inject these environment variables. Let's go ahead and do that:
+
+```
+docker run -it -p 5000:5000 `
+  --net redis `
+  -v ${PWD}:/work `
+  -e REDIS_SENTINELS="sentinel-0:5000,sentinel-1:5000,sentinel-2:5000" `
+  -e REDIS_MASTER_NAME="mymaster" `
+  -e REDIS_PASSWORD="a-very-complex-password-here" `
+  python sh
+
+# re-install our dependencies
+pip install -r requirements.txt
+```
+
+Now we can setup a client:
+
+```
+from redis.sentinel import Sentinel
+
+sentinels = []
+
+for s in redis_sentinels.split(","):
+  sentinels.append((s.split(":")[0], s.split(":")[1]))
+
+redis_sentinel = Sentinel(sentinels, socket_timeout=5)
+redis_master = redis_sentinel.master_for(redis_master_name,password = redis_password, socket_timeout=5)
+```
+
+## Retry logic
+
+Now we noticed that if we have a master that fails, the sentinels will choose and assign a new master. We can see this by simply retrying our redis command. </br>
+
+When talking to redis we need to have some retry capability to be able to recover from this scenario. </br>
+
+Let's build a retry function at the top of our application, that runs a redis command:
+
+```
+def redis_command(command, *args):
+  max_retries = 3
+  count = 0
+  backoffSeconds = 5
+  while True:
+    try:
+      return command(*args)
+    except (redis.exceptions.ConnectionError, redis.exceptions.TimeoutError):
+      count += 1
+      if count > max_retries:
+        raise
+    print('Retrying in {} seconds'.format(backoffSeconds))
+    time.sleep(backoffSeconds)
+```
+
+We can test out our `redis_command` by calling it and printing the result 
+to the screen
+
+```
+print(redis_command(redis_master.set, 'foo', 'bar'))
+print(redis_command(redis_master.get, 'foo'))
+```
+
+We can simulate failure again, by finding and stopping the current master.
+
+Once we're done with our tests, we can `exec` into the current master and run `FLUSHALL` to remove our test records from redis.
+
+## Saving our data to Redis
+
+Now let's change our customer functions to point to Redis instead of file </br>
+
+Starting with `getCustomer` to retrieve a single customer
+```
+def getCustomer(customerID):
+  customer = redis_command(redis_master.get, customerID)
+  
+  if customer == None:
+    return {}
+  else:
+    c = str(customer, 'utf-8')
+    return json.loads(c)
+```
+Now we can use that to return all our customers by updating the `getCustomers` function:
+
+```
+def getCustomers():
+  customers = {}
+  customerIDs = redis_command(redis_master.scan_iter, "*")
+  for customerID in customerIDs:
+    customer = getCustomer(customerID)
+    customers[customer["customerID"]] = customer
+  
+  return customers
+```
+
+Let's improve our functions by adding a new function to update a single customer:
+
+```
+def updateCustomer(customer):
+  redis_command(redis_master.set, customer.customerID, json.dumps(customer.__dict__))
+
+```
+
+And finally we can use that function to update all customers by tweaking our `updateCustomers` function:
+
+```
+def updateCustomers(customers):
+  for customer in customers:
+    updateCustomer(customer)
+```
+
+Now our simple functions are done, let's hook them up to our endpoints
+
+```
+# firstly delete these test lines
+print(redis_command(redis_master.set, 'foo', 'bar'))
+print(redis_command(redis_master.get, 'foo'))
+```
+
+Our simple Get all 
+
+```
+@app.route("/", methods=['GET'])
+def get_customers():
+  customers = getCustomers()
+  return json.dumps(customers)
+```
+
+Our Get by ID
+
+```
+@app.route("/get/<string:customerID>", methods=['GET'])
+def get_customer(customerID):
+    customer = getCustomer(customerID)
+
+    if customer == {}:
+      return {}, 404
+    else:
+      return customer
+```
+
+And our update endpoint to update a customer
+
+```
+@app.route("/set", methods=['POST'])
+def add_customer():
+    jsonData = request.json
+
+    if "customerID" not in jsonData:
+      return "customerID required", 400
+    if "firstName" not in jsonData:
+      return "firstName required", 400
+    if "lastName" not in jsonData:
+      return "lastName required", 400
+    
+    customer = Customer( jsonData["customerID"], jsonData["firstName"], jsonData["lastName"])
+    updateCustomer(customer)
+    return "success", 200
+```
+## Docker
+
+Let's build our container image and run it while mounting our customer file
+
+Our final `dockerfile`
+```
+FROM python:3.9.6-alpine3.13 as dev
+
+WORKDIR /work
+
+FROM dev as runtime
+WORKDIR /app
+
+COPY ./requirements.txt /app/
+RUN pip install -r /app/requirements.txt
+
+COPY ./src/app.py /app/app.py
+ENV FLASK_APP=app.py
+
+CMD flask run -h 0.0.0 -p 5000
+
+```
+
+Build our container.
+
+```
+cd python\introduction\part-5.database.redis
+
+docker build . -t customer-app
+
+```
+
+Now we can run our production container:
+
+```
+docker build . -t customer-app
+
+docker run -it -p 5000:5000 `
+  --net redis `
+  -e REDIS_SENTINELS="sentinel-0:5000,sentinel-1:5000,sentinel-2:5000" `
+  -e REDIS_MASTER_NAME="mymaster" `
+  -e REDIS_PASSWORD="a-very-complex-password-here" `
+  customer-app
+```
\ No newline at end of file
diff --git a/python/introduction/part-5.database.redis/dockerfile b/python/introduction/part-5.database.redis/dockerfile
new file mode 100644
index 0000000..05fcb85
--- /dev/null
+++ b/python/introduction/part-5.database.redis/dockerfile
@@ -0,0 +1,14 @@
+FROM python:3.9.6-alpine3.13 as dev
+
+WORKDIR /work
+
+FROM dev as runtime
+WORKDIR /app
+
+COPY ./requirements.txt /app/
+RUN pip install -r /app/requirements.txt
+
+COPY ./src/app.py /app/app.py
+ENV FLASK_APP=app.py
+
+CMD flask run -h 0.0.0 -p 5000
\ No newline at end of file
diff --git a/python/introduction/part-5.database.redis/requirements.txt b/python/introduction/part-5.database.redis/requirements.txt
new file mode 100644
index 0000000..44daf50
--- /dev/null
+++ b/python/introduction/part-5.database.redis/requirements.txt
@@ -0,0 +1,2 @@
+Flask == 2.0.2
+Redis == 3.5.3
\ No newline at end of file
diff --git a/python/introduction/part-5.database.redis/src/app.py b/python/introduction/part-5.database.redis/src/app.py
new file mode 100644
index 0000000..c02a496
--- /dev/null
+++ b/python/introduction/part-5.database.redis/src/app.py
@@ -0,0 +1,99 @@
+import os.path
+import csv
+import os
+import json
+import time
+from flask import Flask
+from flask import request
+import redis
+from redis.sentinel import Sentinel
+
+redis_sentinels = os.environ.get('REDIS_SENTINELS')
+redis_master_name = os.environ.get('REDIS_MASTER_NAME')
+redis_password = os.environ.get('REDIS_PASSWORD')
+
+class Customer:
+  def __init__(self, c="",f="",l=""):
+    self.customerID = c
+    self.firstName  = f
+    self.lastName   = l
+  def fullName(self):
+    return self.firstName + " " + self.lastName
+
+def redis_command(command, *args):
+  max_retries = 3
+  count = 0
+  backoffSeconds = 5
+  while True:
+    try:
+      return command(*args)
+    except (redis.exceptions.ConnectionError, redis.exceptions.TimeoutError):
+      count += 1
+      if count > max_retries:
+        raise
+    print('Retrying in {} seconds'.format(backoffSeconds))
+    time.sleep(backoffSeconds)
+
+def getCustomers():
+  customers = {}
+  customerIDs = redis_command(redis_master.scan_iter, "*")
+  for customerID in customerIDs:
+    customer = getCustomer(customerID)
+    customers[customer["customerID"]] = customer
+  
+  return customers
+
+def getCustomer(customerID):
+  customer = redis_command(redis_master.get, customerID)
+  
+  if customer == None:
+    return {}
+  else:
+    c = str(customer, 'utf-8')
+    return json.loads(c)
+
+def updateCustomer(customer):
+  redis_command(redis_master.set, customer.customerID, json.dumps(customer.__dict__))
+
+def updateCustomers(customers):
+  for customer in customers:
+    updateCustomer(customer)
+  
+app = Flask(__name__)
+
+sentinels = []
+
+for s in redis_sentinels.split(","):
+  sentinels.append((s.split(":")[0], s.split(":")[1]))
+
+redis_sentinel = Sentinel(sentinels, socket_timeout=5)
+redis_master = redis_sentinel.master_for(redis_master_name,password = redis_password, socket_timeout=5)
+
+@app.route("/", methods=['GET'])
+def get_customers():
+  customers = getCustomers()
+  return json.dumps(customers)
+
+@app.route("/get/<string:customerID>", methods=['GET'])
+def get_customer(customerID):
+    customer = getCustomer(customerID)
+
+    if customer == {}:
+      return {}, 404
+    else:
+      return customer
+
+@app.route("/set", methods=['POST'])
+def add_customer():
+    jsonData = request.json
+
+    if "customerID" not in jsonData:
+      return "customerID required", 400
+    if "firstName" not in jsonData:
+      return "firstName required", 400
+    if "lastName" not in jsonData:
+      return "lastName required", 400
+    
+    customer = Customer( jsonData["customerID"], jsonData["firstName"], jsonData["lastName"])
+    updateCustomer(customer)
+    return "success", 200
\ No newline at end of file

From 23767a56440103a609a821a95e7da4a61b234218 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 5 Dec 2021 08:25:53 +1100
Subject: [PATCH 02/16] video player side edit

---
 golang/introduction/part-5.database.redis/readme.md | 2 +-
 python/introduction/part-5.database.redis/README.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/golang/introduction/part-5.database.redis/readme.md b/golang/introduction/part-5.database.redis/readme.md
index 2cac64b..125dddc 100644
--- a/golang/introduction/part-5.database.redis/readme.md
+++ b/golang/introduction/part-5.database.redis/readme.md
@@ -17,7 +17,7 @@ This is important for learning Go, however there are a few challenges for using
 
 Follow my Redis clustering Tutorial </br>
 
-<a href="https://youtube.com/playlist?list=PLHq1uqvAteVtlgFkmOlIqWro3XP26y_oW" title="Redis"><img src="https://i.ytimg.com/vi/L3zp347cWNw/hqdefault.jpg" width="50%" alt="Redis Guide" /></a>
+<a href="https://youtube.com/playlist?list=PLHq1uqvAteVtlgFkmOlIqWro3XP26y_oW" title="Redis"><img src="https://i.ytimg.com/vi/L3zp347cWNw/hqdefault.jpg" width="30%" alt="Redis Guide" /></a>
 
 Code is over [here](../../../storage/redis/clustering/readme.md)
 
diff --git a/python/introduction/part-5.database.redis/README.md b/python/introduction/part-5.database.redis/README.md
index eefa375..6e94949 100644
--- a/python/introduction/part-5.database.redis/README.md
+++ b/python/introduction/part-5.database.redis/README.md
@@ -11,7 +11,7 @@ Be sure to checkout: </br>
 
 Follow my Redis clustering Tutorial </br>
 
-<a href="https://youtube.com/playlist?list=PLHq1uqvAteVtlgFkmOlIqWro3XP26y_oW" title="Redis"><img src="https://i.ytimg.com/vi/L3zp347cWNw/hqdefault.jpg" width="50%" alt="Redis Guide" /></a>
+<a href="https://youtube.com/playlist?list=PLHq1uqvAteVtlgFkmOlIqWro3XP26y_oW" title="Redis"><img src="https://i.ytimg.com/vi/L3zp347cWNw/hqdefault.jpg" width="30%" alt="Redis Guide" /></a>
 
 Code is over [here](../../../storage/redis/clustering/readme.md)
 

From 9449f8a2a749b1da83c918fb289e3e3111269824 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Wed, 26 Jan 2022 10:32:43 +1100
Subject: [PATCH 03/16] wip

---
 .gitignore                   |   1 +
 kubernetes/rancher/README.md | 230 +++++++++++++++++++++++++++++++++++
 2 files changed, 231 insertions(+)
 create mode 100644 kubernetes/rancher/README.md

diff --git a/.gitignore b/.gitignore
index 05f0edd..8fc8891 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,3 +11,4 @@ __pycache__/
 security/letsencrypt/introduction/certs/**
 kubernetes/shipa/installs/shipa-helm-chart-1.1.1/
 messaging/kafka/data/*
+kubernetes/rancher/volume/*
diff --git a/kubernetes/rancher/README.md b/kubernetes/rancher/README.md
new file mode 100644
index 0000000..3d59625
--- /dev/null
+++ b/kubernetes/rancher/README.md
@@ -0,0 +1,230 @@
+# Introduction to Rancher: On-prem Kubernetes
+
+This guide follows the general instructions of running a [manual rancher install](https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/) and running our own infrastructure on Hyper-v
+
+# Hyper-V : Prepare our infrastructure
+
+In this demo, we will use Hyper-V to create our infrastructure.  </br>
+For on-premise, many companies use either Hyper-V, VMWare Vsphere and other technologies to create virtual infrastructure on bare metal. </br>
+
+Few points to note here:
+
+* Benefit of Virtual infrastructure is that it's immutable
+  a) We can add and throw away virtual machines at will.
+  b) This makes maintenance easier as we can roll updated virtual machines instead of 
+     patching existing machines and turning them to long-living snowflakes.
+  c) Reduce lifespan of machines
+
+* Bare Metal provides the compute. 
+  a) We don't want Kubernetes directly on bare metal as we want machines to be immutable.
+  b) This goes back to the previous point on immutability.
+
+* Every virtual machine needs to be able to reach each other on the network
+  a) This is a kubernetes networking requirements that all nodes can communicate with one another
+
+# Hyper-V : Create our network
+
+In order for us to create virtual machines all on the same network, I am going to create a virtual switch in Hyper-v </br>
+Open Powershell in administrator
+
+```
+# get our network adapter where all virtual machines will run on
+# grab the name we want to use
+Get-NetAdapter
+
+Import-Module Hyper-V
+$ethernet = Get-NetAdapter -Name "Ethernet 2"
+New-VMSwitch -Name "virtual-network" -NetAdapterName $ethernet.Name -AllowManagementOS $true -Notes "shared virtual network interface"
+```
+
+# Hyper-V : Create our machines
+
+We firstly need harddrives for every VM. </br>
+Let's create three:
+
+```
+mkdir c:\temp\vms\linux-0\
+mkdir c:\temp\vms\linux-1\
+mkdir c:\temp\vms\linux-2\
+
+New-VHD -Path c:\temp\vms\linux-0\linux-0.vhdx -SizeBytes 20GB
+New-VHD -Path c:\temp\vms\linux-1\linux-1.vhdx -SizeBytes 20GB
+New-VHD -Path c:\temp\vms\linux-2\linux-2.vhdx -SizeBytes 20GB
+```
+
+```
+New-VM `
+-Name "linux-0" `
+-Generation 1 `
+-MemoryStartupBytes 2048MB `
+-SwitchName "virtual-network" `
+-VHDPath "c:\temp\vms\linux-0\linux-0.vhdx" `
+-Path "c:\temp\vms\linux-0\"
+
+New-VM `
+-Name "linux-1" `
+-Generation 1 `
+-MemoryStartupBytes 2048MB `
+-SwitchName "virtual-network" `
+-VHDPath "c:\temp\vms\linux-1\linux-1.vhdx" `
+-Path "c:\temp\vms\linux-1\"
+
+New-VM `
+-Name "linux-2" `
+-Generation 1 `
+-MemoryStartupBytes 2048MB `
+-SwitchName "virtual-network" `
+-VHDPath "c:\temp\vms\linux-2\linux-2.vhdx" `
+-Path "c:\temp\vms\linux-2\"
+
+```
+
+Setup a DVD drive that holds the `iso` file for Ubuntu Server
+
+```
+Set-VMDvdDrive -VMName "linux-0" -ControllerNumber 1 -Path "C:\temp\ubuntu-20.04.3-live-server-amd64.iso"
+Set-VMDvdDrive -VMName "linux-1" -ControllerNumber 1 -Path "C:\temp\ubuntu-20.04.3-live-server-amd64.iso"
+Set-VMDvdDrive -VMName "linux-2" -ControllerNumber 1 -Path "C:\temp\ubuntu-20.04.3-live-server-amd64.iso"
+```
+
+Start our VM's
+
+```
+Start-VM -Name "linux-0"
+Start-VM -Name "linux-1"
+Start-VM -Name "linux-2"
+```
+
+Now we can open up Hyper-v Manager and see our infrastructure. </br>
+In this video we'll connect to each server, and run through the initial ubuntu setup. </br>
+Once finished, select the option to reboot and once it starts, you will notice an `unmount` error on CD-Rom </br>
+This is ok, just shut down the server and start it up again.
+
+# Hyper-V : Setup SSH for our machines
+
+Now in this demo, because I need to copy rancher bootstrap commands to each VM, it would be easier to do so
+using SSH. So let's connect to each VM in Hyper-V and setup SSH. </br>
+This is because `copy+paste` does not work without `Enhanced Session` mode in Ubuntu Server. </br>
+
+Let's temporarily turn on SSH on each server:
+
+```
+sudo apt update
+sudo apt install -y nano net-tools openssh-server
+sudo systemctl enable ssh
+sudo ufw allow ssh
+sudo systemctl start ssh
+```
+
+Record the IP address of each VM so we can SSH to it:
+
+```
+sudo ifconfig
+# record eth0
+linux-0 IP=192.168.0.22
+linux-1 IP=192.168.0.23
+linux-2 IP=192.168.0.24
+```
+
+In new Powershell windows, let's SSH to our VMs
+
+```
+ssh linux-0@192.168.0.22
+ssh linux-1@192.168.0.23
+ssh linux-2@192.168.0.24
+```
+
+# Setup Docker
+
+It is required that every machine that needs to join our cluster, has docker running on it. </br>
+Firstly, rancher will use docker to run it's agent as well as bootstrap the cluster.</br>
+
+Install docker on each VM:
+```
+curl -sSL https://get.docker.com/ | sh
+sudo usermod -aG docker $(whoami)
+sudo service docker start
+```
+
+# Running Rancher in Docker
+
+So Rancher can be [deployed](https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/) almost anywhere. </br>
+We can run it in Kubernetes on-prem or the cloud.  </br>
+
+Now because we want Rancher to manage kubernetes clusters, we dont want it running in the clusters we are managing. </br>
+So I would like to keep my Rancher server outside and separate from my Kubernetes clusters.</br>
+
+So let's setup a single server with [docker](https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/)
+
+## Persist data
+
+We will want to persist ranchers data across reboots. </br>
+Rancher stores its data under `/var/lib/rancher` 
+
+Let's create some space to save data:
+
+```
+cd kubernetes/rancher
+mkdir volume
+
+```
+
+## Run Rancher
+
+```
+docker run -d --name rancher-server  -v ${PWD}/volume:/var/lib/rancher --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher
+```
+
+## Unlock Rancher
+
+Once its up and running we can extract the Rancher initial boostrap password from the logs 
+
+```
+docker logs rancher-server > rancher.log
+```
+
+## Get Rancher IP
+
+It's important that our servers can reach the Rancher server. </br>
+As all the VMs and my machine are on the same network, we can use my machine IP as the server IP so the VM's can reach it. </br>
+let's grab the IP:
+
+```
+ipconfig
+```
+
+We can now access Rancher on [localhost](https://localhost)
+
+## Deploy Sample Workloads
+
+To deploy some sample basic workloads, let's get the `kubeconfig` for our cluster </br>
+
+Set kubeconfig:
+
+```
+$ENV:KUBECONFIG="<path-to-kubeconfig>"
+```
+
+Deploy 2 pods, and a service:
+
+```
+kubectl create ns marcel
+kubectl -n marcel apply -f .\kubernetes\configmaps\configmap.yaml
+kubectl -n marcel apply -f .\kubernetes\secrets\secret.yaml
+kubectl -n marcel apply -f .\kubernetes\deployments\deployment.yaml
+kubectl -n marcel apply -f .\kubernetes\services\service.yaml
+```
+
+One caveat is because we are not a cloud provider, Kubernetes does not support our service `type=LoadBalancer`. </br>
+For that, we need something like `metallb`. </br>
+However - we can `port-forward`
+
+```
+kubectl -n marcel get svc 
+NAME              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
+example-service   LoadBalancer   10.43.235.240   <pending>     80:31310/TCP   13s
+
+kubectl -n marcel port-forward svc/example-service 81:80
+```
+
+We can access our example-app on port 81
\ No newline at end of file

From 633f8f2865f73df293f37e1d3a51d5ba2fcc9388 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 13 Feb 2022 19:42:42 +1100
Subject: [PATCH 04/16] updates

---
 kubernetes/deployments/deployment.yaml |  4 ++++
 kubernetes/rancher/README.md           | 12 ++++++------
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/kubernetes/deployments/deployment.yaml b/kubernetes/deployments/deployment.yaml
index 1bb0605..88a96c5 100644
--- a/kubernetes/deployments/deployment.yaml
+++ b/kubernetes/deployments/deployment.yaml
@@ -43,6 +43,10 @@ spec:
           limits:
             memory: "256Mi"
             cpu: "500m"
+      tolerations:
+      - key: "cattle.io/os"
+        value: "linux"
+        effect: "NoSchedule"      
 #NOTE: comment out `volumeMounts` section for configmap and\or secret guide
         # volumeMounts:
         # - name: secret-volume
diff --git a/kubernetes/rancher/README.md b/kubernetes/rancher/README.md
index 3d59625..ad60de8 100644
--- a/kubernetes/rancher/README.md
+++ b/kubernetes/rancher/README.md
@@ -121,17 +121,17 @@ Record the IP address of each VM so we can SSH to it:
 ```
 sudo ifconfig
 # record eth0
-linux-0 IP=192.168.0.22
-linux-1 IP=192.168.0.23
-linux-2 IP=192.168.0.24
+linux-0 IP=192.168.0.16
+linux-1 IP=192.168.0.17
+linux-2 IP=192.168.0.18
 ```
 
 In new Powershell windows, let's SSH to our VMs
 
 ```
-ssh linux-0@192.168.0.22
-ssh linux-1@192.168.0.23
-ssh linux-2@192.168.0.24
+ssh linux-0@192.168.0.16
+ssh linux-1@192.168.0.17
+ssh linux-2@192.168.0.18
 ```
 
 # Setup Docker

From 4d02dbe534763b8abc2f42dccda391155a2794cf Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 18:07:05 +1100
Subject: [PATCH 05/16] example-app

---
 .gitignore                                    |  1 +
 .../example-application/configmap.yaml        | 10 +++++
 .../example-application/deployment.yaml       | 43 +++++++++++++++++++
 .../example-application/ingress.yaml          | 17 ++++++++
 .../example-application/service.yaml          | 15 +++++++
 5 files changed, 86 insertions(+)
 create mode 100644 kubernetes/portainer/example-application/configmap.yaml
 create mode 100644 kubernetes/portainer/example-application/deployment.yaml
 create mode 100644 kubernetes/portainer/example-application/ingress.yaml
 create mode 100644 kubernetes/portainer/example-application/service.yaml

diff --git a/.gitignore b/.gitignore
index 05f0edd..e76baf6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,3 +11,4 @@ __pycache__/
 security/letsencrypt/introduction/certs/**
 kubernetes/shipa/installs/shipa-helm-chart-1.1.1/
 messaging/kafka/data/*
+kubernetes/portainer/volume*
diff --git a/kubernetes/portainer/example-application/configmap.yaml b/kubernetes/portainer/example-application/configmap.yaml
new file mode 100644
index 0000000..e9e3124
--- /dev/null
+++ b/kubernetes/portainer/example-application/configmap.yaml
@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: example-config
+data:
+  config.json: |
+    {
+      "environment" : "dev"
+    }
+# kubectl create configmap example-config --from-file ./golang/configs/config.json
diff --git a/kubernetes/portainer/example-application/deployment.yaml b/kubernetes/portainer/example-application/deployment.yaml
new file mode 100644
index 0000000..b66f5d1
--- /dev/null
+++ b/kubernetes/portainer/example-application/deployment.yaml
@@ -0,0 +1,43 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: example-deploy
+  labels:
+    app: example-app
+    test: test
+spec:
+  selector:
+    matchLabels:
+      app: example-app
+  replicas: 2
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+      maxUnavailable: 0
+  template:
+    metadata:
+      labels:
+        app: example-app
+    spec:
+      containers:
+      - name: example-app
+        image: aimvector/python:1.0.3
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 5000
+        resources:
+          requests:
+            memory: "64Mi"
+            cpu: "50m"
+          limits:
+            memory: "256Mi"
+            cpu: "500m"
+        volumeMounts:
+        - name: config-volume
+          mountPath: /configs/
+      volumes:
+      - name: config-volume
+        configMap:
+          name: example-config
diff --git a/kubernetes/portainer/example-application/ingress.yaml b/kubernetes/portainer/example-application/ingress.yaml
new file mode 100644
index 0000000..1b49a62
--- /dev/null
+++ b/kubernetes/portainer/example-application/ingress.yaml
@@ -0,0 +1,17 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: example-ingress
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: marcel.test
+    http:
+      paths:
+      - path: /hello
+        pathType: Prefix
+        backend:
+          service:
+            name: example-service
+            port:
+              number: 80
\ No newline at end of file
diff --git a/kubernetes/portainer/example-application/service.yaml b/kubernetes/portainer/example-application/service.yaml
new file mode 100644
index 0000000..7302b6d
--- /dev/null
+++ b/kubernetes/portainer/example-application/service.yaml
@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: example-service
+  labels:
+    app: example-app
+spec:
+  type: ClusterIP
+  selector:
+    app: example-app
+  ports:
+    - protocol: TCP
+      name: http
+      port: 80
+      targetPort: 5000
\ No newline at end of file

From a8ba3bba7c697727d41e4965f28717950f8484a0 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 18:21:24 +1100
Subject: [PATCH 06/16] ingress update

---
 kubernetes/portainer/example-application/ingress.yaml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kubernetes/portainer/example-application/ingress.yaml b/kubernetes/portainer/example-application/ingress.yaml
index 1b49a62..7b36be0 100644
--- a/kubernetes/portainer/example-application/ingress.yaml
+++ b/kubernetes/portainer/example-application/ingress.yaml
@@ -2,13 +2,15 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: example-ingress
+  annotations:
+    nginx.ingress.kubernetes.io/rewrite-target: /$2
 spec:
   ingressClassName: nginx
   rules:
   - host: marcel.test
     http:
       paths:
-      - path: /hello
+      - path: /hello(/|$)(.*)
         pathType: Prefix
         backend:
           service:

From 5481fb7c46ff92f704a11f3fa1ed1f1a92fdf7a8 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 18:30:51 +1100
Subject: [PATCH 07/16] update

---
 kubernetes/ingress/ingress-nginx-example.yaml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/kubernetes/ingress/ingress-nginx-example.yaml b/kubernetes/ingress/ingress-nginx-example.yaml
index 4e83911..aa87f00 100644
--- a/kubernetes/ingress/ingress-nginx-example.yaml
+++ b/kubernetes/ingress/ingress-nginx-example.yaml
@@ -2,7 +2,6 @@ apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   annotations:
-    kubernetes.io/ingress.class: "nginx"
     nginx.ingress.kubernetes.io/rewrite-target: /   #new 
   name: example-app
   namespace: example-app

From da8749be1068348322810496af66f7f289e83021 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 18:32:58 +1100
Subject: [PATCH 08/16] update

---
 kubernetes/ingress/ingress-nginx-example.yaml         | 1 +
 kubernetes/portainer/example-application/ingress.yaml | 1 -
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/kubernetes/ingress/ingress-nginx-example.yaml b/kubernetes/ingress/ingress-nginx-example.yaml
index aa87f00..4e83911 100644
--- a/kubernetes/ingress/ingress-nginx-example.yaml
+++ b/kubernetes/ingress/ingress-nginx-example.yaml
@@ -2,6 +2,7 @@ apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   annotations:
+    kubernetes.io/ingress.class: "nginx"
     nginx.ingress.kubernetes.io/rewrite-target: /   #new 
   name: example-app
   namespace: example-app
diff --git a/kubernetes/portainer/example-application/ingress.yaml b/kubernetes/portainer/example-application/ingress.yaml
index 7b36be0..f7fbcf8 100644
--- a/kubernetes/portainer/example-application/ingress.yaml
+++ b/kubernetes/portainer/example-application/ingress.yaml
@@ -5,7 +5,6 @@ metadata:
   annotations:
     nginx.ingress.kubernetes.io/rewrite-target: /$2
 spec:
-  ingressClassName: nginx
   rules:
   - host: marcel.test
     http:

From 733553237350869570029d2727878c907f55e106 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 18:48:30 +1100
Subject: [PATCH 09/16] update

---
 kubernetes/portainer/example-application/ingress.yaml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kubernetes/portainer/example-application/ingress.yaml b/kubernetes/portainer/example-application/ingress.yaml
index f7fbcf8..fd4522a 100644
--- a/kubernetes/portainer/example-application/ingress.yaml
+++ b/kubernetes/portainer/example-application/ingress.yaml
@@ -3,13 +3,14 @@ kind: Ingress
 metadata:
   name: example-ingress
   annotations:
-    nginx.ingress.kubernetes.io/rewrite-target: /$2
+    nginx.ingress.kubernetes.io/rewrite-target: /
 spec:
+  ingressClassName: nginx
   rules:
   - host: marcel.test
     http:
       paths:
-      - path: /hello(/|$)(.*)
+      - path: /hello
         pathType: Prefix
         backend:
           service:

From a2d82e4dc440ebe9df7fe0c078947b9d63d63c5b Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 18:49:53 +1100
Subject: [PATCH 10/16] update

---
 kubernetes/portainer/example-application/ingress.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kubernetes/portainer/example-application/ingress.yaml b/kubernetes/portainer/example-application/ingress.yaml
index fd4522a..7b36be0 100644
--- a/kubernetes/portainer/example-application/ingress.yaml
+++ b/kubernetes/portainer/example-application/ingress.yaml
@@ -3,14 +3,14 @@ kind: Ingress
 metadata:
   name: example-ingress
   annotations:
-    nginx.ingress.kubernetes.io/rewrite-target: /
+    nginx.ingress.kubernetes.io/rewrite-target: /$2
 spec:
   ingressClassName: nginx
   rules:
   - host: marcel.test
     http:
       paths:
-      - path: /hello
+      - path: /hello(/|$)(.*)
         pathType: Prefix
         backend:
           service:

From 99058f0d8984574d68fd7ac29cca5f8d2ff80602 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 21:36:14 +1100
Subject: [PATCH 11/16] test

---
 kubernetes/portainer/example-application/deployment.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kubernetes/portainer/example-application/deployment.yaml b/kubernetes/portainer/example-application/deployment.yaml
index b66f5d1..5f2fa7a 100644
--- a/kubernetes/portainer/example-application/deployment.yaml
+++ b/kubernetes/portainer/example-application/deployment.yaml
@@ -23,7 +23,7 @@ spec:
     spec:
       containers:
       - name: example-app
-        image: aimvector/python:1.0.3
+        image: aimvector/python:1.0.4
         imagePullPolicy: Always
         ports:
         - containerPort: 5000

From ac0e5cf053bd1c3a76be839656de70d2fe349d43 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sun, 20 Feb 2022 22:08:37 +1100
Subject: [PATCH 12/16] test

---
 kubernetes/portainer/example-application/deployment.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kubernetes/portainer/example-application/deployment.yaml b/kubernetes/portainer/example-application/deployment.yaml
index 5f2fa7a..b66f5d1 100644
--- a/kubernetes/portainer/example-application/deployment.yaml
+++ b/kubernetes/portainer/example-application/deployment.yaml
@@ -23,7 +23,7 @@ spec:
     spec:
       containers:
       - name: example-app
-        image: aimvector/python:1.0.4
+        image: aimvector/python:1.0.3
         imagePullPolicy: Always
         ports:
         - containerPort: 5000

From 4b42ca91c51ef3af554b36b240688081d1723d90 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Sat, 26 Feb 2022 12:02:03 +1100
Subject: [PATCH 13/16] update

---
 kubernetes/portainer/example-application/deployment.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kubernetes/portainer/example-application/deployment.yaml b/kubernetes/portainer/example-application/deployment.yaml
index b66f5d1..5f2fa7a 100644
--- a/kubernetes/portainer/example-application/deployment.yaml
+++ b/kubernetes/portainer/example-application/deployment.yaml
@@ -23,7 +23,7 @@ spec:
     spec:
       containers:
       - name: example-app
-        image: aimvector/python:1.0.3
+        image: aimvector/python:1.0.4
         imagePullPolicy: Always
         ports:
         - containerPort: 5000

From 79052248fad591eea33a79227d72110387a5a269 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Mon, 7 Mar 2022 22:28:16 +1100
Subject: [PATCH 14/16] portainer files

---
 kubernetes/portainer/README.md                | 124 ++++++++++++++++++
 kubernetes/portainer/kind.yaml                |  12 ++
 .../portainer-agent-ce211-k8s-nodeport.yaml   |  81 ++++++++++++
 .../portainer/portainer-agent-edge-k8s.yaml   | 100 ++++++++++++++
 4 files changed, 317 insertions(+)
 create mode 100644 kubernetes/portainer/README.md
 create mode 100644 kubernetes/portainer/kind.yaml
 create mode 100644 kubernetes/portainer/portainer-agent-ce211-k8s-nodeport.yaml
 create mode 100644 kubernetes/portainer/portainer-agent-edge-k8s.yaml

diff --git a/kubernetes/portainer/README.md b/kubernetes/portainer/README.md
new file mode 100644
index 0000000..21e02d7
--- /dev/null
+++ b/kubernetes/portainer/README.md
@@ -0,0 +1,124 @@
+# Introduction to Portainer
+
+Start here 👉🏽[https://www.portainer.io/](https://www.portainer.io/) </br>
+Documentation 👉🏽[https://docs.portainer.io/](https://docs.portainer.io/)
+
+## Portainer installation
+
+In this demo, I will be running Kubernetes 1.22 using `kind` </br>
+Which is compatible with portainer 2.11.1 </br>
+
+Let's go ahead with a local docker install: 
+
+```
+cd kubernetes\portainer
+mkdir volume-ce 
+
+docker run -d -p 9443:9443 -p 8000:8000 --name portainer-ce `
+--restart=always `
+-v /var/run/docker.sock:/var/run/docker.sock `
+-v ${PWD}/volume-ce:/data `
+portainer/portainer-ce:2.11.1
+```
+
+## SSL & DOMAIN
+
+We can also upload SSL certificates for our portainer.</br> 
+In this demo, portainer will issue self signed certificates. </br> 
+We will need a domain for our portainer server so our clusters can contact it. </br>
+Let's use [nip.io](https://nip.io/) to create a public endpoint for portainer.
+
+## Create Kubernetes Cluster
+
+Let's start by creating a local `kind` [cluster](https://kind.sigs.k8s.io/)
+
+For local clusters, we can use the public endpoint Agent. </br>
+We can get a public endpoint for the portainer agent by: </br>
+
+* Ingress
+* LoadBalancer
+* NodePort
+
+So we'll deploy portainer agent with `NodePort` for local </br>
+
+For production environments, I would recommend not to expose the portainer agent. </br>
+In this case, for Production, we'll use the portainer edge agent.  </br>
+
+
+To get `NodePort` exposed in `kind`, we'll open a host port with a [kind.yaml](./kind.yaml) config
+
+```
+kind create cluster --name local --config kind.yaml
+```
+
+## Manage Kubernetes Environments 
+
+The portainer UI gives us a one line command to deploy the portainer agent. </br>
+Note that in the video, we pick the `node port` option.
+
+## Local: Portainer Agent
+
+I download the YAML from [here](https://downloads.portainer.io/portainer-agent-ce211-k8s-nodeport.yaml) to take a closer look at what it is deploying </br>
+
+Deploy the portainer agent in my `kind` cluster:
+
+```
+kubectl apply -f portainer-agent-ce211-k8s-nodeport.yaml
+```
+
+See the agent: 
+
+```
+kubectl -n portainer get pods
+```
+
+See the service with the endpoint it exposes: 
+
+```
+kubectl -n portainer get svc
+```
+
+Now since we dont have a public load balancer and using nodeport, our service will be exposed on the node IP. </br>
+Since the Kubernetes node is our local machine, we should be able to access the portainer agent on `<computer-IP>:30778` </br>
+
+We can obtain our local IP with `ipconfig` </br>
+The IP and NodePort will be used to connect our portainer server to the new agent. </br>
+
+## Production: Portainer Edge Agent
+
+For the Edge agent, we get the command in the portainer UI. </br>
+Once deployed, we can see the egde agent in our AKS cluster:
+
+```
+kubectl -n portainer get pods
+```
+
+## Helm 
+
+Let's showcase how to deploy helm charts. </br>
+Most folks would have helm charts for their ingress controllers, monitoring, logging and other 
+platform dependencies.</br>
+
+Let's add Kubernetes NGINX Ingress repo:
+
+```
+https://kubernetes.github.io/ingress-nginx
+```
+
+## GitOps
+
+So from the Application menu, we can add an application from a `git` repository. </br>
+Let's add this repo:
+
+```
+https://github.com/marcel-dempers/docker-development-youtube-series
+```
+
+We also specify all our manifests path that portainer needs to deploy:
+
+* kubernetes/portainer/example-application/deployment.yaml
+* kubernetes/portainer/example-application/configmap.yaml
+* kubernetes/portainer/example-application/service.yaml
+* kubernetes/portainer/example-application/ingress.yaml
+
+Portainer will now poll our repo and deploy any updates, GitOps style!
diff --git a/kubernetes/portainer/kind.yaml b/kubernetes/portainer/kind.yaml
new file mode 100644
index 0000000..75d4b10
--- /dev/null
+++ b/kubernetes/portainer/kind.yaml
@@ -0,0 +1,12 @@
+apiVersion: kind.x-k8s.io/v1alpha4
+kind: Cluster
+nodes:
+- role: control-plane
+  image: kindest/node:v1.22.0@sha256:b8bda84bb3a190e6e028b1760d277454a72267a5454b57db34437c34a588d047
+  extraPortMappings:
+  - containerPort: 30778
+    hostPort: 30778
+    listenAddress: "0.0.0.0"
+    protocol: tcp
+- role: worker
+  image: kindest/node:v1.22.0@sha256:b8bda84bb3a190e6e028b1760d277454a72267a5454b57db34437c34a588d047
\ No newline at end of file
diff --git a/kubernetes/portainer/portainer-agent-ce211-k8s-nodeport.yaml b/kubernetes/portainer/portainer-agent-ce211-k8s-nodeport.yaml
new file mode 100644
index 0000000..73802a7
--- /dev/null
+++ b/kubernetes/portainer/portainer-agent-ce211-k8s-nodeport.yaml
@@ -0,0 +1,81 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: portainer
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: portainer-sa-clusteradmin
+  namespace: portainer
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: portainer-crb-clusteradmin
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+- kind: ServiceAccount
+  name: portainer-sa-clusteradmin
+  namespace: portainer
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: portainer-agent
+  namespace: portainer
+spec:
+  type: NodePort
+  selector:
+    app: portainer-agent
+  ports:
+    - name: http
+      protocol: TCP
+      port: 9001
+      targetPort: 9001
+      nodePort: 30778
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: portainer-agent-headless
+  namespace: portainer
+spec:
+  clusterIP: None
+  selector:
+    app: portainer-agent
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: portainer-agent
+  namespace: portainer
+spec:
+  selector:
+    matchLabels:
+      app: portainer-agent
+  template:
+    metadata:
+      labels:
+        app: portainer-agent
+    spec:
+      serviceAccountName: portainer-sa-clusteradmin
+      containers:
+      - name: portainer-agent
+        image: portainer/agent:2.11.1
+        imagePullPolicy: Always
+        env:
+        - name: LOG_LEVEL
+          value: DEBUG
+        - name: AGENT_CLUSTER_ADDR
+          value: "portainer-agent-headless"
+        - name: KUBERNETES_POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        ports:
+        - containerPort: 9001
+          protocol: TCP
diff --git a/kubernetes/portainer/portainer-agent-edge-k8s.yaml b/kubernetes/portainer/portainer-agent-edge-k8s.yaml
new file mode 100644
index 0000000..f1c9027
--- /dev/null
+++ b/kubernetes/portainer/portainer-agent-edge-k8s.yaml
@@ -0,0 +1,100 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: portainer
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: portainer-sa-clusteradmin
+  namespace: portainer
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: portainer-crb-clusteradmin
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+  - kind: ServiceAccount
+    name: portainer-sa-clusteradmin
+    namespace: portainer
+# Optional: can be added to expose the agent port 80 to associate an Edge key.
+# ---
+# apiVersion: v1
+# kind: Service
+# metadata:
+#   name: portainer-agent
+#   namespace: portainer
+# spec:
+#   type: LoadBalancer
+#   selector:
+#     app: portainer-agent
+#   ports:
+#     - name: http
+#       protocol: TCP
+#       port: 80
+#       targetPort: 80
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: portainer-agent
+  namespace: portainer
+spec:
+  clusterIP: None
+  selector:
+    app: portainer-agent
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: portainer-agent
+  namespace: portainer
+spec:
+  selector:
+    matchLabels:
+      app: portainer-agent
+  template:
+    metadata:
+      labels:
+        app: portainer-agent
+    spec:
+      serviceAccountName: portainer-sa-clusteradmin
+      containers:
+        - name: portainer-agent
+          image: portainer/agent:2.11.1
+          imagePullPolicy: Always
+          env:
+            - name: LOG_LEVEL
+              value: INFO
+            - name: KUBERNETES_POD_IP
+              valueFrom:
+                fieldRef:
+                  fieldPath: status.podIP
+            - name: EDGE
+              value: "1"
+            - name: AGENT_CLUSTER_ADDR
+              value: "portainer-agent"
+            - name: EDGE_ID
+              valueFrom:
+                configMapKeyRef:
+                  name: portainer-agent-edge
+                  key: edge.id
+            - name: EDGE_INSECURE_POLL
+              valueFrom:
+                configMapKeyRef:
+                  name: portainer-agent-edge
+                  key: edge.insecure_poll
+            - name: EDGE_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: portainer-agent-edge-key
+                  key: edge.key
+          ports:
+            - containerPort: 9001
+              protocol: TCP
+            - containerPort: 80
+              protocol: TCP

From c23f3a0cdc8f77cda007af99442a150b603de084 Mon Sep 17 00:00:00 2001
From: marcel-dempers <aimvec@gmail.com>
Date: Tue, 22 Mar 2022 16:44:09 +1100
Subject: [PATCH 15/16] updates

---
 .../part-5.database.redis/customers.json      |  37 ++++
 .../part-5.database.redis/requirements.txt    |   2 +-
 .../part-5.database.redis/src/app.py          | 200 +++++++++---------
 3 files changed, 139 insertions(+), 100 deletions(-)
 create mode 100644 python/introduction/part-5.database.redis/customers.json

diff --git a/python/introduction/part-5.database.redis/customers.json b/python/introduction/part-5.database.redis/customers.json
new file mode 100644
index 0000000..f79f787
--- /dev/null
+++ b/python/introduction/part-5.database.redis/customers.json
@@ -0,0 +1,37 @@
+{
+  "a": {
+    "customerID": "a",
+    "firstName": "James",
+    "lastName": "Baker"
+  },
+  "b": {
+    "customerID": "b",
+    "firstName": "Jonathan",
+    "lastName": "D"
+  },
+  "c": {
+    "customerID": "c",
+    "firstName": "Aleem",
+    "lastName": "Janmohamed"
+  },
+  "d": {
+    "customerID": "d",
+    "firstName": "Ivo",
+    "lastName": "Galic"
+  },
+  "e": {
+    "customerID": "e",
+    "firstName": "Joel",
+    "lastName": "Griffiths"
+  },
+  "f": {
+    "customerID": "f",
+    "firstName": "Michael",
+    "lastName": "Spinks"
+  },
+  "g": {
+    "customerID": "g",
+    "firstName": "Victor",
+    "lastName": "Savkov"
+  }
+}
\ No newline at end of file
diff --git a/python/introduction/part-5.database.redis/requirements.txt b/python/introduction/part-5.database.redis/requirements.txt
index 44daf50..a61fdfc 100644
--- a/python/introduction/part-5.database.redis/requirements.txt
+++ b/python/introduction/part-5.database.redis/requirements.txt
@@ -1,2 +1,2 @@
 Flask == 2.0.2
-Redis == 3.5.3
\ No newline at end of file
+redis == 3.5.3
\ No newline at end of file
diff --git a/python/introduction/part-5.database.redis/src/app.py b/python/introduction/part-5.database.redis/src/app.py
index c02a496..89d4f78 100644
--- a/python/introduction/part-5.database.redis/src/app.py
+++ b/python/introduction/part-5.database.redis/src/app.py
@@ -1,99 +1,101 @@
-import os.path
-import csv
-import os
-import json
-import time
-from flask import Flask
-from flask import request
-import redis
-from redis.sentinel import Sentinel
-
-redis_sentinels = os.environ.get('REDIS_SENTINELS')
-redis_master_name = os.environ.get('REDIS_MASTER_NAME')
-redis_password = os.environ.get('REDIS_PASSWORD')
-
-class Customer:
-  def __init__(self, c="",f="",l=""):
-    self.customerID = c
-    self.firstName  = f
-    self.lastName   = l
-  def fullName(self):
-    return self.firstName + " " + self.lastName
-
-def redis_command(command, *args):
-  max_retries = 3
-  count = 0
-  backoffSeconds = 5
-  while True:
-    try:
-      return command(*args)
-    except (redis.exceptions.ConnectionError, redis.exceptions.TimeoutError):
-      count += 1
-      if count > max_retries:
-        raise
-    print('Retrying in {} seconds'.format(backoffSeconds))
-    time.sleep(backoffSeconds)
-
-def getCustomers():
-  customers = {}
-  customerIDs = redis_command(redis_master.scan_iter, "*")
-  for customerID in customerIDs:
-    customer = getCustomer(customerID)
-    customers[customer["customerID"]] = customer
-  
-  return customers
-
-def getCustomer(customerID):
-  customer = redis_command(redis_master.get, customerID)
-  
-  if customer == None:
-    return {}
-  else:
-    c = str(customer, 'utf-8')
-    return json.loads(c)
-
-def updateCustomer(customer):
-  redis_command(redis_master.set, customer.customerID, json.dumps(customer.__dict__))
-
-def updateCustomers(customers):
-  for customer in customers:
-    updateCustomer(customer)
-  
-app = Flask(__name__)
-
-sentinels = []
-
-for s in redis_sentinels.split(","):
-  sentinels.append((s.split(":")[0], s.split(":")[1]))
-
-redis_sentinel = Sentinel(sentinels, socket_timeout=5)
-redis_master = redis_sentinel.master_for(redis_master_name,password = redis_password, socket_timeout=5)
-
-@app.route("/", methods=['GET'])
-def get_customers():
-  customers = getCustomers()
-  return json.dumps(customers)
-
-@app.route("/get/<string:customerID>", methods=['GET'])
-def get_customer(customerID):
-    customer = getCustomer(customerID)
-
-    if customer == {}:
-      return {}, 404
-    else:
-      return customer
-
-@app.route("/set", methods=['POST'])
-def add_customer():
-    jsonData = request.json
-
-    if "customerID" not in jsonData:
-      return "customerID required", 400
-    if "firstName" not in jsonData:
-      return "firstName required", 400
-    if "lastName" not in jsonData:
-      return "lastName required", 400
-    
-    customer = Customer( jsonData["customerID"], jsonData["firstName"], jsonData["lastName"])
-    updateCustomer(customer)
-    return "success", 200
\ No newline at end of file
+import os.path
+import csv
+import json
+import time
+from flask import Flask
+from flask import request
+import os
+from redis.sentinel import Sentinel
+
+dataPath = "./customers.json"
+
+redis_sentinels = os.environ.get('REDIS_SENTINELS')
+redis_master_name = os.environ.get('REDIS_MASTER_NAME')
+redis_password = os.environ.get('REDIS_PASSWORD')
+
+def redis_command(command, *args):
+  max_retries = 3
+  count = 0
+  backoffSeconds = 5
+  while True:
+    try:
+      return command(*args)
+    except (redis.exceptions.ConnectionError, redis.exceptions.TimeoutError):
+      count += 1
+      if count > max_retries:
+        raise
+      print('Retrying in {} seconds'.format(backoffSeconds))
+      time.sleep(backoffSeconds)
+
+class Customer:
+  def __init__(self, c="",f="",l=""):
+    self.customerID = c
+    self.firstName  = f
+    self.lastName   = l
+  def fullName(self):
+    return self.firstName + " " + self.lastName
+
+def getCustomers():
+  customers = {}
+  customerIDs = redis_command(redis_master.scan_iter, "*")
+  for customerID in customerIDs:
+    customer = getCustomer(customerID)
+    customers[customer["customerID"]] = customer
+  return customers
+
+def getCustomer(customerID):
+  customer = redis_command(redis_master.get, customerID)
+  
+  if customer == None:
+    return {}
+  else:
+    c = str(customer, 'utf-8')
+    return json.loads(c)
+
+def updateCustomer(customer):
+  redis_command(redis_master.set,customer.customerID, json.dumps(customer.__dict__) )
+
+def updateCustomers(customers):
+  for customer in customers:
+    updateCustomer(customer)
+
+app = Flask(__name__)
+
+sentinels = []
+
+for s in redis_sentinels.split(","):
+  sentinels.append((s.split(":")[0], s.split(":")[1]))
+
+redis_sentinel = Sentinel(sentinels, socket_timeout=5)
+redis_master = redis_sentinel.master_for(redis_master_name,password = redis_password, socket_timeout=5)
+
+
+@app.route("/", methods=['GET'])
+def get_customers():
+    customers = getCustomers()
+    return json.dumps(customers)
+
+@app.route("/get/<string:customerID>", methods=['GET'])
+def get_customer(customerID):
+    customer = getCustomer(customerID)
+
+    if customer == {}:
+      return {}, 404
+    else:
+      return customer
+
+@app.route("/set", methods=['POST'])
+def add_customer():
+  jsonData = request.json
+
+  if "customerID" not in jsonData:
+      return "customerID required", 400
+  if "firstName" not in jsonData:
+      return "firstName required", 400
+  if "lastName" not in jsonData:
+      return "lastName required", 400
+
+  customer = Customer( jsonData["customerID"], jsonData["firstName"], jsonData["lastName"])
+  updateCustomer(customer)
+  return "success", 200
+

From 8dd82ada77c6094d12a212c59c84360932301514 Mon Sep 17 00:00:00 2001
From: Marcel Dempers <34320559+marcel-dempers@users.noreply.github.com>
Date: Tue, 22 Mar 2022 16:48:36 +1100
Subject: [PATCH 16/16] Update README.md

---
 kubernetes/rancher/README.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kubernetes/rancher/README.md b/kubernetes/rancher/README.md
index ad60de8..c27ae9c 100644
--- a/kubernetes/rancher/README.md
+++ b/kubernetes/rancher/README.md
@@ -159,9 +159,9 @@ So let's setup a single server with [docker](https://rancher.com/docs/rancher/v2
 ## Persist data
 
 We will want to persist ranchers data across reboots. </br>
-Rancher stores its data under `/var/lib/rancher` 
+Rancher stores its data under `/var/lib/rancher` </br> 
 
-Let's create some space to save data:
+In this repo, let's create a space to persist data:
 
 ```
 cd kubernetes/rancher
@@ -227,4 +227,4 @@ example-service   LoadBalancer   10.43.235.240   <pending>     80:31310/TCP   13
 kubectl -n marcel port-forward svc/example-service 81:80
 ```
 
-We can access our example-app on port 81
\ No newline at end of file
+We can access our example-app on port 81