Introduction to Go: Storing data in Redis Database
Up until now, we've learned the fundamentals of Go and built a small web microservice that handles our video data.
Our service has a / GET endpoint for returning all videos, as well as a simple /update endpoint for updating our list of videos.
We've learnt how to read and write from files and learn how to work with json data.
This is important for learning Go, however there are a few challenges for using a file as storage.
- It can be problematic if we have more than one instance of our service writing to the same file
- It's important to keep state out of our application, so if our application crashes, we don't lose our data
In part 1, we covered the fundamentals of writing basic Go
In part 2, we've learnt how to use basic data structures like json so we can send\receive data.
In part 3, we've learnt how to write a HTTP service to expose our videos data
Start up a Redis Cluster
Follow my Redis clustering Tutorial
Code is over here
Go Dev Environment
The same as Part 1+2+3, we start with a dockerfile where we declare our version of go.
The dockerfile:
FROM golang:1.15-alpine as dev
WORKDIR /work
Let's build and start our container:
cd golang\introduction\part-5.database.redis
docker build --target dev . -t go
docker run -it -p 80:80 -v ${PWD}:/work go sh
go version
Our application
Create a new directory that holds defines our repository and holds our module
mkdir videos
- Define a module path
# change directory to your application source code
cd videos
# create a go module file
go mod init videos
Create our base code
We start out all our applications with a main.go defining our package, declaring our import dependencies
and our entrypoint main() function
package main
import (
"net/http"
"encoding/json"
"io/ioutil"
"fmt"
)
func main() {
http.HandleFunc("/", HandleGetVideos)
http.HandleFunc("/update", HandleUpdateVideos)
http.ListenAndServe(":8080", nil)
}
Now before we write these handler functions, we need our videos application
Create our Videos app
Firstly, we create a separate code file videos.go that deals with our YouTube videos
The videos.go file defines what a video struct looks like, a getVideos() function to retrieve
videos list as a slice and a saveVideos() function to save videos to a file locally.
We start with our dependencies.
We import 2 packages, 1 for reading and writing files, and another for dealing with json
package main
import (
"io/ioutil"
"encoding/json"
)
Then we define what a video struct looks like:
type video struct {
Id string
Title string
Description string
Imageurl string
Url string
}
We have a function for retrieving video objects as a list of type slice :
func getVideos()(videos []video){
fileBytes, err := ioutil.ReadFile("./videos.json")
if err != nil {
panic(err)
}
err = json.Unmarshal(fileBytes, &videos)
if err != nil {
panic(err)
}
return videos
}
We also need to copy our videos.json file which contains our video data.
And finally, we have a function that accepts a list of type slice and stores the videos to a local file
func saveVideos(videos []video)(){
videoBytes, err := json.Marshal(videos)
if err != nil {
panic(err)
}
err = ioutil.WriteFile("./videos.json", videoBytes, 0644)
if err != nil {
panic(err)
}
}
HTTP Handlers
Now we have to define our handler functions HandleGetVideos and HandleUpdateVideos,
func HandleGetVideos(w http.ResponseWriter, r *http.Request){
videos := getVideos()
videoBytes, err := json.Marshal(videos)
if err != nil {
panic(err)
}
w.Write(videoBytes)
}
func HandleUpdateVideos(w http.ResponseWriter, r *http.Request){
if r.Method == "POST" {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
var videos []video
err = json.Unmarshal(body, &videos)
if err != nil {
w.WriteHeader(400)
fmt.Fprintf(w, "Bad request")
}
saveVideos(videos)
} else {
w.WriteHeader(405)
fmt.Fprintf(w, "Method not Supported!")
}
}
Now so far, we have a web HTTP application that can list and update our youtube videos.
Let's build and run it
go build
./videos
If we head over to http://localhost in the browser we can see our application returns our 2 videos.
.
We can use tools like PostMan to generate a POST request to update our videos too.
Redis Go Package
Now instead of reading and writing our records to a json file, we are going to read and write records to Redis.
https://github.com/go-redis/redis/
go get github.com/go-redis/redis/v8
And to use the library, we have to import it
import (
"github.com/go-redis/redis/v8"
)
Now to connect to any database, you're going to need a bit of information:
- Database Address
- Database Port
- Database Username\Password
We can define these as environment variables and read those in our code
import (
"os"
)
//main.go (global variables)
var redis_sentinels = os.Getenv("REDIS_SENTINELS")
var redis_master = os.Getenv("REDIS_MASTER_NAME")
var redis_password = os.Getenv("REDIS_PASSWORD")
We define an empty context and Redis client Context can be used to control timeouts and deadlines for our application. We can set up an empty context for now.
import (
"context"
)
var ctx = context.Background()
var redisClient *redis.Client
Redis Sentinel Client
https://redis.uptrace.dev/guide/sentinel.html#redis-server-client
import (
"strings"
)
sentinelAddrs := strings.Split(redis_sentinels, ",")
rdb := redis.NewFailoverClient(&redis.FailoverOptions{
MasterName: redis_master,
SentinelAddrs: sentinelAddrs,
Password: redis_password,
DB: 0,
})
redisClient = rdb
rdb.Ping(ctx)
We can also add the Ping() to our handler functions to ensure it can connect using the global redis client variable
Networking
Now we need to remember our go container may not be able to talk to the redis containers because they are running on different networks.
If you took note, we started our Redis containers on a redis network by passing --net redis as a flag to our docker run commands.
Let's restart our Go container on the same network
We also need to set our ENV variables to point our container to the redis sentinels.
If we look at our sentinel configuration, our master alias is set to mymaster
docker run -it -p 80:80 `
--net redis `
-e REDIS_SENTINELS="sentinel-1:5000,sentinel-2:5000,sentinel-3:5000" `
-e REDIS_MASTER_NAME="mymaster" `
-e REDIS_PASSWORD="a-very-complex-password-here" `
-v ${PWD}:/work go sh
We can now observe our container is connected to Redis.
Our application: http://localhost
Store our Data
So now we can store our video data in Redis instead of a local json file.
We'll write the json document of a video (struct) to Redis.
Let's create a saveVideo() function that stores a single record.
func saveVideo(video video)(){
videoBytes, err := json.Marshal(video)
if err != nil {
panic(err)
}
err = redisClient.Set(ctx, video.Id, videoBytes, 0).Err()
if err != nil {
panic(err)
}
}
Now since we have one endpoint that saves all videos, we need to update it to save each video it has.
func saveVideos(videos []video)(){
for _, video := range videos {
saveVideo(video)
}
}
To get the videos, let's create a function to retrieve a single record:
func getVideo(id string)(video video) {
value, err := redisClient.Get(ctx, id).Result()
if err != nil {
panic(err)
}
if err != redis.Nil {
err = json.Unmarshal([]byte(value), &video)
}
return video
}
And finally we need to update our GetVideos() function to loop all the video keys and return all videos
func getVideos()(videos []video){
keys, err := redisClient.Keys(ctx,"*").Result()
if err != nil {
panic(err)
}
for _, key := range keys {
video := getVideo(key)
videos = append(videos, video)
}
return videos
}
And we need all our imports
import (
"encoding/json"
"github.com/go-redis/redis/v8"
)
Now if we rebuild our all and access it, we get null as there are no videos in Redis. Let's add two!
Let's POST the following using PostMan to url http://localhost/update
[
{
"id" : "QThadS3Soig",
"title" : "Kubernetes on Amazon",
"imageurl" : "https://i.ytimg.com/vi/QThadS3Soig/sddefault.jpg",
"url" : "https://youtu.be/QThadS3Soig",
"description" : "TEST"
},
{
"id" : "eyvLwK5C2dw",
"title" : "Kubernetes on Azure",
"imageurl" : "https://i.ytimg.com/vi/eyvLwK5C2dw/mqdefault.jpg?sqp=CISC_PoF&rs=AOn4CLDo7kizrJozB0pxBhxL9JbyiW_EPw",
"url" : "https://youtu.be/eyvLwK5C2dw",
"description" : "TEST"
}
]
If we refresh our page, we can now see two records!
Improvements
Now that you have the fundamental knowledge of HTTP and Redis,
you can update the code to retrieve 1 video by ID, or delete a video by ID.
You can add search functionality and more!
Let's update our / GET handler to be able to return a single video
func HandleGetVideos(w http.ResponseWriter, r *http.Request){
id, ok := r.URL.Query()["id"]
if ok {
videoID := id[0]
video := getVideo(videoID)
if video.Id == "" { //video not found, or empty ID
w.WriteHeader(http.StatusNotFound)
w.Write([]byte("{}"))
return
}
videoBytes, err := json.Marshal(video)
if err != nil {
panic(err)
}
w.Write(videoBytes)
return
}
videos := getVideos()
videoBytes, err := json.Marshal(videos)
if err != nil {
panic(err)
}
w.Write(videoBytes)
}
We can also update our /update POST endpoint to be able to update a single video
func HandleUpdateVideos(w http.ResponseWriter, r *http.Request){
if r.Method == "POST" {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
_, ok := r.URL.Query()["id"]
if ok {
var video video
err = json.Unmarshal(body, &video)
if err != nil {
w.WriteHeader(400)
fmt.Fprintf(w, "Bad request")
}
saveVideo(video)
return
}
var videos []video
err = json.Unmarshal(body, &videos)
if err != nil {
w.WriteHeader(400)
fmt.Fprintf(w, "Bad request")
}
saveVideos(videos)
return
} else {
w.WriteHeader(405)
fmt.Fprintf(w, "Method not Supported!")
}
}
Build our Docker container
Full dockerfile :
FROM golang:1.15-alpine as dev
WORKDIR /work
FROM golang:1.15-alpine as build
WORKDIR /videos
COPY ./videos/* /videos/
RUN go build -o videos
FROM alpine as runtime
COPY --from=build /videos/videos /
CMD ./videos
Build :
cd golang\introduction\part-5.database.redis
docker build . -t videos
Run :
docker run -it -p 80:80 `
--net redis `
-e REDIS_SENTINELS="sentinel-1:5000,sentinel-2:5000,sentinel-3:5000" `
-e REDIS_MASTER_NAME="mymaster" `
-e REDIS_PASSWORD="a-very-complex-password-here" `
videos