From 547ac52fa69d8082b9f0e249a624992a5c40a01f Mon Sep 17 00:00:00 2001 From: marcel-dempers Date: Thu, 27 Oct 2022 18:27:10 +1100 Subject: [PATCH] postgres replication --- .../postgres/3-replication/README.md | 140 +++++++++++++++++- 1 file changed, 132 insertions(+), 8 deletions(-) diff --git a/storage/databases/postgres/3-replication/README.md b/storage/databases/postgres/3-replication/README.md index 0a4befd..b650fe2 100644 --- a/storage/databases/postgres/3-replication/README.md +++ b/storage/databases/postgres/3-replication/README.md @@ -7,13 +7,9 @@ In this chapter we will setup a second PostgreSQL instance.
Then we will learn how to configure our first PostgreSQL instance to replicate its data to the new second instance.
This will essentially give us a primary and a secondary server for better availability in case we lose our primary server.
- -* Documentation on HA \ replication \ failover modes - - ## Get our Primary PostgreSQL up and running -Let's start by running our PostgreSQL in docker
+Let's start by running our primary PostgreSQL in docker
Few things to note here:
* We start our instance with a different name to identify it as the first instance with the `--name postgres-1` flag and `2` for the second instance @@ -40,11 +36,77 @@ docker run -it --rm --name postgres-1 ` -e PGDATA="/data" ` -v ${PWD}/postgres-1/pgdata:/data ` -v ${PWD}/postgres-1/config:/config ` +-v ${PWD}/postgres-1/archive:/mnt/server/archive ` -p 5000:5432 ` postgres:15.0 -c 'config_file=/config/postgresql.conf' ``` -Start instance 2: +## Create Replication User + +In order to take a backup we will use a new PostgreSQL user account which has the permissions to do replication.
+ +Let's create this user account by logging into `postgres-1`: + +``` +docker exec -it postgres-1 bash + +# create a new user +createuser -U postgresadmin -P -c 5 --replication replicationUser + +exit +``` + +# Enable Write-Ahead Log and Replication + +There is quite a lot to read about PostgreSQL when it comes to high availability.
+ +The first thing we want to take a look at is [WAL](https://www.postgresql.org/docs/current/wal-intro.html)
+Basically PostgreSQL has a mechanism of writing transaction logs to file and does not accept the transaction until its been written to the transaction log and flushed to disk.
+ +This ensures that if there is a crash in the system, that the database can be recovered from the transaction log.
+Hence it is "writing ahead".
+ +More documentation for configuration [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) and [max_wal_senders](https://www.postgresql.org/docs/current/runtime-config-replication.html) +``` +wal_level = hot_standby +max_wal_senders = 3 +``` + +# Enable Archive + +More documentation for configuration [archive_mode](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-MODE) + +``` +archive_mode = on +archive_command = 'test ! -f /mnt/server/archive/%f && cp %p /mnt/server/archive/%f' + +``` + +## Take a base backup + +To take a database backup, we'll be using the [pg_basebackup](https://www.postgresql.org/docs/current/app-pgbasebackup.html) utility.
+ +The utility is in the PostgreSQL docker image, so let's run it without running a database as all we need is the `pg_basebackup` utility.
+Note that we also mount our blank data directory as we will make a new backup in there: + +``` +cd storage/databases/postgres/3-replication + +docker run -it --rm ` +--net postgres ` +-v ${PWD}/postgres-2/pgdata:/data ` +--entrypoint /bin/bash postgres:15.0 +``` + +Take the backup by logging into `postgres-1` with our `replicationUser` and writing the backup to `/data`. + +``` +pg_basebackup -h postgres-1 -p 5432 -U replicationUser -D /data/ -Fp -Xs -R +``` + +Now we should see PostgreSQL data ready for our second instance in `${PWD}/postgres-2/pgdata` + +## Start standby instance ``` cd storage/databases/postgres/3-replication @@ -57,6 +119,68 @@ docker run -it --rm --name postgres-2 ` -e PGDATA="/data" ` -v ${PWD}/postgres-2/pgdata:/data ` -v ${PWD}/postgres-2/config:/config ` --p 5000:5432 ` +-v ${PWD}/postgres-2/archive:/mnt/server/archive ` +-p 5001:5432 ` postgres:15.0 -c 'config_file=/config/postgresql.conf' -``` \ No newline at end of file +``` + +## Test the replication + +Let's test our replication by creating a new table in `postgres-1`
+On our primary instance, lets do that: + +``` +# login to postgres +psql --username=postgresadmin postgresdb + +#create a table +CREATE TABLE customers (firstname text, customer_id serial, date_created timestamp); + +#show the table +\dt +``` + +Now lets log into our `postgres-2` instance and view the table: + +``` +docker exec -it postgres-2 bash + +# login to postgres +psql --username=postgresadmin postgresdb + +#show the tables +\dt +``` + +## Failover + +Now lets say `postgres-1` fails.
+PostgreSQL does not have built-in automated failver and recovery and requires tooling to perform this.
+ +When `postgres-1` fails, we would use a utility called [pg_ctl](https://www.postgresql.org/docs/current/app-pg-ctl.html) to promote our stand-by server to a new primary server.
+ +Then we have to build a new stand-by server just like we did in this guide.
+We would also need to configure replication on the new primary, the same way we did in this guide.
+ +Let's stop the primary server to simulate failure: + +``` +docker rm -f postgres-1 +``` + +Then log into `postgres-2` and promote it to primary: +``` +docker exec -it postgres-2 bash + +# confirm we cannot create a table as its a stand-by server +CREATE TABLE customers (firstname text, customer_id serial, date_created timestamp); + +# run pg_ctl as postgres user (cannot be run as root!) +runuser -u postgres -- pg_ctl promote + +# confirm we can create a table as its a primary server +CREATE TABLE customers (firstname text, customer_id serial, date_created timestamp); +``` + +That's it for chapter three!
+Now we understand how to [run PostgreSQL](../1-introduction/README.md), how to [configure PostgreSQL](../2-configuration/README.md) and how to setup replication for better availability.