Self Hosted Postgres in Kubernetes with PITR Recovery

2025-12-31T00:00:00+00:00
I would like to selfhost a few services, and they require persistence, in the form of a PostgreSQL database. Since my Kubernetes cluster runs on 4 Raspberry Pi 4, each with an SD card for disk, it's only a matter of time until one of them gets corrupted. To avoid the innevitable and predictable data loss, I'll need automated backups, idealy with Point-In-Time Recovery (PITR).
If I'm going to reinvent the wheel, why not go for the whole wheel?
Solution
CloudNativePG</a> seems to be most mature and feature complete out there. It provides an operator that does a lot of the heavy lifting. Backups are handled by the Barman Cloud</a> plugin. It handles aspects like WAL Log archiving, taking regular snapshots and uploading them a cloud object storage, like S3 1</a>. Seems to meet all the needs I need for my project.

Setup
Setting up a Cloud Native PG database
Using the instructions on their website</a>, we'll need to install the operator. We can do it with the following command:
$</span> kubectl apply</span> --server-side -f</span> https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.28/releases/cnpg-1.28.0.yaml</span></span></code></pre>
We can now create our database.
The Barman Cloud Plugin we'll be using later assumes that the DB is in the cnpg-system</code> namespace.
To simply this experiment, we'll work with that assumption, and we'll start by creating the namespace:</p>
$</span> kubectl create namespace cnpg-system</span></span></code></pre>
Then we can define our cluster configuration, with one user so we can connect to it later:</p>
apiVersion</span>:</span> v1</span></span>
type</span>:</span> kubernetes.io/basic-auth</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> postgres-secret</span></span>
  namespace</span>:</span> cnpg-system</span></span>
stringData</span>:</span></span>
  username</span>:</span> user1</span></span>
  password</span>:</span> supersecretpassword</span></span>
---</span></span>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> Cluster</span></span>
metadata</span>:</span></span>
  name</span>:</span> test</span></span>
  namespace</span>:</span> cpgn-system</span></span>
spec</span>:</span></span>
  instances</span>:</span> 1</span></span>
  storage</span>:</span></span>
    size</span>:</span> 1Gi</span></span></code></pre>
And when we apply this, we have a new cluster with 1 instance.
We can forward the port, and connect to it like we normally would to a Postgres instance running locally.</p>
$</span> kubectl port-forward</span> -n</span> cnpg-system service/test-restore-rw 5432:5432</span></span></code></pre>Backups#</span></a>
</h2>
Installing#</span></a>
</h3>
Now, we can configure backups for our new Postgres cluster.</p>
First, let's install the prerequisites that the Barman Cloud plugin requires.
We will need to install Certificate Manager2</a></sup>.</p>
$</span> kubectl apply</span> -f</span> https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml</span></span></code></pre>
Then, we can install the plugin:</p>
$</span> kubectl apply</span> -f</span> https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.9.0/manifest.yaml</span></span></code></pre>Configuration#</span></a>
</h3>
We can now define the backup configuration.
For this example, I'll use S3 as the storage target for the backups.</p>
First, we'll need to store a set AWS credentials with access to the right bucket3</a></sup>.
We'll use an Opaque Secret</a> to store the access and secret keys:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> s3-credentials</span></span>
  namespace</span>:</span> cnpg-system</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  access-key-id</span>:</span> <redacted></span></span>
  secret-key-id</span>:</span> <redacted></span></span></code></pre>
Next, we'll define the storage configuration to our S3 Bucket:</p>
apiVersion</span>:</span> barmancloud.cnpg.io/v1</span></span>
kind</span>:</span> ObjectStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> s3-store</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  configuration</span>:</span></span>
    destinationPath</span>:</span> s3://<bucket>/postgres/</span></span>
    s3Credentials</span>:</span></span>
      accessKeyId</span>:</span></span>
        name</span>:</span> s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKey</span>:</span></span>
        name</span>:</span> s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span>
    wal</span>:</span></span>
      compression</span>:</span> gzip</span></span></code></pre>
Finally, we'll need to tell our Postgres cluster to use this configuration, and enable WAL archiving.
We'll add the following to the Cluster spec</code> field:</p>
plugins</span>:</span></span>
  -</span> name</span>:</span> barman-cloud.cloudnative-pg.io</span></span>
    isWALArchiver</span>:</span> true</span></span>
    parameters</span>:</span></span>
      barmanObjectName</span>:</span> s3-store</span></span></code></pre>
The final piece of the puzzle is setup regular "base" backups.
These will backup the entire dataset, and give us a "base" for the WAL logs to be applied to in order to get our Point-In-Time Restore.</p>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> ScheduledBackup</span></span>
metadata</span>:</span></span>
  name</span>:</span> pg-backup</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  cluster</span>:</span></span>
    name</span>:</span> test</span></span>
  schedule</span>:</span> '0 0 * * * *'</span> # hourly</span></span>
  backupOwnerReference</span>:</span> self</span></span>
  method</span>:</span> plugin</span></span>
  pluginConfiguration</span>:</span></span>
    name</span>:</span> barman-cloud.cloudnative-pg.io</span></span></code></pre>Results#</span></a>
</h3>
Now that everything is configured, we can list our S3 bucket, and sure enough, we have backups.
The top level structure has our base and WAL logs:</p>
$ aws s3 ls s3://<redacted>/postgres/test/</span></span>
                           PRE base/</span></span>
                           PRE wals/</span></span></code></pre>
Digging in deeper, we have our base backups, nicely named by timestamp, one every hour, as we'd expect:</p>
$ aws s3 ls s3://<redacted>/postgres/test/base/</span></span>
                           PRE 20251227T030000/</span></span>
                           PRE 20251227T040000/</span></span>
                           PRE 20251227T050000/</span></span>
                           PRE 20251227T060000/</span></span>
                           PRE 20251227T070000/</span></span>
                           ...</span></span></code></pre>
And finally, we have our WAL logs archived:</p>
$ aws s3 ls s3://<redacted>/postgres/test/wals/0000000100000000/</span></span>
2025-12-26 21:00:02      16944 000000010000000000000013.gz</span></span>
2025-12-26 21:00:04        210 000000010000000000000014.00000028.backup.gz</span></span>
2025-12-26 21:00:03      16513 000000010000000000000014.gz</span></span>
2025-12-26 21:05:03      17139 000000010000000000000015.gz</span></span>
2025-12-26 21:30:03      17276 000000010000000000000016.gz</span></span>
2025-12-26 22:00:02      16416 000000010000000000000017.gz</span></span>
2025-12-26 22:00:05        209 000000010000000000000018.00000028.backup.gz</span></span>
2025-12-26 22:00:04      16510 000000010000000000000018.gz</span></span>
...</span></span></code></pre>
So we should have all the pieces needed to perform a PITR.
We'll tacke this in the next section.</p>
Restoring#</span></a>
</h2>
For testing the restore, I've created a simple table, with timestamps for easy reasoning:</p>
CREATE TABLE</span> test</span>(</span></span>
  id </span>int</span>,</span></span>
  created_at </span>timestamp</span>(</span>6</span>)</span></span>
);</span></span></code></pre>
And we inserted different values, and this is the end state of the table:</p>
test=> select * from test;</span></span>
 id |         created_at         </span></span>
----+----------------------------</span></span>
 16 | 2025-12-27 03:00:17.601392</span></span>
 17 | 2025-12-27 03:00:20.438822</span></span>
 18 | 2025-12-27 03:28:06.64914</span></span>
 19 | 2025-12-27 20:42:25.563207</span></span>
 20 | 2025-12-27 20:42:32.992066</span></span>
 21 | 2025-12-27 20:45:58.634257</span></span>
(6 rows)</span></span></code></pre>
We'll restore the table to 2025-12-27, at 20:43:00 UTC.
For this, we'll create a new cluster, and we'll point it at the backups we have.
We'll also need to give it the target time we want the cluster restored to.
We arrive at this configuration:</p>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> Cluster</span></span>
metadata</span>:</span></span>
  name</span>:</span> test-restore</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  instances</span>:</span> 1</span></span>
  imagePullPolicy</span>:</span> IfNotPresent</span></span>
  bootstrap</span>:</span></span>
    recovery</span>:</span></span>
      source</span>:</span> source</span></span>
  externalClusters</span>:</span></span>
  -</span> name</span>:</span> source</span></span>
    plugin</span>:</span></span>
      name</span>:</span> barman-cloud.cloudnative-pg.io</span></span>
      parameters</span>:</span></span>
        barmanObjectName</span>:</span> s3-store</span></span>
        serverName</span>:</span> test</span></span>
        targetTime</span>:</span> "2025-12-27T20:43:00Z"</span></span>
  storage</span>:</span></span>
    size</span>:</span> 1Gi</span></span></code></pre>
Once the restore is done (it was pretty much instant for this example), we con connect and check our test table:</p>
test=> select * from test;</span></span>
 id |         created_at         </span></span>
----+----------------------------</span></span>
 16 | 2025-12-27 03:00:17.601392</span></span>
 17 | 2025-12-27 03:00:20.438822</span></span>
 18 | 2025-12-27 03:28:06.64914</span></span>
 19 | 2025-12-27 20:42:25.563207</span></span>
 20 | 2025-12-27 20:42:32.992066</span></span>
(5 rows)</span></span></code></pre>
As we'd expect, we're missing row with id</code> 21, as it's after</em> the target restore time.</p>
Conclusions and Final Remarks#</span></a>
</h1>
All in all, this was an easy setup, and it works fine for at least the basic use case.
The true test is once this sees some "production" loads, and testing with an actual live data base.</p>
A note on S3 cots#</span></a>
</h2>
For this example, I used S3 as the backup target destination.
The amount of data stored is small, however, WAL archiving could end up writing a lot of objects.
This could incur significant S3 API charges, so it's something I'm keeping an eye on.
With hourly base backups, I didn't see any cost increases for my AWS account.
But inserting 21 records, and deleting a few is not exactly a representative use case, but at least the "baseline" cost is not absurd.</p>
Retention policies#</span></a>
</h2>
Barman has the option of specifying retention policies</a>.
However, for this experiment, I've gone with specifying a lifecycle policy on the S3 bucket.
Everything under postgres/</code> will be deleted after 7 days.
This will give me a one week recovery window.</p>
Using the Barman retention policy will cause the plugin to list S3, and then delete the objects.
This also will incur some S3 API charges, and I think that using the S3 lifecycle rule is probably good enough.</p>




The plugin also supports Google Cloud Storage, or Azure Blobs or some other services that implementation a compatible API</a>. ↩</a></p>
</li>

You can skip this step if your cluster already has it installed on the cluster. My test cluster did not. ↩</a></p>
</li>

For reference, I granted the user full access to the S3 bucket where the backups are stored. ↩</a></p>
</li>
</ol>
</section>
Caius' Lab - PostgreSQL

Self Hosted Postgres in Kubernetes with PITR Recovery
