Caius' Lab - Self-Hosting

Backing up Persistent Volume Claims with k8up

2026-05-03T00:00:00+00:00

Previously, I wrote about backing up Postgres to S3 using CNPG and Barman Cloud</a>. This is enough if all the data you want to persist is in Postgres. However, if you have any other volumes, this will not be enough. In this post, I'll go over setting up backups with k8up</a>, and what the restore process looks like.</p>

Why k8up?
k8up (pronounced "ketchup") is a Kubernetes Operator distributed via a Helm chart. By default, it backs up all Persistent Volume Claims (PVCs) marked as `ReadWriteMany</code>, ReadWriteOnce</code>, or with a certain label. It usesrestic</a>, so it can write to object storage. For this blog post, I'm using Amazon S3 as the destination, but self-hosted options like Garage</a> should also work.</p>`
`As to why I chose it, it's more mature than VolSync</a>, and I'm interested only in backing up PVCs. The cluster definition is managed with Helm</a>, so Velero</a> seems a bit too heavy for my limited use case.</p>`
`Finally, there's Longhorn</a>, which can also do backups. My cluster will be quite resource-constrained, so I'm not sure the extra overhead of Longhorn will be worth it in the long run. I'm OK with an "outage" if I need to restore a PVC from S3, and the extra redundancy doesn't seem worth the overhead.</p>`

Test service#</span></a>
</h1>
To evaluate this, I've created a very simple service that writes the current time to a PVC once a second.
It will give me an idea of how backups perform, and how recovery works.
Here is the Helm template for this service:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> PersistentVolumeClaim</span></span>
metadata</span>:</span></span>
  name</span>:</span> time-writer-pvc</span></span>
  namespace</span>:</span> backup-test</span></span>
  labels</span>:</span></span>
    must-backup</span>:</span> "true"</span></span>
spec</span>:</span></span>
  accessModes</span>:</span></span>
    -</span> ReadWriteOnce</span></span>
  resources</span>:</span></span>
    requests</span>:</span></span>
      storage</span>:</span> 100Mi</span></span>
---</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Pod</span></span>
metadata</span>:</span></span>
  name</span>:</span> time-writer</span></span>
  namespace</span>:</span> backup-test</span></span>
spec</span>:</span></span>
  containers</span>:</span></span>
    -</span> name</span>:</span> time-writer</span></span>
      image</span>:</span> busybox:1.37</span></span>
      imagePullPolicy</span>:</span> IfNotPresent</span></span>
      command</span>: [</span>"/bin/sh"</span>,</span> "-c"</span>]</span></span>
      args</span>:</span></span>
        -</span> while true; do date >> /data/time.log; sleep 1; done</span></span>
      volumeMounts</span>:</span></span>
        -</span> name</span>:</span> data</span></span>
          mountPath</span>:</span> /data</span></span>
  volumes</span>:</span></span>
    -</span> name</span>:</span> data</span></span>
      persistentVolumeClaim</span>:</span></span>
        claimName</span>:</span> time-writer-pvc</span></span></code></pre>Backup setup#</span></a>
</h1>
Once we have this setup, we can start with the backups.
First, we need to create 2 secrets: one for the restic encryption key, and one for the AWS credentials.</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
  namespace</span>:</span> backup-test</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  secretKey</span>: {{</span> .Values.secrets.resticSecretKey</span> }}</span></span>
---</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
  namespace</span>:</span> backup-test</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  access-key-id</span>: {{</span> .Values.secrets.awsAccessKey</span> }}</span></span>
  secret-key-id</span>: {{</span> .Values.secrets.awsSecretKey</span> }}</span></span></code></pre>
Now, we can define the backup we want to take.
For s3</code>, I'll use the regional S3 endpoint, s3.us-west-2.amazonaws.com</code>.
For the bucket name, we can also specify a prefix.
In this example, the restic repo lives under the test</code> prefix.</p>
apiVersion</span>:</span> k8up.io/v1</span></span>
kind</span>:</span> Backup</span></span>
metadata</span>:</span></span>
  name</span>:</span> backup-test</span></span>
  namespace</span>:</span> backup-test</span></span>
spec</span>:</span></span>
  failedJobsHistoryLimit</span>:</span> 2</span></span>
  successfulJobsHistoryLimit</span>:</span> 2</span></span>
  backend</span>:</span></span>
    repoPasswordSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
        key</span>:</span> secretKey</span></span>
    s3</span>:</span></span>
      endpoint</span>:</span> https://s3.us-west-2.amazonaws.com</span></span>
      bucket</span>:</span> bucket-name/test</span></span>
      accessKeyIDSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKeySecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span></code></pre>
Once this is done, we can run helm upgrade</code> and we get our backup.
Checking the jobs, they have completed:</p>
$ kubectl get jobs -n backup-test</span></span>
NAME                                  STATUS     COMPLETIONS   DURATION   AGE</span></span>
backup-schedule-test-backup-c9wz6-0   Complete   1/1           9s         12m</span></span></code></pre>
And looking at the logs, we get confirmation that it backed up our only file:</p>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup    starting backup</span></span>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup    starting backup for folder    {"foldername": "time-writer-pvc"}</span></span>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup.command    restic command    {"path": "/usr/local/bin/restic", "args": ["backup", "--host", "backup-test", "--json</span></span>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}</span></span>
2026-05-03T19:45:37Z    INFO    k8up.restic.restic.backup.progress    backup finished    {"new files": 1, "changed files": 0, "errors": 0} </span></span></code></pre>
For the final confirmation, we can see the restic repo structure in S3:</p>
$ aws s3 ls s3://bucket-names/test/</span></span>
                           PRE data/</span></span>
                           PRE index/</span></span>
                           PRE keys/</span></span>
                           PRE snapshots/</span></span>
2026-05-03 12:45:35        155 config</span></span></code></pre>Recovery#</span></a>
</h1>
A backup is only useful if we can recover our data from it.
So let's see how this would work.
First, let's create a PVC that's the destination for our data:</p>
kind</span>:</span> PersistentVolumeClaim</span></span>
apiVersion</span>:</span> v1</span></span>
metadata</span>:</span></span>
  name</span>:</span> restore-test</span></span>
  namespace</span>:</span> backup-test</span></span>
  annotations</span>:</span></span>
    # Setting this to false to exclude from future backups</span></span>
    k8up.io/backup</span>:</span> "false"</span></span>
spec</span>:</span></span>
  accessModes</span>:</span></span>
    -</span> ReadWriteOnce</span></span>
  resources</span>:</span></span>
    requests</span>:</span></span>
      storage</span>:</span> 100Mi</span></span></code></pre>
Then, we can define our restore:</p>
apiVersion</span>:</span> k8up.io/v1</span></span>
kind</span>:</span> Restore</span></span>
metadata</span>:</span></span>
  name</span>:</span> restore-test</span></span>
spec</span>:</span></span>
  restoreMethod</span>:</span></span>
    folder</span>:</span></span>
      claimName</span>:</span> restore-test</span></span>
  backend</span>:</span></span>
    repoPasswordSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
        key</span>:</span> secretKey</span></span>
    s3</span>:</span></span>
      endpoint</span>:</span> https://s3.us-west-2.amazonaws.com</span></span>
      bucket</span>:</span> bucket-name/test</span></span>
      accessKeyIDSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKeySecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span></code></pre>
This will create a new job, and checking the logs, we can see that it succeeded:</p>
$ kubectl logs -n backup-test jobs/restore-restore-test -f</span></span>
<...truncated...></span></span>
2026-05-03T20:04:04Z    INFO    k8up.restic.restic.restore.command      restic command  {"path": "/usr/local/bin/restic", "args": ["restore", "be8f37f009f0c910ad5ce8aa067d3dbd4050e7de9c41fde14394555a73adee06:/data/time-writer-pvc", "--target", "/restore"]}</span></span>
2026-05-03T20:04:04Z    INFO    k8up.restic.restic.restore.command      Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}</span></span>
2026-05-03T20:04:05Z    INFO    k8up.restic.restic.restore.restic.stdout        restoring snapshot be8f37f0 of [/data/time-writer-pvc] at 2026-05-03 19:45:35.978401924 +0000 UTC by @backup-test to /restore</span></span>
2026-05-03T20:04:05Z    INFO    k8up.restic.restic.restore.restic.stdout        Summary: Restored 1 files/dirs (3.144 KiB) in 0:00</span></span></code></pre>
Finally, we can check the PVC directly, and confirm we have data until 19:45:36.
The backup was started at 19:45:35 and finished by 19:45:37.
This gives us a very good RPO of a few seconds, which is great for my use case!</p>
$ kubectl exec -n backup-test -it pvc-inspect -- tail /data/time.log</span></span>
Sun May  3 19:45:27 UTC 2026</span></span>
Sun May  3 19:45:28 UTC 2026</span></span>
Sun May  3 19:45:29 UTC 2026</span></span>
Sun May  3 19:45:30 UTC 2026</span></span>
Sun May  3 19:45:31 UTC 2026</span></span>
Sun May  3 19:45:32 UTC 2026</span></span>
Sun May  3 19:45:33 UTC 2026</span></span>
Sun May  3 19:45:34 UTC 2026</span></span>
Sun May  3 19:45:35 UTC 2026</span></span>
Sun May  3 19:45:36 UTC 2026</span></span></code></pre>Scheduling#</span></a>
</h1>
The above example only covers a "one-time" backup.
For this to be useful, I need a regular schedule.
This is where the Schedule</code> comes in.</p>
I'll start with the example provided in the documentation</a>.
We'll back up every 5 minutes, and every hour we'll do checking and pruning.
Data retention can be tuned depending on the use case.</p>
apiVersion</span>:</span> k8up.io/v1</span></span>
kind</span>:</span> Schedule</span></span>
metadata</span>:</span></span>
  name</span>:</span> schedule-test</span></span>
spec</span>:</span></span>
  backend</span>:</span></span>
    repoPasswordSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
        key</span>:</span> secretKey</span></span>
    s3</span>:</span></span>
      endpoint</span>:</span> https://s3.us-west-2.amazonaws.com</span></span>
      bucket</span>:</span> bucket-name/test</span></span>
      accessKeyIDSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKeySecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span>
  backup</span>:</span></span>
    schedule</span>:</span> '*/5 * * * *'</span></span>
    failedJobsHistoryLimit</span>:</span> 2</span></span>
    successfulJobsHistoryLimit</span>:</span> 2</span></span>
  check</span>:</span></span>
    schedule</span>:</span> '0 * * * *'</span></span>
  prune</span>:</span></span>
    schedule</span>:</span> '30 * * * *'</span></span>
    retention</span>:</span></span>
      keepLast</span>:</span> 5</span></span>
      keepDaily</span>:</span> 7</span></span></code></pre>
This will result in one job every 5 minutes:</p>
$ kubectl get jobs -n backup-test</span></span>
NAME                                  STATUS     COMPLETIONS   DURATION   AGE</span></span>
backup-schedule-test-backup-7l7sj-0   Complete   1/1           10s        3m17s</span></span>
backup-schedule-test-backup-l7jkq-0   Complete   1/1           10s        8m17s</span></span></code></pre>
And for each job, the backup was successful:</p>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup       starting backup</span></span>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup       starting backup for folder      {"foldername": "time-writer-pvc"}</span></span>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup.command       restic command  {"path": "/usr/local/bin/restic", "args": ["backup", "--json", "--host", "backup-test", "/data/time-writer-pvc"]}</span></span>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup.command       Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}</span></span>
2026-05-03T21:40:31Z    INFO    k8up.restic.restic.backup.progress      backup finished {"new files": 0, "changed files": 1, "errors": 0}</span></span>
2026-05-03T21:40:31Z    INFO    k8up.restic.restic.backup.progress      stats   {"time": 1.727354075, "bytes added": 51912, "bytes processed": 50837}</span></span></code></pre>Conclusion#</span></a>
</h1>
Overall, k8up seems to meet my use case.
The setup is straightforward and easy to reason about.
The restore process is also straightforward.</p>
While the example here is very limited, it gives me confidence that the basics work.
The real test will come once it's operating on real data, but regular restores and testing should give me the confidence I need there.</p>



Setting up a Homelab: Step 1
2026-04-19T00:00:00+00:00
I am building a Kubernetes cluster out of second-hand Lenovo ThinkCenter machines.
As part of this project, I want to streamline the setup as much as possible, and I'll try to automate most steps.
The goal is to have a reproducible setup that will simplify modifications, adding more hardware, etc.</p>
In this post, I will go over what I call the "bootstrap step."
How do you start with a fresh install of an OS, as easily as possible without having to "manually" install it from a USB stick on each machine?
The solution is to create a pre-configured NVMe drive, from a cloud OS image, and set up cloud init</a> to do the configuration that we need.</p>
Theory of Operation#</span></a>
</h1>

I am running this process on macOS.
This comes with a few extra complications, compared to running this from a Linux machine1</a></sup>.
Should this be run from Linux, as it makes life easier?
Probably, but where's the fun in that.</p>
</blockquote>
Before I dive into the details, let's look at what needs to be done:</p>
First, we need to write the image on the NVMe drive.
That's easy enough.
However, I want the machines to be headless, so I don't have any inputs or output connected.
Ideally when they boot, they are accessible from the network, and ready to set up.
This is where cloud-init comes in.</p>
Cloud-init runs on a cloud VM during first boot, to handle machine-specific configuration.
That's our exact use case!
It reads its configuration from a fairly straightforward yaml configuration format</a>.
While we aren't running in "the cloud," the abstraction makes sense, and cloud-init is flexible enough to work on a bare-metal machine.</p>
The goal of the setup is to do 2 things:</p>

Set up SSH key for remote access</li>
Set up a hostname, so we can easily find it on the network using DNS.</li>
</ol>
Step 1: Writing the image#</span></a>
</h1>
First, we need to choose an OS that we want to run.
For my setup I chose to run Ubuntu 25.10, the latest at the time of writing.
You can find releases on this page.</a></p>
Once we have that, we can't write that directly to the NVMe drive.
They are in the QEMU Qcow2 image format (they are designed for cloud use).
We'll need to convert them to raw.
We'll do this with qemu2</a></sup>.</p>
qemu-img</span> convert</span> -f</span> qcow2</span> -O</span> raw ubuntu-25.10-server-cloudimg-amd64.img ubuntu-25.10-server-cloudimg-amd64.raw</span></span></code></pre>
Now we have an image we can write to the NVMe drive.
We'll use Ansible</a> to automate the whole process, so it's easily repeatable.</p>

Doing any of these following steps on the wrong disk WILL wipe out that disk, and you will lose any and all stored data!
Double check all paths, and your assumptions!
Proceed at your own risk!</p>
</blockquote>
We'll assume that the following variables are defined:</p>

target_disk: /dev/disk2</code> - the buffered block device that our NVMe is mounted to</li>
target_raw_disk: /dev/rdisk2</code> - the unbuffered block device for faster writes using dd</code></li>
</ul>
First, we need to make sure that the drive is unmounted and ready to be written to:</p>
-</span> name</span>:</span> Unmount disk</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> diskutil unmountDisk {{ target_disk }}</span></span></code></pre>
Then we can copy the image to the NVMe drive:</p>
-</span> name</span>:</span> Write image to disk</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> dd if={{ image_src }} of={{ target_raw_disk }} bs=4m</span></span></code></pre>
We now have an NVMe drive that will boot on our machine.
However, it doesn't have the configuration needed for headless remote access.
So we'll have to tackle that next, and this is where cloud-init comes into play.</p>
Step 2: Cloud-init configuration#</span></a>
</h1>
Cloud-init has different ways in which it can get its initial scripts.
On a cloud machine, it's usually through some kind of metadata service.
For example, on AWS it's from http://169.254.169.254/latest/</code>.
However, this isn't a cloud machine, so we'll have to do something different.</p>
Luckily, Cloud-init supports the NoCloud</code> datasource</a>.
This allows us to get the user data from a specific partition called cidata</code> or CIDATA</code>.
This is the option we'll go with next3</a></sup>.</p>
Creating the partition#</span></a>
</h2>
Since I'm managing a Linux file system, I decided to use gptfdisk</code>4</a></sup> for all the partition management.
First, we need to make sure that we see the whole disk, otherwise we'll have no new space to create the new partition.</p>
-</span> name</span>:</span> Make GPT see the full disk</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> sgdisk --move-second-header {{ target_disk }}</span></span></code></pre>
The default root (/</code>) partition will expand to fill all the available space, so we don't want to create our new partition immediately after.
This would prevent the partition from growing.
So we'll have to create the partition at the end.
For this example, I chose a size of 50MB.
This is probably overkill for the 2 tiny files we're writing, but it gives room to grow if we need to.</p>
-</span> name</span>:</span> "Add CIDATA partition at the end"</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> sgdisk --new=0:-51M:+50M --typecode=0:0700 --change-name=0:CIDATA {{ target_disk }}</span></span></code></pre>
Next, we'll format it as FAT32, and mount it:</p>
-</span> name</span>:</span> "Format CIDATA to FAT32"</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> diskutil eraseVolume MS-DOS CIDATA {{ target_disk }}s2</span></span>
</span>
-</span> name</span>:</span> Mount CIDATA partition</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> diskutil mount {{ target_disk }}s2</span></span></code></pre>
At the end, if you run diskutil</code>, you'll get something like this, if all went well.</p>
/dev/disk2 (external, physical):</span></span>
   #:                       TYPE NAME                    SIZE       IDENTIFIER</span></span>
   0:      GUID_partition_scheme                        *256.1 GB   disk2</span></span>
   1: BC13C2FF-59E6-4262-A352-B275FD6F7172               1.1 GB     disk2s13</span></span>
   2:        Bios Boot Partition                         4.2 MB     disk2s14</span></span>
   3:                        EFI UEFI                    111.1 MB   disk2s15</span></span>
   4:           Linux Filesystem                         254.8 GB   disk2s1</span></span>
   5:       Microsoft Basic Data CIDATA                  52.8 MB    disk2s2</span></span></code></pre>Writing the cloud-init config#</span></a>
</h2>
Now, we're ready to write our cloud-init files.
We'll need two of them:</p>

meta-data</code> for setting the hostname</li>
user-data</code> for setting up SSH access.</li>
</ul>
-</span> name</span>:</span> Write cloud-init meta-data</span></span>
  ansible.builtin.copy</span>:</span></span>
    content</span>:</span> |</span></span>
      local-hostname: {{ target_hostname }}</span></span>
    dest</span>:</span> /Volumes/CIDATA/meta-data</span></span>
</span>
-</span> name</span>:</span> Write cloud-init user-data</span></span>
  ansible.builtin.copy</span>:</span></span>
    content</span>:</span> |</span></span>
      #cloud-config</span></span>
      users:</span></span>
      - name: ubuntu</span></span>
        ssh_authorized_keys:</span></span>
        {% for key in public_keys -%}</span></span>
          - {{ key }}</span></span>
        {% endfor -%}</span></span>
          sudo: ALL=(ALL) NOPASSWD:ALL</span></span>
          shell: /bin/bash</span></span>
    dest</span>:</span> /Volumes/CIDATA/user-data</span></span>
  become</span>:</span> true</span></span></code></pre>
We'll unmount, eject, and we're done.</p>
-</span> name</span>:</span> Unmount disk</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> diskutil unmountDisk {{ target_disk }}s2</span></span>
</span>
-</span> name</span>:</span> Eject disk</span></span>
  ansible.builtin.command</span>:</span></span>
    cmd</span>:</span> diskutil eject {{ target_disk }}</span></span></code></pre>
To create our new configured boot disk, all we need to do is run:</p>
ansible-playbook</span> playbooks/ubuntu-start.yml</span> --ask-become-pass -e</span> "target_hostname=machine1"</span></span></code></pre>
This makes setting up multiple machines for the same cluster really easy, as we only need to change the target_hostname</code> variable when running the playbook.</p>
Conclusion#</span></a>
</h1>
In this post, I've detailed how to create a bootable drive for a new PC, that will boot up, and configure itself with the correct hostname and SSH keys for remote access.
Using cloud-init allows us to specify a custom bootstrap step for each machine, so we can use the same "base" image in all cases.
This has greatly simplified adding new machines to the cluster, or reprovisioning old ones if we replace the SSD.</p>
To state the obvious, this would be a lot simpler if done using Linux, as we could mount the root file system, and make our changes there.
In that case, using cloud-init is probably superfluous, and this post would be 1/4 of the size it is now.
However, using cloud-init gives us more flexibility on what's run.
More importantly, it presented a great opportunity for experimenting with a novel (to me) way of bootstrapping Linux on a bare-metal headless machine.</p>
Appendix#</span></a>
</h1>
The full playbook:</p>
-</span> name</span>:</span> Flash Ubuntu disk image</span></span>
  hosts</span>:</span> localhost</span></span>
  become</span>:</span> true</span></span>
  vars</span>:</span></span>
    image_src</span>:</span> /tmp/ubuntu-25.10-server-cloudimg-amd64.raw</span></span>
    target_raw_disk</span>:</span> /dev/rdisk2</span></span>
    target_disk</span>:</span> /dev/disk2</span></span>
    target_hostname</span>:</span> ubuntu</span></span>
    public_keys</span>:</span></span>
      -</span> ssh-rsa AAAAB3N....</span></span>
  </span></span>
  tasks</span>:</span></span>
    -</span> name</span>:</span> Check running on macOS</span></span>
      ansible.builtin.assert</span>:</span></span>
        that</span>:</span> ansible_facts['os_family'] == "Darwin"</span></span>
        fail_msg</span>:</span> "This playbook can only be run on macOS"</span></span>
  </span></span>
    -</span> name</span>:</span> Unmount disk</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> diskutil unmountDisk {{ target_disk }}</span></span>
  </span></span>
    -</span> name</span>:</span> Write image to disk</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> dd if={{ image_src }} of={{ target_raw_disk }} bs=4m</span></span>
  </span></span>
    -</span> name</span>:</span> Make GPT see the full disk</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> sgdisk --move-second-header {{ target_disk }}</span></span>
  </span></span>
    -</span> name</span>:</span> "Add CIDATA partition at the end"</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> sgdisk --new=0:-51M:+50M --typecode=0:0700 --change-name=0:CIDATA {{ target_disk }}</span></span>
  </span></span>
    -</span> name</span>:</span> "Format CIDATA to FAT32"</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> diskutil eraseVolume MS-DOS CIDATA {{ target_disk }}s2</span></span>
  </span></span>
    -</span> name</span>:</span> Mount CIDATA partition</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> diskutil mount {{ target_disk}}s2</span></span>
  </span></span>
    -</span> name</span>:</span> Write cloud-init meta-data</span></span>
      ansible.builtin.copy</span>:</span></span>
        content</span>:</span> |</span></span>
          local-hostname: {{ target_hostname }}</span></span>
        dest</span>:</span> /Volumes/CIDATA/meta-data</span></span>
  </span></span>
    -</span> name</span>:</span> Write cloud-init user-data</span></span>
      ansible.builtin.copy</span>:</span></span>
        content</span>:</span> |</span></span>
          #cloud-config</span></span>
          users:</span></span>
            - name: ubuntu</span></span>
              ssh_authorized_keys:</span></span>
              {% for key in public_keys -%}</span></span>
                - {{ key }}</span></span>
              {% endfor -%}</span></span>
              sudo: ALL=(ALL) NOPASSWD:ALL</span></span>
              shell: /bin/bash</span></span>
        dest</span>:</span> /Volumes/CIDATA/user-data</span></span>
      become</span>:</span> true</span></span>
  </span></span>
    -</span> name</span>:</span> Unmount disk</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> diskutil unmountDisk {{ target_disk }}s2</span></span>
    </span></span>
    -</span> name</span>:</span> Eject disk</span></span>
      ansible.builtin.command</span>:</span></span>
        cmd</span>:</span> diskutil eject {{ target_disk }}</span></span></code></pre>Errata#</span></a>
</h1>

2026.06.09</strong> - Updated Ansible snippet and appendix playbook to fix hard-coded reference to /dev/disk2</code> in the "Mount CIDATA partition" task.</li>
</ul>



You can't reliably mount an ext4 file system</a>, and using cloud-init gets around this limitation. ↩</a></p>
</li>

On a Mac, you can install the right package with brew install qemu</code> if you're using Homebrew</a>. On a Debian-based Linux distro, you can use apt install qemu-utils</code>. ↩</a></p>
</li>

There's also the option of passing in kernel arguments, and using a service on the local network. However, since mounting ext4</code> partitions is not a feasible option, I've decided against this approach. ↩</a></p>
</li>

You can install it with brew install gptfdisk</code> ↩</a></p>
</li>
</ol>
</section>


Why I Self Host
2026-03-28T00:00:00+00:00
I have been running my own servers, on and off since college.
They have been a great experimentation platform, and given me resources to tap into for my professional life.
It wasn't always smooth sailing, but here's some of my takes, and why it's always been worth the hassle, at least for me.</p>
A short history#</span></a>
</h1>
The early days#</span></a>
</h2>
The first  "server" I built was during college when I took an old desktop, put Linux on it, and used it mostly as a network file store.
No redundancy, but it was fun.</p>
Later on during grad school, I had to setup the research group server1</a></sup>.
Using DynDNS</a>2</a></sup>, it was publicly accessible.
That worked fine for a while, until a vulnerable Apache module turned it into a spam bot.
I only found out when I got an annoyed email from a sysadmin in Bucharest, Romania, that it was spamming their network.
Took it down for the day, cleaned it up, and made a note to apply patches more often in the future.</p>
Today#</span></a>
</h2>
Currently, I am running a Pi Cluster with 4 Raspberry Pi 4 Nodes, with 4 GB each, running K3s</a>.
While not a very powerful cluster, and only using SD Card for storage, it's enough to serve a static website, grafana, prometheus, and any random experiments.
I also have a 1 node Framework Desktop that will run more "critical" systems, like a Postgres database for services that need it.</p>
Why?#</span></a>
</h1>
A lot of the services I run (or plan to) are usually available for free (or a reasonable subscription) else where.
You can host a website cheaply on AWS with S3 and Cloudfront (assuming low traffic).
You can host your 3D prints on Printables and the like, and I do.
But there are a few advantages to having control of where your data ends up.</p>
Learning#</span></a>
</h2>
This has been the primary reason for this. The Pi Cluster in particular was a great learning experience. Testing a repeatable Ansible set of playbooks, getting K3S to work and deploying a few services using both plain Kubernetes templates and Helm is not something I get to do in my day job.</p>
I find that I learn best when working on something practical. This is true from programming languages to infrastructure and networking. Having a "disposable" setup makes this a lot easier when learning.</p>
Enshittification#</span></a>
</h2>
It's an unfortunate reality of today's world.
Free services get more restrictive, open source alternatives get yanked3</a></sup>, or the prices get constantly increased, with "AI" as the excuse4</a></sup>.</p>
Having your own services protects from this reality, and you don't need to worry as much about increased costs, lost access, etc.</p>
Private access#</span></a>
</h2>
From the grad school days, I've learned to be wary when exposing anything over the internet.
To this end, the cluster is currently private and I access it from anywhere using Tailscale.
The free tier5</a></sup> is enough for me and my partner, and we can access the services from anywhere.
Of course, locally we can always use the local network.</p>
Using Cert Manager</a>,
I have publicly valid certificates and DNS that points to the Tailscale IP address of a node in the cluster.
This works great, and we don't get any annoying browser warnings.
Making everything public is only a matter of updating the DNS records.</p>
Your data, your rules#</span></a>
</h2>
Keeping the data private means it's not going to be used to train AI models, and target ads at you.
I'd rather have my data truly private, not private with an asterisk and a disclaimer, as the service can still read it but promises to play nice.</p>
The downsides#</span></a>
</h1>
Reliability#</span></a>
</h2>
My setup is only as reliable as my internet and power are.
A UPS helps with the power, but no backup to the internet connection.
If Comcast is being its usual self, we'll get a few periods of downtime a week.
For now this is not an issue, as the total downtime is less than 10 minutes a week, and mostly at night.</p>
Data safety#</span></a>
</h2>
If anything were to happen to the hardware, well, the data is gone.
That is solved by having automatic backups to S36</a></sup>, and having a good 3-2-1 backup policy</a> is critical.</p>
Also, test your backups regularly.
You don't want to find that your backups are broken when you really need them.</p>
Cost#</span></a>
</h2>
In particular, the upfront costs of getting the hardware.
This is only getting more expensive these days7</a></sup>.
However, if you've got some old hardware lying around, or go the used option, it's going to be less of a concern.</p>
Security#</span></a>
</h2>
If the spambot incident taught me anything, it's that if it's open to the internet it's a target.
Data leaks could be more serious if it has private data, and you're the only one in charge and responsible for securing it.
Keeping up to date with patches becomes even more critical the more sensitive the stored data.
A solution like Tailscale sidesteps the issue by keeping everything private.</p>
Conclusion#</span></a>
</h1>
Is this for everyone?
No. But if you want to learn, or care about where data ends up living, this is something worth looking into.</p>




In our living room, because the university policies for a "hard-wired" server were beyond draconian, and made the hardware useless for the research we were doing ↩</a></p>
</li>

It was free in those days ↩</a></p>
</li>

Minio archiving their open source repo</a> ↩</a></p>
</li>

Google Workspace price increase due to new AI features</a> ↩</a></p>
</li>

I am aware that this could enshittify in the future. However, Headscale</a> is an option, so I'm confident there's an offramp if I need it ↩</a></p>
</li>

For data durability, I haven't found anything that beats S3, or other key store offerings by a large cloud vendor. ↩</a></p>
</li>

RAM price hikes</a> ↩</a></p>
</li>
</ol>
</section>


Self Hosted Postgres in Kubernetes with PITR Recovery
2025-12-31T00:00:00+00:00
Introduction#</span></a>
</h1>
I would like to selfhost a few services, and they require persistence, in the form of a PostgreSQL database.
Since my Kubernetes cluster runs on 4 Raspberry Pi 4, each with an SD card for disk, it's only a matter of time until one of them gets corrupted.
To avoid the innevitable and predictable data loss, I'll need automated backups, idealy with Point-In-Time Recovery (PITR).</p>
If I'm going to reinvent the wheel, why not go for the whole wheel?</p>
Solution#</span></a>
</h1>
CloudNativePG</a> seems to be most mature and feature complete out there.
It provides an operator that does a lot of the heavy lifting.
Backups are handled by the Barman Cloud</a> plugin.
It handles aspects like WAL Log archiving, taking regular snapshots and uploading them a cloud object storage, like S31</a></sup>.
Seems to meet all the needs I need for my project.</p>
Setup#</span></a>
</h1>
Setting up a Cloud Native PG database#</span></a>
</h2>
Using the instructions on their website</a>, we'll need to install the operator.
We can do it with the following command:</p>
$</span> kubectl apply</span> --server-side -f</span> https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.28/releases/cnpg-1.28.0.yaml</span></span></code></pre>
We can now create our database.
The Barman Cloud Plugin we'll be using later assumes that the DB is in the cnpg-system</code> namespace.
To simply this experiment, we'll work with that assumption, and we'll start by creating the namespace:</p>
$</span> kubectl create namespace cnpg-system</span></span></code></pre>
Then we can define our cluster configuration, with one user so we can connect to it later:</p>
apiVersion</span>:</span> v1</span></span>
type</span>:</span> kubernetes.io/basic-auth</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> postgres-secret</span></span>
  namespace</span>:</span> cnpg-system</span></span>
stringData</span>:</span></span>
  username</span>:</span> user1</span></span>
  password</span>:</span> supersecretpassword</span></span>
---</span></span>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> Cluster</span></span>
metadata</span>:</span></span>
  name</span>:</span> test</span></span>
  namespace</span>:</span> cpgn-system</span></span>
spec</span>:</span></span>
  instances</span>:</span> 1</span></span>
  storage</span>:</span></span>
    size</span>:</span> 1Gi</span></span></code></pre>
And when we apply this, we have a new cluster with 1 instance.
We can forward the port, and connect to it like we normally would to a Postgres instance running locally.</p>
$</span> kubectl port-forward</span> -n</span> cnpg-system service/test-restore-rw 5432:5432</span></span></code></pre>Backups#</span></a>
</h2>
Installing#</span></a>
</h3>
Now, we can configure backups for our new Postgres cluster.</p>
First, let's install the prerequisites that the Barman Cloud plugin requires.
We will need to install Certificate Manager2</a></sup>.</p>
$</span> kubectl apply</span> -f</span> https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml</span></span></code></pre>
Then, we can install the plugin:</p>
$</span> kubectl apply</span> -f</span> https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.9.0/manifest.yaml</span></span></code></pre>Configuration#</span></a>
</h3>
We can now define the backup configuration.
For this example, I'll use S3 as the storage target for the backups.</p>
First, we'll need to store a set AWS credentials with access to the right bucket3</a></sup>.
We'll use an Opaque Secret</a> to store the access and secret keys:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> s3-credentials</span></span>
  namespace</span>:</span> cnpg-system</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  access-key-id</span>:</span> <redacted></span></span>
  secret-key-id</span>:</span> <redacted></span></span></code></pre>
Next, we'll define the storage configuration to our S3 Bucket:</p>
apiVersion</span>:</span> barmancloud.cnpg.io/v1</span></span>
kind</span>:</span> ObjectStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> s3-store</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  configuration</span>:</span></span>
    destinationPath</span>:</span> s3://<bucket>/postgres/</span></span>
    s3Credentials</span>:</span></span>
      accessKeyId</span>:</span></span>
        name</span>:</span> s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKey</span>:</span></span>
        name</span>:</span> s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span>
    wal</span>:</span></span>
      compression</span>:</span> gzip</span></span></code></pre>
Finally, we'll need to tell our Postgres cluster to use this configuration, and enable WAL archiving.
We'll add the following to the Cluster spec</code> field:</p>
plugins</span>:</span></span>
  -</span> name</span>:</span> barman-cloud.cloudnative-pg.io</span></span>
    isWALArchiver</span>:</span> true</span></span>
    parameters</span>:</span></span>
      barmanObjectName</span>:</span> s3-store</span></span></code></pre>
The final piece of the puzzle is setup regular "base" backups.
These will backup the entire dataset, and give us a "base" for the WAL logs to be applied to in order to get our Point-In-Time Restore.</p>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> ScheduledBackup</span></span>
metadata</span>:</span></span>
  name</span>:</span> pg-backup</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  cluster</span>:</span></span>
    name</span>:</span> test</span></span>
  schedule</span>:</span> '0 0 * * * *'</span> # hourly</span></span>
  backupOwnerReference</span>:</span> self</span></span>
  method</span>:</span> plugin</span></span>
  pluginConfiguration</span>:</span></span>
    name</span>:</span> barman-cloud.cloudnative-pg.io</span></span></code></pre>Results#</span></a>
</h3>
Now that everything is configured, we can list our S3 bucket, and sure enough, we have backups.
The top level structure has our base and WAL logs:</p>
$ aws s3 ls s3://<redacted>/postgres/test/</span></span>
                           PRE base/</span></span>
                           PRE wals/</span></span></code></pre>
Digging in deeper, we have our base backups, nicely named by timestamp, one every hour, as we'd expect:</p>
$ aws s3 ls s3://<redacted>/postgres/test/base/</span></span>
                           PRE 20251227T030000/</span></span>
                           PRE 20251227T040000/</span></span>
                           PRE 20251227T050000/</span></span>
                           PRE 20251227T060000/</span></span>
                           PRE 20251227T070000/</span></span>
                           ...</span></span></code></pre>
And finally, we have our WAL logs archived:</p>
$ aws s3 ls s3://<redacted>/postgres/test/wals/0000000100000000/</span></span>
2025-12-26 21:00:02      16944 000000010000000000000013.gz</span></span>
2025-12-26 21:00:04        210 000000010000000000000014.00000028.backup.gz</span></span>
2025-12-26 21:00:03      16513 000000010000000000000014.gz</span></span>
2025-12-26 21:05:03      17139 000000010000000000000015.gz</span></span>
2025-12-26 21:30:03      17276 000000010000000000000016.gz</span></span>
2025-12-26 22:00:02      16416 000000010000000000000017.gz</span></span>
2025-12-26 22:00:05        209 000000010000000000000018.00000028.backup.gz</span></span>
2025-12-26 22:00:04      16510 000000010000000000000018.gz</span></span>
...</span></span></code></pre>
So we should have all the pieces needed to perform a PITR.
We'll tacke this in the next section.</p>
Restoring#</span></a>
</h2>
For testing the restore, I've created a simple table, with timestamps for easy reasoning:</p>
CREATE TABLE</span> test</span>(</span></span>
  id </span>int</span>,</span></span>
  created_at </span>timestamp</span>(</span>6</span>)</span></span>
);</span></span></code></pre>
And we inserted different values, and this is the end state of the table:</p>
test=> select * from test;</span></span>
 id |         created_at         </span></span>
----+----------------------------</span></span>
 16 | 2025-12-27 03:00:17.601392</span></span>
 17 | 2025-12-27 03:00:20.438822</span></span>
 18 | 2025-12-27 03:28:06.64914</span></span>
 19 | 2025-12-27 20:42:25.563207</span></span>
 20 | 2025-12-27 20:42:32.992066</span></span>
 21 | 2025-12-27 20:45:58.634257</span></span>
(6 rows)</span></span></code></pre>
We'll restore the table to 2025-12-27, at 20:43:00 UTC.
For this, we'll create a new cluster, and we'll point it at the backups we have.
We'll also need to give it the target time we want the cluster restored to.
We arrive at this configuration:</p>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> Cluster</span></span>
metadata</span>:</span></span>
  name</span>:</span> test-restore</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  instances</span>:</span> 1</span></span>
  imagePullPolicy</span>:</span> IfNotPresent</span></span>
  bootstrap</span>:</span></span>
    recovery</span>:</span></span>
      source</span>:</span> source</span></span>
  externalClusters</span>:</span></span>
  -</span> name</span>:</span> source</span></span>
    plugin</span>:</span></span>
      name</span>:</span> barman-cloud.cloudnative-pg.io</span></span>
      parameters</span>:</span></span>
        barmanObjectName</span>:</span> s3-store</span></span>
        serverName</span>:</span> test</span></span>
        targetTime</span>:</span> "2025-12-27T20:43:00Z"</span></span>
  storage</span>:</span></span>
    size</span>:</span> 1Gi</span></span></code></pre>
Once the restore is done (it was pretty much instant for this example), we con connect and check our test table:</p>
test=> select * from test;</span></span>
 id |         created_at         </span></span>
----+----------------------------</span></span>
 16 | 2025-12-27 03:00:17.601392</span></span>
 17 | 2025-12-27 03:00:20.438822</span></span>
 18 | 2025-12-27 03:28:06.64914</span></span>
 19 | 2025-12-27 20:42:25.563207</span></span>
 20 | 2025-12-27 20:42:32.992066</span></span>
(5 rows)</span></span></code></pre>
As we'd expect, we're missing row with id</code> 21, as it's after</em> the target restore time.</p>
Conclusions and Final Remarks#</span></a>
</h1>
All in all, this was an easy setup, and it works fine for at least the basic use case.
The true test is once this sees some "production" loads, and testing with an actual live data base.</p>
A note on S3 cots#</span></a>
</h2>
For this example, I used S3 as the backup target destination.
The amount of data stored is small, however, WAL archiving could end up writing a lot of objects.
This could incur significant S3 API charges, so it's something I'm keeping an eye on.
With hourly base backups, I didn't see any cost increases for my AWS account.
But inserting 21 records, and deleting a few is not exactly a representative use case, but at least the "baseline" cost is not absurd.</p>
Retention policies#</span></a>
</h2>
Barman has the option of specifying retention policies</a>.
However, for this experiment, I've gone with specifying a lifecycle policy on the S3 bucket.
Everything under postgres/</code> will be deleted after 7 days.
This will give me a one week recovery window.</p>
Using the Barman retention policy will cause the plugin to list S3, and then delete the objects.
This also will incur some S3 API charges, and I think that using the S3 lifecycle rule is probably good enough.</p>




The plugin also supports Google Cloud Storage, or Azure Blobs or some other services that implementation a compatible API</a>. ↩</a></p>
</li>

You can skip this step if your cluster already has it installed on the cluster. My test cluster did not. ↩</a></p>
</li>

For reference, I granted the user full access to the S3 bucket where the backups are stored. ↩</a></p>
</li>
</ol>
</section>

Step 2: Cloud-init configuration
`Luckily, Cloud-init supports the` `NoCloud</code> datasource</a>. This allows us to get the user data from a specific partition called cidata</code> or CIDATA</code>. This is the option we'll go with next3</a></sup>.</p>`

Cost
In particular, the upfront costs of getting the hardware. This is only getting more expensive these days 7</a></sup>. However, if you've got some old hardware lying around, or go the used option, it's going to be less of a concern.</p>

Caius' Lab - Self-Hosting

Backing up Persistent Volume Claims with k8up

Conclusion

Setting up a Homelab: Step 1

Why I Self Host

Self Hosted Postgres in Kubernetes with PITR Recovery

Caius' Lab - Self-Hosting

Backing up Persistent Volume Claims with k8up

Setting up a Homelab: Step 1

Writing the cloud-init config#</span></a> </h2> Now, we're ready to write our cloud-init files. We'll need two of them:</p> meta-data</code> for setting the hostname</li>

Why I Self Host

Self Hosted Postgres in Kubernetes with PITR Recovery