Backing up Persistent Volume Claims with k8up

#Self-Hosting #Kubernetes

Previously, I wrote about backing up Postgres to S3 using CNPG and Barman Cloud. This is enough if all the data you want to persist is in Postgres. However, if you have any other volumes, this will not be enough. In this post, I'll go over setting up backups with k8up, and what the restore process looks like.

Why k8up?

k8up (pronounced "ketchup") is a Kubernetes Operator distributed via a Helm chart. By default, it backs up all Persistent Volume Claims (PVCs) marked as ReadWriteMany, ReadWriteOnce, or with a certain label. It uses restic, so it can write to object storage. For this blog post, I'm using Amazon S3 as the destination, but self-hosted options like Garage should also work.

As to why I chose it, it's more mature than VolSync, and I'm interested only in backing up PVCs. The cluster definition is managed with Helm, so Velero seems a bit too heavy for my limited use case.

Finally, there's Longhorn, which can also do backups. My cluster will be quite resource-constrained, so I'm not sure the extra overhead of Longhorn will be worth it in the long run. I'm OK with an "outage" if I need to restore a PVC from S3, and the extra redundancy doesn't seem worth the overhead.

Test service

To evaluate this, I've created a very simple service that writes the current time to a PVC once a second. It will give me an idea of how backups perform, and how recovery works. Here is the Helm template for this service:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: time-writer-pvc
  namespace: backup-test
  labels:
    must-backup: "true"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: time-writer
  namespace: backup-test
spec:
  containers:
    - name: time-writer
      image: busybox:1.37
      imagePullPolicy: IfNotPresent
      command: ["/bin/sh", "-c"]
      args:
        - while true; do date >> /data/time.log; sleep 1; done
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: time-writer-pvc

Backup setup

Once we have this setup, we can start with the backups. First, we need to create 2 secrets: one for the restic encryption key, and one for the AWS credentials.

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.clusterName }}-restic-secret-key
  namespace: backup-test
type: Opaque
stringData:
  secretKey: {{ .Values.secrets.resticSecretKey }}
---
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.clusterName }}-s3-credentials
  namespace: backup-test
type: Opaque
stringData:
  access-key-id: {{ .Values.secrets.awsAccessKey }}
  secret-key-id: {{ .Values.secrets.awsSecretKey }}

Now, we can define the backup we want to take. For s3, I'll use the regional S3 endpoint, s3.us-west-2.amazonaws.com. For the bucket name, we can also specify a prefix. In this example, the restic repo lives under the test prefix.

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-test
  namespace: backup-test
spec:
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 2
  backend:
    repoPasswordSecretRef:
        name: {{ .Values.clusterName }}-restic-secret-key
        key: secretKey
    s3:
      endpoint: https://s3.us-west-2.amazonaws.com
      bucket: bucket-name/test
      accessKeyIDSecretRef:
        name: {{ .Values.clusterName }}-s3-credentials
        key: access-key-id
      secretAccessKeySecretRef:
        name: {{ .Values.clusterName }}-s3-credentials
        key: secret-key-id

Once this is done, we can run helm upgrade and we get our backup. Checking the jobs, they have completed:

$ kubectl get jobs -n backup-test
NAME                                  STATUS     COMPLETIONS   DURATION   AGE
backup-schedule-test-backup-c9wz6-0   Complete   1/1           9s         12m

And looking at the logs, we get confirmation that it backed up our only file:

2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup    starting backup
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup    starting backup for folder    {"foldername": "time-writer-pvc"}
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup.command    restic command    {"path": "/usr/local/bin/restic", "args": ["backup", "--host", "backup-test", "--json
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2026-05-03T19:45:37Z    INFO    k8up.restic.restic.backup.progress    backup finished    {"new files": 1, "changed files": 0, "errors": 0}

For the final confirmation, we can see the restic repo structure in S3:

$ aws s3 ls s3://bucket-names/test/
                           PRE data/
                           PRE index/
                           PRE keys/
                           PRE snapshots/
2026-05-03 12:45:35        155 config

Recovery

A backup is only useful if we can recover our data from it. So let's see how this would work. First, let's create a PVC that's the destination for our data:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: restore-test
  namespace: backup-test
  annotations:
    # Setting this to false to exclude from future backups
    k8up.io/backup: "false"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi

Then, we can define our restore:

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore-test
spec:
  restoreMethod:
    folder:
      claimName: restore-test
  backend:
    repoPasswordSecretRef:
        name: {{ .Values.clusterName }}-restic-secret-key
        key: secretKey
    s3:
      endpoint: https://s3.us-west-2.amazonaws.com
      bucket: bucket-name/test
      accessKeyIDSecretRef:
        name: {{ .Values.clusterName }}-s3-credentials
        key: access-key-id
      secretAccessKeySecretRef:
        name: {{ .Values.clusterName }}-s3-credentials
        key: secret-key-id

This will create a new job, and checking the logs, we can see that it succeeded:

$ kubectl logs -n backup-test jobs/restore-restore-test -f
<...truncated...>
2026-05-03T20:04:04Z    INFO    k8up.restic.restic.restore.command      restic command  {"path": "/usr/local/bin/restic", "args": ["restore", "be8f37f009f0c910ad5ce8aa067d3dbd4050e7de9c41fde14394555a73adee06:/data/time-writer-pvc", "--target", "/restore"]}
2026-05-03T20:04:04Z    INFO    k8up.restic.restic.restore.command      Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2026-05-03T20:04:05Z    INFO    k8up.restic.restic.restore.restic.stdout        restoring snapshot be8f37f0 of [/data/time-writer-pvc] at 2026-05-03 19:45:35.978401924 +0000 UTC by @backup-test to /restore
2026-05-03T20:04:05Z    INFO    k8up.restic.restic.restore.restic.stdout        Summary: Restored 1 files/dirs (3.144 KiB) in 0:00

Finally, we can check the PVC directly, and confirm we have data until 19:45:36. The backup was started at 19:45:35 and finished by 19:45:37. This gives us a very good RPO of a few seconds, which is great for my use case!

$ kubectl exec -n backup-test -it pvc-inspect -- tail /data/time.log
Sun May  3 19:45:27 UTC 2026
Sun May  3 19:45:28 UTC 2026
Sun May  3 19:45:29 UTC 2026
Sun May  3 19:45:30 UTC 2026
Sun May  3 19:45:31 UTC 2026
Sun May  3 19:45:32 UTC 2026
Sun May  3 19:45:33 UTC 2026
Sun May  3 19:45:34 UTC 2026
Sun May  3 19:45:35 UTC 2026
Sun May  3 19:45:36 UTC 2026

Scheduling

The above example only covers a "one-time" backup. For this to be useful, I need a regular schedule. This is where the Schedule comes in.

I'll start with the example provided in the documentation. We'll back up every 5 minutes, and every hour we'll do checking and pruning. Data retention can be tuned depending on the use case.

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: schedule-test
spec:
  backend:
    repoPasswordSecretRef:
        name: {{ .Values.clusterName }}-restic-secret-key
        key: secretKey
    s3:
      endpoint: https://s3.us-west-2.amazonaws.com
      bucket: bucket-name/test
      accessKeyIDSecretRef:
        name: {{ .Values.clusterName }}-s3-credentials
        key: access-key-id
      secretAccessKeySecretRef:
        name: {{ .Values.clusterName }}-s3-credentials
        key: secret-key-id
  backup:
    schedule: '*/5 * * * *'
    failedJobsHistoryLimit: 2
    successfulJobsHistoryLimit: 2
  check:
    schedule: '0 * * * *'
  prune:
    schedule: '30 * * * *'
    retention:
      keepLast: 5
      keepDaily: 7

This will result in one job every 5 minutes:

$ kubectl get jobs -n backup-test
NAME                                  STATUS     COMPLETIONS   DURATION   AGE
backup-schedule-test-backup-7l7sj-0   Complete   1/1           10s        3m17s
backup-schedule-test-backup-l7jkq-0   Complete   1/1           10s        8m17s

And for each job, the backup was successful:

2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup       starting backup
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup       starting backup for folder      {"foldername": "time-writer-pvc"}
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup.command       restic command  {"path": "/usr/local/bin/restic", "args": ["backup", "--json", "--host", "backup-test", "/data/time-writer-pvc"]}
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup.command       Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2026-05-03T21:40:31Z    INFO    k8up.restic.restic.backup.progress      backup finished {"new files": 0, "changed files": 1, "errors": 0}
2026-05-03T21:40:31Z    INFO    k8up.restic.restic.backup.progress      stats   {"time": 1.727354075, "bytes added": 51912, "bytes processed": 50837}

Conclusion

Overall, k8up seems to meet my use case. The setup is straightforward and easy to reason about. The restore process is also straightforward.

While the example here is very limited, it gives me confidence that the basics work. The real test will come once it's operating on real data, but regular restores and testing should give me the confidence I need there.

Why k8up?#

Test service#

Backup setup#

Recovery#

Scheduling#

Conclusion#