Caius' Lab - Kubernetes

Installing Coder in Kubernetes with Devcontainer Support

Caius Brindescu — Mon, 15 Jun 2026 00:00:00 +0000

Introduction#</a> </h1>
I have a Framework Desktop whose main purpose is to be a remote development machine. I'm currently using it with remote SSH access thorugh VS Code, and Docker devcontainers to get reproducible dev environments. And this works fine as is. However, I still need to have a laptop accessible that can run VS Code, and I'd like to able to run a quick development session from everywhere, regardlesss of what hardware I have on hand. I can't run this setup from an iPad, or a browser, or some other machine, which would lighten my travel "tech stack" considerably.
This is where Coder</a> comes in. It will allow me to run my development setup from any browser, and still keep my current VSCode worflow when working from a laptop. It also gives me the flexibility you get from GitHub's Codespaces</a>, without having yet another service I need to pay for.

Existing Setup
The desktop runs K3s</a> as a single node cluster and it already runs a couple of services I use in my daily life. As part of these services, it already has a Postgres instance, that has backups configured 1</a>. The goal is to use this Postgres instance for persistence, so I have one less piece of infrastructure I need to worry about.
Finally, I use helm to manage the services running on this "cluster," so I have a good base set up.

The first step is to set up Coder itself. It already comes with a Helm chart</a>. For my setup, I'll wrap it in my own chart, so I can configure it,

apiVersion</span>:</span> v2</span></span>
name</span>:</span> coder</span></span>
version</span>:</span> 0.1.0</span></span>
dependencies</span>:</span></span>
  -</span> name</span>:</span> coder</span></span>
    version</span>:</span> "2.*"</span></span>
    repository</span>:</span> https://helm.coder.com/v2</span></span></code></pre>
And I will need to define a couple of resources, in order to hook it up into the existing infrastructure.</p>
First, a secret for the DB connection string, that will contain the username and password:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> coder-db-secret</span></span>
  namespace</span>: {{</span> .Release.Namespace</span> }}</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  url</span>: {{</span> .Values.dbUrl | quote</span> }}</span></span></code></pre>
Secondly, I need an ingress controller, so I can access it from my custom URL:</p>
apiVersion</span>:</span> networking.k8s.io/v1</span></span>
kind</span>:</span> Ingress</span></span>
metadata</span>:</span></span>
  name</span>:</span> coder</span></span>
  namespace</span>: {{</span> .Release.Namespace</span> }}</span></span>
  annotations</span>:</span></span>
    cert-manager.io/cluster-issuer</span>: {{</span> .Values.clusterIssuer</span> }}</span></span>
spec</span>:</span></span>
  ingressClassName</span>:</span> traefik</span></span>
  tls</span>:</span></span>
    -</span> hosts</span>:</span></span>
        - {{</span> .Values.ingress.host</span> }}</span></span>
        - {{</span> .Values.ingress.wildcardHost | quote</span> }}</span></span>
      secretName</span>:</span> coder-tls</span></span>
  rules</span>:</span></span>
    -</span> host</span>: {{</span> .Values.ingress.host</span> }}</span></span>
      http</span>:</span></span>
        paths</span>:</span></span>
          -</span> path</span>:</span> /</span></span>
            pathType</span>:</span> Prefix</span></span>
            backend</span>:</span></span>
              service</span>:</span></span>
                name</span>:</span> coder</span></span>
                port</span>:</span></span>
                  number</span>:</span> 80</span></span>
    -</span> host</span>: {{</span> .Values.ingress.wildcardHost | quote</span> }}</span></span>
      http</span>:</span></span>
        paths</span>:</span></span>
          -</span> path</span>:</span> /</span></span>
            pathType</span>:</span> Prefix</span></span>
            backend</span>:</span></span>
              service</span>:</span></span>
                name</span>:</span> coder</span></span>
                port</span>:</span></span>
                  number</span>:</span> 80</span></span></code></pre>
Like all my other services, this will use Let's Encrypt</a> and DNS verification to generate valid TLS certficate.
The deployment will only be available in my tailnet</a>, but having a valid certificate makes everything a lot easier.
I don't get annoying browser warnings, and it's one step taken care of if I ever want to make the service publicly accessible.</p>
Finally, I need to define the values for our deployment.</p>
The details for the ingress are defined below.
It's the cluster issuer (defined elsewhere in my helm charts), and the domains that we want to serve coder from.</p>
clusterIssuer</span>:</span> lets-encrypt-prod</span></span>
</span>
ingress</span>:</span></span>
  host</span>:</span> coder.caius.dev</span></span>
  wildcardHost</span>:</span> "*.coder.caius.dev"</span></span></code></pre>
For the DB secret, I've defined the dbUrl</code> inside a separate secrets.yaml</code> file that is not added to source control.
The format of this is:</p>
dbUrl</span>:</span> "<user>@<password>:<db_host>:5432"</span></span></code></pre>
Next, I'll wire up the secret DB access string, the access URL, and disable coder's builtin ingress, as we are using our own.
We'll also need to disable the cluster access URL.
Without this, things like the CLI install script will try and use the Kubernetes internal hostname, and that's not resolvable from outside the cluster.</p>
coder</span>:</span></span>
  coder</span>:</span></span>
    envUseClusterAccessURL</span>:</span> false</span></span>
</span>
    env</span>:</span></span>
      -</span> name</span>:</span> CODER_PG_CONNECTION_URL</span></span>
        valueFrom</span>:</span></span>
          secretKeyRef</span>:</span></span>
            name</span>:</span> coder-db-secret</span></span>
            key</span>:</span> url</span></span>
      -</span> name</span>:</span> CODER_ACCESS_URL</span></span>
        value</span>:</span> https://coder.caius.dev</span></span>
      -</span> name</span>:</span> CODER_WILDCARD_ACCESS_URL</span></span>
        value</span>:</span> "*.coder.caius.dev"</span></span>
</span>
    ingress</span>:</span></span>
      enable</span>:</span> false</span></span></code></pre>
Finally, let's set up some resource limits:</p>
    resources</span>:</span></span>
      requests</span>:</span></span>
        cpu</span>:</span> 250m</span></span>
        memory</span>:</span> 512Mi</span></span>
      limits</span>:</span></span>
        cpu</span>:</span> "2"</span></span>
        memory</span>:</span> 2Gi</span></span></code></pre>
Applying this will bring up the deployment.
It takes a few minutes for the cert to be provisioned, but once that was done, I was able to log in and everything worked!</p>
Workspaces namespace#</span></a>
</h2>
For the workspaces themselves, I decided to use a separate namespace, coder-workspaces</code>.
For this, I created a simple namespace resource using helm:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Namespace</span></span>
metadata</span>:</span></span>
  name</span>:</span> coder-workspaces</span></span>
  labels</span>:</span></span>
    pod-security.kubernetes.io/enforce</span>:</span> privileged</span></span></code></pre>
We also need an RBAC authorization, so the coder service account can provision the resources we need when creating a new workspace in the coder-workspace</code> namespace:</p>
apiVersion</span>:</span> rbac.authorization.k8s.io/v1</span></span>
kind</span>:</span> Role</span></span>
metadata</span>:</span></span>
  name</span>:</span> coder-workspaces</span></span>
  namespace</span>:</span> coder-workspaces</span></span>
rules</span>:</span></span>
-</span> apiGroups</span>: [</span>""</span>]</span></span>
  resources</span>:</span></span>
  -</span> pods</span></span>
  -</span> pods/exec</span></span>
  -</span> pods/log</span></span>
  -</span> persistentvolumeclaims</span></span>
  -</span> secrets</span></span>
  -</span> services</span></span>
  -</span> serviceaccounts</span></span>
  verbs</span>: [</span>"get"</span>,</span> "list"</span>,</span> "watch"</span>,</span> "create"</span>,</span> "update"</span>,</span> "patch"</span>,</span> "delete"</span>]</span></span>
-</span> apiGroups</span>: [</span>"apps"</span>]</span></span>
  resources</span>:</span></span>
  -</span> deployments</span></span>
  -</span> replicasets</span></span>
  verbs</span>: [</span>"get"</span>,</span> "list"</span>,</span> "watch"</span>,</span> "create"</span>,</span> "update"</span>,</span> "patch"</span>,</span> "delete"</span>]</span></span>
---</span></span>
apiVersion</span>:</span> rbac.authorization.k8s.io/v1</span></span>
kind</span>:</span> RoleBinding</span></span>
metadata</span>:</span></span>
  name</span>:</span> coder-workspaces</span></span>
  namespace</span>:</span> coder-workspaces</span></span>
roleRef</span>:</span></span>
  apiGroup</span>:</span> rbac.authorization.k8s.io</span></span>
  kind</span>:</span> Role</span></span>
  name</span>:</span> coder-workspaces</span></span>
subjects</span>:</span></span>
-</span> kind</span>:</span> ServiceAccount</span></span>
  name</span>:</span> coder</span></span>
  namespace</span>:</span> coder</span></span></code></pre>Workspace Template#</span></a>
</h2>
For the template, I started off with the default kubernetes-devcontainer</code> template.
However there were a couple of things that needed to be addressed.</p>
First, and the most obvious, was changing the namespace from default</code> to coder-workspaces</code>.
This was a single line change.</p>
I also wanted to use SSH keys for repo authentication, as I didn't want to install the coder app in GitHub.
Finally, there were are few quality of life issues, like the directory that code server (and VS Code) opened in, as well as always trusting GitHub's SSH keys.
I will go into detail for each of them in the following sections.</p>
Using SSH keys for git repo authentication#</span></a>
</h3>
To access my private repos, I wanted to use SSH authentication.
I didn't want to install the coder github app, as I didn't really know what info it would syphon out.
Coder already defines a key pair that can be used for this purpose, one for each user.
However, I still need to inject the private key inside the container.
To accomplish this, I created a kubernetes secret with the private key, and then used that to inject it into the container.</p>
resource</span> "kubernetes_secret_v1" "ssh_key" {</span></span>
  </span>metadata</span> {</span></span>
    name</span>      =</span> "coder-</span>${</span>lower</span>(</span>data</span>.</span>coder_workspace</span>.</span>me</span>.</span>id</span>)</span>}</span>-ssh-key"</span></span>
    namespace</span> =</span> var</span>.</span>namespace</span></span>
  }</span></span>
  data</span> =</span> {</span></span>
    id_rsa</span> =</span> data.coder_workspace_owner.me.ssh_private_key</span></span>
  }</span></span>
}</span></span></code></pre>
I then set the correct enviroment variable, so the containers knows where to get the private key from:</p>
"ENVBUILDER_GIT_SSH_PRIVATE_KEY_PATH"</span> :</span> "/tmp/ssh/id_rsa"</span>,</span></span></code></pre>
Finally, I need to mount the newly created secret at the right path inside the container.
I need to tell the container where to mount it to:</p>
volume_mount</span> {</span></span>
    mount_path</span> =</span> "/tmp/ssh"</span></span>
    name</span>       =</span> "ssh-key"</span></span>
    read_only</span>  =</span> true</span></span>
}</span></span></code></pre>
And I'll need to configure the deployment to create the volume, with the secret contents:</p>
volume</span> {</span></span>
    name</span> =</span> "ssh-key"</span></span>
    </span>secret</span> {</span></span>
        secret_name</span>  =</span> kubernetes_secret_v1</span>.</span>ssh_key</span>.</span>metadata[</span>0</span>]</span>.</span>name</span></span>
        default_mode</span> =</span> "0400"</span></span>
    }</span></span>
}</span></span></code></pre>
I can now allow access to coder's public key, and use SSH to authenticate with GitHub, or any other forge.</p>
With this setup, I have the basic dev workflow sorted out.
I can create a new workspace, and it will clone the repo, and spin up the devcontainer.</p>
Mounting the correct path in the editor#</span></a>
</h3>
The default config will mount /workspaces</code> as the "root" path for the editor.
This isn't ideal, as I'm used to working in the repo root.
Some of the VS code tasks also assume this, and will fail otherwise.
Luckily, I can fix this by tweaking the templates a bit.</p>
First, I need to define a local variable that's the repo name.
This can be easily inferred from the repo url:</p>
repo_dir</span> =</span> "/workspaces/</span>${</span>replace</span>(</span>basename</span>(</span>local</span>.</span>repo_url</span>), ".git", "")</span>}</span>"</span></span></code></pre>
I'll then assign this to the folder</code> parameter in our code_server</code> module:</p>
folder</span> =</span> local</span>.</span>repo_dir</span></span></code></pre>
The coder agent will, by default, expose a VS Code Desktop connection option.
However, this will always point to "home", which in this case it's /workspaces</code>, and this is what we're trying to avoid.
The easier solution around this, was to disable it in the agent definition, by adding the following to the terraform config:</p>
display_apps</span> {</span></span>
  vscode</span> =</span> false</span></span>
}</span></span></code></pre>
Then, I can define an explicit vscode-desktop</code> module that will open the correct folder.</p>
module "vscode" {                                                                                </span></span>
  count   = data.coder_workspace.me.start_count                                                  </span></span>
  source  = "registry.coder.com/coder/vscode-desktop/coder"                                              </span></span>
  version = "~> 1.0"                                                                             </span></span>
                                                                                                 </span></span>
  agent_id   = coder_agent.main.id                                                               </span></span>
  folder     = local.repo_dir                                                                    </span></span>
  order      = 2                                                                                 </span></span>
}</span></span></code></pre>Automatically installing extensions#</span></a>
</h3>
The extensions specified in the devcontainer.json</code> file are not automatically installed when I start up the code server, or the desktop VS Code.
While it is possible to specify extensions to install in the terraform templates, I wanted a generic solution, that would install only the required extensions for each specific repo.
The solution is a script that runs at startup, parses the devcontainer.json</code> file, extracts the list of extensions and installs them.
After the install, it generates a .vscode/extensions.json</code> file, so that any remote VS Code desktop instances will pick them up, and suggest that they are installed2</a></sup>.</p>
The final result is the following coder_script</code> resource:</p>
resource</span> "coder_script" "devcontainer_extensions" {</span></span>
  count</span>              =</span> data</span>.</span>coder_workspace</span>.</span>me</span>.</span>start_count</span></span>
  agent_id</span>           =</span> coder_agent</span>.</span>main</span>.</span>id</span></span>
  display_name</span>       =</span> "Install devcontainer extensions"</span></span>
  run_on_start</span>       =</span> true</span></span>
  start_blocks_login</span> =</span> true</span></span>
  script</span>             = <<-EOT</span></span>
    #!/bin/bash</span></span>
    REPO_DIR="</span>${</span>local</span>.</span>repo_dir</span>}</span>"</span></span>
</span>
    DEVCONTAINER_JSON=""</span></span>
    for candidate in "$REPO_DIR/.devcontainer/devcontainer.json" "$REPO_DIR/devcontainer.json"; do</span></span>
      if [ -f "$candidate" ]; then</span></span>
        DEVCONTAINER_JSON="$candidate"</span></span>
        break</span></span>
      fi</span></span>
    done</span></span>
    [ -z "$DEVCONTAINER_JSON" ] && exit 0</span></span>
</span>
    until command -v code-server &>/dev/null; do sleep 1; done</span></span>
</span>
    command -v jq &>/dev/null || { echo "jq not found, skipping extension install"; exit 0; }</span></span>
    EXTENSIONS=$(jq -r '.customizations?.vscode?.extensions[]?' "$DEVCONTAINER_JSON" 2>/dev/null || true)</span></span>
    [ -z "$EXTENSIONS" ] && exit 0</span></span>
</span>
    echo "$EXTENSIONS" | while IFS= read -r ext; do</span></span>
      [ -z "$ext" ] && continue</span></span>
      code-server --install-extension "$ext" 2>&1 || true</span></span>
    done</span></span>
</span>
    # Write .vscode/extensions.json so VS Code Desktop prompts to install the same extensions</span></span>
    VSCODE_EXTS_FILE="$REPO_DIR/.vscode/extensions.json"</span></span>
    if [ ! -f "$VSCODE_EXTS_FILE" ]; then</span></span>
      mkdir -p "$REPO_DIR/.vscode"</span></span>
      jq -n --argjson exts \</span></span>
        "$(jq -c '[.customizations?.vscode?.extensions[]?]' "$DEVCONTAINER_JSON")" \</span></span>
        '{"recommendations": $exts}' > "$VSCODE_EXTS_FILE"</span></span>
    fi</span></span>
  EOT</span></span>
}</span></span></code></pre>Trusting GitHub's public keys#</span></a>
</h3>
The last nagging problem was having to trust GitHub's keys after I restarted a workspace, during the first interaction with git.
To fix this, I added another script that will trust GitHub's keys, so it always restarts in a "good" state.
The scripts will run in parallel, so this adds nothing to the startup time.</p>
resource</span> "coder_script" "github_known_hosts" {</span></span>
  count</span>              =</span> data</span>.</span>coder_workspace</span>.</span>me</span>.</span>start_count</span></span>
  agent_id</span>           =</span> coder_agent</span>.</span>main</span>.</span>id</span></span>
  display_name</span>       =</span> "Trust github hosts"</span></span>
  run_on_start</span>       =</span> true</span></span>
  start_blocks_login</span> =</span> true</span></span>
  script</span>             = <<-EOT</span></span>
mkdir -p /root/.ssh/</span></span>
cat <<'EOF' >> /root/.ssh/known_hosts</span></span>
|1|YcSxwNwqHecLiBTEXmhb3JcKMbg=|LDlb4p4OoJafZDnkMspTODpHdlY= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl</span></span>
|1|qf+ciiHWdVTQzkK389LmH/RM/Wo=|UaXlxsvDYssHCiqHv9q+aHyf01w= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=</span></span>
|1|nXxxrVNnRIHD8UKpOVlDGR9oYX4=|+FflYHXVajJijO9egAkU5wPQ+Ac= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=</span></span>
EOF</span></span>
echo "Done"</span></span>
EOT</span></span>
}</span></span></code></pre>Conclusions#</span></a>
</h1>
The templates I arrived at are available in this GitHub repo</a>.</p>
The setup works well, and this blog post was written using an iPad and code server.
The goal was to have more flexibility when travelling, to avoid having to carry too many devices.
And I think, so far, I've achieved this goal.</p>
The setup uses the standard devcontainers I've developed, without requiring any custom modfications to be made.
This should make it easy to adapt in any future project, or start up new workspaces as I realize I need to make changes to other projects.</p>



See this blog post</a> on how I set this up. ↩</a></p>
</li>

A simpler option would probably be to check in the extension.json</code> file into version control. But I'm too far down this rabbit hole to change course. ↩</a></p>
</li>
</ol>
</section>



Backing up Persistent Volume Claims with k8up
Caius Brindescu — Sun, 03 May 2026 00:00:00 +0000
Previously, I wrote about backing up Postgres to S3 using CNPG and Barman Cloud</a>.
This is enough if all the data you want to persist is in Postgres.
However, if you have any other volumes, this will not be enough.
In this post, I'll go over setting up backups with k8up</a>, and what the restore process looks like.</p>
Why k8up?#</span></a>
</h1>
k8up (pronounced "ketchup") is a Kubernetes Operator distributed via a Helm chart.
By default, it backs up all Persistent Volume Claims (PVCs) marked as ReadWriteMany</code>, ReadWriteOnce</code>, or with a certain label.
It uses restic</a>, so it can write to object storage.
For this blog post, I'm using Amazon S3 as the destination, but self-hosted options like Garage</a> should also work.</p>
As to why I chose it, it's more mature than VolSync</a>, and I'm interested only in backing up PVCs.
The cluster definition is managed with Helm</a>, so Velero</a> seems a bit too heavy for my limited use case.</p>
Finally, there's Longhorn</a>, which can also do backups.
My cluster will be quite resource-constrained, so I'm not sure the extra overhead of Longhorn will be worth it in the long run.
I'm OK with an "outage" if I need to restore a PVC from S3, and the extra redundancy doesn't seem worth the overhead.</p>
Test service#</span></a>
</h1>
To evaluate this, I've created a very simple service that writes the current time to a PVC once a second.
It will give me an idea of how backups perform, and how recovery works.
Here is the Helm template for this service:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> PersistentVolumeClaim</span></span>
metadata</span>:</span></span>
  name</span>:</span> time-writer-pvc</span></span>
  namespace</span>:</span> backup-test</span></span>
  labels</span>:</span></span>
    must-backup</span>:</span> "true"</span></span>
spec</span>:</span></span>
  accessModes</span>:</span></span>
    -</span> ReadWriteOnce</span></span>
  resources</span>:</span></span>
    requests</span>:</span></span>
      storage</span>:</span> 100Mi</span></span>
---</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Pod</span></span>
metadata</span>:</span></span>
  name</span>:</span> time-writer</span></span>
  namespace</span>:</span> backup-test</span></span>
spec</span>:</span></span>
  containers</span>:</span></span>
    -</span> name</span>:</span> time-writer</span></span>
      image</span>:</span> busybox:1.37</span></span>
      imagePullPolicy</span>:</span> IfNotPresent</span></span>
      command</span>: [</span>"/bin/sh"</span>,</span> "-c"</span>]</span></span>
      args</span>:</span></span>
        -</span> while true; do date >> /data/time.log; sleep 1; done</span></span>
      volumeMounts</span>:</span></span>
        -</span> name</span>:</span> data</span></span>
          mountPath</span>:</span> /data</span></span>
  volumes</span>:</span></span>
    -</span> name</span>:</span> data</span></span>
      persistentVolumeClaim</span>:</span></span>
        claimName</span>:</span> time-writer-pvc</span></span></code></pre>Backup setup#</span></a>
</h1>
Once we have this setup, we can start with the backups.
First, we need to create 2 secrets: one for the restic encryption key, and one for the AWS credentials.</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
  namespace</span>:</span> backup-test</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  secretKey</span>: {{</span> .Values.secrets.resticSecretKey</span> }}</span></span>
---</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
  namespace</span>:</span> backup-test</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  access-key-id</span>: {{</span> .Values.secrets.awsAccessKey</span> }}</span></span>
  secret-key-id</span>: {{</span> .Values.secrets.awsSecretKey</span> }}</span></span></code></pre>
Now, we can define the backup we want to take.
For s3</code>, I'll use the regional S3 endpoint, s3.us-west-2.amazonaws.com</code>.
For the bucket name, we can also specify a prefix.
In this example, the restic repo lives under the test</code> prefix.</p>
apiVersion</span>:</span> k8up.io/v1</span></span>
kind</span>:</span> Backup</span></span>
metadata</span>:</span></span>
  name</span>:</span> backup-test</span></span>
  namespace</span>:</span> backup-test</span></span>
spec</span>:</span></span>
  failedJobsHistoryLimit</span>:</span> 2</span></span>
  successfulJobsHistoryLimit</span>:</span> 2</span></span>
  backend</span>:</span></span>
    repoPasswordSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
        key</span>:</span> secretKey</span></span>
    s3</span>:</span></span>
      endpoint</span>:</span> https://s3.us-west-2.amazonaws.com</span></span>
      bucket</span>:</span> bucket-name/test</span></span>
      accessKeyIDSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKeySecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span></code></pre>
Once this is done, we can run helm upgrade</code> and we get our backup.
Checking the jobs, they have completed:</p>
$ kubectl get jobs -n backup-test</span></span>
NAME                                  STATUS     COMPLETIONS   DURATION   AGE</span></span>
backup-schedule-test-backup-c9wz6-0   Complete   1/1           9s         12m</span></span></code></pre>
And looking at the logs, we get confirmation that it backed up our only file:</p>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup    starting backup</span></span>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup    starting backup for folder    {"foldername": "time-writer-pvc"}</span></span>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup.command    restic command    {"path": "/usr/local/bin/restic", "args": ["backup", "--host", "backup-test", "--json</span></span>
2026-05-03T19:45:35Z    INFO    k8up.restic.restic.backup.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}</span></span>
2026-05-03T19:45:37Z    INFO    k8up.restic.restic.backup.progress    backup finished    {"new files": 1, "changed files": 0, "errors": 0} </span></span></code></pre>
For the final confirmation, we can see the restic repo structure in S3:</p>
$ aws s3 ls s3://bucket-names/test/</span></span>
                           PRE data/</span></span>
                           PRE index/</span></span>
                           PRE keys/</span></span>
                           PRE snapshots/</span></span>
2026-05-03 12:45:35        155 config</span></span></code></pre>Recovery#</span></a>
</h1>
A backup is only useful if we can recover our data from it.
So let's see how this would work.
First, let's create a PVC that's the destination for our data:</p>
kind</span>:</span> PersistentVolumeClaim</span></span>
apiVersion</span>:</span> v1</span></span>
metadata</span>:</span></span>
  name</span>:</span> restore-test</span></span>
  namespace</span>:</span> backup-test</span></span>
  annotations</span>:</span></span>
    # Setting this to false to exclude from future backups</span></span>
    k8up.io/backup</span>:</span> "false"</span></span>
spec</span>:</span></span>
  accessModes</span>:</span></span>
    -</span> ReadWriteOnce</span></span>
  resources</span>:</span></span>
    requests</span>:</span></span>
      storage</span>:</span> 100Mi</span></span></code></pre>
Then, we can define our restore:</p>
apiVersion</span>:</span> k8up.io/v1</span></span>
kind</span>:</span> Restore</span></span>
metadata</span>:</span></span>
  name</span>:</span> restore-test</span></span>
spec</span>:</span></span>
  restoreMethod</span>:</span></span>
    folder</span>:</span></span>
      claimName</span>:</span> restore-test</span></span>
  backend</span>:</span></span>
    repoPasswordSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
        key</span>:</span> secretKey</span></span>
    s3</span>:</span></span>
      endpoint</span>:</span> https://s3.us-west-2.amazonaws.com</span></span>
      bucket</span>:</span> bucket-name/test</span></span>
      accessKeyIDSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKeySecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span></code></pre>
This will create a new job, and checking the logs, we can see that it succeeded:</p>
$ kubectl logs -n backup-test jobs/restore-restore-test -f</span></span>
<...truncated...></span></span>
2026-05-03T20:04:04Z    INFO    k8up.restic.restic.restore.command      restic command  {"path": "/usr/local/bin/restic", "args": ["restore", "be8f37f009f0c910ad5ce8aa067d3dbd4050e7de9c41fde14394555a73adee06:/data/time-writer-pvc", "--target", "/restore"]}</span></span>
2026-05-03T20:04:04Z    INFO    k8up.restic.restic.restore.command      Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}</span></span>
2026-05-03T20:04:05Z    INFO    k8up.restic.restic.restore.restic.stdout        restoring snapshot be8f37f0 of [/data/time-writer-pvc] at 2026-05-03 19:45:35.978401924 +0000 UTC by @backup-test to /restore</span></span>
2026-05-03T20:04:05Z    INFO    k8up.restic.restic.restore.restic.stdout        Summary: Restored 1 files/dirs (3.144 KiB) in 0:00</span></span></code></pre>
Finally, we can check the PVC directly, and confirm we have data until 19:45:36.
The backup was started at 19:45:35 and finished by 19:45:37.
This gives us a very good RPO of a few seconds, which is great for my use case!</p>
$ kubectl exec -n backup-test -it pvc-inspect -- tail /data/time.log</span></span>
Sun May  3 19:45:27 UTC 2026</span></span>
Sun May  3 19:45:28 UTC 2026</span></span>
Sun May  3 19:45:29 UTC 2026</span></span>
Sun May  3 19:45:30 UTC 2026</span></span>
Sun May  3 19:45:31 UTC 2026</span></span>
Sun May  3 19:45:32 UTC 2026</span></span>
Sun May  3 19:45:33 UTC 2026</span></span>
Sun May  3 19:45:34 UTC 2026</span></span>
Sun May  3 19:45:35 UTC 2026</span></span>
Sun May  3 19:45:36 UTC 2026</span></span></code></pre>Scheduling#</span></a>
</h1>
The above example only covers a "one-time" backup.
For this to be useful, I need a regular schedule.
This is where the Schedule</code> comes in.</p>
I'll start with the example provided in the documentation</a>.
We'll back up every 5 minutes, and every hour we'll do checking and pruning.
Data retention can be tuned depending on the use case.</p>
apiVersion</span>:</span> k8up.io/v1</span></span>
kind</span>:</span> Schedule</span></span>
metadata</span>:</span></span>
  name</span>:</span> schedule-test</span></span>
spec</span>:</span></span>
  backend</span>:</span></span>
    repoPasswordSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-restic-secret-key</span></span>
        key</span>:</span> secretKey</span></span>
    s3</span>:</span></span>
      endpoint</span>:</span> https://s3.us-west-2.amazonaws.com</span></span>
      bucket</span>:</span> bucket-name/test</span></span>
      accessKeyIDSecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKeySecretRef</span>:</span></span>
        name</span>: {{</span> .Values.clusterName</span> }}</span>-s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span>
  backup</span>:</span></span>
    schedule</span>:</span> '*/5 * * * *'</span></span>
    failedJobsHistoryLimit</span>:</span> 2</span></span>
    successfulJobsHistoryLimit</span>:</span> 2</span></span>
  check</span>:</span></span>
    schedule</span>:</span> '0 * * * *'</span></span>
  prune</span>:</span></span>
    schedule</span>:</span> '30 * * * *'</span></span>
    retention</span>:</span></span>
      keepLast</span>:</span> 5</span></span>
      keepDaily</span>:</span> 7</span></span></code></pre>
This will result in one job every 5 minutes:</p>
$ kubectl get jobs -n backup-test</span></span>
NAME                                  STATUS     COMPLETIONS   DURATION   AGE</span></span>
backup-schedule-test-backup-7l7sj-0   Complete   1/1           10s        3m17s</span></span>
backup-schedule-test-backup-l7jkq-0   Complete   1/1           10s        8m17s</span></span></code></pre>
And for each job, the backup was successful:</p>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup       starting backup</span></span>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup       starting backup for folder      {"foldername": "time-writer-pvc"}</span></span>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup.command       restic command  {"path": "/usr/local/bin/restic", "args": ["backup", "--json", "--host", "backup-test", "/data/time-writer-pvc"]}</span></span>
2026-05-03T21:40:29Z    INFO    k8up.restic.restic.backup.command       Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}</span></span>
2026-05-03T21:40:31Z    INFO    k8up.restic.restic.backup.progress      backup finished {"new files": 0, "changed files": 1, "errors": 0}</span></span>
2026-05-03T21:40:31Z    INFO    k8up.restic.restic.backup.progress      stats   {"time": 1.727354075, "bytes added": 51912, "bytes processed": 50837}</span></span></code></pre>Conclusion#</span></a>
</h1>
Overall, k8up seems to meet my use case.
The setup is straightforward and easy to reason about.
The restore process is also straightforward.</p>
While the example here is very limited, it gives me confidence that the basics work.
The real test will come once it's operating on real data, but regular restores and testing should give me the confidence I need there.</p>


Self Hosted Postgres in Kubernetes with PITR Recovery
Caius Brindescu — Wed, 31 Dec 2025 00:00:00 +0000
Introduction#</span></a>
</h1>
I would like to selfhost a few services, and they require persistence, in the form of a PostgreSQL database.
Since my Kubernetes cluster runs on 4 Raspberry Pi 4, each with an SD card for disk, it's only a matter of time until one of them gets corrupted.
To avoid the innevitable and predictable data loss, I'll need automated backups, idealy with Point-In-Time Recovery (PITR).</p>
If I'm going to reinvent the wheel, why not go for the whole wheel?</p>
Solution#</span></a>
</h1>
CloudNativePG</a> seems to be most mature and feature complete out there.
It provides an operator that does a lot of the heavy lifting.
Backups are handled by the Barman Cloud</a> plugin.
It handles aspects like WAL Log archiving, taking regular snapshots and uploading them a cloud object storage, like S31</a></sup>.
Seems to meet all the needs I need for my project.</p>
Setup#</span></a>
</h1>
Setting up a Cloud Native PG database#</span></a>
</h2>
Using the instructions on their website</a>, we'll need to install the operator.
We can do it with the following command:</p>
$</span> kubectl apply</span> --server-side -f</span> https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.28/releases/cnpg-1.28.0.yaml</span></span></code></pre>
We can now create our database.
The Barman Cloud Plugin we'll be using later assumes that the DB is in the cnpg-system</code> namespace.
To simply this experiment, we'll work with that assumption, and we'll start by creating the namespace:</p>
$</span> kubectl create namespace cnpg-system</span></span></code></pre>
Then we can define our cluster configuration, with one user so we can connect to it later:</p>
apiVersion</span>:</span> v1</span></span>
type</span>:</span> kubernetes.io/basic-auth</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> postgres-secret</span></span>
  namespace</span>:</span> cnpg-system</span></span>
stringData</span>:</span></span>
  username</span>:</span> user1</span></span>
  password</span>:</span> supersecretpassword</span></span>
---</span></span>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> Cluster</span></span>
metadata</span>:</span></span>
  name</span>:</span> test</span></span>
  namespace</span>:</span> cpgn-system</span></span>
spec</span>:</span></span>
  instances</span>:</span> 1</span></span>
  storage</span>:</span></span>
    size</span>:</span> 1Gi</span></span></code></pre>
And when we apply this, we have a new cluster with 1 instance.
We can forward the port, and connect to it like we normally would to a Postgres instance running locally.</p>
$</span> kubectl port-forward</span> -n</span> cnpg-system service/test-restore-rw 5432:5432</span></span></code></pre>Backups#</span></a>
</h2>
Installing#</span></a>
</h3>
Now, we can configure backups for our new Postgres cluster.</p>
First, let's install the prerequisites that the Barman Cloud plugin requires.
We will need to install Certificate Manager2</a></sup>.</p>
$</span> kubectl apply</span> -f</span> https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml</span></span></code></pre>
Then, we can install the plugin:</p>
$</span> kubectl apply</span> -f</span> https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.9.0/manifest.yaml</span></span></code></pre>Configuration#</span></a>
</h3>
We can now define the backup configuration.
For this example, I'll use S3 as the storage target for the backups.</p>
First, we'll need to store a set AWS credentials with access to the right bucket3</a></sup>.
We'll use an Opaque Secret</a> to store the access and secret keys:</p>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> s3-credentials</span></span>
  namespace</span>:</span> cnpg-system</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  access-key-id</span>:</span> <redacted></span></span>
  secret-key-id</span>:</span> <redacted></span></span></code></pre>
Next, we'll define the storage configuration to our S3 Bucket:</p>
apiVersion</span>:</span> barmancloud.cnpg.io/v1</span></span>
kind</span>:</span> ObjectStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> s3-store</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  configuration</span>:</span></span>
    destinationPath</span>:</span> s3://<bucket>/postgres/</span></span>
    s3Credentials</span>:</span></span>
      accessKeyId</span>:</span></span>
        name</span>:</span> s3-credentials</span></span>
        key</span>:</span> access-key-id</span></span>
      secretAccessKey</span>:</span></span>
        name</span>:</span> s3-credentials</span></span>
        key</span>:</span> secret-key-id</span></span>
    wal</span>:</span></span>
      compression</span>:</span> gzip</span></span></code></pre>
Finally, we'll need to tell our Postgres cluster to use this configuration, and enable WAL archiving.
We'll add the following to the Cluster spec</code> field:</p>
plugins</span>:</span></span>
  -</span> name</span>:</span> barman-cloud.cloudnative-pg.io</span></span>
    isWALArchiver</span>:</span> true</span></span>
    parameters</span>:</span></span>
      barmanObjectName</span>:</span> s3-store</span></span></code></pre>
The final piece of the puzzle is setup regular "base" backups.
These will backup the entire dataset, and give us a "base" for the WAL logs to be applied to in order to get our Point-In-Time Restore.</p>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> ScheduledBackup</span></span>
metadata</span>:</span></span>
  name</span>:</span> pg-backup</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  cluster</span>:</span></span>
    name</span>:</span> test</span></span>
  schedule</span>:</span> '0 0 * * * *'</span> # hourly</span></span>
  backupOwnerReference</span>:</span> self</span></span>
  method</span>:</span> plugin</span></span>
  pluginConfiguration</span>:</span></span>
    name</span>:</span> barman-cloud.cloudnative-pg.io</span></span></code></pre>Results#</span></a>
</h3>
Now that everything is configured, we can list our S3 bucket, and sure enough, we have backups.
The top level structure has our base and WAL logs:</p>
$ aws s3 ls s3://<redacted>/postgres/test/</span></span>
                           PRE base/</span></span>
                           PRE wals/</span></span></code></pre>
Digging in deeper, we have our base backups, nicely named by timestamp, one every hour, as we'd expect:</p>
$ aws s3 ls s3://<redacted>/postgres/test/base/</span></span>
                           PRE 20251227T030000/</span></span>
                           PRE 20251227T040000/</span></span>
                           PRE 20251227T050000/</span></span>
                           PRE 20251227T060000/</span></span>
                           PRE 20251227T070000/</span></span>
                           ...</span></span></code></pre>
And finally, we have our WAL logs archived:</p>
$ aws s3 ls s3://<redacted>/postgres/test/wals/0000000100000000/</span></span>
2025-12-26 21:00:02      16944 000000010000000000000013.gz</span></span>
2025-12-26 21:00:04        210 000000010000000000000014.00000028.backup.gz</span></span>
2025-12-26 21:00:03      16513 000000010000000000000014.gz</span></span>
2025-12-26 21:05:03      17139 000000010000000000000015.gz</span></span>
2025-12-26 21:30:03      17276 000000010000000000000016.gz</span></span>
2025-12-26 22:00:02      16416 000000010000000000000017.gz</span></span>
2025-12-26 22:00:05        209 000000010000000000000018.00000028.backup.gz</span></span>
2025-12-26 22:00:04      16510 000000010000000000000018.gz</span></span>
...</span></span></code></pre>
So we should have all the pieces needed to perform a PITR.
We'll tacke this in the next section.</p>
Restoring#</span></a>
</h2>
For testing the restore, I've created a simple table, with timestamps for easy reasoning:</p>
CREATE TABLE</span> test</span>(</span></span>
  id </span>int</span>,</span></span>
  created_at </span>timestamp</span>(</span>6</span>)</span></span>
);</span></span></code></pre>
And we inserted different values, and this is the end state of the table:</p>
test=> select * from test;</span></span>
 id |         created_at         </span></span>
----+----------------------------</span></span>
 16 | 2025-12-27 03:00:17.601392</span></span>
 17 | 2025-12-27 03:00:20.438822</span></span>
 18 | 2025-12-27 03:28:06.64914</span></span>
 19 | 2025-12-27 20:42:25.563207</span></span>
 20 | 2025-12-27 20:42:32.992066</span></span>
 21 | 2025-12-27 20:45:58.634257</span></span>
(6 rows)</span></span></code></pre>
We'll restore the table to 2025-12-27, at 20:43:00 UTC.
For this, we'll create a new cluster, and we'll point it at the backups we have.
We'll also need to give it the target time we want the cluster restored to.
We arrive at this configuration:</p>
apiVersion</span>:</span> postgresql.cnpg.io/v1</span></span>
kind</span>:</span> Cluster</span></span>
metadata</span>:</span></span>
  name</span>:</span> test-restore</span></span>
  namespace</span>:</span> cnpg-system</span></span>
spec</span>:</span></span>
  instances</span>:</span> 1</span></span>
  imagePullPolicy</span>:</span> IfNotPresent</span></span>
  bootstrap</span>:</span></span>
    recovery</span>:</span></span>
      source</span>:</span> source</span></span>
  externalClusters</span>:</span></span>
  -</span> name</span>:</span> source</span></span>
    plugin</span>:</span></span>
      name</span>:</span> barman-cloud.cloudnative-pg.io</span></span>
      parameters</span>:</span></span>
        barmanObjectName</span>:</span> s3-store</span></span>
        serverName</span>:</span> test</span></span>
        targetTime</span>:</span> "2025-12-27T20:43:00Z"</span></span>
  storage</span>:</span></span>
    size</span>:</span> 1Gi</span></span></code></pre>
Once the restore is done (it was pretty much instant for this example), we con connect and check our test table:</p>
test=> select * from test;</span></span>
 id |         created_at         </span></span>
----+----------------------------</span></span>
 16 | 2025-12-27 03:00:17.601392</span></span>
 17 | 2025-12-27 03:00:20.438822</span></span>
 18 | 2025-12-27 03:28:06.64914</span></span>
 19 | 2025-12-27 20:42:25.563207</span></span>
 20 | 2025-12-27 20:42:32.992066</span></span>
(5 rows)</span></span></code></pre>
As we'd expect, we're missing row with id</code> 21, as it's after</em> the target restore time.</p>
Conclusions and Final Remarks#</span></a>
</h1>
All in all, this was an easy setup, and it works fine for at least the basic use case.
The true test is once this sees some "production" loads, and testing with an actual live data base.</p>
A note on S3 cots#</span></a>
</h2>
For this example, I used S3 as the backup target destination.
The amount of data stored is small, however, WAL archiving could end up writing a lot of objects.
This could incur significant S3 API charges, so it's something I'm keeping an eye on.
With hourly base backups, I didn't see any cost increases for my AWS account.
But inserting 21 records, and deleting a few is not exactly a representative use case, but at least the "baseline" cost is not absurd.</p>
Retention policies#</span></a>
</h2>
Barman has the option of specifying retention policies</a>.
However, for this experiment, I've gone with specifying a lifecycle policy on the S3 bucket.
Everything under postgres/</code> will be deleted after 7 days.
This will give me a one week recovery window.</p>
Using the Barman retention policy will cause the plugin to list S3, and then delete the objects.
This also will incur some S3 API charges, and I think that using the S3 lifecycle rule is probably good enough.</p>




The plugin also supports Google Cloud Storage, or Azure Blobs or some other services that implementation a compatible API</a>. ↩</a></p>
</li>

You can skip this step if your cluster already has it installed on the cluster. My test cluster did not. ↩</a></p>
</li>

For reference, I granted the user full access to the S3 bucket where the backups are stored. ↩</a></p>
</li>
</ol>
</section>