Introduction#

I have a Framework Desktop whose main purpose is to be a remote development machine. I'm currently using it with remote SSH access thorugh VS Code, and Docker devcontainers to get reproducible dev environments. And this works fine as is. However, I still need to have a laptop accessible that can run VS Code, and I'd like to able to run a quick development session from everywhere, regardlesss of what hardware I have on hand. I can't run this setup from an iPad, or a browser, or some other machine, which would lighten my travel "tech stack" considerably.

This is where Coder comes in. It will allow me to run my development setup from any browser, and still keep my current VSCode worflow when working from a laptop. It also gives me the flexibility you get from GitHub's Codespaces, without having yet another service I need to pay for.

Existing Setup#

The desktop runs K3s as a single node cluster and it already runs a couple of services I use in my daily life. As part of these services, it already has a Postgres instance, that has backups configured¹. The goal is to use this Postgres instance for persistence, so I have one less piece of infrastructure I need to worry about.

Finally, I use helm to manage the services running on this "cluster," so I have a good base set up.

Coder setup#

The first step is to set up Coder itself. It already comes with a Helm chart. For my setup, I'll wrap it in my own chart, so I can configure it,

apiVersion: v2
name: coder
version: 0.1.0
dependencies:
  - name: coder
    version: "2.*"
    repository: https://helm.coder.com/v2

And I will need to define a couple of resources, in order to hook it up into the existing infrastructure.

First, a secret for the DB connection string, that will contain the username and password:

apiVersion: v1
kind: Secret
metadata:
  name: coder-db-secret
  namespace: {{ .Release.Namespace }}
type: Opaque
stringData:
  url: {{ .Values.dbUrl | quote }}

Secondly, I need an ingress controller, so I can access it from my custom URL:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: coder
  namespace: {{ .Release.Namespace }}
  annotations:
    cert-manager.io/cluster-issuer: {{ .Values.clusterIssuer }}
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - {{ .Values.ingress.host }}
        - {{ .Values.ingress.wildcardHost | quote }}
      secretName: coder-tls
  rules:
    - host: {{ .Values.ingress.host }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: coder
                port:
                  number: 80
    - host: {{ .Values.ingress.wildcardHost | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: coder
                port:
                  number: 80

Like all my other services, this will use Let's Encrypt and DNS verification to generate valid TLS certficate. The deployment will only be available in my tailnet, but having a valid certificate makes everything a lot easier. I don't get annoying browser warnings, and it's one step taken care of if I ever want to make the service publicly accessible.

Finally, I need to define the values for our deployment.

The details for the ingress are defined below. It's the cluster issuer (defined elsewhere in my helm charts), and the domains that we want to serve coder from.

clusterIssuer: lets-encrypt-prod

ingress:
  host: coder.caius.dev
  wildcardHost: "*.coder.caius.dev"

For the DB secret, I've defined the dbUrl inside a separate secrets.yaml file that is not added to source control. The format of this is:

dbUrl: "<user>@<password>:<db_host>:5432"

Next, I'll wire up the secret DB access string, the access URL, and disable coder's builtin ingress, as we are using our own. We'll also need to disable the cluster access URL. Without this, things like the CLI install script will try and use the Kubernetes internal hostname, and that's not resolvable from outside the cluster.

coder:
  coder:
    envUseClusterAccessURL: false

    env:
      - name: CODER_PG_CONNECTION_URL
        valueFrom:
          secretKeyRef:
            name: coder-db-secret
            key: url
      - name: CODER_ACCESS_URL
        value: https://coder.caius.dev
      - name: CODER_WILDCARD_ACCESS_URL
        value: "*.coder.caius.dev"

    ingress:
      enable: false

Finally, let's set up some resource limits:

    resources:
      requests:
        cpu: 250m
        memory: 512Mi
      limits:
        cpu: "2"
        memory: 2Gi

Applying this will bring up the deployment. It takes a few minutes for the cert to be provisioned, but once that was done, I was able to log in and everything worked!

Workspaces namespace#

For the workspaces themselves, I decided to use a separate namespace, coder-workspaces. For this, I created a simple namespace resource using helm:

apiVersion: v1
kind: Namespace
metadata:
  name: coder-workspaces
  labels:
    pod-security.kubernetes.io/enforce: privileged

We also need an RBAC authorization, so the coder service account can provision the resources we need when creating a new workspace in the coder-workspace namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: coder-workspaces
  namespace: coder-workspaces
rules:
- apiGroups: [""]
  resources:
  - pods
  - pods/exec
  - pods/log
  - persistentvolumeclaims
  - secrets
  - services
  - serviceaccounts
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources:
  - deployments
  - replicasets
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: coder-workspaces
  namespace: coder-workspaces
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: coder-workspaces
subjects:
- kind: ServiceAccount
  name: coder
  namespace: coder

Workspace Template#

For the template, I started off with the default kubernetes-devcontainer template. However there were a couple of things that needed to be addressed.

First, and the most obvious, was changing the namespace from default to coder-workspaces. This was a single line change.

I also wanted to use SSH keys for repo authentication, as I didn't want to install the coder app in GitHub. Finally, there were are few quality of life issues, like the directory that code server (and VS Code) opened in, as well as always trusting GitHub's SSH keys. I will go into detail for each of them in the following sections.

Using SSH keys for git repo authentication#

To access my private repos, I wanted to use SSH authentication. I didn't want to install the coder github app, as I didn't really know what info it would syphon out. Coder already defines a key pair that can be used for this purpose, one for each user. However, I still need to inject the private key inside the container. To accomplish this, I created a kubernetes secret with the private key, and then used that to inject it into the container.

resource "kubernetes_secret_v1" "ssh_key" {
  metadata {
    name      = "coder-${lower(data.coder_workspace.me.id)}-ssh-key"
    namespace = var.namespace
  }
  data = {
    id_rsa = data.coder_workspace_owner.me.ssh_private_key
  }
}

I then set the correct enviroment variable, so the containers knows where to get the private key from:

"ENVBUILDER_GIT_SSH_PRIVATE_KEY_PATH" : "/tmp/ssh/id_rsa",

Finally, I need to mount the newly created secret at the right path inside the container. I need to tell the container where to mount it to:

volume_mount {
    mount_path = "/tmp/ssh"
    name       = "ssh-key"
    read_only  = true
}

And I'll need to configure the deployment to create the volume, with the secret contents:

volume {
    name = "ssh-key"
    secret {
        secret_name  = kubernetes_secret_v1.ssh_key.metadata[0].name
        default_mode = "0400"
    }
}

I can now allow access to coder's public key, and use SSH to authenticate with GitHub, or any other forge.

With this setup, I have the basic dev workflow sorted out. I can create a new workspace, and it will clone the repo, and spin up the devcontainer.

Mounting the correct path in the editor#

The default config will mount /workspaces as the "root" path for the editor. This isn't ideal, as I'm used to working in the repo root. Some of the VS code tasks also assume this, and will fail otherwise. Luckily, I can fix this by tweaking the templates a bit.

First, I need to define a local variable that's the repo name. This can be easily inferred from the repo url:

repo_dir = "/workspaces/${replace(basename(local.repo_url), ".git", "")}"

I'll then assign this to the folder parameter in our code_server module:

folder = local.repo_dir

The coder agent will, by default, expose a VS Code Desktop connection option. However, this will always point to "home", which in this case it's /workspaces, and this is what we're trying to avoid. The easier solution around this, was to disable it in the agent definition, by adding the following to the terraform config:

display_apps {
  vscode = false
}

Then, I can define an explicit vscode-desktop module that will open the correct folder.

module "vscode" {                                                                                
  count   = data.coder_workspace.me.start_count                                                  
  source  = "registry.coder.com/coder/vscode-desktop/coder"                                              
  version = "~> 1.0"                                                                             
                                                                                                 
  agent_id   = coder_agent.main.id                                                               
  folder     = local.repo_dir                                                                    
  order      = 2                                                                                 
}

Automatically installing extensions#

The extensions specified in the devcontainer.json file are not automatically installed when I start up the code server, or the desktop VS Code. While it is possible to specify extensions to install in the terraform templates, I wanted a generic solution, that would install only the required extensions for each specific repo. The solution is a script that runs at startup, parses the devcontainer.json file, extracts the list of extensions and installs them. After the install, it generates a .vscode/extensions.json file, so that any remote VS Code desktop instances will pick them up, and suggest that they are installed².

The final result is the following coder_script resource:

resource "coder_script" "devcontainer_extensions" {
  count              = data.coder_workspace.me.start_count
  agent_id           = coder_agent.main.id
  display_name       = "Install devcontainer extensions"
  run_on_start       = true
  start_blocks_login = true
  script             = <<-EOT
    #!/bin/bash
    REPO_DIR="${local.repo_dir}"

    DEVCONTAINER_JSON=""
    for candidate in "$REPO_DIR/.devcontainer/devcontainer.json" "$REPO_DIR/devcontainer.json"; do
      if [ -f "$candidate" ]; then
        DEVCONTAINER_JSON="$candidate"
        break
      fi
    done
    [ -z "$DEVCONTAINER_JSON" ] && exit 0

    until command -v code-server &>/dev/null; do sleep 1; done

    command -v jq &>/dev/null || { echo "jq not found, skipping extension install"; exit 0; }
    EXTENSIONS=$(jq -r '.customizations?.vscode?.extensions[]?' "$DEVCONTAINER_JSON" 2>/dev/null || true)
    [ -z "$EXTENSIONS" ] && exit 0

    echo "$EXTENSIONS" | while IFS= read -r ext; do
      [ -z "$ext" ] && continue
      code-server --install-extension "$ext" 2>&1 || true
    done

    # Write .vscode/extensions.json so VS Code Desktop prompts to install the same extensions
    VSCODE_EXTS_FILE="$REPO_DIR/.vscode/extensions.json"
    if [ ! -f "$VSCODE_EXTS_FILE" ]; then
      mkdir -p "$REPO_DIR/.vscode"
      jq -n --argjson exts \
        "$(jq -c '[.customizations?.vscode?.extensions[]?]' "$DEVCONTAINER_JSON")" \
        '{"recommendations": $exts}' > "$VSCODE_EXTS_FILE"
    fi
  EOT
}

Trusting GitHub's public keys#

The last nagging problem was having to trust GitHub's keys after I restarted a workspace, during the first interaction with git. To fix this, I added another script that will trust GitHub's keys, so it always restarts in a "good" state. The scripts will run in parallel, so this adds nothing to the startup time.

resource "coder_script" "github_known_hosts" {
  count              = data.coder_workspace.me.start_count
  agent_id           = coder_agent.main.id
  display_name       = "Trust github hosts"
  run_on_start       = true
  start_blocks_login = true
  script             = <<-EOT
mkdir -p /root/.ssh/
cat <<'EOF' >> /root/.ssh/known_hosts
|1|YcSxwNwqHecLiBTEXmhb3JcKMbg=|LDlb4p4OoJafZDnkMspTODpHdlY= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
|1|qf+ciiHWdVTQzkK389LmH/RM/Wo=|UaXlxsvDYssHCiqHv9q+aHyf01w= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
|1|nXxxrVNnRIHD8UKpOVlDGR9oYX4=|+FflYHXVajJijO9egAkU5wPQ+Ac= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
EOF
echo "Done"
EOT
}

Conclusions#

The templates I arrived at are available in this GitHub repo.

The setup works well, and this blog post was written using an iPad and code server. The goal was to have more flexibility when travelling, to avoid having to carry too many devices. And I think, so far, I've achieved this goal.

The setup uses the standard devcontainers I've developed, without requiring any custom modfications to be made. This should make it easy to adapt in any future project, or start up new workspaces as I realize I need to make changes to other projects.