Playing with DigitalOcean Kubernetes

Just for fun, I’ve been working on migrating my personal infrastructure to run on top of Kubernetes. This post documents some of the hacks and tricks it took to do some less conventional things with it.

After poking around with the different Cloud k8s providers, I ended up deciding to go with Digitalocean just for ease of use and predictable billing.

Enabling SWAP on DigitalOcean Kubernetes

The first problem I ran into was that Kubernetes is very memory intensive. And at cloud provider prices, that gets expensive very quickly. So I wanted to enable swap. But it turns out this is an awful idea and very very not recommended. But, I’m a hobbyist so why not. Let’s hack around it.

The first problem is that Digitalocean k8s doesn’t provide any way of getting a shell on the actual node, which is where we’d want to enable swap. But, we can run privileged containers so it really isn’t that hard to escape to the actual node. I just followed along with this blog post to use nsenter to run commands on the host and from there set up SSH.

From there, we can then easily create some swap. By default, kubelet will refuse to use this swap partition (and will even refuse to start), so we have to add the --fail-swap-on=false flag to Kubernetes. DigitalOcean’s nodes manage kubelet through docker in a systemd service. So if we just open up /etc/systemd/system/kubelet.service we can add the flag to the end of the ExecStart command. Then just reloading systemd and restarting kubelet is enough to get it going with swap.

Wireguard VPN on Kubernetes

Next up, I wanted to migrate my VPN into Kubernetes. I’ve been using Wireguard deployed with Algo VPN but I figured it would be nice to move this server into the cluster also.

Using the SSH access I set up previously, I noticed that the nodes were still running Debian Stretch (9.12) which doesn’t have a new enough kernel for Wireguard to be included. So I had to start by installing the latest kernel I could get from the testing repos and then installing Wireguard on the host itself:

# Assumes Debian Stretch 
echo "deb unstable main" > /etc/apt/sources.list.d/unstable-wireguard.list
printf 'Package: *\nPin: release a=unstable\nPin-Priority: 150\n' > /etc/apt/preferences.d/limit-unstable
echo "deb stretch-backports main" >> /etc/apt/sources.list
apt update
apt install -t stretch-backports linux-image-amd64 linux-headers-amd64
apt install -y wireguard-dkms wireguard-tools

After that, we can then actually start a pod to run Wireguard. Since DigitalOcean’s load balancer doesn’t support UDP load balancing, we can’t actually run Wireguard as a properly load balanced service. So instead I decided to just run it on a single node (pinned with an affinity) and expose it via a hostPort. So my final config for running Wireguard in DigitalOcean Kubernetes is:

apiVersion: v1
kind: PersistentVolumeClaim
  creationTimestamp: null
    io.kompose.service: wireguard-claim0
  name: wireguard-claim0
  - ReadWriteOnce
      storage: 1Gi
status: {}
apiVersion: apps/v1
kind: Deployment
  creationTimestamp: null
    io.kompose.service: wireguard
  name: wireguard
  replicas: 1
      io.kompose.service: wireguard
    type: Recreate
      creationTimestamp: null
        io.kompose.service: wireguard
      # Expose the service directly on the host network 
      hostNetwork: true 
      # Pin to the node with `vpn=true`
        vpn: "true"
      - env:
        - name: INTERNAL_SUBNET
        - name: PEERDNS
          value: auto
        - name: PEERS
          value: "3"
        - name: PGID
          value: "1000"
        - name: PUID
          value: "1000"
        - name: SERVERPORT
          value: "51820"
        - name: TZ
          value: America/Los_Angeles
        image: linuxserver/wireguard
        imagePullPolicy: Always
        name: wireguard
        - containerPort: 51820
          hostPort: 51820
          protocol: UDP
        resources: {}
            - NET_ADMIN
            - SYS_MODULE
        - mountPath: /config
          name: wireguard-claim0
      restartPolicy: Always
      serviceAccountName: ""
      - name: wireguard-claim0
          claimName: wireguard-claim0
status: {}

I used a PVC to hold the Wireguard config data and config files for peers.


I also deployed a whole bunch of other services to the cluster, but none of them were too interesting. 3 nginx containers serving various static resources, 2 caddy containers, 2 python containers, postgres, and Datadog. All in all, I currently have 13 pods running in my Cluster on top of 2 nodes each with 4 GB of memory and 8GB of swap.

NAME                                                READY   STATUS        RESTARTS   AGE
blog-867bcf9d5-9t4m7                                1/1     Running       2          5d4h
cascara-binaries-5f9955f99c-9vx7n                   1/1     Running       0          16h
cascara-keys-7cff68c858-bs8zs                       1/1     Running       0          5d
cascara-static-6d7ffbdd79-fcn44                     1/1     Running       1          5d5h
datadog-agent-8qzv9                                 2/2     Running       2          5d21h
datadog-agent-bcrwd                                 2/2     Running       4          5d5h
datadog-agent-kube-state-metrics-66b476ff78-zv6p8   1/1     Running       1          5d21h
daviddworken-57d56745b9-lxzj9                       1/1     Running       0          24h
monitor-f7664c764-4hlm9                             1/1     Running       10         4d20h
nginx-ingress-controller-57897d87cd-tdkcf           1/1     Running       0          5d
nginx-ingress-default-backend-7c868597f4-k9blp      1/1     Running       1          6d2h
postgres-postgresql-0                               1/1     Running       1          5d5h
wireguard-6957986dff-cjnp5                          1/1     Running       0          18h

So far it has been running stably and with no issues. But I’ll have to wait and see how it goes once I need to upgrade my Kubernetes version. :)