Playing with DigitalOcean Kubernetes
Just for fun, I’ve been working on migrating my personal infrastructure to run on top of Kubernetes. This post documents some of the hacks and tricks it took to do some less conventional things with it.
After poking around with the different Cloud k8s providers, I ended up deciding to go with Digitalocean just for ease of use and predictable billing.
Enabling SWAP on DigitalOcean Kubernetes
The first problem I ran into was that Kubernetes is very memory intensive. And at cloud provider prices, that gets expensive very quickly. So I wanted to enable swap. But it turns out this is an awful idea and very very not recommended. But, I’m a hobbyist so why not. Let’s hack around it.
The first problem is that Digitalocean k8s doesn’t provide any way of getting a shell on the actual node, which is where we’d want to enable swap. But, we can run privileged containers so it really isn’t that hard to escape to the actual node. I just followed along with this blog post to use nsenter to run commands on the host and from there set up SSH.
From there, we can then easily create some swap. By default, kubelet will refuse to use this swap partition (and will even refuse to start), so we have to add the --fail-swap-on=false flag to Kubernetes. DigitalOcean’s nodes manage kubelet through docker in a systemd service. So if we just open up /etc/systemd/system/kubelet.service we can add the flag to the end of the ExecStart command. Then just reloading systemd and restarting kubelet is enough to get it going with swap.
Wireguard VPN on Kubernetes
Next up, I wanted to migrate my VPN into Kubernetes. I’ve been using Wireguard deployed with Algo VPN but I figured it would be nice to move this server into the cluster also.
Using the SSH access I set up previously, I noticed that the nodes were still running Debian Stretch (9.12) which doesn’t have a new enough kernel for Wireguard to be included. So I had to start by installing the latest kernel I could get from the testing repos and then installing Wireguard on the host itself:
# Assumes Debian Stretch
echo "deb http://deb.debian.org/debian/ unstable main" > /etc/apt/sources.list.d/unstable-wireguard.list
printf 'Package: *\nPin: release a=unstable\nPin-Priority: 150\n' > /etc/apt/preferences.d/limit-unstable
echo "deb http://deb.debian.org/debian stretch-backports main" >> /etc/apt/sources.list
apt update
apt install -t stretch-backports linux-image-amd64 linux-headers-amd64
apt install -y wireguard-dkms wireguard-tools
After that, we can then actually start a pod to run Wireguard. Since DigitalOcean’s load balancer doesn’t support UDP load balancing, we can’t actually run Wireguard as a properly load balanced service. So instead I decided to just run it on a single node (pinned with an affinity) and expose it via a hostPort. So my final config for running Wireguard in DigitalOcean Kubernetes is:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
io.kompose.service: wireguard-claim0
name: wireguard-claim0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
status: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
io.kompose.service: wireguard
name: wireguard
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: wireguard
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: wireguard
spec:
# Expose the service directly on the host network
hostNetwork: true
# Pin to the node with `vpn=true`
nodeSelector:
vpn: "true"
containers:
- env:
- name: INTERNAL_SUBNET
value: 10.13.13.0
- name: PEERDNS
value: auto
- name: PEERS
value: "3"
- name: PGID
value: "1000"
- name: PUID
value: "1000"
- name: SERVERPORT
value: "51820"
- name: TZ
value: America/Los_Angeles
image: linuxserver/wireguard
imagePullPolicy: Always
name: wireguard
ports:
- containerPort: 51820
hostPort: 51820
protocol: UDP
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- SYS_MODULE
volumeMounts:
- mountPath: /config
name: wireguard-claim0
restartPolicy: Always
serviceAccountName: ""
volumes:
- name: wireguard-claim0
persistentVolumeClaim:
claimName: wireguard-claim0
status: {}
I used a PVC to hold the Wireguard config data and config files for peers.
Misc.
I also deployed a whole bunch of other services to the cluster, but none of them were too interesting. 3 nginx containers serving various static resources, 2 caddy containers, 2 python containers, postgres, and Datadog. All in all, I currently have 13 pods running in my Cluster on top of 2 nodes each with 4 GB of memory and 8GB of swap.
NAME READY STATUS RESTARTS AGE
blog-867bcf9d5-9t4m7 1/1 Running 2 5d4h
cascara-binaries-5f9955f99c-9vx7n 1/1 Running 0 16h
cascara-keys-7cff68c858-bs8zs 1/1 Running 0 5d
cascara-static-6d7ffbdd79-fcn44 1/1 Running 1 5d5h
datadog-agent-8qzv9 2/2 Running 2 5d21h
datadog-agent-bcrwd 2/2 Running 4 5d5h
datadog-agent-kube-state-metrics-66b476ff78-zv6p8 1/1 Running 1 5d21h
daviddworken-57d56745b9-lxzj9 1/1 Running 0 24h
monitor-f7664c764-4hlm9 1/1 Running 10 4d20h
nginx-ingress-controller-57897d87cd-tdkcf 1/1 Running 0 5d
nginx-ingress-default-backend-7c868597f4-k9blp 1/1 Running 1 6d2h
postgres-postgresql-0 1/1 Running 1 5d5h
wireguard-6957986dff-cjnp5 1/1 Running 0 18h
So far it has been running stably and with no issues. But I’ll have to wait and see how it goes once I need to upgrade my Kubernetes version. :)