Notes on Building a Raspberry Pi Kubernetes Cluster (Part 4: TLS, Ingress, and the Dashboard)

For the first post in this series, see Part 1: the Hardware.

While this wasn’t the first thing that I did with my cluster, it was the first thing that I wanted to do, and now having reset and reinstalled everything a few times, it’s become the first thing that I do when I start a fresh cluster.

I’m going to describe setting up the dashboard to serve on a domain using TLS. This has security risks, so think carefully before you do this, and while you might be comfortable with this for your home lab, please don’t do this for production clusters. (Use a cloud Kubernetes provider for those!)

You enable the dashboard by running microk8s enable dashboard. Use microk8s status to verify that it’s enabled. Then (optionally), if you want to temporarily set up the dashboard, follow the instructions in “Accessing the Dashboard” here (also linked above) to use kubectl proxy.

If you’re wise, you might stop here.

But if you want to serve your dashboard on a domain, you know that the first thing that you’ll need is… a domain. Get one from your favorite registrar, and then configure its DNS records with several subdomains, pointing all of them to the address where you’ll be running your cluster (My cluster is running at home behind a Google WiFi). Why so many subdomains? One will be for the dashboard, but you might want to set up your cluster to serve several different apps (websites, APIs, etc). It’s easiest to do that when you can assign each to its own subdomain, and it’s easiest to get all the certs for those at once.

Once you have your domains configured, and a machine running on that address, you can use LetsEncrypt to get SSL certificates. I used certbot, and my command-line invocation looked like this:

ubuntu@fury:~$ sudo certbot certonly --standalone
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Please enter in your domain name(s) (comma and/or space separated) (Enter ‘c’ to cancel): a.domain.me, b.domain.me, c.domain.me, d.domain.me, e.domain.me
Requesting a certificate for a.domain.me and 4 more domains
Performing the following challenges:
http-01 challenge for a.domain.me
http-01 challenge for b.domain.me
http-01 challenge for c.domain.me
http-01 challenge for d.domain.me
http-01 challenge for e.domain.me
Waiting for verification…
Cleaning up challenges
Subscribe to the EFF mailing list (email: me@domain.me).
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/a.domain.me/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/a.domain.me/privkey.pem
Your certificate will expire on 2021–05–28. To obtain a new or
tweaked version of this certificate in the future, simply run
certbot again. To non-interactively renew *all* of your
certificates, run “certbot renew”
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

Note that by running certbot standalone, I can’t have any other server running on port 80 of my host. (There are other ways to do this, but this is the easiest!)

After this, you’ll have a certificate and key for the subdomains that you requested with certbot. You’ll need to tell your Kubernetes installation about them, and you can do that by creating a TLS Secret. I did that using kubectl from the command-line, or actually the microk8s installation of kubectl:

microk8s kubectl create secret tls home-tls \
  --cert=fullchain.pem \
  --key=privkey.pem \
  --namespace=kube-system

This created a secret named home-tls (you can name yours whatever you want) using the cert and key files created by certbot. The --namespace argument defines the secret in the same namespace where the dashboard will run, which is necessary to set up the ingress (coming next).

You can verify that your secret was created using kubectl describe (your object sizes may vary):

microk8s kubectl describe secret --namespace kube-system home-tls
Name: home-tls
Namespace: kube-system
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.key: 1704 bytes
tls.crt: 3566 bytes

In Kubernetes, an ingress is a mapping that makes a service running inside your cluster available to the outside network. We’ll see later that it makes your service available on every node of your cluster, which means that you could set up a load balancer to distribute requests across your cluster and avoid calling failed nodes. But for now let’s just set it up by first enabling ingress with microk8s enable ingress and then using kubectl apply to create the ingress. Create a local file called system-ingress.yml with these contents:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: system-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_ssl_verify off;
spec:
  tls:
  - hosts:
      - d.domain.me
    secretName: home-tls
  rules:
  - host: d.domain.me
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port: 
              number: 443

Replace d.domain.me with the subdomain where you want to run your dashboard, and note that the secretName: value needs to match the name of the TLS secret that you created. You don’t need to change the namespace — just notice that it is kube-system, which should be the namespace of your secret and is also where the kubernetes-dashboard service is defined (this was created by microk8s enable dashboard).

Then when you’re ready to publish your dashboard, run kubectl apply -f system-ingress.yml. After that, you should be able to access your dashboard at your domain (using HTTPS).

If that didn’t work, don’t forget to make one of your cluster nodes available to the outside world with port forwarding rule on your router. I don’t think it matters which one you choose — as I mentioned earlier, the ingress seems to publish services on every node in your cluster. I checked this by using curl to call the dashboard service on every node in my cluster:

curl https://d.domain.me --resolve d.domain.me:443:fury     
curl https://d.domain.me --resolve d.domain.me:443:stark    
curl https://d.domain.me --resolve d.domain.me:443:parker   
curl https://d.domain.me --resolve d.domain.me:443:murdock  
curl https://d.domain.me --resolve d.domain.me:443:strange  
curl https://d.domain.me --resolve d.domain.me:443:banner   
curl https://d.domain.me --resolve d.domain.me:443:romanova 
curl https://d.domain.me --resolve d.domain.me:443:coulsen  
curl https://d.domain.me --resolve d.domain.me:443:vers     

Each request returned the same thing: the home page for the Kubernetes dashboard.

If you want to be safe, you can disable the dashboard by deleting the ingress:

kubectl delete ingress/system-ingress --namespace kube-system

Also, we haven’t said anything about how to log into the dashboard! Assuming you haven’t enabled role-based access controls (rbac), you’ll want to log in with a token that you get by running the following:

token=$(microk8s kubectl -n kube-system get secret \
 | grep default-token \
 | cut -d “ “ -f1)
microk8s kubectl -n kube-system describe secret $token

You’ll get a result like this:

Name: default-token-4hvl9
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: default
 kubernetes.io/service-account.uid: 9ed370e4–9591–4b4c-bf8f-0ceb4c9218d9
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1123 bytes
namespace: 11 bytes
token: VERY-LONG-STRING

Copy that VERY-LONG-STRING and paste it into the sign-in form of the dashboard and enjoy exploring!

Update: What to do when your certificate expires

LetsEncrypt certificates expire after three months. If you search online, you can find ways to configure your cluster to automatically update its certs, but manual updates are easy if you don’t mind taking your cluster offline for a few minutes.

Assuming you’re running on the system that serves your cluster’s external IP (your router port-forwards web requests to it), first stop your kubernetes services by running microk8s stop. Then rerun certbot with sudo certbot certonly --standalone and enter the domain names for the cert that you are renewing. When this finishes, you can restart kubernetes with microk8s start. Your regenerated certs will be in /etc/letsencrypt/live/a.domain.me (substitute your domain for a.domain.me). To update your secret, run kubectl create secret in “dry run” mode to generate a YAML file that you can apply with kubectl apply:

kubectl create secret tls home-tls \
  --namespace=kube-system \
  --cert=fullchain.pem --key=privkey.pem \
  --dry-run=client \
  -o yaml | kubectl apply -f -

You might need to reload your ingress (by deleting and recreating it). Verify your updated cert by using curl to fetch the dashboard root. If this works, try in your browser, and if necessary, clear the cached SSL certs in your browser.