Software Installations: Install MicroK8s on MacOS or Ubuntu

MacOS v10.15.4 (Catalina)

Ubuntu 20.04.4 (Focal)
MicroK8s v1.24

Goals:

Install and run a Kubernetes Cluster on MacOS or on Ubuntu using MicroK8s.

Install:

Before installing Microk8s:

Make sure the machine's hostname does not contain capital letters and/or underscores. This is not a valid name for a Kubernetes node, causing node registration to fail.
Make sure you have the host network configured. I mean, try to ping some site on the internet (e.g. www.google.com), ping the gateway and ping the dns configured to the machine.

MacOS:

Download and install a version of Multipass, a VM system for running Ubuntu and other packages required by MicroK8s:
Open a terminal window:

brew install ubuntu/microk8s/microk8s
microk8s install

Ubuntu:

Open a terminal window to install MicroK8s. The --channel parameter is optional.

sudo apt update
sudo snap install microk8s --classic --channel=1.24/stable

Change permissions to run MicroK8s without sudo:

sudo usermod -aG microk8s $USER
sudo chown -f -R $USER ~/.kube
sudo shutdown -r now

[Optional] You may need to configure the Ubuntu firewall to allow pod-to-pod and pod-to-internet communications:

sudo ufw allow in on cni0
sudo ufw allow out on cni0
sudo ufw default allow routed

The following steps are the same for both OS

Check the MicroK8s status:

microk8s status --wait-ready

Enable MicroK8s Addons:

microk8s enable rbac dns ha-cluster

Wait until you see that the new addons we just installed are already enabled:

microk8s status --wait-ready

Install and configure a default kubernetes Storage Class:

So far, we have not yet installed a Storage Class for our MicroK8s installation.
Depending on the installation method, your kubernetes cluster may be deployed with an existing StorageClass that is marked as default. This default StorageClass is then used to dynamically provision storage for PersistentVolumeClaims that do not require any specific storage class. However, the pre-installed default StorageClass may not fit well with your expected workload.
To see the Storage Classes available, run the command:

kubectl get storageclasses

Option 1 - The MicroK8s default storage class:

Pros: Less cpu and memory resources are used by kubernetes;
Cons: We can't include these persistent volumes in the Velero Backup.
How to install:

microk8s enable hostpath-storage

Option 2 - The Openebs storage classes (recommended):

Pros: We can include these persistent volumes in the Velero Backup;
Cons: More cpu and memory resources will be used by kubernetes.
How to install:

[Optional] sudo apt install open-iscsi
sudo systemctl enable iscsid
microk8s enable openebs

Mark a StorageClass as default:

microk8s kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

[Optional] Enable Kubernetes Dashboard:

microk8s enable dashboard

Again, wait until you see that the dashboard addon we just installed is already enabled:

microk8s status --wait-ready

Assuming that Role-Based Access Control (RBAC) is enabled in your microk8s installation, we need to create an Administrative Service Account.

Create the administrative service account in the kube-system namespace:
microk8s kubectl -n kube-system create serviceaccount admin-user

Grant permissions to administrative service account:
microk8s kubectl create clusterrolebinding --clusterrole=cluster-admin --serviceaccount=kube-system:admin-user admin-user-rolebinding

Create the access token:
microk8s kubectl -n kube-system create token admin-user

We can access the Dashboard using its Cluster IP address:
The kubernetes-dashboard service in the kube-system namespace has a Cluster IP. To get the Cluster IP and port of kubernetes-dashboard we need to run the command below:
microk8s kubectl get endpoints -A | grep kubernetes-dashboard

Point your browser to https:/<kubernetes-dashboard-endpoint-ip>:<port>

At this point you can access kubernetes Dashboard only on your cluster host machine. That's because your cluster host machine can reach the cluster internal IPs.

However, if we need to access the Dashboard from another machine we have two options to do that. We can expose the Dashboard as an external service or we can configure Kubernetes Ingress to redirect external requests to the internal service.

Option 1 - Expose Dashboard as an external Service:

In order to expose Dashboard to external access we need to do some extra configuration. One downside of the approach is that you have to use Firefox browser to access the Dashboard.

microk8s kubectl -n kube-system patch service kubernetes-dashboard -p '{"spec":{"type":"NodePort","ports":[{"port":443,"nodePort":31234}]}}'

After this command, the Kubernetes Dashboard is exposed in your microk8s installation on port 31234. Note that Service NodePorts can only be in a "special" port range, 30000~32767 by default.

Point your Firefox browser to https://<kubernetes-host-machine>:31234

Option 2 - Configure Kubernetes Ingress Addon:

microk8s enable ingress

Once again, wait until you see that the Ingress addon we just installed is already enabled:
microk8s status --wait-ready

Create a file (eg. ingress-dashboard-config.yml) with the content below. Remember to use the right port for the Dashboard in the content below (at the last line of the file), the port number we got from `microk8s kubectl get endpoints` command above:

microk8s kubectl apply -f ingress-dashboard-config.yml

Point your browser (anyone) to https://<kubernetes-host-machine>/dashboard/

And finally we can login to Dashboard using the token created for the admin-user:

[Optional] Enable MicroK8s built-in insecure registry. The registry is hosted within the Kubernetes cluster and is exposed as a NodePort service on port 32000 of the localhost. The size of the registry should be >= 20Gi:

microk8s enable registry:size=20Gi

Enabling the private registry
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
The registry is enabled
The size of the persistent volume is 20Gi

To disable the built-in registry:

microk8s disable registry
microk8s disable storage:destroy-storage

Pushing to this insecure registry may fail in some versions of Docker unless the daemon is explicitly configured to trust this registry. To address this we need to open Docker Desktop -> Preferences -> Docker Engine -> then add the follow configuration:

sudo nano /etc/docker/daemon.json

{ "insecure-registries" : ["localhost:32000"] }

Click on `Apply & Restart` button

[Optional] Use a custom network domain for Kubernetes, other than cluster.local

Edit the CoreDNS configmap:

microk8s kubectl -n kube-system edit configmap coredns

Add this line below to the CoreDNS configmap:

rewrite name substring my-custom-domain.com cluster.local

Restart the CoreDNS service, or restart the MicroK8s.

Check the MicroK8s installation:

microk8s inspect

Important: Pay attention to possible `configuration needed` messages.

microk8s config
microk8s kubectl cluster-info
microk8s kubectl version

microk8s kubectl get nodes

Check the status of the node. It must be "Ready".

Create an alias for kubectl:

sudo snap alias microk8s.kubectl kubectl
sudo snap alias microk8s.kubectl k8s

Ubuntu only:

Install Docker:

sudo snap install docker
Change permissions to run Docker without sudo:

sudo groupadd docker
sudo usermod -aG docker $USER

Or, in case you executed docker using sudo before:

sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
sudo chmod g+rwx "$HOME/.docker" -R

sudo chown root:docker /var/run/docker.sock
docker version
docker-compose version
sudo reboot

Tests:

Check the Kubernetes DNS service:

microk8s kubectl get endpoints
microk8s kubectl -n kube-system get pod
microk8s kubectl -n kube-system get svc kube-dns
microk8s kubectl -n kube-system get endpoints kube-dns

Create a simple Pod to use as a DNS test environment:

microk8s kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
microk8s kubectl exec -i -t dnsutils -- nslookup kubernetes.default
microk8s kubectl exec -i -t dnsutils -- nslookup www.google.com

If we have the SERVFAIL in the response of the last command, then we need to reconfigure the Kubernetes DNS:

Check the local DNS configuration first. Take a look inside the resolv.conf file. (See Customizing DNS Service and Known issues below for more information)

microk8s kubectl exec -ti dnsutils -- cat /etc/resolv.conf

Troubleshooting - Check for errors on the DNS pod:

microk8s kubectl logs --namespace=kube-system -l k8s-app=kube-dns

Check definition of CoreDNS:

microk8s kubectl get configmap -n kube-system coredns -o yaml

Fetch your actual DNS using the command:

nmcli dev show 2>/dev/null | grep DNS | sed 's/^.*:\s*//'

Change forward address in CoreDNS config map from default (8.8.8.8 8.8.4.4) to your actual DNS:

microk8s kubectl -n kube-system edit configmap coredns

Change the following line, on the `Corefile` session, from this (using the <i> key):

forward . 8.8.8.8 8.8.4.4

To this:

forward . your.dns.ips.here.separated.by.space

And save (using <Esc> and <:> <x> <Enter> keys)

After saving the changes, it may take up to a minute or two for Kubernetes to propagate these changes to the CoreDNS pods
Now test the DNS resolution again:

microk8s kubectl exec -i -t dnsutils -- nslookup www.google.com

When the DNS is running:

If we have for example a service called my-service running on a namespace called my-namespace, and the domain name for our cluster is cluster.local, then the service can be accessed with the address:

my-service.my-namespace.svc.cluster.local

If a Pod in the default namespace has the IP address 172.17.0.3, and the domain name for our cluster is cluster.local, then the Pod has a DNS name:
- 172-17-0-3.default.pod.cluster.local

Deploy an application:

microk8s kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

deployment.apps/kubernetes-bootcamp created

See the application Pod status

microk8s kubectl get pods

NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-6f6656d949-z82zm 1/1 Running 0 1m2s

microk8s kubectl describe pod/<pod-id>

...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m15s default-scheduler Successfully assigned default/kubernetes-bootcamp-6f6656d949-z82zm to microk8s-vm
Normal Pulling 2m13s kubelet, microk8s-vm Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"
Normal Pulled 108s kubelet, microk8s-vm Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1"
Normal Created 108s kubelet, microk8s-vm Created container kubernetes-bootcamp
Normal Started 108s kubelet, microk8s-vm Started container kubernetes-bootcamp

Delete the deployment:

microk8s kubectl delete deployment kubernetes-bootcamp

[Attention - Don't do this if you installed the Kubernetes Ingress Addon] Deploy NGINX server:

microk8s kubectl create deployment nginx --image=nginx
microk8s kubectl get deployments
microk8s kubectl describe deployment nginx
microk8s kubectl create service nodeport nginx --tcp=80:80
microk8s kubectl get svc

service/nginx NodePort 10.152.183.133 <none> 80:30618/TCP 7s

Open the nginx web page on a browser or using curl / wget:

http://<kubernetes-host-machine>:<nginx-service-port>

Delete the NGINX deployment

microk8s kubectl delete deployment nginx

Configure Access to Multiple Clusters (Work-in-progress):

Kubeconfig file:

echo $KUBECONFIG
Use multiple kubeconfig files at the same time and view merged config

export KUBECONFIG=~/.kube/config:~/.kube/config2

Show merged kubeconfig settings:

microk8s kubectl config view
microk8s config
microk8s kubectl config view --raw

Display list of contexts:

microk8s kubectl config get-contexts

kubectl config get-contexts:

microk8s kubectl config current-context

Get a list of users:

microk8s kubectl config view -o jsonpath='{.users[*].name}'

To add a new cluster, we need to add a user to the cluster that supports basic auth, that will be used when connecting the kubeconf:

microk8s kubectl config set-credentials cluster-admin --username=admin --password=<password>

Set the default context to <context-name>:

microk8s kubectl config use-context <context-name>

Set a context utilizing a specific username and namespace:

Add another Node to the MicroK8s Cluster:

The MicroK8s instance on which this command is run will be the master of the cluster and will host the Kubernetes control plane:

microk8s add-node

Join node with: microk8s join 192.168.64.2:25000/19c8a4677a2f03ea738749e9baecec88

The `add-node` command prints a microk8s join command which should be executed on the MicroK8s instance that you wish to join to the cluster:

microk8s join 192.168.64.2:25000/19c8a4677a2f03ea738749e9baecec88

Joining a node to the cluster should only take a few seconds. Afterwards you should be able to see the node has joined running the above command on master:

microk8s kubectl get nodes

NAME STATUS ROLES AGE VERSION
192.168.1.110 Ready <none> 35s v1.18.2-41+4706dd1a7d2b25
microk8s-vm Ready <none> 2d2h v1.18.2-41+b5cdb79a4060a3

Running the `get nodes` command on secondary (leaf) node will get the message:

This MicroK8s deployment is acting as a node in a cluster. Please use the microk8s kubectl on the master.

Notes:

The pods already running on the secondary (leaf) node will be stopped and will not be started anywhere, I mean, not on the leaf node anymore and also not on the master.
The Kubernetes Config file is saved by default at /var/snap/microk8s/current/credentials/client.config file on the master node.
How to show the default port used by the cluster:

cat /var/snap/microk8s/current/credentials/client.config | grep server

server: https://127.0.0.1:16443

ToDo: Configure k8s command to access the remote Master.

To remove a node from the cluster, we have two steps. First, use the command bellow on Master:

microk8s remove-node <node-name>
Then use the following command on Node:

microk8s leave

Stopped.
Started.
Enabling pod scheduling
node/ubuntu already uncordoned

microk8s kubectl get nodes

See other important commands at Kubernetes Knowledge Base

Clean-up

Remove Microk8s on Ubuntu:

sudo snap remove microk8s
sudo snap saved
sudo snap forget <snapshot-set-id>

Remove Microk8s on macOS:

brew uninstall ubuntu/microk8s/microk8s

References:

Wednesday, May 27, 2020

Install MicroK8s on MacOS or Ubuntu