Kubernetes 1.26.3 Install On Ubuntu Jammy | Fredeirck's Personal Blog

install kubernetes 1.26.3 on ubuntu jammy.

Prepare the environment

upgrade your Ubuntu servers

install HWE kernel

sudo apt install linux-generic-hwe-22.04

or just full upgrade all package and kernel

sudo apt update
sudo apt -y full-upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f

synchronize time to timezone, mine is UTC+8

sudo apt install ntpdate \
  && sudo ntpdate cn.pool.ntp.org \
  && sudo hwclock --systohc \
  && sudo cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

if is work node, add master node DNS record

$ vim /etc/hosts

192.168.1.102 private.k8sapi.finuks

# test it 
ping private.k8sapi.finuks

Install kubelet, kubeadm and kubectl

1、 Kubernetes Community-Owned Package Repositories
Guide Links

sudo apt install curl apt-transport-https -y

# Make sure to replace the Kubernetes minor version in the command below with the minor version that you're currently using, example I use 1.26:
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.26/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories, so you can disregard the version in the URL:
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.26/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

Update: In releases older than Debian 12 and Ubuntu 22.04, the folder /etc/apt/keyrings does not exist by default, and it should be created before the curl command.

(optional) install the Kubernetes package repository (aliyun)

sudo apt -y install curl apt-transport-https
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

2、Then install required packages.

sudo apt update
sudo apt install wget curl vim git kubelet=1.26.3-00 kubeadm=1.26.3-00 kubectl=1.26.3-00

# use `hold` preventing it from being upgraded
sudo apt-mark hold kubelet kubeadm kubectl
sudo apt-mark unhold <package-name>

3、Confirm installation by checking the version of kubectl.

finuks@k8s-master:~$ kubectl version --client && kubeadm version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
kubeadm version: &version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:38:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

4、(option) delete the k8s update repository.

Disable Swap Space

Disable all swaps from /proc/swaps.

sudo swapoff -a

# Check if swap has been disabled by running the free command.
free -h

Now disable Linux swap space permanently in /etc/fstab. Search for a swap line and add # (hashtag) sign in front of the line.

$ sudo vim /etc/fstab
#/swap.img	none	swap	sw	0	0

Confirm setting is correct

sudo mount -a
free -h

Enable kernel modules and configure sysctl.

# Enable kernel modules
sudo modprobe overlay
sudo modprobe br_netfilter

# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload sysctl
sudo sysctl --system

Install Container runtime (Master and Worker nodes)

To run containers in Pods, Kubernetes uses a container runtime. Supported container runtimes are:

Docker
CRI-O
Containerd

NOTE: You have to choose one runtime at a time.

Initialize control plane (run on first master node)

finuks@k8s-master:~$ sudo lsmod | grep br_netfilter
br_netfilter           32768  0
bridge                331776  1 br_netfilter

Enable kubelet service.

sudo systemctl enable kubelet

We now want to initialize the machine that will run the control plane components which includes etcd (the cluster database) and the API Server.

If you have multiple CRI sockets, please use –cri-socket to select one:

# CRI-O
sudo kubeadm config images pull --cri-socket /var/run/crio/crio.sock

# Containerd
sudo kubeadm config images pull --cri-socket /run/containerd/containerd.sock

# Docker
sudo kubeadm config images pull --cri-socket /run/cri-dockerd.sock

Pull container images:

finuks@k8s-master:/etc/kubernetes$ sudo kubeadm config images pull \
--kubernetes-version=1.26.3 \
--cri-socket unix:///run/cri-dockerd.sock \
--image-repository registry.aliyuncs.com/google_containers
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.26.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.26.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.26.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.26.3
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.6-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.9.3

[!!!]Must check have images below before execute kubeadm init:

finuks@k8s-master:~$ docker images
REPOSITORY                                                        TAG       IMAGE ID       CREATED         SIZE
registry.aliyuncs.com/google_containers/kube-apiserver            v1.26.3   1d9b3cbae03c   3 weeks ago     134MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.26.3   5a7904736932   3 weeks ago     56.4MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.26.3   ce8c2293ef09   3 weeks ago     123MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.26.3   92ed2bec97a6   3 weeks ago     65.6MB
registry.aliyuncs.com/google_containers/etcd                      3.5.6-0   fce326961ae2   4 months ago    299MB
registry.aliyuncs.com/google_containers/pause                     3.9       e6f181688397   5 months ago    744kB
registry.aliyuncs.com/google_containers/coredns                   v1.9.3    5185b96f0bec   10 months ago   48.8MB
registry.k8s.io/pause                                             3.6       6270bb605e12   19 months ago   683kB

Note: the registry.k8s.io/pause:3.6 not include in kubeadm config images pull,please download it, or api-server will not start to run.

Bootstrap with shared endpoint (DNS name for control plane API)

sudo kubeadm init \
	--kubernetes-version=1.26.3 \
	--image-repository registry.aliyuncs.com/google_containers \
	--cri-socket unix:///run/cri-dockerd.sock \
	--upload-certs \
	--service-cidr=10.96.0.0/12 \
	--pod-network-cidr=10.244.0.0/16 \
	--apiserver-advertise-address=192.168.1.102 \
	--control-plane-endpoint=private.k8sapi.finuks

kubernetes-version: prevent version check from internet
image-repository: because host server in China, use china registry, GFW reason, you know.
cri-socket: I have multiple socket, so specific one.
upload-certs: upload the certificates to a Kubernetes secret, then distributed to worker nodes.By default, these certificates are stored locally on the master node.
service-cidr: specify the IP address range that Kubernetes will use for service IP addresses. By default, Kubernetes uses the IP address range 10.96.0.0/12 for service IP addresses.
pod-network-cidr: specify the IP address range that Kubernetes will use for pod IP addresses. By default, Kubernetes uses the IP address range 10.244.0.0/16 for pod IP addresses.
apiserver-advertise-address: must be ipv4 or ipv6 address. should be set to the IP address of the load balancer, rather than the individual nodes for HA. tell API server to advertise its address to other nodes in the cluster. ensure worker nodes can connect to the API server.
control-plane-endpoint: tell control plane to use IP/DNS as the primary endpoint for API requests. while have multiple control plane nodes, configure to a load balancer or DNS server for HA.

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

NOTE: remember to record below join info to a text:

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join private.k8sapi.finuks:6443 --token eb51l2.clhcckbw8ev5xoa3 \
        --discovery-token-ca-cert-hash sha256:27c5e0b6c38f79ab479736af4294bd327c1ae223467b85474313ac542f50d700 \
        --control-plane --certificate-key f62f15aeb1de503e7788cc04e188503759520a72737435aed2bed4ff282e69af

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join private.k8sapi.finuks:6443 --token eb51l2.clhcckbw8ev5xoa3 \
        --discovery-token-ca-cert-hash sha256:27c5e0b6c38f79ab479736af4294bd327c1ae223467b85474313ac542f50d700

if above token has expired, retrieve join token like below:

kubeadm token create --print-join-command

Deploy calico network plugin to the cluster

Install Calico. The article link

Install Calico using the kubectl create command and the tigera-operator.yaml

curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml --output ./tigera-operator.yaml
kubectl create -f ./tigera-operator.yaml

After deploying Calico, configure it to work with your network. Provide additional configuration by editing the custom-resources.yaml file.

curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml --output ./custom-resources.yaml

Edit the ipPools section in the file to reflect your configuration. Put the server’s IP address in the cidr field.
Apply the configuration with kubectl create.

kubectl create -f ./custom-resources.yaml

Customize the manifest as necessary.
If successful, Calico deployment shows three running pods in the calico-system namespace or all. Check the status of the pods by typing:

watch kubectl get pods -n calico-system
watch kubectl get pods --all-namespaces

Remove the taints on the control plane so that you can schedule pods on it.

kubectl taint nodes --all node-role.kubernetes.io/control-plane-
kubectl taint nodes --all node-role.kubernetes.io/master-

NOTE: if have multiple network interface, like OpenVPN’s tun, please set the kubernetes used interface router to default. Here, I use NetworkManager to set openvpn’s IPv4 and IPv6 router to default. if not do this, I found the calico will not work well, something like “calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket”.

nmcli connection modify openvpn-rgsoft-k8s-worker1 ipv4.never-default false
nmcli connection modify openvpn-rgsoft-k8s-worker1 ipv6.never-default false

Record my installation used images

master node

quay.io/tigera/operator                                           v1.29.3   bc738edacda6   8 weeks ago     72MB
calico/typha                                                      v3.25.1   dfed532bc8bd   8 weeks ago     68.2MB
calico/kube-controllers                                           v3.25.1   212faac284a2   8 weeks ago     72.8MB
calico/apiserver                                                  v3.25.1   44b3b862e431   8 weeks ago     84.5MB
calico/cni                                                        v3.25.1   a0138614e609   8 weeks ago     201MB
calico/node-driver-registrar                                      v3.25.1   2dc09e59a52a   8 weeks ago     24MB
calico/csi                                                        v3.25.1   e04b5a6490df   8 weeks ago     21.3MB
calico/pod2daemon-flexvol                                         v3.25.1   958f4d519550   8 weeks ago     14.7MB
calico/node                                                       v3.25.1   cae61b85e9b4   8 weeks ago     248MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.26.3   1d9b3cbae03c   2 months ago    134MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.26.3   ce8c2293ef09   2 months ago    123MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.26.3   5a7904736932   2 months ago    56.4MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.26.3   92ed2bec97a6   2 months ago    65.6MB
registry.aliyuncs.com/google_containers/etcd                      3.5.6-0   fce326961ae2   6 months ago    299MB
registry.aliyuncs.com/google_containers/pause                     3.9       e6f181688397   7 months ago    744kB
registry.aliyuncs.com/google_containers/coredns                   v1.9.3    5185b96f0bec   12 months ago   48.8MB
registry.k8s.io/pause                                             3.6       6270bb605e12   21 months ago   683kB

worker node (GPU)

quay.io/tigera/operator                                            v1.29.3           bc738edacda6   8 weeks ago     72MB
calico/typha                                                       v3.25.1           dfed532bc8bd   8 weeks ago     68.2MB
calico/kube-controllers                                            v3.25.1           212faac284a2   8 weeks ago     72.8MB
calico/apiserver                                                   v3.25.1           44b3b862e431   8 weeks ago     84.5MB
calico/cni                                                         v3.25.1           a0138614e609   8 weeks ago     201MB
calico/node-driver-registrar                                       v3.25.1           2dc09e59a52a   8 weeks ago     24MB
calico/csi                                                         v3.25.1           e04b5a6490df   8 weeks ago     21.3MB
calico/pod2daemon-flexvol                                          v3.25.1           958f4d519550   8 weeks ago     14.7MB
calico/node                                                        v3.25.1           cae61b85e9b4   8 weeks ago     248MB
registry.aliyuncs.com/google_containers/kube-proxy                 v1.26.3           92ed2bec97a6   2 months ago    65.6MB
kubernetesui/dashboard                                             v2.7.0            07655ddf2eeb   8 months ago    246MB
kubernetesui/metrics-scraper                                       v1.0.8            115053965e86   12 months ago   43.8MB
k8simage/kube-scheduler                                            v1.23.3           99a3486be4f2   16 months ago   53.5MB
registry.k8s.io/pause                                              3.6               6270bb605e12   21 months ago   683kB
registry.cn-hangzhou.aliyuncs.com/acs/k8s-gpushare-schd-extender   1.11-d170d8a      1a90345a95a0   4 years ago     99MB
registry.cn-hangzhou.aliyuncs.com/acs/k8s-gpushare-plugin          v2-1.11-aff8a23   e424cbbbfc1b   4 years ago     116MB

Prepare the environment#

Install kubelet, kubeadm and kubectl#

Disable Swap Space#

Install Container runtime (Master and Worker nodes)#

Initialize control plane (run on first master node)#

Deploy calico network plugin to the cluster#

Record my installation used images#