Kubernetes. AlmaLinux 8. Kubernetes Cluster installation with Multi-Masters and many Workers. Etcd. Haproxy balancer
Links
There are some errors
Input data
Balancer Haproxy: haproxy2 192.168.199.161
Kub-master-1: kub2-m-1 192.168.199.171
Kub-master-2: kub2-m-2 192.168.199.172
Kub-master-3: kub2-m-3 192.168.199.173
Kub-worker-1: kub2-w-1 192.168.199.181
Kub-worker-2: kub2-w-2 192.168.199.182
Install
DNS server
On the DNS server that all your hosts look at, add A-records to match the IP and the hostname:
For example:
... kub2-m-1 192.168.199.171 ...
All MASTERS and WORKERS nodes
Hostname
hostnamectl set-hostname <kub2-m-<NUMBER> for master or kub2-w-<NUMBER> for worker>
Utils install
dnf install -y iproute-tc
SWAP off
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab
Docker install
dnf config-manager -y --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf remove -y podman buildah
dnf install -y docker-ce docker-ce-cli containerd.io
systemctl start docker.service
systemctl enable docker.service
docker version
Kubernetes install
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl EOF
yum repolist
yum update -y
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet
systemctl start kubelet
Balancer HAPROXY host
Haproxy install
dnf install -y haproxy
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.ORIG
nano /etc/haproxy/haproxy.cfg
global
....
defaults
....
#------
#KUBERNETES
#------
frontend kubernetes
bind 192.168.199.161:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server kub2-m-1 192.168.199.171:6443 check fall 3 rise 2
server kub2-m-2 192.168.199.172:6443 check fall 3 rise 2
server kub2-m-3 192.168.199.173:6443 check fall 3 rise 2systemctl restart haproxy
systemctl status haproxy
Cfssl install
You can do it at any Linux host, but lets's do it at the Haproxy host as well, since we have this Linux host. We have to generate certs and store them here
It's a tool for certs generating
Download the binaries
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
Add the execution permission to the binaries
chmod +x cfssl*
Move the binaries to /usr/local/bin
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
Certs generating
Creating a certificate authority
Create the certificate authority configuration file
nano ca-config.json
{
"signing": {
"default": {
"expiry": "262800h"
},
"profiles": {
"kubernetes": {
"usages": ["signing", "key encipherment", "server auth", "client auth"],
"expiry": "262800h"
}
}
}
}
Create the certificate authority signing request configuration file
nano ca-csr.json
{
"CN": "Kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "IE",
"L": "Cork",
"O": "Kubernetes",
"OU": "CA",
"ST": "Cork Co."
}
]
}
Generate the certificate authority certificate and private key
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
Verify that the ca-key.pem and the ca.pem were generated
ls -la
... ca-key.pem ca.pem ...
Creating the certificate for the Etcd cluster
Create the certificate signing request configuration file
nano kubernetes-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "IE",
"L": "Cork",
"O": "Kubernetes",
"OU": "Kubernetes",
"ST": "Cork Co."
}
]
}
Generate the certificate and private key
-hostname=192.168.199.171,192.168.199.172,192.168.199.173,192.168.199.161 - IP of master nodes and haproxy balancer host
cfssl gencert \ -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -hostname=192.168.199.171,192.168.199.172,192.168.199.173,192.168.199.161,127.0.0.1,kubernetes.default \ -profile=kubernetes kubernetes-csr.json | \ cfssljson -bare kubernetes
Verify that the kubernetes-key.pem and the kubernetes.pem file were generated.
ls -la
... kubernetes-key.pem kubernetes.pem ...
Copy the certificate to each nodes - MASTERS and WORKERS
scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.171:~
scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.172:~
scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.173:~
??? why are they nedded on the WORKERS hosts ??? Maybe it's unnecessary. TEST and write down to this manual
scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.181:~
scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.182:~
All MASTERS hosts
Installing and configuring Etcd
Create a configuration directory for Etcd
mkdir /etc/etcd /var/lib/etcd
Move the certificates to the configuration directory
mv ~/ca.pem ~/kubernetes.pem ~/kubernetes-key.pem /etc/etcd
Download the etcd binaries
wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz
Extract the etcd archive
tar xvzf etcd-v3.3.13-linux-amd64.tar.gz
Move the etcd binaries to /usr/local/bin
mv etcd-v3.3.13-linux-amd64/etcd* /usr/local/bin/
Create an etcd systemd unit file
nano /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos
[Service]
ExecStart=/usr/local/bin/etcd \
--name <THIS-MASTER-IP> \
--cert-file=/etc/etcd/kubernetes.pem \
--key-file=/etc/etcd/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/kubernetes.pem \
--peer-key-file=/etc/etcd/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--initial-advertise-peer-urls https://<THIS-MASTER-IP>:2380 \
--listen-peer-urls https://<THIS-MASTER-IP>:2380 \
--listen-client-urls https://<THIS-MASTER-IP>:2379,http://127.0.0.1:2379 \
--advertise-client-urls https://<THIS-MASTER-IP>:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster <MASTER-1-IP>=https://<MASTER-1-IP>:2380,<MASTER-2-IP>=https://<MASTER-2-IP>:2380,<MASTER-3-IP>=https://<MASTER-3-IP>:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetReload the daemon configuration
systemctl daemon-reload
Start service
systemctl enable --now etcd
systemctl status etcd
Show etcd cluster status
ETCDCTL_API=3 etcdctl member list
Initializing MASTERS nodes
MASTER-1 node (192.168.199.171)
Bash completion install
yum install bash-completion -y
curl https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker -o /etc/bash_completion.d/docker.sh
echo 'source <(kubectl completion bash)' >>~/.bashrc
kubectl completion bash >/etc/bash_completion.d/kubectl
echo 'source <(kubeadm completion bash)' >>~/.bashrc
kubeadm completion bash >/etc/bash_completion.d/kubeadm
Enable kubelet
systemctl enable kubelet.service
Create config to ini kubernetes cluster
nano config.yaml
apiserver-count: "3" - quantity of masters in cluster
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
controlPlaneEndpoint: "<BALANCER-HAPROXY-IP>:6443"
etcd:
external:
endpoints:
- https://<MASTER-1-IP>:2379
- https://<MASTER-2-IP>:2379
- https://<MASTER-3-IP>:2379
caFile: /etc/etcd/ca.pem
certFile: /etc/etcd/kubernetes.pem
keyFile: /etc/etcd/kubernetes-key.pem
networking:
podSubnet: 10.30.0.0/24
apiServer:
certSANs:
- "<BALANCER-HAPROXY-IP>"
extraArgs:
apiserver-count: "3"
Init kubernetes cluster
kubeadm init --config=config.yaml
Most likely this command will cause an error:
W1030 11:25:49.110660 80056 common.go:84] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta2". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
In this case you have to convert config to newer version:
kubeadm config migrate --old-config config.yaml --new-config config-new.yaml
It will create a new config:
nano config-new.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: vgyip6.2qes2wgwlnjkjp67
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.199.171
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
name: kub2-m-1
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
---
apiServer:
certSANs:
- 192.168.199.161
extraArgs:
apiserver-count: "3"
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.199.161:6443
controllerManager: {}
dns: {}
etcd:
external:
caFile: /etc/etcd/ca.pem
certFile: /etc/etcd/kubernetes.pem
endpoints:
- https://192.168.199.171:2379
- https://192.168.199.172:2379
- https://192.168.199.173:2379
keyFile: /etc/etcd/kubernetes-key.pem
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
networking:
dnsDomain: cluster.local
podSubnet: 10.30.0.0/24
serviceSubnet: 10.96.0.0/12
scheduler: {}
Now you can try to run init command again:
kubeadm init --config=config-new.yaml
It may cause an another ERROR:
[ERROR CRI]: container runtime is not running: output: time="2022-05-20T02:06:28Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Than you have to RUN:
rm -f /etc/containerd/config.toml
systemctl restart containerd
And finally run init command again:
kubeadm init --config=config-new.yaml
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \
--discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \
--discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe
...Write down commands for join cluster MASTERS and WORKERS:
"
kubeadm join..." for Master join"
kubeadm join..." for Workers joinBut if you somehow miss the output from the kubeadm init command, nothing to worry about as you can always run following command to generate the kubeadm join token again:
kubeadm token create --print-join-command
If you will run kubectl as root - Point config path in shell
export KUBECONFIG=/etc/kubernetes/admin.conf
And to .bashrc
nano .bashrc
... export KUBECONFIG=/etc/kubernetes/admin.conf
Show cluster status
kubectl cluster-info
Copy the certificates to the other master(s)
scp -r /etc/kubernetes/pki root@192.168.199.172:~
scp -r /etc/kubernetes/pki root@192.168.199.173:~
MASTER-2 (192.168.199.172) and MASTER-3 (192.168.199.173) nodes
Bash completion
yum install bash-completion -y
curl https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker -o /etc/bash_completion.d/docker.sh
echo 'source <(kubectl completion bash)' >>~/.bashrc
kubectl completion bash >/etc/bash_completion.d/kubectl
echo 'source <(kubeadm completion bash)' >>~/.bashrc
kubeadm completion bash >/etc/bash_completion.d/kubeadm
Enable kubelet
systemctl enable kubelet.service
Remove original certs
rm -f ~/pki/apiserver.*
Move the certificates to the /etc/kubernetes directory.
mv ~/pki /etc/kubernetes/
Remove containerd config
rm -f /etc/containerd/config.toml
systemctl restart containerd
Join kubernetes cluster, that we have init on MASTER-1 host
kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \
--discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe \
--control-plane
...
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
...If you will run kubectl as root - Point config path in shell
export KUBECONFIG=/etc/kubernetes/admin.conf
And to .bashrc
nano .bashrc
... export KUBECONFIG=/etc/kubernetes/admin.conf
Show cluster status
kubectl cluster-info
WORKER-1 add (and any other additional WORKERS add)
Remove containerd config
rm -f /etc/containerd/config.toml
systemctl restart containerd
kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \
--discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe
ANY-MASTER
Set node(s) role "node"
kubectl label node kub-node-1 node-role.kubernetes.io/worker=
Cluster info
kubectl cluster-info
Nodes info
kubectl get node
Master or Worker node info
kubectl describe node kub-master-1
kubectl describe node kub-node-1
Now we need to setup cluster networking
Setup cluster networking
We need to deploy the pod's internal network so that containers on different nodes can communicate with each other. POD network is the overlay network between the worker nodes.
An overlay network is a virtual network of nodes and logical links, which are built on top of an existing network.
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
Wait about 1 minute!
kubectl get nodes
Ready means everything is OK
... NAME STATUS ROLES AGE VERSION kub2-m-1 Ready control-plane 105d v1.24.3 kub2-m-2 Ready control-plane 105d v1.24.3 kub2-w-1 Ready node 105d v1.24.3 kub2-w-2 Ready node 21d v1.25.2 ...
Certificates
Check certs expiration
kubeadm certs check-expiration
...
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Feb 20, 2023 19:11 UTC 364d ca no
apiserver Feb 20, 2023 19:10 UTC 364d ca no
apiserver-etcd-client Feb 20, 2023 19:11 UTC 364d etcd-ca no
apiserver-kubelet-client Feb 20, 2023 19:10 UTC 364d ca no
controller-manager.conf Feb 20, 2023 19:11 UTC 364d ca no
etcd-healthcheck-client Feb 20, 2023 19:10 UTC 364d etcd-ca no
etcd-peer Feb 20, 2023 19:10 UTC 364d etcd-ca no
etcd-server Feb 20, 2023 19:10 UTC 364d etcd-ca no
front-proxy-client Feb 20, 2023 19:10 UTC 364d front-proxy-ca no
scheduler.conf Feb 20, 2023 19:11 UTC 364d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Feb 18, 2032 19:10 UTC 9y no
etcd-ca Feb 18, 2032 19:10 UTC 9y no
front-proxy-ca Feb 18, 2032 19:10 UTC 9y no
...Renew all certs
!!! You have to do it at every MASTER node
kubeadm certs renew all
reboot
After reboot you'll see that Certificates have actual date and time