Kubernetes. AlmaLinux 8. Kubernetes Cluster installation with Multi-Masters and many Workers. Etcd. Haproxy balancer

Links

There are some errors

https://dockerlabs.collabnix.com/kubernetes/beginners/Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.html

Input data

Balancer Haproxy: haproxy2 192.168.199.161

Kub-master-1: kub2-m-1 192.168.199.171

Kub-master-2: kub2-m-2 192.168.199.172

Kub-master-3: kub2-m-3 192.168.199.173

Kub-worker-1: kub2-w-1 192.168.199.181

Kub-worker-2: kub2-w-2 192.168.199.182

Install

DNS server

On the DNS server that all your hosts look at, add A-records to match the IP and the hostname:

For example:

...
kub2-m-1 192.168.199.171
...

All MASTERS and WORKERS nodes

Hostname

hostnamectl set-hostname <kub2-m-<NUMBER> for master or kub2-w-<NUMBER> for worker>

Utils install

dnf install -y iproute-tc

SWAP off

swapoff -a

sed -i '/ swap / s/^/#/' /etc/fstab

Docker install

dnf config-manager -y --add-repo https://download.docker.com/linux/centos/docker-ce.repo

dnf remove -y podman buildah

dnf install -y docker-ce docker-ce-cli containerd.io

systemctl start docker.service

systemctl enable docker.service

docker version

Kubernetes install

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum repolist

yum update -y

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable kubelet

systemctl start kubelet

Balancer HAPROXY host

Haproxy install

dnf install -y haproxy

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.ORIG

nano /etc/haproxy/haproxy.cfg

global
       ....
defaults
       ....
#------
#KUBERNETES
#------

frontend kubernetes
bind 192.168.199.161:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server kub2-m-1 192.168.199.171:6443 check fall 3 rise 2
server kub2-m-2 192.168.199.172:6443 check fall 3 rise 2
server kub2-m-3 192.168.199.173:6443 check fall 3 rise 2

systemctl restart haproxy

systemctl status haproxy

Cfssl install

You can do it at any Linux host, but lets's do it at the Haproxy host as well, since we have this Linux host. We have to generate certs and store them here

It's a tool for certs generating

Download the binaries

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

Add the execution permission to the binaries

chmod +x cfssl*

Move the binaries to /usr/local/bin

mv cfssl_linux-amd64 /usr/local/bin/cfssl

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

Certs generating

Creating a certificate authority

Create the certificate authority configuration file

nano ca-config.json

{
  "signing": {
    "default": {
      "expiry": "262800h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "262800h"
      }
    }
  }
}

Create the certificate authority signing request configuration file nano ca-csr.json

{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "IE",
    "L": "Cork",
    "O": "Kubernetes",
    "OU": "CA",
    "ST": "Cork Co."
  }
 ]
}

Generate the certificate authority certificate and private key

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

Verify that the ca-key.pem and the ca.pem were generated

ls -la

    ...
    ca-key.pem
    ca.pem
    ...

Creating the certificate for the Etcd cluster

Create the certificate signing request configuration file

nano kubernetes-csr.json

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "IE",
    "L": "Cork",
    "O": "Kubernetes",
    "OU": "Kubernetes",
    "ST": "Cork Co."
  }
 ]
}

Generate the certificate and private key

-hostname=192.168.199.171,192.168.199.172,192.168.199.173,192.168.199.161 - IP of master nodes and haproxy balancer host

cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-hostname=192.168.199.171,192.168.199.172,192.168.199.173,192.168.199.161,127.0.0.1,kubernetes.default \
-profile=kubernetes kubernetes-csr.json | \
cfssljson -bare kubernetes

Verify that the kubernetes-key.pem and the kubernetes.pem file were generated.

ls -la

...
kubernetes-key.pem
kubernetes.pem
...

Copy the certificate to each nodes - MASTERS and WORKERS

scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.171:~

scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.172:~

scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.173:~

??? why are they nedded on the WORKERS hosts ??? Maybe it's unnecessary. TEST and write down to this manual

scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.181:~

scp ca.pem kubernetes.pem kubernetes-key.pem root@192.168.199.182:~

All MASTERS hosts

Installing and configuring Etcd

Create a configuration directory for Etcd

mkdir /etc/etcd /var/lib/etcd

Move the certificates to the configuration directory

mv ~/ca.pem ~/kubernetes.pem ~/kubernetes-key.pem /etc/etcd

Download the etcd binaries

wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz

Extract the etcd archive

tar xvzf etcd-v3.3.13-linux-amd64.tar.gz

Move the etcd binaries to /usr/local/bin

mv etcd-v3.3.13-linux-amd64/etcd* /usr/local/bin/

Create an etcd systemd unit file

nano /etc/systemd/system/etcd.service

[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
ExecStart=/usr/local/bin/etcd \
  --name <THIS-MASTER-IP> \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://<THIS-MASTER-IP>:2380 \
  --listen-peer-urls https://<THIS-MASTER-IP>:2380 \
  --listen-client-urls https://<THIS-MASTER-IP>:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://<THIS-MASTER-IP>:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster <MASTER-1-IP>=https://<MASTER-1-IP>:2380,<MASTER-2-IP>=https://<MASTER-2-IP>:2380,<MASTER-3-IP>=https://<MASTER-3-IP>:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Reload the daemon configuration

systemctl daemon-reload

Start service

systemctl enable --now etcd

systemctl status etcd

Show etcd cluster status

ETCDCTL_API=3 etcdctl member list

Initializing MASTERS nodes

MASTER-1 node (192.168.199.171)

Bash completion install

yum install bash-completion -y

curl https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker -o /etc/bash_completion.d/docker.sh

echo 'source <(kubectl completion bash)' >>~/.bashrc

kubectl completion bash >/etc/bash_completion.d/kubectl

echo 'source <(kubeadm completion bash)' >>~/.bashrc

kubeadm completion bash >/etc/bash_completion.d/kubeadm

Enable kubelet

systemctl enable kubelet.service

Create config to ini kubernetes cluster

nano config.yaml

apiserver-count: "3" - quantity of masters in cluster

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
controlPlaneEndpoint: "<BALANCER-HAPROXY-IP>:6443"
etcd:
  external:
    endpoints:
      - https://<MASTER-1-IP>:2379
      - https://<MASTER-2-IP>:2379
      - https://<MASTER-3-IP>:2379
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/24
apiServer:
  certSANs:
    - "<BALANCER-HAPROXY-IP>"
  extraArgs:
    apiserver-count: "3"

Init kubernetes cluster

kubeadm init --config=config.yaml

Most likely this command will cause an error:

   W1030 11:25:49.110660   80056 common.go:84] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta2".    Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

In this case you have to convert config to newer version:

kubeadm config migrate --old-config config.yaml --new-config config-new.yaml

It will create a new config:

nano config-new.yaml

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: vgyip6.2qes2wgwlnjkjp67
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.199.171
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: kub2-m-1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
---
apiServer:
  certSANs:
  - 192.168.199.161
  extraArgs:
    apiserver-count: "3"
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.199.161:6443
controllerManager: {}
dns: {}
etcd:
  external:
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    endpoints:
    - https://192.168.199.171:2379
    - https://192.168.199.172:2379
    - https://192.168.199.173:2379
    keyFile: /etc/etcd/kubernetes-key.pem
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
networking:
  dnsDomain: cluster.local
  podSubnet: 10.30.0.0/24
  serviceSubnet: 10.96.0.0/12
scheduler: {}

Now you can try to run init command again:

kubeadm init --config=config-new.yaml

It may cause an another ERROR:

[ERROR CRI]: container runtime is not running: output: time="2022-05-20T02:06:28Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

Than you have to RUN:

rm -f /etc/containerd/config.toml

systemctl restart containerd

And finally run init command again:

kubeadm init --config=config-new.yaml

    ...
    Your Kubernetes control-plane has initialized successfully!

    To start using your cluster, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

    Alternatively, if you are the root user, you can run:

    export KUBECONFIG=/etc/kubernetes/admin.conf

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
    https://kubernetes.io/docs/concepts/cluster-administration/addons/

    You can now join any number of control-plane nodes by copying certificate authorities
    and service account keys on each node and then running the following as root:

    kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \
            --discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe \
            --control-plane

    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \
            --discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe
    ...

Write down commands for join cluster MASTERS and WORKERS:

"kubeadm join..." for Master join

"kubeadm join..." for Workers join

But if you somehow miss the output from the kubeadm init command, nothing to worry about as you can always run following command to generate the kubeadm join token again:

kubeadm token create --print-join-command

If you will run kubectl as root - Point config path in shell

export KUBECONFIG=/etc/kubernetes/admin.conf

And to .bashrc

nano .bashrc

...
export KUBECONFIG=/etc/kubernetes/admin.conf

Show cluster status

kubectl cluster-info

Copy the certificates to the other master(s)

scp -r /etc/kubernetes/pki root@192.168.199.172:~

scp -r /etc/kubernetes/pki root@192.168.199.173:~

MASTER-2 (192.168.199.172) and MASTER-3 (192.168.199.173) nodes

Bash completion

yum install bash-completion -y

curl https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker -o /etc/bash_completion.d/docker.sh

echo 'source <(kubectl completion bash)' >>~/.bashrc

kubectl completion bash >/etc/bash_completion.d/kubectl

echo 'source <(kubeadm completion bash)' >>~/.bashrc

kubeadm completion bash >/etc/bash_completion.d/kubeadm

Enable kubelet

systemctl enable kubelet.service

Remove original certs

rm -f ~/pki/apiserver.*

Move the certificates to the /etc/kubernetes directory.

mv ~/pki /etc/kubernetes/

Remove containerd config

rm -f /etc/containerd/config.toml

systemctl restart containerd

Join kubernetes cluster, that we have init on MASTER-1 host

kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \

       --discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe \

       --control-plane

    ...
    This node has joined the cluster and a new control plane instance was created:

    * Certificate signing request was sent to apiserver and approval was received.
    * The Kubelet was informed of the new secure connection details.
    * Control plane label and taint were applied to the new node.
    * The Kubernetes control plane instances scaled up.


    To start administering your cluster from this node, you need to run the following as a regular user:

            mkdir -p $HOME/.kube
            sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
            sudo chown $(id -u):$(id -g) $HOME/.kube/config

    Run 'kubectl get nodes' to see this node join the cluster.
    ...

If you will run kubectl as root - Point config path in shell

export KUBECONFIG=/etc/kubernetes/admin.conf

And to .bashrc

nano .bashrc

...
export KUBECONFIG=/etc/kubernetes/admin.conf

Show cluster status

kubectl cluster-info

WORKER-1 add (and any other additional WORKERS add)

Remove containerd config

rm -f /etc/containerd/config.toml

systemctl restart containerd

kubeadm join 192.168.199.161:6443 --token vgyip6.2qes2wgwlnjkjp67 \

           --discovery-token-ca-cert-hash sha256:e94027b5cb382eb2021c680aa8758dcfa244b217c88fa511ab7ed3b5ff330abe

ANY-MASTER

Set node(s) role "node"

kubectl label node kub-node-1 node-role.kubernetes.io/node=

Cluster info

kubectl cluster-info

Nodes info

kubectl get node

Master or Worker node info

kubectl describe node kub-master-1

kubectl describe node kub-node-1

Now we need to setup cluster networking

Setup cluster networking

We need to deploy the pod's internal network so that containers on different nodes can communicate with each other. POD network is the overlay network between the worker nodes.

An overlay network is a virtual network of nodes and logical links, which are built on top of an existing network.

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Wait about 1 minute!

kubectl get nodes

Ready means everything is OK

...
NAME         STATUS   ROLES           AGE    VERSION
kub2-m-1     Ready    control-plane   105d   v1.24.3
kub2-m-2     Ready    control-plane   105d   v1.24.3
kub2-w-1     Ready    node            105d   v1.24.3
kub2-w-2     Ready    node            21d    v1.25.2
...

Certificates

Check certs expiration

kubeadm certs check-expiration

    ...
    [check-expiration] Reading configuration from the cluster...
    [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
    admin.conf                 Feb 20, 2023 19:11 UTC   364d            ca                      no
    apiserver                  Feb 20, 2023 19:10 UTC   364d            ca                      no
    apiserver-etcd-client      Feb 20, 2023 19:11 UTC   364d            etcd-ca                 no
    apiserver-kubelet-client   Feb 20, 2023 19:10 UTC   364d            ca                      no
    controller-manager.conf    Feb 20, 2023 19:11 UTC   364d            ca                      no
    etcd-healthcheck-client    Feb 20, 2023 19:10 UTC   364d            etcd-ca                 no
    etcd-peer                  Feb 20, 2023 19:10 UTC   364d            etcd-ca                 no
    etcd-server                Feb 20, 2023 19:10 UTC   364d            etcd-ca                 no
    front-proxy-client         Feb 20, 2023 19:10 UTC   364d            front-proxy-ca          no
    scheduler.conf             Feb 20, 2023 19:11 UTC   364d            ca                      no
    CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
    ca                      Feb 18, 2032 19:10 UTC   9y              no
    etcd-ca                 Feb 18, 2032 19:10 UTC   9y              no
    front-proxy-ca          Feb 18, 2032 19:10 UTC   9y              no
    ...

Renew all certs

!!! You have to do it at every MASTER node

kubeadm certs renew all

reboot

After reboot you'll see that Certificates have actual date and time