Просмотр кода страницы Kubernetes. AlmaLinux 8. Kubernetes Cluster installation with the only one Master-Node and many Worker-Nodes

[[Категория:Kubernetes]]
__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

= About =
Install Kubernetes Master + Node at AlmaLinux 8

= Install =

== MASTER '''kub-master-1''' ==

=== Selinux OFF ===
<code>setenforce 0</code>

<code>sed -i 's/#\?\(SELINUX\=\).*$/\1disabled/' /etc/sysconfig/selinux</code>

<code>reboot</code>

=== Set hostname ===
<code>hostnamectl set-hostname kub-master-1</code>

Add IP address and hostname resolve to hosts
 cat <<EOF>> /etc/hosts
 192.168.199.184 kub-master-1
 192.168.199.186 kub-node-1
 192.168.199.187 kub-node-2
 EOF

=== Swap OFF ===
<code>swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab</code>

=== Bridge settings ===
<code>modprobe br_netfilter</code>

<code>echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables</code>

=== Docker install ===
<code>dnf config-manager --add-repo=<nowiki>https://download.docker.com/linux/centos/docker-ce.repo</nowiki></code>

<code>dnf install -y containerd.io</code>

<code>dnf install -y --nobest docker-ce</code>

<code>systemctl enable docker</code>

<code>systemctl start docker</code>

=== Kubernetes install ===
 cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
 [kubernetes]
 name=Kubernetes
 baseurl=<nowiki>https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch</nowiki>
 enabled=1
 gpgcheck=1
 repo_gpgcheck=1
 gpgkey=<nowiki>https://packages.cloud.google.com/yum/doc/yum-key.gpg</nowiki> <nowiki>https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg</nowiki>
 exclude=kubelet kubeadm kubectl
 EOF
<code>yum repolist</code>

<code>yum update -y</code>

<code>yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes</code>

Autocompletion kubadm

<code>yum install bash-completion -y</code>

<code>curl <nowiki>https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker</nowiki> -o /etc/bash_completion.d/docker.sh</code>

<code>echo 'source <(kubectl completion bash)' >>~/.bashrc</code>

<code>kubectl completion bash >/etc/bash_completion.d/kubectl</code>

<code>echo 'source <(kubeadm completion bash)' >>~/.bashrc</code>

<code>kubeadm completion bash >/etc/bash_completion.d/kubeadm</code><blockquote>'''Relogin to terminal!'''</blockquote>

=== Docker setup ===
<code>mkdir /etc/docker</code>
 cat <<EOF | sudo tee /etc/docker/daemon.json
 {
   "exec-opts": ["native.cgroupdriver=systemd"],
   "log-driver": "json-file",
   "log-opts": {
     "max-size": "100m"
   },
   "storage-driver": "overlay2"
 }
 EOF
<code>systemctl daemon-reload</code>

<code>systemctl restart docker</code>

<code>kubeadm reset</code>

<code>systemctl status docker</code><blockquote># init

   '''ERROR:'''

       [ERROR CRI]: container runtime is not running: output: time="2022-05-20T02:06:28Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

   '''SOLUTION:'''

       sudo rm /etc/containerd/config.toml

       sudo systemctl restart containerd

       sudo kubeadm init</blockquote>

=== Kubernetes init ===
'''''--apiserver-advertise-address=192.168.199.184''''' - LAN interface address, if it is not specified, the default address will be used

'''''--pod-network-cidr=172.16.213.0/24''''' - container virtual network

<code>kubeadm init --apiserver-advertise-address=192.168.199.184 --pod-network-cidr=172.16.213.0/24</code>
     ...
     Your Kubernetes control-plane has initialized successfully!
     To start using your cluster, you need to run the following as a regular user:
     mkdir -p $HOME/.kube
     sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
     sudo chown $(id -u):$(id -g) $HOME/.kube/config
     Alternatively, if you are the root user, you can run:
     export KUBECONFIG=/etc/kubernetes/admin.conf
     You should now deploy a pod network to the cluster.
     Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
     <nowiki>https://kubernetes.io/docs/concepts/cluster-administration/addons/</nowiki>
     Then you can join any number of worker nodes by running the following on each as root:
     kubeadm join 192.168.199.184:6443 --token gn2gva.2h73qvdhk0bg6krn \
             --discovery-token-ca-cert-hash sha256:418ca3a0003c8945a1cb9c108992c33aedcebbd1a4ba3960b43230a71eb8cda5
     ...
   

<code>systemctl enable kubelet</code>

<code>mkdir -p $HOME/.kube</code>

<code>cp -i /etc/kubernetes/admin.conf $HOME/.kube/config</code>

<code>chown $(id -u):$(id -g) $HOME/.kube/config</code>

=== Setup cluster networking ===
We need to deploy the pod's internal network so that containers on different nodes can communicate with each other. POD network is the overlay network between the worker nodes.

An overlay network is a virtual network of nodes and logical links, which are built on top of an existing network.

<code>kubectl apply -f <nowiki>https://docs.projectcalico.org/manifests/calico.yaml</nowiki></code>

Wait about 1 minute

<code>kubectl get nodes</code>

<code>touch /etc/default/kubelet</code>

<code>echo "KUBELET_EXTRA_ARGS=--node-ip=192.168.199.184" > /etc/default/kubelet</code>

If you somehow miss the output from the kubeadm init command, nothing to worry about as you can always run following command to generate the kubeadm join token again.

<code>kubeadm token create --print-join-command</code>
     ...
     kubeadm join 192.168.199.184:6443 --token 2qw1q0.hnmu7eggccf7172y --discovery-token-ca-cert-hash sha256:c7832b18f117af05346cf019aa7a9d3ca0029c65a69320f7cb807147ee975efb
     ...

== WORKERS '''kub-node-1 (you can add additional nodes this way, for example "kub-node-2")''' ==
<code>setenforce 0</code>

<code>sed -i 's/#\?\(SELINUX\=\).*$/\1disabled/' /etc/sysconfig/selinux</code>

<code>reboot</code>

<code>hostnamectl set-hostname kub-node-1</code>

Add host and IP match to <code>/etc/hosts</code><blockquote>Also you need to add new node host to <code>/etc/hosts</code> of all k8s nodes and masters in cluster. Or use a DNS server to match IP and hostname.</blockquote>
 cat <<EOF>> /etc/hosts
 192.168.199.184 kub-master-1
 192.168.199.186 kub-node-1
 192.168.199.187 kub-node-2
 EOF
<code>swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab</code>

<code>modprobe br_netfilter</code>

<code>echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables</code>

<code>dnf config-manager --add-repo=<nowiki>https://download.docker.com/linux/centos/docker-ce.repo</nowiki></code>

<code>dnf install -y containerd.io</code>

<code>dnf install -y --nobest docker-ce</code>

<code>systemctl enable docker</code>

<code>systemctl start docker</code>
 cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
 [kubernetes]
 name=Kubernetes
 baseurl=<nowiki>https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch</nowiki>
 enabled=1
 gpgcheck=1
 repo_gpgcheck=1
 gpgkey=<nowiki>https://packages.cloud.google.com/yum/doc/yum-key.gpg</nowiki> <nowiki>https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg</nowiki>
 exclude=kubelet kubeadm kubectl
 EOF
<code>yum repolist</code>

<code>yum update -y</code>

<code>yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes</code>

<code>yum install bash-completion -y</code>

<code>curl <nowiki>https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker</nowiki> -o /etc/bash_completion.d/docker.sh</code>

<code>echo 'source <(kubectl completion bash)' >>~/.bashrc</code>

<code>kubectl completion bash >/etc/bash_completion.d/kubectl</code>

<code>echo 'source <(kubeadm completion bash)' >>~/.bashrc</code>

<code>kubeadm completion bash >/etc/bash_completion.d/kubeadm</code>

<code>echo 'alias k=kubectl' >>~/.bashrc</code>

<code>echo 'complete -o default -F __start_kubectl k' >>~/.bashrc</code><blockquote>'''Relogin to terminal!'''</blockquote><code>touch /etc/default/kubelet</code>

'''''--node-ip=192.168.199.186''''' - kub-node-1 IP address

<code>echo "KUBELET_EXTRA_ARGS=--node-ip=192.168.199.186" > /etc/default/kubelet</code>

<code>mkdir /etc/docker</code>
 cat <<EOF | sudo tee /etc/docker/daemon.json
 {
   "exec-opts": ["native.cgroupdriver=systemd"],
   "log-driver": "json-file",
   "log-opts": {
     "max-size": "100m"
   },
   "storage-driver": "overlay2"
 }
 EOF
<code>systemctl enable docker</code>

<code>systemctl daemon-reload</code>

<code>systemctl restart docker</code>

<code>kubeadm reset</code><blockquote>   '''ERROR:'''

       [ERROR CRI]: container runtime is not running: output: time="2022-05-20T02:06:28Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

   '''SOLUTION:'''

       sudo rm /etc/containerd/config.toml

       sudo systemctl restart containerd</blockquote><code>kubeadm join 192.168.199.184:6443 --token 2qw1q0.hnmu7eggccf7172y --discovery-token-ca-cert-hash sha256:c7832b18f117af05346cf019aa7a9d3ca0029c65a69320f7cb807147ee975efb</code>

<code>systemctl enable kubelet</code>

= Postinstall and info =

== MASTER '''kub-master-1''' ==

=== Set node(s) role "node" ===
<code>kubectl label node kub-node-1 node-role.kubernetes.io/node=</code>

=== Cluster info ===
<code>kubectl cluster-info</code>

=== Master or node info ===
<code>kubectl describe node kub-master-1</code>

<code>kubectl describe node kub-node-1</code>

=== Get nodes ===
<code>kubectl get nodes -o wide</code>
     ...
     NAME     STATUS   ROLES                  AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
     master   Ready    control-plane,master   65m     v1.23.1   192.168.199.169   <none>        CentOS Linux 8   4.18.0-348.7.1.el8_5.x86_64    docker://20.10.12
     node-1   Ready    <none>                 2m28s   v1.23.1   192.168.199.179   <none>        CentOS Linux 8   4.18.0-305.25.1.el8_4.x86_64   docker://20.10.12
     ...

=== Get pods ===
<code>kubectl get pods --all-namespaces</code>
     ...
     NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
     kube-system   calico-kube-controllers-85b5b5888d-lrsq8   1/1     Running   0             58m
     kube-system   calico-node-gwtl6                          1/1     Running   0             2m32s
     kube-system   calico-node-lbbbv                          1/1     Running   0             58m
     kube-system   coredns-64897985d-6jv5w                    1/1     Running   0             65m
     kube-system   coredns-64897985d-n6c9l                    1/1     Running   0             65m
     kube-system   etcd-master                                1/1     Running   1 (64m ago)   65m
     kube-system   kube-apiserver-master                      1/1     Running   1 (64m ago)   65m
     kube-system   kube-controller-manager-master             1/1     Running   1 (64m ago)   65m
     kube-system   kube-proxy-44l8g                           1/1     Running   1 (64m ago)   65m
     kube-system   kube-proxy-ljfsd                           1/1     Running   0             2m32s
     kube-system   kube-scheduler-master                      1/1     Running   1 (64m ago)   65m
     ...

=== Certificates ===
Check certs expiration

<code>kubeadm certs check-expiration</code>
     ...
     [check-expiration] Reading configuration from the cluster...
     [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
     CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
     admin.conf                 Feb 20, 2023 19:11 UTC   364d            ca                      no
     apiserver                  Feb 20, 2023 19:10 UTC   364d            ca                      no
     apiserver-etcd-client      Feb 20, 2023 19:11 UTC   364d            etcd-ca                 no
     apiserver-kubelet-client   Feb 20, 2023 19:10 UTC   364d            ca                      no
     controller-manager.conf    Feb 20, 2023 19:11 UTC   364d            ca                      no
     etcd-healthcheck-client    Feb 20, 2023 19:10 UTC   364d            etcd-ca                 no
     etcd-peer                  Feb 20, 2023 19:10 UTC   364d            etcd-ca                 no
     etcd-server                Feb 20, 2023 19:10 UTC   364d            etcd-ca                 no
     front-proxy-client         Feb 20, 2023 19:10 UTC   364d            front-proxy-ca          no
     scheduler.conf             Feb 20, 2023 19:11 UTC   364d            ca                      no
     CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
     ca                      Feb 18, 2032 19:10 UTC   9y              no
     etcd-ca                 Feb 18, 2032 19:10 UTC   9y              no
     front-proxy-ca          Feb 18, 2032 19:10 UTC   9y              no
     ...
Renew all certs

<code>kubeadm certs renew all</code>

<code>reboot</code>