쿠버네티스 1.18 설치 Docker

[따배쿠] 0. 따라하면서 배우는 쿠버네티스 입문편 - OT

https://confluence.curvc.com/pages/releaseview.action?pageId=98048155

Category : Kubernetes
Tag : Azure

실습 예제

실습 환경 : 직접설치

Windows10에 Virtual Box 설치, 네트워크 구성 및 가상머신에 Ubuntu20.04설치

Master, Node1, Node2 시스템에 도커 설치

쿠버네티스 설치 : v1.18

온라인 K8S Playground 사용하기

https://www.katacoda.com/courses/kube… https://labs.play-with-k8s.com/

쿠버네티스 환경 구성

1. 가상환경 구성

가상환경 도구
- vmware fusion 13
OS : ubuntu 20.04 LTS Desktop
환경 구성
- master
- node1
- node2

2. 가상환경 세팅

apt-get 명령어 update
```
sudo apt-get update
```
ssh 설치
```
sudo apt-get install openssh-server
```
ip정보를 위한 net-tools 설치
```
sudo apt install net-tools
```

3. 접속테스트

ssh 접속
```
ssh <계정명>@<server ip>
```
ssh 접속시 다음과 같은 경고가 발생하는 경우
- 1. 다음 명령어 입력하여 known_hosts 내용 갱신
    ssh-keygen -R <접속서버ip>
  2. 접속 시도
- 정상접속 확인

4. Swap 메모리를 비활성화

swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab #Error 

// 아래와 같이 접근 불허
master@master:~$ swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
sed: couldn't open temporary file /etc/sed0PMt8C: Permission denied


// 위 명령어 대신 아래 명령어로 실행
sudo swapoff -a && sudo sed -i '/swap/s/^/#/' /etc/fstab

5. iptable 설정

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

6. 방화벽 해제

방화벽 상태 확인
```
sudo ufw status
```
방화벽 활성화
```
sudo ufw enable
```

명령어 실행 참고

master@master:~$ sudo ufw status
Status: inactive
master@master:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with 
operation (y|n)? y
Firewall is active and enabled on system startup
master@master:~$ sudo ufw status
Status: active

ip 접근 허용

sudo ufw allow from <master ip>
sudo ufw allow from <node1 ip>
sudo ufw allow from <node2 ip>

실행 명령어[예시]

sudo ufw allow from 192.168.123.120
sudo ufw allow from 192.168.123.121
sudo ufw allow from 192.168.123.122
sudo ufw allow from 10.100.0.4
sudo ufw allow from 10.100.1.4
sudo ufw allow from 10.100.2.4

방화벽 상태 확인 후 reload
```
sudo ufw status
sudo ufw reload
```

7. telnet 테스트

telnet 설치
```
sudo apt-get install telnetd
```

부팅시 자동 시작 구성

sudo systemctl stop inetutils-inetd
sudo systemctl start inetutils-inetd
sudo systemctl enable inetutils-inetd
//status 확인
sudo systemctl status inetutils-inetd

telnet 테스트

  # MASTER
telnet 10.100.1.4
telnet 10.100.2.4

  # NODE1
telnet 10.100.0.4
telnet 10.100.2.4

  # NODE2
telnet 10.100.0.4
telnet 10.100.1.4

4. [master, node1, node2] Docker 설치

Docker Docs

1. repository를 이용한 설치(Install using the apt repository)

설치전 필요한 패키지설치

# Update the apt package index and install packages to allow apt to use a repository over HTTPS:
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

도커에 GPG키 설치

# Add Docker’s official GPG key:
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

repository등록

# Use the following command to set up the repository:
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

2. Docker Engine 설치(Install Docker Engine)

apt 업데이트
```
sudo apt-get update
```

docker-ce docker-ce-cli containerd.io(도커엔진) 설치 진행

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Docker 설치명령어 통합[master, node1, node2]

별도로 실행
```
sudo apt-get update
```

Docker 설치 확인

docker version
systemctl enable docker
systemctl start docker
systemctl status docker

5. [master, node1, node2] Kubernetes 설치

Kubernetes Docs

[2024-04-29 신규 업데이트사항]

sudo mkdir /etc/apt/keyrings

5.1. 설치 전 확인사항

1. 패키지 인덱스를 업데이트 apt하고 Kubernetes apt리포지토리를 사용하는 데 필요한 패키지를 설치합니다.

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

2. Google Cloud 공개 서명 키를 다운로드합니다.

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

3. Kubernetes apt리포지토리를 추가합니다.

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list 

4. 패키지 인덱스를 업데이트하고 aptkubelet, kubeadm 및 kubectl을 설치하고 해당 버전을 고정합니다.

4. [0] apt update를 건너뛰는 경우 패키지가 없다고 나올 수 있다.

sudo apt-get update

sudo apt-get update 404 에러

쿠버네티스 공식 리포지토리 경로 변경으로 인한 에러

master@MASTER:~$ sudo apt-get update
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease
Ign:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Err:7 https://packages.cloud.google.com/apt kubernetes-xenial Release
  404  Not Found [IP: 142.250.76.142 443]
Reading package lists... Done
E: The repository 'https://apt.kubernetes.io kubernetes-xenial Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

아래 과정 수행

sudo su -
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl

[4-1]. 1.18버전 설치

sudo apt install -y kubelet=1.18.15-00 kubeadm=1.18.15-00 kubectl=1.18.15-00
sudo apt install -y kubelet=1.18.16-00 kubeadm=1.18.16-00 kubectl=1.18.16-00

버전이슈 발생

sudo apt install -y kubelet=1.18.15-00 kubeadm=1.18.15-00 kubectl=1.18.15-00
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '1.18.15-00' for 'kubelet' was not found
E: Version '1.18.15-00' for 'kubeadm' was not found
E: Version '1.18.15-00' for 'kubectl' was not found

다음 과정 진행

1. kubectl 바이너리 다운로드: curl을 사용하여 kubectl 바이너리를 다운로드합니다. 다음 명령을 실행하십시오:

curl -LO "https://dl.k8s.io/release/v1.18.20/bin/linux/amd64/kubectl"

2. 다운로드한 바이너리 실행 권한 부여: 다운로드한 kubectl 바이너리 파일에 실행 권한을 부여해야 합니다. 다음 명령을 사용하여 실행 권한을 부여하십시오:

chmod +x kubectl

3. 바이너리 설치 위치 이동: 바이너리를 설치할 디렉토리로 이동합니다. 보통은 시스템 PATH에 포함된 디렉토리(/usr/local/bin 등)에 설치합니다. 예를 들어:

sudo mv kubectl /usr/local/bin/

4. 설치 확인: kubectl이 올바르게 설치되었는지 확인하려면 다음 명령을 실행하십시오:

kubectl version --client

[4-2]. 1.19버전 설치 [flannal_test]

sudo apt-get install -y kubelet=1.19.15-00 kubeadm=1.19.15-00 kubectl=1.19.15-00

[4-3] 최신 버전 설치

sudo apt-get install -y kubelet kubeadm kubectl

5. 버전 고정

sudo apt-mark hold kubelet kubeadm kubectl

6. 활성화

systemctl start kubelet && systemctl enable kubelet

비밀번호 입력

7. systemd 와 cgroup 설정 맞추기, 꼭!! 꼭!! 하자

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
    "max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

8. 재시작

sudo systemctl enable docker 
sudo systemctl daemon-reload 
sudo systemctl restart docker

~~[4-2]에 있는 최신버전으로 설치시 docker를 인식하지 못하는 이슈가 있어 1.18버젼으로 설치 진행~~

apt를 update해주지 않아 생긴 문제였다.

[tip] 다음 아래의 이미지와 같은 문제가 발생하는 경우 다음 명령어를 순차 진행한다.[해결됨]

(1) 공개키 재등록

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

(2) /etc/apt/sources.list.d/kubernetes.list 파일을 삭제
```
sudo rm /etc/apt/sources.list.d/kubernetes.list
```
(3) 최종적으로 패키지 목록을 업데이트하고 패키지를 설치
```
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
```

5.2. Control-plane 구성

control-plane은 마스터노드에서만 구현하고 워커노드에서 하지 않는다.

1. 패키지 설치 후 데몬 실행

sudo systemctl daemon-reload
sudo systemctl restart kubelet

2. kubelet 명령어 실행 및 활성화

sudo systemctl start kubelet
sudo systemctl enable kubelet

3. [master] kubeadm 초기화 실행

마스터노드에 필요한 컴포넌트를 구성하기 위해
다음 명령어를 실행한다.

[3-1]. 일반적인 kubeadm 초기화 명령어

kubeadm init

[3-2]. CNI를 flannel로 설치 한 경우 다음 명령어로 실행

kubeadm init --apiserver-advertise-address=<서버ip> --pod-network-cidr=10.244.0.0/16

memo

kubeadm init --apiserver-advertise-address=10.100.0.4 --pod-network-cidr=10.244.0.0/16
kubeadm init --apiserver-advertise-address=192.168.123.120 --pod-network-cidr=10.244.0.0/16

[3-3]. Kubeadm 초기화 및 Calico 설치

kubeadm init --pod-network-cidr=<master node가 속한 네트워크 주소 범위>
kubeadm init --pod-network-cidr=10.100.0.0/16

# Calico 설치
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

kubeadm init 실행

  master@master:~$ kubeadm init
  I0615 01:09:40.042008    2117 version.go:252] remote version is much newer: v1.27.2; falling back to: stable-1.18
  W0615 01:09:40.461387    2117 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
  [init] Using Kubernetes version: v1.18.20
  [preflight] Running pre-flight checks
  error execution phase preflight: [preflight] Some fatal errors occurred:
          [ERROR IsPrivilegedUser]: user is not running as root
  [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
  To see the stack trace of this error execute with --v=5 or higher
  master@master:~$ sudo kubeadm init
  I0615 01:09:46.558519    2131 version.go:252] remote version is much newer: v1.27.2; falling back to: stable-1.18
  W0615 01:09:46.965384    2131 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
  [init] Using Kubernetes version: v1.18.20
  [preflight] Running pre-flight checks
          [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
          [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.2. Latest validated version: 19.03
  [preflight] Pulling images required for setting up a Kubernetes cluster
  [preflight] This might take a minute or two, depending on the speed of your internet connection
  [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
  [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  [kubelet-start] Starting the kubelet
  [certs] Using certificateDir folder "/etc/kubernetes/pki"
  [certs] Generating "ca" certificate and key
  [certs] Generating "apiserver" certificate and key
  [certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.123.120]
  [certs] Generating "apiserver-kubelet-client" certificate and key
  [certs] Generating "front-proxy-ca" certificate and key
  [certs] Generating "front-proxy-client" certificate and key
  [certs] Generating "etcd/ca" certificate and key
  [certs] Generating "etcd/server" certificate and key
  [certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.123.120 127.0.0.1 ::1]
  [certs] Generating "etcd/peer" certificate and key
  [certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.123.120 127.0.0.1 ::1]
  [certs] Generating "etcd/healthcheck-client" certificate and key
  [certs] Generating "apiserver-etcd-client" certificate and key
  [certs] Generating "sa" key and public key
  [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
  [kubeconfig] Writing "admin.conf" kubeconfig file
  [kubeconfig] Writing "kubelet.conf" kubeconfig file
  [kubeconfig] Writing "controller-manager.conf" kubeconfig file
  [kubeconfig] Writing "scheduler.conf" kubeconfig file
  [control-plane] Using manifest folder "/etc/kubernetes/manifests"
  [control-plane] Creating static Pod manifest for "kube-apiserver"
  [control-plane] Creating static Pod manifest for "kube-controller-manager"
  W0615 01:09:50.445790    2131 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
  [control-plane] Creating static Pod manifest for "kube-scheduler"
  W0615 01:09:50.446638    2131 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
  [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
  [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
  [apiclient] All control plane components are healthy after 23.007847 seconds
  [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
  [kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
  [upload-certs] Skipping phase. Please see --upload-certs
  [mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
  [mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
  [bootstrap-token] Using token: bjybd0.kgkhkpl52fjpzv21
  [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
  [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
  [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
  [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
  [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
  [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
  [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
  [addons] Applied essential addon: CoreDNS
  [addons] Applied essential addon: kube-proxy

  Your Kubernetes control-plane has initialized successfully!

  To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

  You should now deploy a pod network to the cluster.
  Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

  Then you can join any number of worker nodes by running the following on each as root:

  kubeadm join 192.168.123.120:6443 --token bjybd0.kgkhkpl52fjpzv21 \
      --discovery-token-ca-cert-hash sha256:06d1c91b6d3dc0c5ccab57b9f28f42fb59f25bb7c5273d1f4b106214418df974

초기화 에러가 발생한 경우

  root@master:~# kubeadm init --apiserver-advertise-address=192.168.123.120 --pod-network-cidr=10.244.0.0/16
  [init] Using Kubernetes version: v1.27.3
  [preflight] Running pre-flight checks
  error execution phase preflight: [preflight] Some fatal errors occurred:
          [ERROR CRI]: container runtime is not running: output: time="2023-06-21T21:01:20+09:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
  , error: exit status 1
  [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
  To see the stack trace of this error execute with --v=5 or higher

해결방법

아래 명령어 실행 후 다시 초기화 진행

  sudo rm /etc/containerd/config.toml
  sudo systemctl restart containerd

초기화

  kubeadm init
  kubeadm init --apiserver-advertise-address=192.168.123.120 --pod-network-cidr=10.244.0.0/16

초기화 방법

Docker 초기화

  $ docker rm -f `docker ps -aq`
  $ docker volume rm `docker volume ls -q`
  $ sudo umount /var/lib/docker/volumes
  $ sudo rm -rf /var/lib/docker/
  $ sudo systemctl restart docker

kubeadm 초기화

  $ sudo kubeadm reset
  $ sudo systemctl restart kubelet
  $ sudo reboot

4. 토큰값 저장 [컨트롤+d로 저장]

cat > token.txt
kubeadm join 192.168.123.120:6443 --token bjybd0.kgkhkpl52fjpzv21 \
    --discovery-token-ca-cert-hash sha256:06d1c91b6d3dc0c5ccab57b9f28f42fb59f25bb7c5273d1f4b106214418df974

5. 계정에 권한 할당

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

권한이 없으면 아래 명령어가 동작하지 않음

  root@master:~# kubectl get nodes
  The connection to the server localhost:8080 was refused - did you specify the right host or port?
  root@master:~# mkdir -p $HOME/.kube
  root@master:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  root@master:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config
  root@master:~# kubectl get nodes
  NAME     STATUS     ROLES    AGE   VERSION
  master   NotReady   master   10m   v1.18.15
  root@master:~# exit

  로그아웃

  master@master:~$ kubectl get nodes
  The connection to the server localhost:8080 was refused - did you specify the right host or port?
  master@master:~$ mkdir -p $HOME/.kube
  master@master:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  master@master:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
  master@master:~$ kubectl get nodes
  NAME     STATUS     ROLES    AGE   VERSION
  master   NotReady   master   10m   v1.18.15

kubectl get nodes 커맨드를 입력하면 아래와 같이 status가 NotReady로 나오는 것을 볼 수 있다.
이는 POD 사이의 통신을 지원하는 CNI가 설정되어있지 않기 때문이다.

6. 컨테이너 네트워크 인터페이스(CNI) 설치

[6-1]~~아래 링크를 참고해 CALICO CNI를 설정함~~ https://projectcalico.docs.tigera.io/getting-started/kubernetes/quickstart

  kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml
  kubectl create -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml
  watch kubectl get pods -n calico-system

[6-2]~~아래 링크를 참고해 WEAVENET CNI를 설정함~~ https://www.weave.works/docs/net/latest/kubernetes/kube-addon/
```
  kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
```

[6-3]Flannel 설치 [https://github.com/flannel-io/flannel#flannel]

설치안됨

  kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

이걸로 진행

  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

[6-3]Flannel 설치 후 STATUS 변화

  master@master:~$ kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
  namespace/kube-flannel created
  serviceaccount/flannel created
  clusterrole.rbac.authorization.k8s.io/flannel created
  clusterrolebinding.rbac.authorization.k8s.io/flannel created
  configmap/kube-flannel-cfg created
  daemonset.apps/kube-flannel-ds created
  master@master:~$ kubectl get nodes
  NAME     STATUS     ROLES    AGE   VERSION
  master   NotReady   master   25m   v1.18.15
  master@master:~$ kubectl get nodes
  NAME     STATUS   ROLES    AGE   VERSION
  master   Ready    master   26m   v1.18.15

[6-4]Calico 설치

Calico CNI의 매니페스트 파일을 다운

  wget https://docs.projectcalico.org/manifests/calico.yaml

다운로드한 매니페스트 파일 (calico.yaml)을 편집기로 열고, “PodDisruptionBudget”과 관련된 라인을 주석 처리 또는 삭제

  # - apiVersion: policy/v1
  #   kind: PodDisruptionBudget
  #   metadata:
  #     name: calico-kube-controllers
  #   spec:
  #     maxUnavailable: 1
  #     selector:
  #       matchLabels:
  #         k8s-app: calico-kube-controllers

수정된 Calico CNI 매니페스트 파일을 클러스터에 적용
```
  kubectl apply -f calico.yaml
```

7. Worker node 구성, 클러스터 조인

0. 조인 삭제방법

kubectl drain <NODE_NAME> --delete-local-data --force --ignore-daemonsets
kubectl delete node <NODE_NAME>

kubectl drain node1 --delete-local-data --force --ignore-daemonsets
kubectl delete node node1

1. master node에서 token.txt에 저장한 토큰값 복사

kubeadm join 192.168.123.120:6443 --token bjybd0.kgkhkpl52fjpzv21 \
    --discovery-token-ca-cert-hash sha256:06d1c91b6d3dc0c5ccab57b9f28f42fb59f25bb7c5273d1f4b106214418df974

2. node1, node2에 붙여넣고 실행

Node1

root@node1:~# kubeadm join 192.168.123.120:6443 --token bjybd0.kgkhkpl52fjpzv21 \
>     --discovery-token-ca-cert-hash sha256:06d1c91b6d3dc0c5ccab57b9f28f42fb59f25bb7c5273d1f4b106214418df974
W0615 01:49:57.785100   14785 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.2. Latest validated version: 19.03
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Node2

root@node2:~# kubeadm join 192.168.123.120:6443 --token 7wq5vw.pwqf8h3urgy5jyt1 \
>     --discovery-token-ca-cert-hash sha256:0aba913bca4ceb4414b7b2518d8a914082a3525777543c83f86c065203cd2a8c
W0615 01:41:54.520077   13609 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.2. Latest validated version: 19.03
error execution phase preflight: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID "7wq5vw"
To see the stack trace of this error execute with --v=5 or higher
root@node2:~# kubeadm join 192.168.123.120:6443 --token bjybd0.kgkhkpl52fjpzv21 \
>     --discovery-token-ca-cert-hash sha256:06d1c91b6d3dc0c5ccab57b9f28f42fb59f25bb7c5273d1f4b106214418df974
W0615 01:50:02.167192   14045 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.2. Latest validated version: 19.03
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

master에서 worker node가 붙었는지 확인

 kubectl get nodes

확인

 root@master:~# kubectl get nodes -o wide
 NAME     STATUS     ROLES    AGE   VERSION    INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
 master   Ready      master   40m   v1.18.15   192.168.123.120   <none>        Ubuntu 20.04.6 LTS   5.15.0-73-generic   docker://24.0.2
 node1    NotReady   <none>   6s    v1.18.15   192.168.123.121   <none>        Ubuntu 20.04.6 LTS   5.15.0-73-generic   docker://24.0.2
 node2    NotReady   <none>   2s    v1.18.15   192.168.123.122   <none>        Ubuntu 20.04.6 LTS   5.15.0-73-generic   docker://24.0.2

상태보기

 kubectl get pod --all-namespaces

STATUS가 생성중이면 Worker node도 NotReady라고 뜬다.

 root@master:~# kubectl get pod --all-namespaces
 NAMESPACE      NAME                             READY   STATUS              RESTARTS   AGE
 kube-flannel   kube-flannel-ds-k88r7            0/1     CrashLoopBackOff    5          4m18s
 kube-flannel   kube-flannel-ds-q2czc            0/1     CrashLoopBackOff    8          18m
 kube-flannel   kube-flannel-ds-vt7t2            0/1     CrashLoopBackOff    5          4m22s
 kube-system    coredns-66bff467f8-r9xhg         0/1     ContainerCreating   0          44m
 kube-system    coredns-66bff467f8-xpnfz         0/1     ContainerCreating   0          44m
 kube-system    etcd-master                      1/1     Running             0          44m
 kube-system    kube-apiserver-master            1/1     Running             0          44m
 kube-system    kube-controller-manager-master   1/1     Running             0          44m
 kube-system    kube-proxy-5ht77                 1/1     Running             0          4m22s
 kube-system    kube-proxy-f7qm2                 1/1     Running             0          44m
 kube-system    kube-proxy-zs2tn                 1/1     Running             0          4m18s
 kube-system    kube-scheduler-master            1/1     Running             0          44m

명령어 자동화 구성

링크참조 [https://kubernetes.io/pt-br/docs/reference/kubectl/cheatsheet/]

root계정, 일반계정 둘다 실행

 source <(kubectl completion bash) 
 echo "source <(kubectl completion bash)" >> ~/.bashrc
 source <(kubeadm completion bash) 
 echo "source <(kubeadm completion bash)" >> ~/.bashrc

명령어 참고

 root@master:~# source <(kubectl completion bash)
 root@master:~# echo "source <(kubectl completion bash)" >> ~/.bashrc
 root@master:~# source <(kubeadm completion bash)
 root@master:~# echo "source <(kubeadm completion bash)" >> ~/.bashrc
 root@master:~# exit
 로그아웃
 master@master:~$ source <(kubectl completion bash)
 master@master:~$ echo "source <(kubectl completion bash)" >> ~/.bashrc
 master@master:~$ source <(kubeadm completion bash)
 master@master:~$ echo "source <(kubeadm completion bash)" >> ~/.bashrc

설치 완료

root@master:~# kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION    INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master   Ready    master   63m   v1.18.15   192.168.123.120   <none>        Ubuntu 20.04.6 LTS   5.15.0-73-generic   docker://24.0.2
node1    Ready    <none>   23m   v1.18.15   192.168.123.121   <none>        Ubuntu 20.04.6 LTS   5.15.0-73-generic   docker://24.0.2
node2    Ready    <none>   23m   v1.18.15   192.168.123.122   <none>        Ubuntu 20.04.6 LTS   5.15.0-73-generic   docker://24.0.2

2023-06-23

root@master:~# kubectl get pod --all-namespaces
NAMESPACE      NAME                             READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-29897            1/1     Running   0          93s
kube-flannel   kube-flannel-ds-fjchp            1/1     Running   0          49s
kube-flannel   kube-flannel-ds-ktwh6            1/1     Running   0          52s
kube-system    coredns-66bff467f8-dfdq8         1/1     Running   0          2m48s
kube-system    coredns-66bff467f8-lf6tn         1/1     Running   0          2m48s
kube-system    etcd-master                      1/1     Running   0          2m57s
kube-system    kube-apiserver-master            1/1     Running   0          2m57s
kube-system    kube-controller-manager-master   1/1     Running   0          2m57s
kube-system    kube-proxy-86zlk                 1/1     Running   0          52s
kube-system    kube-proxy-knjw5                 1/1     Running   0          49s
kube-system    kube-proxy-tdpqz                 1/1     Running   0          2m48s
kube-system    kube-scheduler-master            1/1     Running   0          2m57s
root@master:~# kubectl get po -A -o wide
NAMESPACE      NAME                             READY   STATUS    RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
kube-flannel   kube-flannel-ds-29897            1/1     Running   0          107s    192.168.123.120   master   <none>           <none>
kube-flannel   kube-flannel-ds-fjchp            1/1     Running   0          63s     192.168.123.122   node2    <none>           <none>
kube-flannel   kube-flannel-ds-ktwh6            1/1     Running   0          66s     192.168.123.121   node1    <none>           <none>
kube-system    coredns-66bff467f8-dfdq8         1/1     Running   0          3m2s    10.244.0.3        master   <none>           <none>
kube-system    coredns-66bff467f8-lf6tn         1/1     Running   0          3m2s    10.244.0.2        master   <none>           <none>
kube-system    etcd-master                      1/1     Running   0          3m11s   192.168.123.120   master   <none>           <none>
kube-system    kube-apiserver-master            1/1     Running   0          3m11s   192.168.123.120   master   <none>           <none>
kube-system    kube-controller-manager-master   1/1     Running   0          3m11s   192.168.123.120   master   <none>           <none>
kube-system    kube-proxy-86zlk                 1/1     Running   0          66s     192.168.123.121   node1    <none>           <none>
kube-system    kube-proxy-knjw5                 1/1     Running   0          63s     192.168.123.122   node2    <none>           <none>
kube-system    kube-proxy-tdpqz                 1/1     Running   0          3m2s    192.168.123.120   master   <none>           <none>
kube-system    kube-scheduler-master            1/1     Running   0          3m11s   192.168.123.120   master   <none>           <none>

kubernetes 명령어 자동완성 설정

kubernetes docs 참고

일반계정과 루트계정 둘다 실행시켜준다.

kubectl 명령어 자동화

source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.

kubeadm 명령어 자동화

source <(kubeadm completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubeadm completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.

Kubernetes를 설치하면서 리포팅을 동시에 하고있던중
가상으로 돌아가는 vm에 접속하기 위하여 포트포워딩을 해둔 상태였다.
비밀번호도 너무 단순하게 했던건지
모르는 외부 아이피로부터 접속시도중이라는 팝업창을 발견
불안감이 커지면서 귀찮아서 내린 방화벽을 다시 올리게 되었다.

방화벽 일부 특정 ip 허용

sudo ufw allow from <접근중인 ip>   # 특정ip만 접근 허용
sudo ufw enable                     # 방화벽 활성화
sudo ufw status verbose             # 상태 확인

sudo ufw allow from 192.168.123.120
sudo ufw allow from 192.168.123.121
sudo ufw allow from 192.168.123.122

root@master:~# sudo ufw allow from ***.***.***.***
규칙이 업데이트됐습니다
root@master:~# sudo ufw enable
명령은 존재하는 ssh 연결에 피해를 줄 수 있습니다. 이 작업과 함께 진행하시겠습니까(y|n)? y
방화벽이 활성 상태이며 시스템이 시작할 때 사용됩니다
root@master:~# sudo ufw status verbose
상태: 활성
로깅: on (low)
기본 설정: deny (내부로 들어옴), allow (외부로 나감), deny (라우팅 된)
새 프로필: skip

목적                         동작          출발
--                         --          --
Anywhere                   ALLOW IN    ***.***.***.***

By 김민서

The BlackCode got Dream