[ubuntu] Kubernetes 1.28 설치

Azure, VMWARE 설치검증 완료

https://domdom.tistory.com/591
  • Category : Kubernetes
  • Tag : Azure

사전작업

해당 작업은 vmware나 virtual box로 진행할때 필요한 진행과정임 Azure 등 작업필요 없음

1. 방화벽 해제

# 아래 포트를 허용시켜준다.
ufw enable
sudo ufw allow 22/tcp
sudo ufw allow 8080/tcp
sudo ufw allow 6443/tcp
sudo ufw status

2. ip 접근 허용

sudo ufw allow from <master ip>
sudo ufw allow from <node1 ip>
sudo ufw allow from <node2 ip>
  • memo 
    sudo ufw allow from 192.168.123.120
sudo ufw allow from 192.168.123.121
sudo ufw allow from 192.168.123.122

[ubuntu] Docker 설치

1-1. Docker 자동설치 스크립트

sudo wget -qO- http://get.docker.com/ | sh

1-2 자동설치 스크립트로 설치가 안되는 경우

  1. 설치전 필요한 패키지설치 
    # Update the apt package index and install packages to allow apt to use a repository over HTTPS:
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
  2. 도커에 GPG키 설치 
    # Add Docker’s official GPG key:
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
  3. repository등록 
    # Use the following command to set up the repository:
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  4. apt 업데이트 
    sudo apt-get update
  5. docker-ce docker-ce-cli containerd.io(도커엔진) 설치 진행 
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

2. Docker 활성화

sudo systemctl enable docker
sudo systemctl start docker
sudo systemctl enable containerd
sudo systemctl start containerd

3. Docker 컨테이너 실행 테스트

sudo docker run --rm hello-world

4. Docker의 cgroup driver를 cgroupfs에서 systemd로 변경

#sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# 재시작
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker  

# 변경된 cgroup driver 확인
sudo docker info | grep -i cgroup

5. Swap 메모리를 비활성화 [root 권한 필요]

# swap disable
swapoff -a
echo 0 > /proc/sys/vm/swappiness
sed -e '/swap/ s/^#*/#/' -i /etc/fstab

Kubernetes 설치

1. kubeadm, kubelet, kubectl을 설치

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update

1번을 진행 후 아래와 같은 문제가 발생하는 경우

  • 404 Not Found
master@master:~$ sudo apt-get update
Hit:1 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:5 https://download.docker.com/linux/ubuntu focal InRelease
Ign:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Err:7 https://packages.cloud.google.com/apt kubernetes-xenial Release
  404  Not Found [IP: 142.250.206.206 443]
Reading package lists... Done
E: The repository 'https://apt.kubernetes.io kubernetes-xenial Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
  • 다음 과정 수행
sudo su -
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update

[2-1]. 1.18버전 설치

sudo apt install -y kubelet=1.18.15-00 kubeadm=1.18.15-00 kubectl=1.18.15-00

[2-2]. 1.19버전 설치 [flannal_test]

sudo apt-get install -y kubelet=1.19.15-00 kubeadm=1.19.15-00 kubectl=1.19.15-00

[2-3] 최신 버전 설치

sudo apt-get install -y kubelet kubeadm kubectl

3. 자동으로 업데이트 되지 않도록 패키지 버전을 고정

sudo apt-mark hold kubelet kubeadm kubectl

4. [MASTER] 마스터노드와 워커노드를 설정

다음으로, 마스터노드와 워커노드를 설정해줍니다.

kubeadm init 명령어를 통해 마스터 노드를 초기화 및 실행할 수 있습니다.

마스터 노드 세팅을 위해 필요한 옵션은 아래와 같습니다.

명령어 내용
–pod-network-cidr Pod 네트워크를 설정합니다.
–apiserver-advertise-address 마스터노드의 API Server 주소를 설정합니다.

쿠버네티스 네트워크 모델 중 하나인 Flannel을 사용하기 위해
–pod-network-cidr 옵션에 10.244.0.0/16을 넣어줍니다.
Flannel은 서로다른 노드에 있는 Pod간 통신을 원활히 하기 위해 필요한 네트워크 플러그인 이며,
Flannel의 기본 네트워크 대역은 10.244.0.0/16입니다.
apiserver-advertise-address 에는 ifconfig 명령어를 입력하면 나오는 eth0 IP를 넣어주면 됩니다.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address={마스터 노드 IP}
  • memo 
    sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.0.0.4
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.123.120

* 다음과 같은 에러 발생시

master@master:~$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.0.0.4
[init] Using Kubernetes version: v1.28.3
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2023-11-01T15:17:07Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc 
error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
  • /etc/containerd/config.toml을 삭제하고 containerd 서비스를 다시 시작 
    sudo rm /etc/containerd/config.toml
systemctl restart containerd

2024-05-17 아래 로그처럼 클러스터 조인 실패처리

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2024-05-17T06:19:52Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

1. worker node1,2에서 config.toml 확인

sudo nano /etc/containerd/config.toml

2. “cir” 삭제 후 저장

  • GNU nano 4.8 화면창 참고

    disabled_plugins = [“cri”]에서 cri를 삭제 후 Ctrl+o, Ctrl+x를 눌러 저장 후 나온다

#   Copyright 2018-2022 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

disabled_plugins = []

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

3. 컨테이너 런타임 재시작

sudo systemctl restart containerd

4. 클러스터 조인 재실행

root@node1:~# sudo systemctl restart containerd
root@node1:~# kubeadm join 10.0.0.4:6443 --token jvelw6.y3jp89wc1s3zkf52         --discovery-token-ca-cert-hash sha256:e42cb2a4262e8106142b3dba64713d13b9757aa648775e9a841e42b3432589a2
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

5. [MASTER] 토큰 복사

cat > token.txt
sudo kubeadm join 10.0.0.4:6443 --token fz8v03.ouh96b4fenr45xbw \
        --discovery-token-ca-cert-hash sha256:9ffebbe31308a9698a357f28349a194d7e1ebf03efdd8003e5502d6f6767cacf

sudo kubeadm join 192.168.123.120:6443 --token 76r7q8.kj3508ymnonx63zd \
        --discovery-token-ca-cert-hash sha256:cea33b0f64bd4c561a8e09a2bbe6f18c411b0178837541ab83f2506bc12e8fa6

6. [MASTER] 클러스터를 사용하기 위한 권한 변경작업 [일반, 관리자 둘다해줌]

sudo mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

7. [MASTER] Pod 끼리의 통신을 도와주는 Flannel Pod 네트워크를 클러스터에 배포

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

8. 마스터 서버의 포트설정 후 접속테스트

https://<master node public ip>:6443

# 접속시 아래처럼 웹 페이지를 볼 수 있음
Client sent an HTTP request to an HTTPS server.

9. [NODE1, NODE2] 워커노드 조인

복사했던 토큰을 각 노드에 실행해준다.

  • 만약 다음과 같은 에러가 발생한 경우 
    node1@node1:~$ kubeadm join 10.0.0.4:6443 --token fz8v03.ouh
>         --discovery-token-ca-cert-hash sha256:9ffebbe31308ebf03efdd8003e5502d6f6767cacf
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal erro
        [ERROR IsPrivilegedUser]: user is not running as roo
[preflight] If you know what you are doing, you can make a cignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or h
  • kubeadm을 root 사용자 권한으로 실행하지 않아서 발생한 것
  • 다음과 같이 실행 
    sudo kubeadm join 10.0.0.4:6443 --token fz8v03.ouh96b4fenr45xbw --discovery-token-ca-cert-hash sha256:9ffebbe31308a9698a357f28349a194d7e1ebf03efdd8003e5502d6f6767cacf

kubeadm join 192.168.123.120:6443 --token 76r7q8.kj3508ymnonx63zd --discovery-token-ca-cert-hash sha256:cea33b0f64bd4c561a8e09a2bbe6f18c411b0178837541ab83f2506bc12e8fa6 --node-name new-node-name
sudo kubeadm join 192.168.123.120:6443 --token ion4b3.32c033cpyif8su15 --discovery-token-ca-cert-hash sha256:75af9c3c175620235728ed1c3ad4bffc32fc0c042ceefab264e18cddf15ffce8
  • 토큰이 만료되어 재발행이 필요한 경우
    1. 발행된 토큰 리스트 확인 
        kubeadm token list
    2. 기존 토큰 삭제 
        kubeadm token delete <TOKEN>
    3. 신규 토큰 발행 
        kubeadm token create --print-join-command
  • 조인 자체가 안되는 경우 
    node1@node1:~$ sudo kubeadm join 192.168.123.120:6443 --token xgb8m3.h7hbxmpt6zpnq1pq --discovery-token-ca-cert-hash sha256:be0caf1a69e12f7742294b3d07bfa2d1c8fe5ab8c1cd3c105442b0010a78c057
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2023-11-05T22:43:00+09:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
    1. [WORKERNODE] vi 편집기로 config.toml파일 수정 
        sudo vi /etc/containerd/config.toml
    2. [WORKERNODE] disabled_plugins 라인 주석처리 후 저장 
        # disabled_plugins = ["cri"]
    3. [WORKERNODE] 컨테이너 재실행 
        sudo systemctl restart containerd
    4. [WORKERNODE] 다시 조인 
        sudo kubeadm join 192.168.123.120:6443 --token xgb8m3.h7hbxmpt6zpnq1pq --discovery-token-ca-cert-hash sha256:be0caf1a69e12f7742294b3d07bfa2d1c8fe5ab8c1cd3c105442b0010a78c057
  • 정상적으로 조인이 되는것을 확인할 수 있다. 
    node1@node1:~$ sudo kubeadm join 10.0.0.4:6443 --token fz8v0
>         --discovery-token-ca-cert-hash sha256:9ffebbe31308ebf03efdd8003e5502d6f6767cacf
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kube
cm kubeadm-config -o yaml'

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

설치 확인

master@master:~$ kubectl get pod --all-namespaces
NAMESPACE      NAME                             READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-gm8tj            1/1     Running   0          83s
kube-flannel   kube-flannel-ds-w9j26            1/1     Running   0          86s
kube-flannel   kube-flannel-ds-zwzp4            1/1     Running   0          14m
kube-system    coredns-5dd5756b68-2wht8         1/1     Running   0          18m
kube-system    coredns-5dd5756b68-89ns4         1/1     Running   0          18m
kube-system    etcd-master                      1/1     Running   0          18m
kube-system    kube-apiserver-master            1/1     Running   0          18m
kube-system    kube-controller-manager-master   1/1     Running   0          18m
kube-system    kube-proxy-2qv8m                 1/1     Running   0          86s
kube-system    kube-proxy-nwfnn                 1/1     Running   0          18m
kube-system    kube-proxy-xvg7w                 1/1     Running   0          83s
kube-system    kube-scheduler-master            1/1     Running   0          18m
master@master:~$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   19m     v1.28.2
node1    Ready    <none>          2m36s   v1.28.2
node2    Ready    <none>          2m33s   v1.28.2

Share this post

김민서's photo

By

The BlackCode got Dream