kubernetes03-部署k8s集群基本环境

00:文章简介

介绍k8s集群部署的环境准备。

01: 了解k8s基础环境组成及相关要求

1.1:基本环境组成

OS:Ubuntu 20.04.2 LTS

容器技术:docker

批量管理工具:ansible

内网DNS环境:bind

SSL证书:阿里云

相关域名: linux98.com

资源配置文件存放目录: /root/mykube

1.2:系统详情

网络配置

Text
1
2
3
4
5
6
7
8
9
network:
version: 2
ethernets:
eth0:
addresses: [172.20.200.201/16]
gateway4: 172.20.1.1
nameservers:
search: [linux98.com]
addresses: [172.20.254.10, 172.20.254.11]

软件源配置

Text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse

1.3:相关要求

关闭swap分区

Text
1
2
echo "vm.swappiness=0" >> /etc/sysctl.d/k8s.conf
sed -i 's/.*swap/#&/' /etc/fstab

配置内核参数

Text
1
2
3
echo "net.bridge.bridge-nf-call-ip6tables=1" >> /etc/sysctl.d/k8s.conf
echo "net.bridge.bridge-nf-call-iptables=1" >> /etc/sysctl.d/k8s.conf
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.d/k8s.conf

启动相关模块

Text
1
2
modprobe br_netfilter
modprobe overlay

02:使用cobbler批量部署基础架构

这里只安装底层操作系统,初始化配置在下面手动做

03:配置master节点环境

3.1:基础环境

Text
1
2
3
4
5
6
7
8
9
10
cat >> /etc/sysctl.d/k8s.conf <<EOF
vm.swappiness=0
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
EOF

modprobe br_netfilter
modprobe overlay
sysctl -p /etc/sysctl.d/k8s.conf

3.2:docker环境

安装略,前面有

修改docker启动方式和设置本地镜像仓库

Text
1
2
3
4
5
6
7
8
9
cat /etc/docker/daemon.json

{
"registry-mirrors": ["https://73yi6cz9.mirror.aliyuncs.com"],
"insecure-registries": ["harbor.linux98.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}

systemctl restart docker

3.3:安装k8s软件源

Text
1
2
3
4
5
6
7
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

04:相关要求编排为playbook

gitee

05:在harbor节点上部署harbor镜像仓库

docker基础环境略

Text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apt-get install docker-compose -y
wget https://github.com/goharbor/harbor/releases/download/v2.3.2/harbor-offlineinstaller-v2.3.2.tgz
tar -xf harbor-offlineinstaller-v2.3.2.tgz -C /usr/local/
cd /usr/local/harbor

docker load < harbor.v2.3.2.tar.gz
cp harbor.yml.tmpl harbor.yml

# 修改如下部分
hostname: harbor.linux98.com
https:
port: 443
# nginx ssl认证
certificate: /usr/local/harbor/cert/6442545_harbor.linux98.com
private_key: /usr/local/harbor/cert/6442545_harbor.linux98.com.key
harbor_admin_password: 123456
data_volume: /data/harbor

# 执行检查
./prepare

# 安装
./install.sh

# systemd文件
cat > /lib/systemd/system/harbor.service << EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
#需要注意harbor的安装位置
ExecStart=/usr/bin/docker-compose -f /usr/local/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /usr/local/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

# 设置开机自启动
systemctl start harbor
systemctl enable harbor

使用浏览器登陆harbor.linux98l.com,用户名:admin,密码:123456

左侧导航->系统管理->用户管理->创建用户-> 创建普通用户:用户名cpli,密码:A12345678a

使用普通用户登陆创建公开项目:google_containers、appimages

image

06:在master节点下载并上传k8s的镜像

查看k8s v1.22.1所依赖的镜像及版本

Text
1
2
3
4
5
6
7
8
9
10
# 查看
kubeadm config images list --kubernetes-version=v1.22.1

k8s.gcr.io/kube-apiserver:v1.22.1
k8s.gcr.io/kube-controller-manager:v1.22.1
k8s.gcr.io/kube-scheduler:v1.22.1
k8s.gcr.io/kube-proxy:v1.22.1
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4

登陆harbor并使用脚本下载上面的镜像,然后上传到harbor

Text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 执行脚本
#!/bin/bash
#login
expect -c "
spawn docker login harbor.linux98.com
expect {
\"*Username:*\" {send \"cpli\r\"; exp_continue}
\"*Password:*\" {send \"A12345678a\r\"; exp_continue}
} "

# download and upload
images=$(kubeadm config images list --kubernetes-version=v1.22.1 | grep -o '[a-z|-]*:.*')

for i in ${images}
do
docker pull registry.aliyuncs.com/google_containers/$i
docker tag registry.aliyuncs.com/google_containers/$i harbor.linux98.com/google_containers/$i
docker rmi registry.aliyuncs.com/google_containers/$i
docker push harbor.linux98.com/google_containers/$i
done

下载dashboard和flannel插件的镜像,上传到harbor

flannel的github主页:https://github.com/flannel-io/flannel 这里使用的是v0.14.0

dashboard的github主页:https://github.com/kubernetes/dashboard

Text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 下载flannel和dashboard镜像并上传
# wget https://github.com/flannel-io/flannel/blob/v0.14.0/Documentation/kube-flannel.yml

docker pull quay.io/coreos/flannel:v0.14.0
docker tag quay.io/coreos/flannel:v0.14.0 harbor.linux98.com/google_containers/flannel:v0.14.0
docker rmi quay.io/coreos/flannel:v0.14.0

# dashboard-github https://github.com/kubernetes/dashboard
# wget https://github.com/kubernetes/dashboard/blob/v2.3.1/aio/deploy/recommended.yaml

docker pull kubernetesui/dashboard:v2.3.1
docker pull kubernetesui/metrics-scraper:v1.0.6

docker tag kubernetesui/dashboard:v2.3.1 harbor.linux98.com/google_containers/dashboard:v2.3.1
docker tag kubernetesui/metrics-scraper:v1.0.6 harbor.linux98.com/google_containers/metrics-scraper:v1.0.6

docker rmi kubernetesui/dashboard:v2.3.1
docker rmi kubernetesui/metrics-scraper:v1.0.6

最后登陆harbor查看效果

image