如何用k3s和ZeroTier无公网ip优雅地部署Kubernetes集群

发布于 2022-08-31  344 次阅读


自己Host的云服务使用起来比付费购买的更爽。

——沃兹基·硕德

一:什么是ZeroTier?什么是k3s?

关于ZeroTier,请见我的另一篇博客:如何用zerotier优雅地异地组网 -「桃花源」

k3s是一个轻量级的Kubernetes版本,由Rancher公司开发/维护,因为其体积小、性能要求不高,故而成为自建Kubernetes集群的好选择。

想要异地部署自己的集群,你至少需要2台不同的机器,他们可以有公网ip,也可以没有。(反正我们要用ZeroTier异地组网)

   “如果你只有一台机器,还要什么集群呢(笑”  ==>  展开 / 收缩

二:部署集群之前的必要准备

任务一:使用ZeroTier给所有的机器组网

这一步非常的简单,跟随上面提到的那篇博客就可以了。下文默认你异地组网这一步已经完成了。

任务二:安装k3s的前置工作

首先,我们需要在所有的机器上都禁用缓存、开启必要的端口、调整ZeroTier的设置。这些都是k3s的必要要求,或者是保证k3s和ZeroTier不冲突的前置条件。在每台机器上执行以下指令:

# 禁用缓存
sudo swapoff -a
# 去/etc/fstab文件中把包含"swap"字样的行统统注释掉
sudo vim /etc/fstab
# 如果什么都没注释,不需要重启,否则重启
sudo reboot

# 打开k3s默认端口
sudo ufw allow 6443
# 打开k3s自带的flannel VXLAN默认端口
sudo ufw allow 8472
# 打开k3s自带的metrics server默认端口
sudo ufw allow 10250

# 更改ZeroTier的设置,去/var/lib/zerotier-one/local.conf文件
sudo vim /var/lib/zerotier-one/local.conf

然后在这个文件的末尾添加如下内容:

{
	"settings": {
		"interfacePrefixBlacklist": [ "flannel" ]
	}
}

添加完后保存文件。这些设置的目的是让ZeroTier无视flannel即将建立的虚拟网络设备上的任何包。这是因为flannel的虚拟网络即将建立在ZeroTier之上,如果ZeroTier将包转发到ZeroTier网络,会造成Storming(风暴)。

接下来,我们执行以下指令:

# 重启ZeroTier服务
sudo systemctl daemon-reload
sudo systemctl restart zerotier-one

# 下载官方安装脚本
sudo curl -L "https://get.k3s.io" -o "./install-k3s.sh"
sudo chmod +x "./install-k3s.sh"

三:重要的选择——在线安装还是离线安装?

任务三:决定在线还是离线安装

首先,你需要选一台机器当作你的主节点(master node)。其他的节点都会是工作节点(worker node)。根据你的主节点的网络环境,你可以选择在线安装或者离线安装。如果你的主节点可以访问GoogleGithub,那么你可以跳过这一步。否则,你需要进行离线安装

离线安装的额外准备步骤:

  1. 在另一个能够访问Github的节点下载最新的k3s release,然后传到你的主节点。
  2. 把该文件命名为“k3s”并且放进“/usr/local/bin/”目录。
  3. 执行以下指令:
sudo chmod +x /usr/local/bin/k3s
export INSTALL_K3S_SKIP_DOWNLOAD=true

接下来,你就可以进入下一步安装过程了。

四:开始安装部署k3s!

任务四:正式部署k3s!

首先,你需要找出你的主节点上的ZeroTier所在的虚拟网络设备。你可以用以下指令列出网络设备和相关信息:

ip address

ZeroTier所创建的虚拟网络设备通常以字母“zt”开头,很好认出。下文中假设你的ZeroTier虚拟网络设备名称为“ztr0”。在你的主节点运行以下指令:

# 指定flannel的底层网络设备为ZeroTier的虚拟设备
export INSTALL_K3S_EXEC="server --flannel-iface=ztr0"
# 继承环境变量并提升权限执行安装脚本
sudo -E sh -c "./install-k3s.sh"
# 输出重要信息,务必做好记录,后文会用到
sudo cat "/etc/rancher/k3s/k3s.yaml"
sudo cat "/var/lib/rancher/k3s/server/node-token"
# 检查k3s状态。internal ip(内部ip)应该与ZeroTier中该节点的ip地址匹配
sudo kubectl get nodes -o wide

接下来我们去设置工作节点。此时需要用到你的主节点在ZeroTier中的地址和刚才输出的node-token。下文中,我们假设你的主节点在ZeroTier内部的地址为“master.zero”,刚才输出的node-token为“MY_TOKEN”。在你的工作节点上运行以下指令:

# 准备环境变量
export K3S_URL=https://master.zero:6443
export K3S_TOKEN=MY_TOKEN
export INSTALL_K3S_EXEC="agent --flannel-iface=ztr0 --server=${K3S_URL} --token=${K3S_TOKEN}"
# 继承环境变量并提升权限执行安装脚本
sudo -E sh -c "./install-k3s.sh"

等待几分钟后,重新回到你的主节点,在上面执行如下指令:

sudo kubectl get nodes

这时,如果列出的nodes(节点)不止一个,就意味着你的工作节点成功加入了集群。你接下来可以重复刚才的步骤,直到你的所有结点都加入了集群。

五:我不小心装错了,怎样卸载k3s?

在你的节点上运行下面的指令,就可以卸载k3s了:

# 停止k3s服务(需要等几秒种)
cd /usr/local/bin/
sudo sh -c "./k3s-killall.sh"

# 如果该节点是主节点:
sudo sh -c "./k3s-uninstall.sh"

# 如果该节点是工作节点:
sudo sh -c "./k3s-agent-uninstall.sh"

# 清理残留的目录和文件
rm -rf /var/lib/rancher
rm -rf /etc/rancher

六:进阶 - 安装Rancher控制台管理你的k3s集群(可选)

任务五:安装Rancher控制台GUI

如果你觉得用命令行管理k3s实在是太麻烦,那么你可以选择安装Rancher控制台,然后用GUI来管理你的集群。在你的主节点上执行以下的指令:

# 切换成root用户
sudo su -

# 安装helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
sh -c "./get_helm.sh"

# 安装cert-manager(请自行选择合适的版本)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.8.0 --kubeconfig /etc/rancher/k3s/k3s.yaml --set startupapicheck.timeout=10m
kubectl get pods --namespace cert-manager

# 安装rancher控制台,假设密码为password(建议强密码)
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
kubectl create namespace cattle-system
helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=master.zero --set bootstrapPassword=password --kubeconfig /etc/rancher/k3s/k3s.yaml
kubectl -n cattle-system rollout status deploy/rancher
kubectl -n cattle-system get deploy rancher

这一步操作比较慢,请耐心等待。等完成之后,你就可以在你的主节点地址(本文假设是master.zero)看到你的Rancher控制台登陆界面啦!登陆进去之后可以调整各种设置,也可以很方便地运行容器、部署服务等等。

七:帮助你快速部署k3s的保姆级脚本!

最后,如果你觉得按照教程部署还是太麻烦了,你也可以尝试下面这个脚本。这个脚本不包含Rancher控制台,只包含ZeroTier的安装与自动配置、k3s所需的自动配置,和k3s的部署。它支持在线/离线部署,也支持主节点/工作节点的部署。

另注:请不要盲目相信互联网的脚本,包括此脚本。强烈建议您仔细阅读或者作必要的改动。此脚本只经过有限的测试,如果因为脚本过时/脚本有bug而导致直接运行出了问题,会得不偿失。

#!/bin/bash
# prepare -------------------------------------------
Y=$'\e[33m';
G=$'\e[32m';
C=$'\e[36m';
N=$'\e[0m';
isEmptyInput(){
    input=$1
    if [ -z "$input" ]; then
        echo "${Y}Empty user input. Terminated.${N}"
        exit
    fi
}
simpleContinue(){
    read -p "${G}Continue? ${Y}(y/n)${N} " v_continue
    isEmptyInput $v_continue
    if [ $v_continue != "y" ]; then
        echo "${Y}User Aborted.${N}"
        exit
    fi
}
v_is_root=$(id -u)
if [ $v_is_root -ne 0 ]; then
    echo "${C}This script needs root privileges."
    simpleContinue
    sudo "$0" "[email protected]"
    exit $?
else
    echo "${C}You are running as root now."
fi
echo "${C}This script will perform the following tasks:"
echo "${C}- ${N}Install / Configure Zerotier"
echo "${C}- ${N}Disable swap permanently"
echo "${C}- ${N}Open ports necessary for k3s"
echo "${C}- ${N}Install and configure k3s (this script does ${Y}NOT${N} support HA cluster)"
echo "${C}Please make sure that ${Y}k3s${C} has ${Y}NOT${C} been installed on your system."
echo "${C}Please also make sure that you have ${Y}Internet Access${C} and can access github."
echo "${C}Otherwise, please abort by typing ${Y}n${C}."
simpleContinue
# disable swap permantly on all nodes -------------------
echo "${C}Turning off swap...${N}"
swapoff -a
sed -i 's/^[^#]*.*swap.*$/#&/' /etc/fstab
mount -a
echo "${G}...Done."
# open port on all nodes ---------------------
echo "${C}Detecting ufw installation..."
v_ufw_str=$( dpkg --get-selections | grep ufw )
if [[ "$v_ufw_str" == *"install"* ]]; then
    echo "${G}...Done."
    echo "${C}Configuring ports...${N}"
    ufw allow 6443
    ufw allow 8472
    ufw allow 10250
	ufw allow 7946
    echo "${G}...Done."
else
    echo "${G}...Done."
    echo "${Y}Warning: ufw is not detected. Automatic port configuration aborted."
    echo "${C}Make sure the following ports are open before continuing:"
    echo "${C}- ${N}6443 (tcp/udp) ${C}- dedault port for k3s"
    echo "${C}- ${N}8472 (tcp/udp) ${C}- dedault port for flannel VXLAN"
    echo "${C}- ${N}10250 (tcp/udp) ${C}- dedault port for metrics server"
    simpleContinue
fi
# zerotier installation -------------------------
echo "${C}Install ZeroTier? (if you have already installed, type ${Y}n${C} to skip installation)"
read -p "${G}Continue? ${Y}(y/n)${N} " v_zt_install
isEmptyInput $v_zt_install
if [ $v_zt_install != "y" ]; then
    echo "${C}ZeroTier installation skipped."
else
    echo "${C}Installing ZeroTier..."
    curl -s 'https://raw.githubusercontent.com/zerotier/ZeroTierOne/master/doc/contact%40zerotier.com.gpg' | gpg --import && if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi
    echo "${G}...Done."
fi
echo "${C}Join ZeroTier Network? (if you have already joined a network, type ${Y}n${C} to skip)"
read -p "${G}Continue? ${Y}(y/n)${N} " v_zt_join
isEmptyInput $v_zt_join
if [ $v_zt_join != "y" ]; then
    echo "${C}Joining ZeroTier Network skipped."
else
    echo "${C}Joining ZeroTier Network..."
    v_zt_id_loop=1
    while [ $v_zt_id_loop -eq 1 ]; do
        read -p "${C}Input your ZeroTier Network ID:${N} " v_zt_id
        isEmptyInput $v_zt_id
        echo "${C}Your ZeroTier ID is ${G}$v_zt_id${C}. If incorrect, type ${Y}n${C}."
        read -p "${G}Continue? ${Y}(y/n)${N} " v_zt_id_lp_c
        isEmptyInput $v_zt_id_lp_c
        if [ $v_zt_id_lp_c == "y" ]; then
            zerotier-cli join $v_zt_id
            v_zt_id_loop=0
        fi
    done
    echo "${G}...Done."
fi
# prepare zerotier on all nodes ----------------
echo "${C}Configuring Zerotier..."
zt_config_path="/var/lib/zerotier-one/local.conf"
touch $zt_config_path
mv $zt_config_path ${zt_config_path}.backup
touch $zt_config_path
cat <<EOF >> $zt_config_path
{
    "settings": {
        "interfacePrefixBlacklist": [ "flannel" ]
    }
}
EOF
systemctl daemon-reload
systemctl restart zerotier-one
echo "${G}...Done. ${C}(backup file at ${Y}${zt_config_path}${C}.backup)"
# k3s install preparation -------------------------
echo "${C}Online or Offline Installation?"
echo "${C}Online: if you have no trouble accessing git or google, type ${Y}y${C}."
echo "${C}Offline: if you have trouble accessing git or google, type ${Y}n${C}."
read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_inst
isEmptyInput $v_k3s_inst
if [ $v_k3s_inst != "y" ]; then
	v_k3s_off_loop=1
	while [ $v_k3s_off_loop -eq 1 ]; do
		echo "${C}Preparing for offline install..."
		echo "${C}Please download the latest k3s release binary \"${Y}k3s${C}\" and put it under the directory \"${Y}/usr/local/bin/${C}\" before continuing."
		read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_off_lp_c
		isEmptyInput $v_k3s_off_lp_c
		if [ $v_k3s_off_lp_c == "y" ]; then
			v_k3s_off_loop=0
		fi
	done
	chmod +x /usr/local/bin/k3s
	export INSTALL_K3S_SKIP_DOWNLOAD=true
fi
echo "${C}Preparing installation script...${N}"
curl -L "https://get.k3s.io" -o "./install-k3s.sh"
chmod +x "./install-k3s.sh"
echo "${G}...Done."
# k3s installation ------------------------------
echo "${C}Server or Agent Installation?"
echo "${C}Server (unique master node and control plane for the cluster): type ${Y}y${C}."
echo "${C}Agent (worker node): type ${Y}n${C}."
read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_type
isEmptyInput $v_k3s_type
echo "${C}Looking for a ZeroTier interface..."
v_zt_iface=$(ls /sys/class/net | grep zt | head -1)
if [ -z "$v_zt_iface" ]; then
	echo "${Y}Fatal Error: Cannot find a ZeroTier network interface. Terminated.${N}"
	exit
fi
echo "${G}...Done."
echo "${C}Using ZeroTier network interface ${G}$v_zt_iface${C}. If you have more than one ZeroTier network interface and would like to use another one, please type ${Y}n${C}."
read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_if_sel
isEmptyInput $v_k3s_if_sel
if [ $v_k3s_if_sel != "y" ]; then
	echo "${C}Preparing for user-specified ZeroTier network interface..."
	v_k3s_if_loop=1
	while [ $v_k3s_if_loop -eq 1 ]; do
		read -p "${C}Input ZeroTier network interface name (starts with ${Y}zt${C}): " v_zt_if
		isEmptyInput $v_zt_if
		echo "${C}Using ZeroTier network interface ${G}$v_zt_if${C}. If incorrect, type ${Y}n${C}."
		read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_if_lp_c
		isEmptyInput $v_k3s_if_lp_c
		if [ $v_k3s_if_lp_c == "y" ]; then
			v_zt_iface=$v_zt_if
			v_k3s_if_loop=0
		fi
	done
	echo "${G}...Done."
fi
if [ $v_k3s_type != "y" ]; then
	echo "${C}Installing agent..."
	v_k3s_url_loop=1
	while [ $v_k3s_url_loop -eq 1 ]; do
        read -p "${C}Input k3s server ip/name (the server url will be: ${Y}https://${C}[ip/name]${Y}:6443${C}): " v_k3s_url
        isEmptyInput $v_k3s_url
        echo "${C}Your k3s server ip/name is ${G}$v_k3s_url${C}. If incorrect, type ${Y}n${C}."
        read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_url_lp_c
        isEmptyInput $v_k3s_url_lp_c
        if [ $v_k3s_url_lp_c == "y" ]; then
            export K3S_URL="https://$v_k3s_url:6443"
            v_k3s_url_loop=0
        fi
	done
	v_k3s_tok_loop=1
	while [ $v_k3s_tok_loop -eq 1 ]; do
        read -p "${C}Input your k3s node-token (stored in the file \"${Y}/var/lib/rancher/k3s/server/node-token${C}\" on the machine running k3s ${Y}server${C}):${N} " v_k3s_tok
        isEmptyInput $v_k3s_tok
        echo "${C}Your k3s node-token is ${G}$v_k3s_tok${C}. If incorrect, type ${Y}n${C}."
        read -p "${G}Continue? ${Y}(y/n)${N} " v_k3s_tok_lp_c
        isEmptyInput $v_k3s_tok_lp_c
        if [ $v_k3s_tok_lp_c == "y" ]; then
            export K3S_TOKEN=$v_k3s_tok
            v_k3s_tok_loop=0
        fi
	done
	export INSTALL_K3S_EXEC="agent --flannel-iface=$v_zt_iface --server=${K3S_URL} --token=${K3S_TOKEN}"
	echo "${C}Install argument: ${Y}$INSTALL_K3S_EXEC${N}"
	sudo -E sh -c "./install-k3s.sh"
	echo "${G}...Done."
else
	echo "${C}Installing server..."
	export INSTALL_K3S_EXEC="server --flannel-iface=$v_zt_iface --disable=servicelb"
	echo "${C}Install argument: ${Y}$INSTALL_K3S_EXEC${N}"
	sudo -E sh -c "./install-k3s.sh"
	echo "${G}...Done."
	echo "${C}Printing node-token (from ${Y}/var/lib/rancher/k3s/server/node-token${C})...${N}"
	cat "/var/lib/rancher/k3s/server/node-token"
	echo "${G}...Done."
	echo "${C}Checking k3s status...${N}"
	kubectl get nodes
	echo "${G}...Done."
fi
echo "${C}=== ${G}Congratulations, installation has completed!${C} ===${N}"

恭喜你看完了这篇文章!撒花花★,°:.☆( ̄▽ ̄)/$:.°★

希望对你有所帮助~

顺便再求个友链/打赏/转发什么的┑( ̄Д  ̄)┍


lbz敬上