0.准备工作

• 这次的安装部署是在arm64上面执行的，板子为RK3328，系统为Ubuntu16.04。

  主要有一个Master和两个Node，主要需要安装以下的组件：

这里Master的组件是：kube-apiserver,kube-controller-manager,kube-scheduler

• 提前下载好相关的组件

  • etcd-v3.3.5-linux-arm64.tar.gz
  • flannel-v0.10.0-linux-arm64.tar.gz
  • kubernetes-node-linux-arm64.tar.gz
  • kubernetes-server-linux-arm64.tar.gz

  具体下载到GitHub上下载，如果不知道网址，可以直接搜 xxx release（比如etcd release），一般出来第一个就是了，然后选择对应版本下载

1.部署Master

1. 部署前的初始化

   • 首先以root用户执行以下动作

     1. 关闭防火墙ufw disable

     2. 安装ntp（如果没有安装) sudo apt-get install ntp

     3. 添加主机名和IP到/etc/hosts

        172.16.32.10 k8s-master
        172.16.32.11 k8s-node1
        172.16.32.12 k8s-node2
        

     4. 新建一个k8s-master用户，并且赋予root权限

        useradd -m -d /home/k8s-master -s /bin/bash k8s-master
         sudo sed -i -r '/root.*ALL=\(ALL.*ALL/a   \k8s-master  ALL=\(ALL\) NOPASSWD: ALL' /etc/sudoers
        

     5. 切换到k8s-master用户 su k8s-master

   • 切换到k8s-master用户执行以下动作

     1. 创建文件夹，用来保存bin和组件的配置文件

        sudo mkdir -p ~/kubernetes/bin && sudo mkdir ~/kubernetes/cfg

     2. 设置环境变量，因为我们的二进制放在自己指定的路径，加入环境变量就方便使用。

        echo "export PATH=\$PATH:/home/k8s-master/kubernetes/bin" >> ~/.bashrc

        source ~/.bashrc

2. 安装ETCD

   • 解压etcd-v3.3.5-linux-arm64.tar.gz

     sudo tar -zxvf etcd-v3.3.5-linux-arm64.tar.gz

   • 复制解压目录下的etcd，etcdctl到~/kubernetes/bin

     sudo cp etcd-v3.3.5-linux-arm64/etcd* ~/kubernetes/bin

   • 创建etcd配置文件 sudo vi ~/kubernetes/cfg/ectd.cfg

     ETCD_NAME="default"
     ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
     ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
     ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
     

     注意这里在启动etcd服务之前，需要先创建好/var/lib/etcd目录，存放etcd的数据

   • 创建etcd服务文件 sudo vi /lib/systemd/system/etcd.service

     [Unit]
     Description=Etcd Server
     After=network.target
     
     [Service]
     Type=simple
     WorkingDirectory=/var/lib/etcd
     Environment=ETCD_UNSUPPORTED_ARCH=arm64
     EnvironmentFile=-/home/k8s-master/kubernetes/cfg/etcd.conf
     # set GOMAXPROCS to number of processors
     ExecStart=/bin/bash -c "GOMAXPROCS=\$(nproc) /home/k8s-master/kubernetes/bin/etcd"
     Type=notify
     
     [Install]
     WantedBy=multi-user.target
     

     注意，这里需要加上一句Environment=ETCD_UNSUPPORTED_ARCH=arm64，因为当前etcd要支持arm必须这样子，否则etcd无法启动。

   • 启动ETCD服务

     sudo systemctl daemon-reload
     sudo systemctl start etcd
     sudo systemctl enable etcd
     

     可以通过systemctl status etcd 查看ETCD的状态，如果出错，可以看log日志：/var/log/syslog

   • 创建ETCD网络

     etcdctl set /coreos.com/network/config "{\"Network\":\"10.1.0.0/16\",\" Backend \"{\"Type\":\"vxlan\"}}"

     如果不指定Backend类型为vxlan,在安装flannel时会报错：不支持UDP Backend,因为flannel默认Backend是UDP，arm不支持，所以在创建ETCD网络的时候需要指定Backend类型

3. 安装Master三个重要组件：kube-apiserver,kube-controller-manager,kube-scheduler

   • 解压kubernetes-server-linux-arm64.tar.gz

     sudo tar -zxvf kubernetes-server-linux-arm64.tar.gz -C kubernetes-server-linux-arm64

   • 复制二进制bin文件到 ~/kubernetes/bin

     sudo cp kubernetes-server-linux-arm64/kubernetes/server/bin/kube-apiserver ~/kubernetes/bin

     sudo cp kubernetes-server-linux-arm64/kubernetes/server/bin/kube-controller-manager ~/kubernetes/bin

     sudo cp kubernetes-server-linux-arm64/kubernetes/server/bin/kube-scheduler ~/kubernetes/bin

   • 安装kube-apiserver

     • 添加kube-apisever配置文件

       sudo vi ~/kubernetes/cfg/kube-apiserver

       # --logtostderr=true: log to standard error instead of files
       KUBE_LOGTOSTDERR="--logtostderr=true"
       
       # --v=0: log level for V logs
       KUBE_LOG_LEVEL="--v=4"
       
       # --etcd-servers=[]: List of etcd servers to watch (http://ip:port),
       # comma separated. Mutually exclusive with -etcd-config
       KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"
       
       # --insecure-bind-address=127.0.0.1: The IP address on which to serve the --insecure-port.
       KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
       
       # --insecure-port=8080: The port on which to serve unsecured, unauthenticated access.
       KUBE_API_PORT="--insecure-port=8080"
       
       # --allow-privileged=false: If true, allow privileged containers.
       KUBE_ALLOW_PRIV="--allow-privileged=false"
       
       # --service-cluster-ip-range=<nil>: A CIDR notation IP range from which to assign service cluster IPs.
       # This must not overlap with any IP ranges assigned to nodes for pods.
       KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=169.169.0.0/16"
       
       # --admission-control="AlwaysAdmit": Ordered list of plug-ins
       # to do admission control of resources into cluster.
       # Comma-delimited list of:
       #   LimitRanger, AlwaysDeny, SecurityContextDeny, NamespaceExists,
       #   NamespaceLifecycle, NamespaceAutoProvision, AlwaysAdmit,
       #   ServiceAccount, DefaultStorageClass, DefaultTolerationSeconds, ResourceQuota
       KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ResourceQuota"
       

     • 添加api-server服务文件

       sudo vi /lib/systemd/system/kube-apiserver.service

       [Unit]
       Description=Kubernetes API Server
       Documentation=https://github.com/kubernetes/kubernetes
       
       [Service]
       EnvironmentFile=-/home/k8s-master/kubernetes/cfg/kube-apiserver
       ExecStart=/home/k8s-master/kubernetes/bin/kube-apiserver    
                          ${KUBE_LOGTOSTDERRR}         \
                               ${KUBE_LOG_LEVEL}           \
                               ${KUBE_ETCD_SERVERS}        \
                               ${KUBE_API_ADDRESS}         \
                               ${KUBE_API_PORT}            \
                               ${KUBE_ALLOW_PRIV}          \
                               ${KUBE_SERVICE_ADDRESSES}   \
                               ${KUBE_ADMISSION_CONTROL}
       Restart=on-failure
       
       [Install]
       WantedBy=multi-user.target
       

     • 启动服务

       sudo systemctl daemon-reload
       sudo systemctl start kube-apiserver
       sudo systemctl enable kube-apiserver
       

   • 安装kube-controller-manager

     • 添加配置文件

       sudo vi ~/kubernetes/cfg/kube-controller-manager

       KUBE_LOGTOSTDERR="--logtostderr=true"
       KUBE_LOG_LEVEL="--v=4"
       KUBE_MASTER="--master=172.16.32.10:8080"
       

     • 添加服务文件

       sudo vi /lib/systemd/system/kube-controller-manager.service

       [Unit]
       Description=Kubernetes Controller Manager
       Documentation=https://github.com/kubernetes/kubernetes
       
       [Service]
       EnvironmentFile=-/home/k8s-master/kubernetes/cfg/kube-controller-manager
       ExecStart=/home/k8s-master/kubernetes/bin/kube-controller-manager 
                        ${KUBE_LOOGTOSTDERR} \
                             ${KUBE_LOG_LEVEL}   \
                             ${KUBE_MASTER}
       Restart=on-failure
       
       [Install]
       WantedBy=multi-user.target
       

     • 启动服务

       sudo systemctl daemon-reload
       sudo systemctl strat kube-controller-manager
       sudo systemctl enable kube-controller-manager
       

   • 安装kube-scheduler

     • 添加配置文件

       sudo vi ~/kubernetes/cfg/kube-scheduler

       # --logtostderr=true: log to standard error instead of files
       KUBE_LOGTOSTDERR="--logtostderr=true"
       
       # --v=0: log level for V logs
       KUBE_LOG_LEVEL="--v=4"
       
       KUBE_MASTER="--master=172.16.32.10:8080"
       

     • 添加服务文件

       sudo vi ~/lib/systemd/system/kube-scheduler.service

       [Unit]
       Description=Kubernetes Scheduler
       Documentation=https://github.com/kubernetes/kubernetes
       
       [Service]
       EnvironmentFile=-/home/k8s-master/kubernetes/cfg/kube-scheduler
       ExecStart=/home/$USER_HOME_DIR/kubernetes/bin/kube-scheduler  
                          ${KUBE_LOGTOSTDERRR}     \
                               ${KUBE_LOG_LEVEL}       \
                               ${KUBE_MASTER}
       Restart=on-failure
       
       [Install]
       WantedBy=multi-user.target
       

     • 启动服务

       sudo systemctl daemon-reload
       sudo systemctl start kube-scheduler
       sudo systemctl enable kube-scheduler
       

4. 安装flannel

   • 解压flannel-v0.10.0-linux-arm64.tar.gz

     sudo tar -zxvf flannel-v0.10.0-linux-arm64.tar.gz -C flannel-v0.10.0-linux-arm64

   • 先复制二进制文件到~/kubernetes/bin

     sudo cp flannel-v0.10.0-linux-arm64/* ~/kubernetes/bin

   • 安装flannel

     • 添加配置文件

       sudo vi ~/kubernetes/cfg/flanneld.conf

       FLANNEL_ETCD_ENDPOINTS="http://172.16.32.10:2379"
       FLANNEL_IFACE="eth0"
       FLANNEL_ETCD_PREFIX="/coreos.com/network"
       FLANNEL_OPTIONS=""
       

       注意，如果有多个网卡，需要指定具体的网卡，要加上FLANNEL_IFACE=xxx选项,在启动时候加上-iface=$(FLANNEL_IFACE)

     • 添加服务文件

       sudo vi /lib/systemd/system/flanneld.service

       [Unit]
       Description=Flanneld
       Documentation=https://github.com/coreos/flannel
       After=network.target
       Before=docker.service
       
       [Service]
       EnvironmentFile=-/home/k8s-master/kubernetes/cfg/flanneld.conf
       ExecStart=/home/k8s-master/kubernetes/bin/flanneld \
                -etcd-endpoints=\${FLANNEL_ETCD_ENDPOINTS} \
                -iface=\${FLANNEL_IFACE} \
                -etcd-prefix=\${FLANNEL_ETCD_PREFIX} \
                \${FLANNEL_OPTIONS}
       Restart=on-failure
       Type=notify
       LimitNOFILE=65536
       
       [Install]
       WantedBy=multi-user.target
       RequiredBy=docker.service
       

     • 启动服务

       sudo systemctl daemon-reload
       sudo systemctl start flanneld
       sudo systemctl enable flanneld
       

   • 配置docker网络，覆盖docker网络

     sudo cat /run/flannel/subnet.env | grep "FLANNEL_SUBNET" | cut -d= -f2
     # 将结果保存起来(10.1.90.1/24)
     
     #创建文件，写入
     sudo echo "
        [Service]
        ExecStart=
        ExecStart=/usr/bin/dockerd --bip=10.1.90.1/24 --mtu=1472
        " > /etc/systemd/system/docker.service.d/docker.conf
     
     #重启docker
     sudo systemctl daemon-reload
     sudo systemctl start docker
     sudo systemctl enable docker 
     

2.部署Node

Node的部署与Master大致一样，都是添加配置文件，服务文件，启动服务。

1. 同样在部署前需要做些初始化

   • 关闭防火墙ufw disable

   • 安装ntp（如果没有安装) sudo apt-get install ntp

   • 添加主机名和IP到/etc/hosts

     172.16.32.10 k8s-master
     172.16.32.11 k8s-node1
     172.16.32.12 k8s-node2
     

   • 新建一个k8s-node1用户，并且赋予root权限

     useradd -m -d /home/k8s-node1 -s /bin/bash k8s-node1
     sudo sed -i -r '/root.*ALL=\(ALL.*ALL/a   \k8s-node1  ALL=\(ALL\) NOPASSWD: ALL' /etc/sudoers
     

   • 切换到k8s-node1 su k8s-node1

   • 切换到k8s-node1用户执行以下动作

     • 创建文件夹，用来保存bin和组件的配置文件

       sudo mkdir -p ~/kubernetes/bin && sudo mkdir ~/kubernetes/cfg

2. 安装kubelet,kube-proxy

   • 解压kubernetes-node-linux-arm64.tar.gz

     sudo tar -zxvf kubernetes-node-linux-arm64.tar.gz kubernetes-node-linux-arm64

   • 复制二进制bin到~/kubernetes/bin

     sudo cp kubernetes-node-linux-arm64/kubernetes/node/bin/kubelet ~/kubernetes/bin

     sudo cp kubernetes-node-linux-arm64/kubernetes/node/bin/kube-proxy ~/kubernetes/bin

   • 安装kubelet

     • 添加配置文件

       sudo vi ~/kubernetes/cfg/kubelet.kubeconfig

       apiVersion: v1
       kind: Config
       clusters:
         - cluster:
             server: http://172.16.32.10:8080/
           name: local
       contexts:
         - context:
             cluster: local
           name: local
       current-context: local
       

       sudo vi ~/kubernetes/cfg/kubelet

       # --logtostderr=true: log to standard error instead of files
       KUBE_LOGTOSTDERR="--logtostderr=true"
       
       #  --v=0: log level for V logs
       KUBE_LOG_LEVEL="--v=4"
       
       # --address=0.0.0.0: The IP address for the Kubelet to serve on (set to 0.0.0.0 for all interfaces)
       NODE_ADDRESS="--address=172.16.32.11"
       
       # --port=10250: The port for the Kubelet to serve on. Note that "kubectl logs" will not work if you set this flag.
       NODE_PORT="--port=10250"
       
       # --hostname-override="": If non-empty, will use this string as identification instead of the actual hostname.
       NODE_HOSTNAME="--hostname-override=ubuntu-node1"
       
       # Path to a kubeconfig file, specifying how to connect to the API server.
       KUBELET_KUBECONFIG="--kubeconfig=~/kubernetes/cfg/kubelet.kubeconfig"
       #KUBELET_KUBECONFIG="--api-servers=http://${MASTER_ADDRESS}:8080"
       
       # --allow-privileged=false: If true, allow containers to request privileged mode. [default=false]
       KUBE_ALLOW_PRIV="--allow-privileged=false"
       
       # DNS info
       KUBELET__DNS_IP="--cluster-dns=169.169.0.2"
       KUBELET_DNS_DOMAIN="--cluster-domain=cluster.local"
       
       KUBELET_SWAP="--fail-swap-on=false"
       
       KUBELET_ARGS="--pod_infra_container_image=hub.c.163.com/allan1991/pause-amd64:3.0"
       

       在kubernetes1.10版本中，KUBELET_KUBECONFIG修改为了用一个yaml的配置文件，而不再是以前的指定--api-server的方式，需要相应更改过来，否则kubelet无法启动。

     • 添加服务文件

       sudo vi /lib/systemd/system/kubelet.service

       [Unit]
       Description=Kubernetes Kubelet
       After=docker.service
       Requires=docker.service
       
       [Service]
       EnvironmentFile=-/home/k8s-node1/kubernetes/cfg/kubelet
       ExecStart=/home/k8s-node1/kubernetes/bin/kubelet       
                      ${KUBE_LOGTOSTDERR}     \
                           ${KUBE_LOG_LEVEL}       \
                           ${NODE_ADDRESS}         \
                           ${NODE_PORT}            \
                           ${NODE_HOSTNAME}        \
                           ${KUBELET_KUBECONFIG}   \
                           ${KUBE_ALLOW_PRIV}      \
                           ${KUBELET__DNS_IP}      \
                           ${KUBELET_DNS_DOMAIN}   \
                           $KUBELET_SWAP
       Restart=on-failure
       KillMode=process
       
       [Install]
       WantedBy=multi-user.target
       

     • 启动服务

       sudo systemctl daemon-reload
       sudo systemctl start kubelet
       sudo systemctl enable kubelet
       

   • 安装kube-proxy

     • 添加配置文件

       sudo vi ~/kubernetes/cfg/kube-proxy

       # --logtostderr=true: log to standard error instead of files
       KUBE_LOGTOSTDERR="--logtostderr=true"
       
       #  --v=0: log level for V logs
       KUBE_LOG_LEVEL="--v=4"
       
       # --hostname-override="": If non-empty, will use this string as identification instead of the actual hostname.
       NODE_HOSTNAME="--hostname-override=k8s-node1"
       
       # --master="": The address of the Kubernetes API server (overrides any value in kubeconfig)
       KUBE_MASTER="--master=http://172.16.32.10:8080"
       

     • 添加服务文件

       sudo vi /lib/systemd/system/kube-proxy.service

       [Unit]
       Description=Kubernetes Proxy
       After=network.target
       
       [Service]
       EnvironmentFile=-/home/k8s-node1/kubernetes/cfg/kube-proxy
       ExecStart=/home/k8s-node1/kubernetes/bin/kube-proxy 
                       ${KUBE_LOGTOSTDERR} \
                           ${KUBE_LOG_LEVEL}   \
                           ${NODE_HOSTNAME}    \
                           ${KUBE_MASTER}
       Restart=on-failure
       
       [Install]
       WantedBy=multi-user.target
       

     • 启动服务

       sudo systemctl daemon-reload
       sudo systemctl start kube-proxy
       sudo systemctl enable kube-proxy
       

   • 安装flannel

     安装方法和在Master中安装的一样，可以参考上面进行安装。

   3.安装过程遇到过的问题记录

   ---

   • failed to start containermanager system validation failed - following cgroup subsystem not mounted:[memory]

     解决方法：

     修/etc/default/grub：

     添加：GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

     更新：update-grub

     重启：sudo systemctl reboot -i

     如果grub没有安装，可以先安装：

     sudo apt install grub-efi-arm64 grub-efi-arm64-bin grub2-common

   • etcdmain: etcd on unsupported platform without ETCD_UNSUPPORTED_ARCH=arm64 set

     解决方法：

     在etcd的服务文件中加上一句：

     Environment=ETCD_UNSUPPORTED_ARCH=arm64 #此行必须添加

   • UDP backend is not supported on this architecture

     解决方法:

     由于flannel预设的backend type是udp，但arm64不支持，所以在etcd中需要指定backend参数：

     etcdctl set /coreos.com/network/config "{\"Network\":\"10.1.0.0/16\",\" Backend \"{\"Type\":\"vxlan\"}}"