1、k8s的架构
k8s架构核心组件
其他组件
2、k8s集群的安装
2.1、准备环境(主机IP、主机名、hosts解析）
10.0.0.11 k8s-master
10.0.0.12 k8s-node-1
10.0.0.13 k8s-node-2
2.2、master节点安装etcd
1）安装etcd
yum install etcd -y vim
2）修改配置文件
vim /etc/etcd/etcd.conf
6行：ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
21行： ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.11:2379"
3）启动etcd
systemctl start etcd.service
systemctl enable etcd.service
4）插入key和值并检查
etcdctl set testdir/testkey0 0
etcdctl get testdir/testkey0
5）检查集群健康状态
etcdctl -C http://10.0.0.11:2379 cluster-health
2.3、master节点安装kubernetes
yum install kubernetes-master.x86_64 -y
vim /etc/kubernetes/apiserver
8行: KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
11行：KUBE_API_PORT="--port=8080"
14 KUBELET_PORT="--kubelet-port=10250"
17行：KUBE_ETCD_SERVERS="--etcd-servers=[http://10.0.0.11:2379](http://10.0.0.11:2379)"
23行：KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
vim /etc/kubernetes/config
22行：KUBE_MASTER="--master=[http://10.0.0.11:8080](http://10.0.0.11:8080)"
systemctl enable kube-apiserver.service
systemctl restart kube-apiserver.service
systemctl enable kube-controller-manager.service
systemctl restart kube-controller-manager.service
systemctl enable kube-scheduler.service
systemctl restart kube-scheduler.service
2.4、 node节点安装kubernetes
yum install kubernetes-node.x86_64 -y
vim /etc/kubernetes/config
22行：KUBE_MASTER="--master=[http://10.0.0.11:8080](http://10.0.0.11:8080)"
vim /etc/kubernetes/kubelet
5行：KUBELET_ADDRESS="--address=0.0.0.0"
8行：KUBELET_PORT="--port=10250"
11行：KUBELET_HOSTNAME="--hostname-override=10.0.0.12"
14行：KUBELET_API_SERVER="--api-servers=[http://10.0.0.11:8080](http://10.0.0.11:8080)"
systemctl enable kubelet.service
systemctl start kubelet.service
systemctl enable kube-proxy.service
systemctl start kube-proxy.service
在10.0.0.11上检查
kubectl get nodes
2.5、所有节点配置flannel网络
yum install flannel -y
sed -i 's#http://127.0.0.1:2379#http://10.0.0.11:2379#g' /etc/sysconfig/flanneld
master节点：
etcdctl mk /atomic.io/network/config '{ "Network": "172.16.0.0/16" }'
yum install docker -y
systemctl enable flanneld.service
systemctl restart flanneld.service
service docker restart
systemctl restart kube-apiserver.service
systemctl restart kube-controller-manager.service
systemctl restart kube-scheduler.service
node节点：
systemctl enable flanneld.service
systemctl restart flanneld.service
service docker restart
systemctl restart kubelet.service
systemctl restart kube-proxy.service
所有节点上传镜像
wget http://192.168.12.201/docker_image/docker_busybox.tar.gz
docker load -i docker_busybox.tar.gz
docker run -it docker_docker.io/busybox:latest
发现主机之间不能ping通，解决方法：
vim /usr/lib/systemd/system/docker.service
18行添加 ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
systemctl daemon-reload
systemctl restart docker
启动容器主机之间互相ping验证
2.6、配置master为镜像仓库
#所有节点
vim /etc/sysconfig/docker
将第四行替换成下面
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --registry-mirror=[https://registry.docker-cn.com](https://registry.docker-cn.com) --insecure-registry=10.0.0.11:5000'
systemctl restart docker
#master节点
wget http://192.168.12.201/docker_image/registry.tar.gz
docker load -i registry.tar.gz
docker run -d -p 5000:5000 --restart=always --name registry -v /opt/myregistry:/var/lib/registry registry
2.7、node节点操作
wget http://192.168.12.201/docker_image/pod-infrastructure-latest.tar.gz
docker load -i pod-infrastructure-latest.tar.gz
docker tag docker.io/tianyebj/pod-infrastructure 10.0.0.72:5000/pod-infrastructure
vim /etc/kubernetes/kubelet
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=10.0.0.72:5000/pod-infrastructure:latest"
3、k8s介绍
3.1、k8s的核心功能
自愈：重新启动失败的容器，在节点不可用时，替换和重新调度节点上的容器，对用户定义的健康检查不响应的容器会被终止，并且在容器准备好服务之前不会把其向客户端广播
弹性伸缩
服务的自动发现和负载均衡
滚动升级和一键回滚
3.2、k8s的历史
2014年 docker容器编排工具
2015年 发布kubernetes 1.0，加入CNCF
2016年 干掉 docker swarm, mesos
2017
2018年 k8s从CNCF毕业
2019年 1.13, 1.14, 1.15
3.3、k8s安装
yum 安装 1.5版本 最容易成功。最适合学习的
源码编译安装 难度最大，可以安装最新版
二进制安装 步骤繁琐 可以安装最新版 shell,ansible,saltstack
kubeadm 安装最容易，需要网络 可以安装最新版
minikube 适合开发人员体验k8s,网络
3.4、k8s的应用场景
k8s最适合跑微服务项目
什么是微服务？
一个功能一个集群
微服务支持更大的并发
减少发布更新的时间
降低开发的难度
集群健壮性更高
4、k8s常用的资源
4.1、创建pod资源
k8s yaml的主要组成
apiVersion: v1 api版本
kind: pod 资源类型
metadata: 属性
spec: 详细
创建pod
vim k8s_pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: web
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
pod资源：至少有两个容器组成，pod基础容器和业务容器
pod是k8s最小的资源单位
4.2、ReplicationController资源
rc：保证指定数量的pod始终存活，rc通过标签选择器来关联pid
k8s资源的常见操作:
kubectl create -f xxx.yaml
kubectl get pod|rc
kubectl describe pod nginx
kubectl delete pod nginx 或者kubectl delete -f xxx.yaml
kubectl edit pod nginx
创建一个rc
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 5
selector:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
rc的滚动升级
新建一个nginx-rc1.15.yaml
升级
kubectl rolling-update nginx -f nginx-rc1.15.yaml --update-period=10s
回滚
kubectl rolling-update nginx2 -f nginx-rc.yaml --update-period=1s
4.3、 service资源
service帮助pod暴露端口
创建一个service
apiVersion: v1
kind: Service
metadata:
name: myweb
spec:
type: NodePort
ports:
- port: 80
nodePort: 30000
targetPort: 80
selector:
app: myweb2
修改nodePort范围
vim /etc/kubernetes/apiserver
KUBE_API_ARGS="--service-node-port-range=3000-50000"
service默认使用iptables来实现负载均衡,新版本中推荐使用lvs(四层负载均衡)
4.3、deployment资源
rc在滚动升级之后，会造成服务访问中断，于是引入了deployment资源
创建deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
resources:
limits:
cpu: 100m
requests:
cpu: 100m
deployment升级和回滚
命令行创建deployment
kubectl run nginx --image=10.0.0.11:5000/nginx:1.13 --replicas=3 --record
命令行升级版本
kubectl set image deploy nginx nginx=10.0.0.11:5000/nginx:1.15
查看deployment所有历史版本
kubectl rollout history deployment nginx
deployment回滚到上一个版本
kubectl rollout undo deployment nginx
deployment回滚到指定版本
kubectl rollout undo deployment nginx --to-revision=2
对外开放端口（随机端口）
kubectl expose deployment nginx --port=80 --type=NodePort
4.5、tomcat+mysql练习
node节点操作
1）上传mysql、tomcat镜像
2）上传镜像
docker load -i docker-mysql-5.7.tar.gz
docker load -i tomcat-app-v2.tar.gz
3）打标签
docker tag docker.io/mysql:5.7 10.0.0.11:5000/mysql:5.7
docker tag docker.io/kubeguide/tomcat-app:v2 10.0.0.11:5000/tomcat-app:v2
master节点操作
1）vim k8s/tomcat_demo/mysql-rc.yml
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
2）kubectl create -f mysql-rc.yml
3）vim k8s/tomcat_demo/mysql-svc.yml
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
targetPort: 3306
selector:
app: mysql
4）kubectl create -f mysql-svc.yml
5）vim k8s/tomcat_demo/tomcat-rc.yml （注意：env下的value配置CLUSTER-IP，vIP可以用kubectl get all -o wide查看svc/mysql下的CLUSTER-IP地址）
apiVersion: v1
kind: ReplicationController
metadata:
name: myweb
spec:
replicas: 1
selector:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: 10.0.0.11:5000/tomcat-app:v2
ports:
- containerPort: 8080
env:
- name: MYSQL_SERVICE_HOST
value: '10.254.43.97 ' <----------------------------------------
- name: MYSQL_SERVICE_PORT
value: '3306'
6）kubectl create -f tomcat-rc.yml
7）vim k8s/tomcat_demo/tomcat-svc.yml
apiVersion: v1
kind: Service
metadata:
name: myweb
spec:
type: NodePort
ports:
- port: 8080
nodePort: 30008
selector:
app: myweb
8） kubectl create -f tomcat-svc.yml
9）kubectl get all -o wide
找到tomcat8080端口对应的对外端口，网页进行访问，有数据就连接上了
网页打不开就执行下面命令
sysctl net.ipv4.ip_forward=1
iptables -P FORWARD ACCEPT
image.png
5、k8s的附加组件
5.1、dns服务
安装dns镜像
1）下载dns_docker镜像包
wget http://192.168.12.201/docker_image/docker_k8s_dns.tar.gz
2）node2节点导入镜像
docker load -i k8s_dns.tar.gz
3）创建dns服务
kubectl create -f skydns-rc.yaml
kubectl create -f skydns-svc.yaml
4）检查
kubectl get all --namespace=kube-system
5）修改所有node节点kubelet的配置文件
vim /etc/kubernetes/kubelet
KUBELET_ARGS="--cluster_dns=10.254.230.254 --cluster_domain=cluster.local"
systemctl restart kubelet
5.2、namespace命令空间
kubectl create namespace lwl
sed -i '3a \ \ namespace: lwl' *.yml|head -5
kubectl get all --namespace lwl
5.3、健康检查
5.3.1、探针的种类
livenessProbe：健康状态检查，周期性检查服务是否存活，检查结果失败，将重启容器
readinessProbe：可用性检查，周期性检查服务是否可用，不可用将从service的endpoints中移除
5.3.2、探针的检测方法
exec：执行一段命令
httpGet：检测某个 http 请求的返回状态码
tcpSocket：测试某个端口是否能够连接
5.3.3 liveness探针的exec使用
vi nginx_pod_exec.yaml
apiVersion: v1
kind: Pod
metadata:
name: exec
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
5.3.4 liveness探针的httpGet使用
vi nginx_pod_httpGet.yaml
apiVersion: v1
kind: Pod
metadata:
name: httpget
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /index.html
port: 80
initialDelaySeconds: 3
periodSeconds: 3
5.3.5 liveness探针的tcpSocket使用
vi nginx_pod_tcpSocket.yaml
apiVersion: v1
kind: Pod
metadata:
name: tcpSocket
spec:
containers:
- name: nginx
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
livenessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 3
periodSeconds: 3
5.3.6 readiness探针的httpGet使用
vi nginx-rc-httpGet.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: readiness
spec:
replicas: 2
selector:
app: readiness
template:
metadata:
labels:
app: readiness
spec:
containers:
- name: readiness
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /qiangge.html
port: 80
initialDelaySeconds: 3
periodSeconds: 3
5.4、dashboard服务
1:上传并导入镜像,打标签
2:创建dashborad的deployment和service
3:访问[http://10.0.0.11:8080/ui/](http://10.0.0.11:8080/ui/)
5.5 通过apiservicer反向代理访问service
第一种：NodePort类型
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 30008
第二种：ClusterIP类型
type: ClusterIP
ports:
- port: 80
targetPort: 80
6、 k8s弹性伸缩
k8s弹性伸缩,需要附加插件heapster监控
6.1 安装heapster监控
1:上传并导入镜像,打标签
ls *.tar.gz
for n in `ls *.tar.gz`;do docker load -i $n ;done
docker tag docker.io/kubernetes/heapster_grafana:v2.6.0 10.0.0.11:5000/heapster_grafana:v2.6.0
docker tag docker.io/kubernetes/heapster_influxdb:v0.5 10.0.0.11:5000/heapster_influxdb:v0.5
docker tag docker.io/kubernetes/heapster:canary 10.0.0.11:5000/heapster:canary
2:上传配置文件,kubectl create -f .
3:打开dashboard验证
image.png
6.2、弹性伸缩
1:修改rc的配置文件
containers:
- name: myweb
image: 10.0.0.11:5000/nginx:1.13
ports:
- containerPort: 80
resources:
limits:
cpu: 100m
requests:
cpu: 100m</pre>
2:创建弹性伸缩规则
kubectl autoscale -n qiangge replicationcontroller myweb --max=8 --min=1 --cpu-percent=8
3:测试
ab -n 1000000 -c 40 http://172.16.28.6/index.html
7、持久化存储
pv: persistent volume
pvc: persistent volume claim
7.1、安装nfs服务端(10.0.0.11)
yum install nfs-utils.x86_64 -y
mkdir /data
vim /etc/exports
/data 10.0.0.0/24(rw,async,no_root_squash,no_all_squash)
systemctl start rpcbind
systemctl start nfs
7.2、在node节点安装nfs客户端
yum install nfs-utils.x86_64 -y
showmount -e 10.0.0.11
7.3、创建pv和pvc
上传yaml配置文件,创建pv和pvc
7.4、创建mysql-rc,pod模板里使用volume
volumeMounts:
- name: mysql
mountPath: /var/lib/mysql
volumes:
- name: mysql
persistentVolumeClaim:
claimName: tomcat-mysql
7.5、 验证持久化
验证方法1:删除mysql的pod,数据库不丢
kubectl delete pod mysql-gt054
验证方法2:查看nfs服务端,是否有mysql的数据文件
7.6、 分布式存储glusterfs
image.png
7.6.1、 什么是glusterfs
Glusterfs是一个开源分布式文件系统，具有强大的横向扩展能力，可支持数PB存储容量和数千客户端，通过网络互联成一个并行的网络文件系统。具有可扩展性、高性能、高可用性等特点。
7.6.2、安装gusterfs
所有节点：
yum install centos-release-gluster -y
yum install install glusterfs-server -y
systemctl start glusterd.service
systemctl enable glusterd.service
mkdir -p /gfs/test1
mkdir -p /gfs/test2
fdisk -l
mkfs.xfs /dev/sdb
mkfs.xfs /dev/sdc
blkid(磁盘UUID）
vim /etc/fatsb(添加sdb和sdc的UUID。。。）
mount -a
df -h
7.6.3、添加存储资源池
master节点：
gluster pool list
gluster peer probe glusterfs02
gluster peer probe glusterfs02
gluster pool list
7.6.4、glusterfs卷管理
创建分布式复制卷
gluster volume create lwl replica 2 glusterfs01:/gfs/test1 glusterfs01:/gfs/test2 glusterfs02:/gfs/test1 glusterfs02:/gfs/test2 force
启动卷
gluster volume start lwl
查看卷
gluster volume info lwl
挂载卷
mount -t glusterfs 10.0.0.14:/lwl /mnt
7.6.5、分布式复制卷详解
pool 池
peer 节点
volume 卷
add-brick 扩容卷
7.6.6、分布式卷扩容
扩容前查看容量：
df -h
扩容命令：
gluster volume add-brick lwl glusterfs:/gfs/test1 glusterfs:/gfs/test2 force
扩容后查看容量:
df -h
7.7、 k8s 对接glusterfs存储
创建endpoint
vi glusterfs-ep.yaml
aapiVersion: v1
kind: Endpoints
metadata:
name: glusterfs
namespace: default
subsets:
- addresses:
- ip: 10.0.0.13
- ip: 10.0.0.14
- ip: 10.0.0.15
ports:
- port: 49152
protocol: TCP
创建service
vi glusterfs-svc.yaml
aapiVersion: v1
kind: Service
metadata:
name: glusterfs
namespace: default
spec:
ports:
- port: 49152
protocol: TCP
targetPort: 49152
sessionAffinity: None
type: ClusterIP
创建gluster类型pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: gluster
labels:
type: glusterfs
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs"
path: "lwl"
readOnly: false
创建pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: gluster
spec:
selector:
matchLabels:
type: glusterfs
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
使用gluster
vim mysql-rc.yml
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
volumes:
- name: mysql
persistentVolumeClaim:
claimName: gluster
containers:
- name: mysql
image: 10.0.0.11:5000/mysql:5.7
volumeMounts:
- name: mysql
mountPath: /var/lib/mysql
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: '123456'
8、与jenkins集成实现ci/cd
image.png
image.png
8.1、安装gitlab并上传代码
#a:安装
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-11.9.11-ce.0.el7.x86_64.rpm
yum localinstall gitlab-ce-11.9.11-ce.0.el7.x86_64.rpm -y
#b:配置
vim /etc/gitlab/gitlab.rb
external_url 'http://10.0.0.15'
prometheus_monitoring['enable'] = false
#c:应用并启动服务
gitlab-ctl reconfigure
#使用浏览器访问http://10.0.0.13,修改root用户密码,创建project
#上传代码到git仓库
cd /srv/
rz -E
unzip xiaoniaofeifei.zip
rm -fr xiaoniaofeifei.zip
git config --global user.name "Administrator"
git config --global user.email "admin@example.com"
git init
git remote add origin http://10.0.0.15/root/xiaoniao.git
git add .
git commit -m "Initial commit"
git push -u origin master
vim dockerfile
FROM 10.0.0.11:5000/nginx:1.13
ADD . /usr/share/nginx/html
docker build -t xiaoniao:v1 .
docker run -d -p 88:80 xiaoniao:v1
git add .
git commit -m "create dockerfile"
git push -u origin master
8.2、访问jenkins
访问[http://10.0.0.12:8080/](http://10.0.0.12:8080/),默认账号密码admin:123456
image.png
8.3、配置jenkins拉去gitlab代码凭据
a:在jenkins上生成秘钥对
ssh-keygen -t rsa
b:复制公钥粘贴gitlab上
c:jenkins上创建全局凭据
image.png
image.png
8.4、拉取代码测试
image.png
image.png
image.png
8.5、点击jenkins立即构建,自动构建docker镜像并上传到私有仓库
修改jenkins 工程配置
docker build -t 10.0.0.11:5000/test:v$BUILD_ID .
docker push 10.0.0.11:5000/test:v$BUILD_ID
image.png
image.png
8.6、jenkins自动部署应用到k8s
#!/bin/bash
if [ -f /tmp/xiaoniao.lock ];then
docker build -t 10.0.0.11:5000/xiaoniao:v$BUILD_ID .
docker push 10.0.0.11:5000/xiaoniao:v$BUILD_ID
kubectl -s 10.0.0.11:8080 set image -n xiaoniao deploy xiaoniao xiaoniao=10.0.0.11:5000/xiaoniao:v$BUILD_ID
echo "更新成功"
else
docker build -t 10.0.0.11:5000/xiaoniao:v$BUILD_ID .
docker push 10.0.0.11:5000/xiaoniao:v$BUILD_ID
kubectl -s 10.0.0.11:8080 create namespace xiaoniao
kubectl -s 10.0.0.11:8080 run xiaoniao -n xiaoniao --image=10.0.0.11:5000/xiaoniao:v$BUILD_ID --replicas=3 --record
kubectl -s 10.0.0.11:8080 expose -n xiaoniao deployment xiaoniao --port=80 --type=NodePort
port=`kubectl -s 10.0.0.11:8080 get svc -n xiaoniao|grep -oP '(?<=80:)\d+'`
echo "你的项目地址访问是http://10.0.0.13:$port"
touch /tmp/xiaoniao.lock
fi
jenkins一键回滚
kubectl -s 10.0.0.11:8080 rollout undo -n xiaoniao deployment xiaoniao
