环境

• kubernetes 1.15
• gitlab 12.6.0-ee
• gitlab runner 12.6.0
• helm 2.14.3

kubernetes、helm和gitlab已提前安装配置好。helm配置可参考https://www.jianshu.com/p/8aec2776f3f1，为gitlab添加kubernetes可参考https://www.jianshu.com/p/1208c132b84c。

安装配置

下载chart

下载gitlab runner helm chart
https://gitlab.com/gitlab-org/charts/gitlab-runner/tree/master

添加访问gitlab的证书

添加gitlab的证书：

kubectl --namespace <NAMESPACE> create secret generic <SECRET_NAME> --from-file=<CERTFICATE_FILENAME>

#xxx为你的证书名
/home/admin(master) # kubectl create secret -n default generic gitlab-cert --from-file=./xxx.crt 

查看

/home/admin(master) # kubectl get secret                                                         
NAME                                 TYPE                                  DATA   AGE
default-token-4w6nw                  kubernetes.io/service-account-token   3      58d
gangly-zorse-gitlab-runner           Opaque                                2      23h
gitlab-cert                          Opaque                                1      14d

修改values.yaml

根据个人要求修改对应配置，以下为我个人修改了的配置

# image: gitlab/gitlab-runner:alpine-v11.6.0 
image: ip:port/gitlab/gitlab-runner:alpine-bleeding-1 #这里镜像我用的自己私有仓库的镜像
gitlabUrl: https://gitlab.example.com/ #这里输入你自己的gitlab的地址。
runnerRegistrationToken: " " #输入你的注册token

你可以把runner安装为共享的或者特定的，token的查看分别在：Admin area > runner > Set up a shared Runner manually

shared.png

Project > Setting > CI / CD Settings > Runners > Expand

specific.png

certsSecretName: gitlab-cert #输入你刚才创建的secret的名字
runners:
  ## Default container image to use for builds when none is specified
  ##
  image: ip:port/ubuntu:16.04 #这里我用的自己私有仓库里的镜像
  imagePullSecrets: [regsecret] #输入私有仓库的secret，可参考https://www.jianshu.com/p/de030582cd75
  helpers:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi
    image: ip:port/gitlab-runner-helper:x86_64-577f813d #这里也是用的我自己私有仓库里的镜像
    # image: gitlab/gitlab-runner-helper:x86_64-latest 

如遇到couldn't execute POST against https://xxx/api/v4/runners: Post https://xxx/api/v4/runners: x509: certificate signed by unknown authority，修改envVars:

envVars:
  - name: RUNNER_EXECUTOR
    value: kubernetes
  - name: CI_SERVER_TLS_CA_FILE
    value: /home/gitlab-runner/.gitlab-runner/certs/<CERTFICATE_FILENAME>.crt
  - name: CONFIG_FILE
    value: /home/gitlab-runner/.gitlab-runner/config.toml

测试的时候ci/cd 遇到unable to get local issuer certificate的问题，不知道怎么在helm的chart改，所以gitlab-runner-helper的镜像是我自己关了git的ssl再打包的镜像，知道怎么改的可以留言一下，感谢。
关闭git ssl认证：git config --global http.sslVerify false

绑定本地 Docker 守护进程

用 volume 绑定的形式把本地 docker.sock 通过 host_path 的方式挂载到 runner 中，修改configmap.yaml文件

# git-lab/runner/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "gitlab-runner.fullname" . }}
  labels:
    app: {{ include "gitlab-runner.fullname" . }}
    chart: {{ include "gitlab-runner.chart" . }}
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
data:
  entrypoint: |
    #!/bin/bash
    set -e
    mkdir -p /home/gitlab-runner/.gitlab-runner/
    cp /scripts/config.toml /home/gitlab-runner/.gitlab-runner/

    # Register the runner
    if [[ -f /secrets/accesskey && -f /secrets/secretkey ]]; then
      export CACHE_S3_ACCESS_KEY=$(cat /secrets/accesskey)
      export CACHE_S3_SECRET_KEY=$(cat /secrets/secretkey)
    fi

    if [[ -f /secrets/gcs-applicaton-credentials-file ]]; then
      export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-applicaton-credentials-file"
    else
      if [[ -f /secrets/gcs-access-id && -f /secrets/gcs-private-key ]]; then
        export CACHE_GCS_ACCESS_ID=$(cat /secrets/gcs-access-id)
        # echo -e used to make private key multiline (in google json auth key private key is oneline with \n)
        export CACHE_GCS_PRIVATE_KEY=$(echo -e $(cat /secrets/gcs-private-key))
      fi
    fi

    if [[ -f /secrets/runner-registration-token ]]; then
      export REGISTRATION_TOKEN=$(cat /secrets/runner-registration-token)
    fi

    if [[ -f /secrets/runner-token ]]; then
      export CI_SERVER_TOKEN=$(cat /secrets/runner-token)
    fi

    if ! sh /scripts/register-the-runner; then
      exit 1
    fi

    # add volume config
    cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF
      [[runners.kubernetes.volumes.host_path]]
            name = "docker"
            mount_path = "/var/run/docker.sock"
    EOF
    # Start the runner
    exec /entrypoint run --user=gitlab-runner \
      --working-directory=/home/gitlab-runner

  config.toml: |
    concurrent = {{ .Values.concurrent }}
    check_interval = {{ .Values.checkInterval }}
    log_level = {{ default "info" .Values.logLevel | quote }}
    {{- if .Values.logFormat }}
    log_format = {{ .Values.logFormat | quote }}
    {{- end }}
    {{- if .Values.metrics.enabled }}
    listen_address = '[::]:9252'
    {{- end }}
  configure: |
    set -e
    cp /init-secrets/* /secrets
  register-the-runner: |
    #!/bin/bash
    MAX_REGISTER_ATTEMPTS=30

    for i in $(seq 1 "${MAX_REGISTER_ATTEMPTS}"); do
      echo "Registration attempt ${i} of ${MAX_REGISTER_ATTEMPTS}"
      /entrypoint register \
        {{- range .Values.runners.imagePullSecrets }}
        --kubernetes-image-pull-secrets {{ . | quote }} \
        {{- end }}
        {{- range $key, $val := .Values.runners.nodeSelector }}
        --kubernetes-node-selector {{ $key | quote }}:{{ $val | quote }} \
        {{- end }}
        {{- range $key, $value := .Values.runners.podLabels }}
        --kubernetes-pod-labels {{ $key | quote }}:{{ $value | quote }} \
        {{- end }}
        {{- range $key, $val := .Values.runners.podAnnotations }}
        --kubernetes-pod-annotations {{ $key | quote }}:{{ $val | quote }} \
        {{- end }}
        {{- range $key, $value := .Values.runners.env }}
        --env {{ $key | quote -}} = {{- $value | quote }} \
        {{- end }}
        {{- if and (hasKey .Values.runners "runUntagged") .Values.runners.runUntagged }}
        --run-untagged=true \
        {{- end }}
        {{- if and (hasKey .Values.runners "protected") .Values.runners.protected }}
        --access-level="ref_protected" \
        {{- end }}
        --non-interactive

      retval=$?

      if [ ${retval} = 0 ]; then
        break
      elif [ ${i} = ${MAX_REGISTER_ATTEMPTS} ]; then
        exit 1
      fi

      sleep 5 
    done

    exit 0

  check-live: |
    #!/bin/bash
    if /usr/bin/pgrep -f .*register-the-runner; then
      exit 0
    elif /usr/bin/pgrep gitlab.*runner; then
      exit 0
    else
      exit 1
    fi

使用helm安装的gitlab runner

安装

/home/admin/git-lab/runner(master) # ls 
CHANGELOG.md  Chart.yaml  CONTRIBUTING.md  LICENSE  NOTICE  README.md  scripts  templates  values.yaml
-------------------------------------------------------------------------------------
/home/admin/git-lab/runner(master) # helm install ./  
/home/admin/git-lab/runner(master) # helm list  
NAME            REVISION    UPDATED                     STATUS      CHART                         APP VERSION   NAMESPACE
gangly-zorse    1           Thu Jan  2 17:32:23 2020    DEPLOYED    gitlab-runner-0.12.0-beta     bleeding      default  
/home/admin/git-lab/runner(master) # kubectl get pods
NAME                                          READY   STATUS    RESTARTS   AGE
gangly-zorse-gitlab-runner-7d4f5f9bbb-th4m6   1/1     Running   0          24h

helm 删除命令helm delete --purge <NAME>

在gitlab上查看runner是否部署成功

runner.png

简单测试

创建Dockerfile文件

FROM ip:port/tomcat:7  #我用的自己私有仓库的jing
RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo "Asia/Shanghai" > /etc/timezone
EXPOSE 8080

创建一个deployment.yaml文件

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tomcat-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tomcat-demo
  template:
    metadata:
      labels:
        app: tomcat-demo
    spec:
      imagePullSecrets:
        - name: regsecret
      containers:
      - name: tomcat-demo
        image: REGISTRY/TAG:7
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-demo
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: tomcat-demo
  selector:
    app: tomcat-demo
  type: NodePort

创建.gitlab-ci.yml文件

创建一个project，并在里面创建一个名为.gitlab-ci.yml的文件，内容如下：

#just a test
image: ip:port/docker:stable-dind
variables:
  DOCKER_DRIVER: overlay2
  REGISTRY: "ip:port" #私有仓库地址
  TAG: "tomcat" 
stages:
  - build
  - deploy
docker-build:
  stage: build
    script:
      - echo "Building Dockerfile-based application..."
      - docker login https://ip:port -u <USER> -p <PASSWORD> #填写自己私有仓库的用户和密码
      - docker pull ip:port/tomcat:7 #拉取tomcat7的镜像
      - docker build -t $REGISTRY/$TAG:$CI_COMMIT_SHORT_SHA .
      - docker push $REGISTRY/$TAG:$CI_COMMIT_SHORT_SHA
k8s-deploy:
  image: ip:port/bitnami/kubectl:latest #使用的自己仓库里的
  stage: deploy
  script:
    - echo "deploy to k8s cluster "
    - sed -i "s/REGISTRY/$REGISTRY/g" deployment.yaml
    - sed -i "s|TAG|$TAG|g" deployment.yaml
    - kubectl apply -f deployment.yaml

保存.gitlab-ci.yml文件并push到已经配置好了runner的project，这里我创建了个空的project进行的runner部署测试。
运行git push -u origin master后查看你的project的ci/cd。

demo.png

如果failed，也可点进build里去查看失败原因。

参考

• https://docs.gitlab.com/runner/executors/kubernetes.html
• https://docs.gitlab.com/runner/install/kubernetes.html
• https://docs.gitlab.com/ee/ci/runners/
• https://gitlab.com/gitlab-org/gitlab-runner/issues/3968
• https://gitlab.com/gitlab-org/gitlab-foss/tree/master/lib/gitlab/ci/templates
• https://www.cnblogs.com/Sinte-Beuve/p/11739196.html