基于Kubernetes搭建企业级可自动伸缩Jenkins slave集群

背景

贝壳在成立后借助链家雄厚的资本实力快速发展,事业部项目数量不断扩增,项目的开发语言涵盖了java、php、nodejs、golang、C等等类型。公司从一开始使用了jenkins来完成编译部署等一些自动化工作,但是随着项目的越来越多,天然单点受限的jenkins的瓶颈越来越突出。业务方大多数是自己挤出一台空闲机器作为编译机,将一些编译环境部署好后,联系jenkins管理员,将这台编译机作为slave节点加入jenkins集群,目前jenkins-slave的节点已经达到了六十多台物理机。但是从jenkins web 上我们可以清晰的看到,jenkins的队列依然保持在十几个任务积压的状态。为什么六十多个slave节点还是不能够满足当前业务方编译的需求呢?首先,编译环境各个业务方独立使用,切编译环境不统一,这是造成当前状况的根因。这种方式造成了节点空闲不均匀,资源利用率极低。而且一旦slave节点遭到破坏,需要人为的进行修复甚至重建,非常麻烦。我们通过kubernetes集群上部署jenkins,并利用Kubernetes插件来调用k8s集群实现动态的按需扩展jenkins-slave,从而实现有任务时自动增加slave节点,任务结束slave节点自动销毁。对于jenkins来说,slave节点(容器)是临时的,任务一结束就会销毁。

原理

Kubernetes插件的目的是能够使用Kubernetes集群动态配置Jenkins slave(使用Kubernetes调度机制来优化负载),运行单个构建,然后销毁该slave。
资源创建:jenkins根据任务属性自动创建临时docker容器,并作为slave节点加入jenkins集群,实现资源的自动扩展;
负载均衡:slave节点的创建是基于kubernetes集群的调度,调度会充分考虑资源负载合理分配到对应节点;
资源释放:任务执行结束后,jenkins自动删除相关节点,并销毁相关docker容器,实现资源的释放;
编译环境:通过插件实现编译环境配置,不能通过插件实现的job也可通过指定节点模版的方式,解决了不同类型编译项目时使用不同的编译容器;
资源扩展:基于kubernetes的优势,在编译压力大资源不够的情况下,扩充kubernetes集群即可实现资源扩展;

部署

原理架构图:

搭建kubernetes集群:

我们使用单独一个Kubernetes集群来部署jenkins服务,如果您没有Kubernetes集群,那么您需要创建一个。参见(使用kubeadm安装单个master的Kubernetes 1.13集群)
jenkins访问通过ingress暴露服务,所以要在kubernetes集群中安装Ingress Controller。参见(kubernetes集群中使用ingress)

部署jenkins到kubernetes集群:

创建master的pv和pvc

注:这里存储使用了10.26.10.47上的nfs,nfs的搭建这里不做详解,也可以将此处改为本地host也可以,但是最好还是将slave的valume使用共享存储实现跨节点。

[root@k8s01-test jenkins]# cat jenkins-master-pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: jenkins-master-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /data0/docker/jenkins-master
    server: 10.26.10.47
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-master-pvc
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
创建slave的pv和pvc
[root@k8s01-test jenkins]# cat jenkins-slave-pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: jenkins-slave-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /data0/docker/jenkins-slave
    server: 10.26.10.47
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-slave-pvc
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
创建认证
[root@k8s01-test jenkins]# cat jenkins-account.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
创建jenkins master
[root@k8s01-test jenkins]# cat jenkins.yaml
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: jenkins
  labels:
    name: jenkins
spec:
  serviceName: jenkins
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts-alpine
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
      securityContext:
        fsGroup: 1000
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-master-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  selector:
    name: jenkins

  ports:
    -
      name: http
      port: 80
      targetPort: 8080
      protocol: TCP
    -
      name: agent
      port: 50000
      protocol: TCP

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/tls-acme: "false"
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/proxy-body-size: 50m
    ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 80
    host: jenkins.intra.ke.com
 # - http:
   #   paths:
   #   - path: /
   #     backend:
   #       serviceName: jenkins
   #       servicePort: 80
   # host: jenkins.intra.ke.com
  #tls:
  #  #- hosts:
  #    #  - jenkins.intra.ke.com
  #      #  secretName: tls-jenkins

yaml可以参考官方文档:
https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes

配置解析

配置本机hosts或解析dns到k8s集群任意IP即可访问jenkins.intra.ke.com

安装配置插件实现自动伸缩

jenkins-slave自动伸缩功能需要借助安装Kubernetes插件来实现

安装插件

Kubernetes plugin
This plugin integrates Jenkins with Kubernetes

配置插件

jenkins》系统管理》系统设置》Add a new cloud 》kubernetes
添加kubernetes
Name: kubernetes
Kubernetes URL: https://kubernetes.default #由于我们的jenkins master 是部署在同一个集群,所以这里填写内部地址
Kubernetes Namespace: default
Jenkins URL: http://jenkins.default

添加Kubernetes Pod Template:
Name: jenkins-agent
Namespace: default
Labels: jenkins-agent

添加container Template
name: jnlp-slave
Docker image: jenkins/jnlp-slave:alpine
Jenkins slave root directory: /home/jenkins

添加卷
type:persistent volume claim
申明值:jenkins-slave-pvc
挂载路径:/home/jenkins




禁用默认Master构建代理

其他插件

GitLab Plugin插件

check out to sub_directory

artifactory插件


{
    "files": [
        {
            "pattern": "docker/${JOB_BASE_NAME}.tar.gz",
            "target": "ke-test-repository/${JOB_BASE_NAME}/${BUILD_NUMBER}/"

        }
    ]
}

docker-build-step插件

配置docker tcp方式 tcp不安全
vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:4343 -H unix:///var/run/docker.sock

execute Docker command 插件实现docker images build pull push

还有一种方式,在 docker 单节点下,可以挂载 docker 到 jenkins 容器中,添加如下参数:
-v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/usr/bin/docker
但是我没有成功,估计是跟jenkins slave基础镜像alphine版有关系

Kubernetes Continuous Deploy 插件

通过SSH从主节点获取群集凭据。您也可以手动配置它。cat /etc/kubernetes/admin.conf
变量替换资源配置,允许您进行动态资源部署。
私有Docker注册表的Docker登录凭据管理。
无需kubectl 在Jenkins从属节点上安装该工具。

Workspace Cleanup Plugin

清理构建数据,减少空间占用

遇到问题及解决方案

slave无法获取到安装的java命令

参考文档

https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes
https://blog.csdn.net/aixiaoyang168/article/details/79767649



如何配置云kubernetes连接K8S集群的验证文件https://blog.csdn.net/mario_hao/article/details/81332546

http://www.updn.cn

发表评论

电子邮件地址不会被公开。 必填项已用*标注