基于Jenkins的Master-slave模式CICD到EKS

一、Jenkins部署

EKS上部署Jenkins，由于EKS自带认证模式，默认官方Kubernetes认证方式是不可行的。踩了坑！！（一定要给足权限！）

首先EKS上已部署 Jenkins 部署就不介绍了可以参考以下连接

https://blog.csdn.net/weixin_42562106/article/details/107565974

另外再提供一种（推荐如上）（如下采用hostpath模式）：

去绑定的node节点 授予jenkins/下的     chmod 777 workspace/

参考文档：https://blog.csdn.net/m0_56444183/article/details/123426731

root@nx-eks-ctl:/home/k8s/jenkins# cat k8s-jenkins.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
  namespace: devops
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: jenkins
  template:
    metadata:
      labels:
        k8s-app: jenkins
      namespace: devops
      name: jenkins
    spec:
      hostAliases:
      - ip: "172.12.18.181"
        hostnames:
          - "updates.jenkins-ci.org"
      nodeSelector:
        key: "devops"
      containers:
        - name: jenkins
          image: harbor.devops.xxxxx.com/demo/jenkinsci/blueocean:v1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 30082
              name: web
              protocol: TCP
            - containerPort: 30081
              name: agent
              protocol: TCP
          resources:
            limits:
              cpu: 1000m
              memory: 4Gi
            requests:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /login
              port: 30082
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 30082
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/lib/jenkins
          env:
            - name: JENKINS_HOME
              value: /var/lib/jenkins
            - name: JENKINS_OPTS
              value: --httpPort=30082
            - name: JENKINS_SLAVE_AGENT_PORT
              value: "30081"
      securityContext:
        runAsUser: 0
      volumes:
        - name: jenkins-home
          hostPath:
            path: /data/devops/jenkins
            type: Directory
      serviceAccountName: jenkins
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
  namespace: devops
---
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
  namespace: devops
spec:
  type: NodePort
  ports:
    - name: web
      port: 30082
      targetPort: 30082
      nodePort: 30082
    - name: slave
      port: 30081
      targetPort: 30081
      nodePort: 30081
  selector:
    k8s-app: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins
  namespace: devops
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
roleRef:
  kind: ClusterRole
  name: jenkins
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: devops
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: devops
spec:
  ingressClassName: kong
  rules:
    - host: jenkins-test-devops.xxxxx.com
      http:
        paths:
          - backend:
              service:
                name: jenkins
                port:
                  number: 30082
            path: /
            pathType: ImplementationSpecific

1.2、插件安装

Chinese
Git
Gitlab
Git Parameter
Extended Choice Parameter
Docker plugin
Docker Pipeline
Groovy
Kubernetes
Pipeline
Config File Provider
Kubernetes Cli
Groovy Postbuild
Kubernetes Continuous Deploy（这个插件有坑，一定要安装1.0.0版本）

CI/CD之后执行 Kubernetes Continuous Deploy，官网插件2.3.1版本有问题不建议使用，我被坑了2天,建议降到1.0版本插件找不到可以直接使用如下

https://www.aliyundrive.com/s/Aq71VchmAKQ

更改时区：

管理--脚本命令行-->System.setProperty('org.apache.commons.jelly.tags.fmt.timeZone', 'Asia/Shanghai')

1.3、在Jenkins上添加k8s

系统管理 => 节点管理 => Configure Cloud

#kubernetes 地址采用了kube的服务器发现：
https://kubernetes.default.svc.cluster.local

#点击Test Connection，如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信

#Jenkins URL地址：
http://jenkins.devops.svc.cluster.local:30082

二、所需镜像制作

Jenkins-Master在构建Job的时候，Kubernetes会创建Jenkins-Slave的Pod来完成Job的构建
我们选择运行Jenkins-Slave的镜像为官方推荐镜像：jenkins/jnlp-slave:latest，但是这个镜像里面并没有Maven 环境，为了方便使用，我们需要自定义一个新的镜像

Dockerﬁle文件内容如下：

maven和settings.xml

链接：https://pan.baidu.com/s/1JsmmfrevDxX-ZNzgZS0nPw?pwd=yism 提取码：yism

FROM jenkins/jnlp-slave:latest

MAINTAINER ZHDYA

# 切换到 root 账户进行操作
USER root

# 安装 maven
COPY apache-maven-3.3.9-bin.tar.gz .

RUN tar -zxf apache-maven-3.3.9-bin.tar.gz && \
    mv apache-maven-3.3.9 /usr/local && \
    rm -f apache-maven-3.3.9-bin.tar.gz && \
    ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/bin/mvn && \
    ln -s /usr/local/apache-maven-3.3.9 /usr/local/apache-maven && \
    mkdir -p /usr/local/apache-maven/repo

COPY settings.xml /usr/local/apache-maven/conf/settings.xml

USER jenkins

构建镜像

docker build -t jenkins-slave-maven:latest .

上传harbor仓库（公共镜像全部放在/library/下）

docker tag jenkins-slave-maven:latest   harbor.devops.xxxxx.com/library/jenkins-slave-maven:latest

另外，在编译镜像时候采用了Docker in Docker模式，还需要镜像：

docker tag docker:stable harbor.devops.xxxxx.com/library/docker:stable

node节点做非安全仓库重启登录测试：

"insecure-registries" : ["harbor.devops.xxxxx.com"],

systemctl daemon-reload
systemctl restart docker

docker login -u admin -p "xxxxx" harbor.devops.xxxxx.com

三、JAVA应用

3.1、pom文件新增

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.5.6</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<groupId>com.xxxxx</groupId>
	<artifactId>xplatform-data-adapter</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>xplatform-data-adapter</name>
	<packaging>jar</packaging>
	<properties>
		<java.version>1.8</java.version>
	</properties>
	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-openfeign</artifactId>
			<version>3.0.5</version>
		</dependency>
		<dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>com.baomidou</groupId>
			<artifactId>mybatis-plus-boot-starter</artifactId>
			<version>3.4.3.4</version>
		</dependency>
		<dependency>
			<groupId>com.baomidou</groupId>
			<artifactId>dynamic-datasource-spring-boot-starter</artifactId>
			<version>3.4.0</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-redis</artifactId>
		</dependency>
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.12</version>
			<optional>true</optional>
		</dependency>

		<dependency>
			<groupId>org.mapstruct</groupId>
			<artifactId>mapstruct-jdk8</artifactId>
			<version>1.2.0.Final</version>
		</dependency>

		<dependency>
			<groupId>org.mapstruct</groupId>
			<artifactId>mapstruct-processor</artifactId>
			<version>1.2.0.Final</version>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>redis.clients</groupId>
			<artifactId>jedis</artifactId>
			<version>3.7.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.guava</groupId>
			<artifactId>guava</artifactId>
			<version>31.0.1-jre</version>
		</dependency>
		<dependency>
			<groupId>com.vdurmont</groupId>
			<artifactId>emoji-java</artifactId>
			<version>4.0.0</version>
		</dependency>
		<dependency>
			<groupId>net.logstash.logback</groupId>
			<artifactId>logstash-logback-encoder</artifactId>
			<version>4.11</version>
		</dependency>
		<dependency>
			<groupId>com.xxxxx.xplatform</groupId>
			<artifactId>xplatform-spring-boot-starter</artifactId>
			<version>v1.0</version>
		</dependency>
	</dependencies>

	<build>
		<finalName>${artifactId}</finalName>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<version>2.6.0</version>
				<configuration>
					<excludes>
						<exclude>
							<groupId>org.projectlombok</groupId>
							<artifactId>lombok</artifactId>
						</exclude>
					</excludes>
				</configuration>
			</plugin>
			//新增dockerfile的编译需求(版本一定为1.3.6)
			<plugin>
            <groupId>com.spotify</groupId>
            <artifactId>dockerfile-maven-plugin</artifactId>
            <version>1.3.6</version>
            <executions>
               <execution>
                  <id>default</id>
                  <goals>
                     <goal>build</goal>
                     <goal>push</goal>
                     <goal>tag</goal>
                  </goals>
               </execution>
            </executions>
            <configuration>
               <repository>${project.artifactId}</repository>
               <tag>latest</tag>
               <buildArgs>
                  <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
               </buildArgs>
            </configuration>
         </plugin>

		</plugins>
	</build>
	<distributionManagement>
		<repository>
			<id>releases</id>
			<url>http://nexus.sparkxmarketing.com/repository/maven-releases/</url>
		</repository>
		<snapshotRepository>
			<id>snapshots</id>
			<url>http://nexus.sparkxmarketing.com/repository/maven-snapshots/</url>
		</snapshotRepository>
	</distributionManagement>
</project>

3.2、dockerfile

FROM harbor.devops.xxxxx.com/library/openjdk:8-jdk
COPY ./target/xplatform-data-adapter.jar /opt/
RUN mkdir -p data/logs/
ENTRYPOINT [ "java", "-jar", "/opt/xplatform-data-adapter.jar" ,"-Dfile.encoding=UTF-8"]

3.3、deploy.yml

apiVersion: v1
kind: Namespace
metadata:
  name: business-api-system
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/http-probe: "true"
    prometheus.io/http-probe-path: /actuator/health/liveness
    prometheus.io/http-probe-port: "8080"
  name: $APP_NAME
  namespace: business-api-system
spec:
  ports:
  - name: bus
    port: 80
    protocol: TCP
    targetPort: 8082
  - name: health
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: $APP_NAME
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: $APP_NAME
  namespace: business-api-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: $APP_NAME
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: $APP_NAME
    spec:
      imagePullSecrets:
      - name: harborsecret
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - preference:
              matchExpressions:
              - key: eks.amazonaws.com/nodegroup
                operator: In
                values:
                - api-node-pool
            weight: 1
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: has.elastic.ip.outer
                operator: In
                values:
                - yeah
      containers:
      - env:
        - name: spring.profiles.active
          value: qa
        image: $IMAGE_NAME
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - curl
              - -X
              - POST
              - 127.0.0.1:8080/actuator/shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /actuator/health/liveness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: $APP_NAME
        ports:
        - containerPort: 8082
          name: project
          protocol: TCP
        - containerPort: 8080
          name: health
          protocol: TCP
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 40
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 2
        resources:
          limits:
            cpu: "1"
            memory: 4Gi
          requests:
            cpu: 500m
            memory: 2Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data/logs/$APP_NAME/
          name: app-logs
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 70
      volumes:
      - hostPath:
          path: /data/logs/$APP_NAME/
          type: ""
        name: app-logs

四、pipelines

def git_address = "https://gitlab.devops.xxxxx.com/xxxxx/${project_name}.git" 
def git_auth = "1043ca6c-88d6-4646-8ec9-d58f2d02a797"

//构建版本的名称
def tag = "latest"
//Harbor私服地址
def harbor_url = "harbor.devops.xxxxx.com"
//Harbor的项目名称
def harbor_project_name = "sparkx"
//Harbor的凭证
def harbor_auth = "402819f2-ebdd-4996-bc18-96a4e6b764db"
//k8s的凭证
def k8s_auth="96042afd-0c1e-40e1-b414-7c072fda9f8a"


//创建一个Pod的模板，label为jenkins-slave
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [ 
    containerTemplate(
        name: 'jnlp',
        image: "harbor.devops.xxxxx.com/library/jenkins-slave-maven"
        ),
	containerTemplate(
        name: 'kubectl',
        image: "harbor.devops.xxxxx.com/library/bitnami/kubectl",
        ttyEnabled: true,
        command: 'cat'
        ),
    containerTemplate( 
        name: 'docker',
        image: "harbor.devops.xxxxx.com/library/docker:stable",
        ttyEnabled: true,
        command: 'cat'
        ),
    ],
    volumes: [
        hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
    ],
)
{
node("jenkins-slave"){
    // 第一步
    stage('PULL'){
        checkout([$class: 'GitSCM', branches: [[name: "${branch}"]], extensions: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
    }
    // 第二步
    stage('Build'){
        //编译并安装公共工程
        sh "mvn clean install '-Dmaven.test.skip=true'"
    }
    // 第三步
    stage('Build&Tag&Push'){
        //把选择的项目信息转为数组
        def selectedProjects = "${project_name}".split(',')

        for(int i=0;i<selectedProjects.size();i++){
            //取出每个项目的名称
            def currentProjectName = selectedProjects[i];

            //定义镜像名称
            def imageName = "${currentProjectName}:${tag}"
			
			//定义newTag
			def newTag = sh(returnStdout: true,script: 'echo `date +"%Y%m%d%H%M"_``git describe --tags --always`').trim()

            //编译，构建本地镜像
            sh "mvn clean package -Dmaven.test.skip=true dockerfile:build"
            container('docker') {

                //给镜像打标签
                sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${currentProjectName}:${newTag}"

                //登录Harbor，并上传镜像
                withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')])
                {
                    //登录
                    sh "docker login -u ${username} -p ${password} ${harbor_url}"
                    //上传镜像
                    sh "docker push ${harbor_url}/${harbor_project_name}/${currentProjectName}:${newTag}"
                }

            //删除本地镜像
            sh "docker rmi -f ${imageName}" 
            sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${currentProjectName}:${newTag}"
            }
            def deploy_image_name = "${harbor_url}/${harbor_project_name}/${currentProjectName}:${newTag}"
            
            sh """
                sed -i 's#\$IMAGE_NAME#${deploy_image_name}#' deploy.yml
                sed -i 's#\$APP_NAME#${currentProjectName}#' deploy.yml
                sed -i 's#\$APP_REPLICAS#${replicas}#' deploy.yml
                cat deploy.yml
            """
        }
    }
	//部署到K8S 
	stage('Deploy') {
            kubernetesDeploy(
                kubeconfigId: '96042afd-0c1e-40e1-b414-7c072fda9f8a',
                configs: 'deploy.yml' 
            )
        }
}
}

问题点：

1、下载镜像 ImagePullBackOff

2种方式（要在应用所在的ns中创建）：
1、kubectl create secret generic harborsecret --from-file=.dockerconfigjson=config.json --type=kubernetes.io/dockerconfigjson
2、kubectl create secret docker-registry registry-auth-secret --docker-server=harbor.devops.xxxxx.com --docker-username=admin --docker-password="xxxxx" -nbusiness-api-system

2、权限问题(https://blog.csdn.net/m0_54852350/article/details/123493674)

目前部署的Jenkins有无密钥都可以连接。因为之前做了"大权限"的RBAC(JENKINS的cluster role 一定要给足权限)

3、插件问题

kubernetesDeploy 一定要选择1.0.0版本，用如上链接下载即可；

4、pom.xml文件版本问题

使用1.4.x不兼容，选用1.3.x

AWS

AWS Jenkins

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议，转载请注明出处！

elasticsearch核心概念上一篇

将加密的 AWS EFS 文件系统挂载到 EKS 的 Pod 中下一篇