K8S之ELK日志收集（多维度）

主要分为4方面：

  • 收集哪些日志？
  • elk Stack日志方案？
  • 容器中的日志怎么收集？
  • k8S平台中应用日志收集(模版可针对spring cloud、dubbo微服务的日志）根据自己的项目需求更改；

一、收集哪些日志

  • k8s系统的组件日志 比如kubectl get cs下面的组件；
  • master节点上的controller-manager，scheduler，apiserver；
  • node节点上的kubelet，kube-proxy；
  • k8s Cluster里面部署的应用程序日志；

最常见的ELK架构如下：

mark
  • Filebeat在各个业务端进行日志采集，然后上传至Logstash；
  • Logstash节点并行（负载均衡，不作为集群），对日志记录进行过滤处理，然后上传至Elasticsearch集群；
  • Elasticsearch构成集群服务，提供日志的索引和存储能力；
  • Kibana负责对Elasticsearch中的日志数据进行检索、分析；

二、日志采集方式

官方文档中提到了三种采集方式，这里简单介绍一下：

mark

三种收集方案的优缺点：

方式 优势 缺点
方案一：Node上部署一个日志收集程序 每个Node仅需部署一个日志收集程序，资源消耗少，对应用无侵入 应用程序日志需要写到标准输出和标准错误输出，不支持多行日志
方案二：Pod中附加专用日志收集的容器 低耦合 每个Pod启动一个日志收集代理，增加资源消耗，并增加运维维护成本
方案三：应用程序直接推送日志 无需额外收集工具 浸入应用，增加应用复杂度

相对来说，方式1在业界使用更为广泛，并且官方也更为推荐。因此，最终我们采用ELK+Filebeat架构，并基于方式1，如下：

mark

三、部署

单节点部署ELK的方法较简单，可以参考下面的yaml编排文件，整体就是创建一个es，然后创建kibana的可视化展示，创建一个es的service服务，然后通过ingress的方式对外暴露域名访问；

3.1、elasticsearch

编写es的yaml，这里部署的是单机版，在k8s集群内中，通常当日志量每天超过20G以上的话，还是建议部署在k8s集群外部，支持分布式集群的架构，这里使用的是有状态部署的方式，并且使用动态存储进行持久化，需要提前创建好存储类，才能运行该yaml； 

# cat elasticsearch.yaml 

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    k8s-app: elasticsearch
spec:
  serviceName: elasticsearch
  selector:
    matchLabels:
      k8s-app: elasticsearch
  template:
    metadata:
      labels:
        k8s-app: elasticsearch
    spec:
      containers:
      - image: elasticsearch:7.3.1
        name: elasticsearch
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 0.5 
            memory: 500Mi
        env:
          - name: "discovery.type"
            value: "single-node"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx2g" 
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 20Gi

---

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch

3.2、kibana

需要部署一个Kibana来对搜集到的日志进行可视化展示，使用Deployment的方式编写一个yaml，使用ingress对外进行暴露访问，直接引用了es； 

# cat kibana.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: kube-system
  labels:
    k8s-app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:7.3.1
        resources:
          limits:
            cpu: 1
            memory: 500Mi
          requests:
            cpu: 0.5 
            memory: 200Mi
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://elasticsearch:9200
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: kube-system
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: kube-system
spec:
  rules:
  - host: kibana.zhdya.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

[root@k8s-master]# ls
elasticsearch.yaml kibana.yaml
[root@k8s-master]# kubectl create -f .
[root@k8s-master]# kubectl get pod -n kube-system
NAME                                    READY   STATUS              RESTARTS   AGE
elasticsearch-0                         1/1     Running             0          7m15s
kibana-b7d98644-cmg9k                   1/1     Running             0          7m15s

绑定本机hosts，访问域名验证 

192.168.171.13 kibana.zhdya.cn

mark

这里选择Explore导入自己的数据

四、Filebeat日志采集

es和kibana部署好了之后，我们如何采集pod日志呢，我们采用方案一的方式，首先在每一个node上中部署一个filebeat的采集器，采用的是7.3.1版本，因为filebeat是对k8s有支持，可以连接api给pod日志打标签，所以yaml中需要进行认证，最后在配置文件中对获取数据采集了之后输入到es中，已在yaml中配置好。

4.1、filebeat

# cat filebeat-kubernetes.yaml 
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      hints.enabled: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: elastic/filebeat:7.3.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---
[root@k8s-master1 elk]# kubectl apply -f filebeat-kubernetes.yaml
configmap/filebeat-config created
configmap/filebeat-inputs created
daemonset.apps/filebeat created
clusterrolebinding.rbac.authorization.k8s.io/filebeat created
clusterrole.rbac.authorization.k8s.io/filebeat created
serviceaccount/filebeat created
[root@k8s-master1 elk]# kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
filebeat-9fv75                            1/1     Running   0          2m42s
filebeat-b86pj                            1/1     Running   0          2m42s
filebeat-zhh6s                            1/1     Running   0          2m42s

4.2、在kibana的web界面进行配置日志可视化

首先打开kibana的web界面，点击左边菜单栏汇中的设置，然后点击在Kibana下面的索引按钮，然后点击左上角的然后根据如图所示分别创建一个 

filebeat-7.3.1-*

mark

然后按照时间过滤，完成创建：

mark

索引匹配创建以后，点击左边最上面的菜单Discove，然后可以在左侧看到我们刚才创建的索引，然后就可以在下面添加要展示的标签，也可以对标签进行筛选，最终效果如图所示，可以看到采集到的日志的所有信息：

mark

当然我日常用来排障 多数是用这个namespace来筛选： mark

挖坑记： 

第一个坑，k8s 1.16以后的版本API变了，详情看
https://www.infoq.cn/article/rrJ5IGIAXy4V-X2Hb0QA
no matches for kind "DaemonSet" in version "extensions/v1beta1"
no matches for kind "Deployment" in version "extensions/v1beta1"

解决：
apiVersion: apps/v1
kind: DaemonSet
=================================================================

第二个坑新版本的 apps.v1 API需要在yaml文件中，selector变为必选项
https://www.cnblogs.com/robinunix/p/11155860.html
error: error validating "prometheus-deployment.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

原有：
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
-------------------------------------------------------------        
更改为：
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat

4.3、k8s组件的日志进行采集

因为我的环境是用的二进制进行部署的，因此我的组件日志都在/var/log/message里面，因此我们还需要部署一个采集k8s组件日志的pod副本，自定义了索引k8s-module-%{+yyyy.MM.dd}，编写yaml如下：

# vim k8s-logs.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: k8s-logs-filebeat-config
  namespace: kube-system 
  
data:
  filebeat.yml: |
    filebeat.inputs:
      - type: log
        paths:
          - /var/log/messages  
        fields:
          app: k8s 
          type: module 
        fields_under_root: true

    setup.ilm.enabled: false
    setup.template.name: "k8s-module"
    setup.template.pattern: "k8s-module-*"

    output.elasticsearch:
      hosts: ['elasticsearch.kube-system:9200']
      index: "k8s-module-%{+yyyy.MM.dd}"

---

apiVersion: apps/v1
kind: DaemonSet 
metadata:
  name: k8s-logs
  namespace: kube-system
spec:
  selector:
    matchLabels:
      project: k8s 
      app: filebeat
  template:
    metadata:
      labels:
        project: k8s
        app: filebeat
    spec:
      containers:
      - name: filebeat
        image: elastic/filebeat:7.3.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 500m
            memory: 500Mi
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        - name: k8s-logs 
          mountPath: /var/log/messages
      volumes:
      - name: k8s-logs
        hostPath: 
          path: /var/log/messages
      - name: filebeat-config
        configMap:
          name: k8s-logs-filebeat-config
[root@k8s-master1 elk]#  kubectl create -f k8s-logs.yaml
configmap/k8s-logs-filebeat-config created
daemonset.apps/k8s-logs created

[root@k8s-master1 elk]# kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
k8s-logs-7znxf                            1/1     Running   0          18s
k8s-logs-jvl7t                            1/1     Running   0          18s
k8s-logs-s6tr9                            1/1     Running   0          18s

还是原来的步骤，最后的一个按钮索引管理这里： 

k8s-module-*

mark

验证测试：

在其中一个node上，输入 

echo hello logs >>/var/log/messages
然后在web上选择k8s-module-*的索引匹配，就可以在采集到的日志中看到刚才输入的hello logs，则证明采集成功，如图所示：

mark

五、方案二：Pod中附加专用日志收集的容器

5.1、前端php-demo的例子：

我们也可以使用方案的方式，通过在pod中注入一个日志收集的容器来采集pod的日志，以一个php-demo的应用为例，使用emptyDir的方式把日志目录共享给采集器的容器收集，编写nginx-deployment.yaml ，直接在pod中加入filebeat的容器，并且自定义索引为nginx-access-%{+yyyy.MM.dd}

# vim nginx-deployment.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: php-demo
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      project: www
      app: php-demo
  template:
    metadata:
      labels:
        project: www
        app: php-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: nginx 
        image: zhdya/nginx-php 
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 256Mi
          limits:
            cpu: 1
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /status.html
            port: 80
          initialDelaySeconds: 20
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /status.html
            port: 80
          initialDelaySeconds: 20
          timeoutSeconds: 20
        volumeMounts:
        - name: nginx-logs 
          mountPath: /usr/local/nginx/logs

      - name: filebeat
        image: elastic/filebeat:7.3.1 
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        - name: nginx-logs 
          mountPath: /usr/local/nginx/logs

      volumes:
      - name: nginx-logs
        emptyDir: {}
      - name: filebeat-config
        configMap:
          name: filebeat-nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-nginx-config
  namespace: kube-system
  
data:
  filebeat.yml: |-
    filebeat.inputs:
      - type: log
        paths:
          - /usr/local/nginx/logs/access.log
        # tags: ["access"]
        fields:
          app: www
          type: nginx-access
        fields_under_root: true

    setup.ilm.enabled: false
    setup.template.name: "nginx-access"
    setup.template.pattern: "nginx-access-*"

    output.elasticsearch:
      hosts: ['elasticsearch.kube-system:9200']
      index: "nginx-access-%{+yyyy.MM.dd}"

创建刚才编写的nginx-deployment.yaml，创建成果之后会在kube-system命名空间下面pod/web-demo-58d89c9bc4-r5692的2个pod副本，还有一个对外暴露的service/web-demo

[root@k8s-master elk]# kubectl apply -f nginx-deployment.yaml
[root@k8s-master fek]# kubectl get pod -n kube-system
NAME                        READY   STATUS    RESTARTS   AGE
php-demo-85849d58df-d98gv   2/2     Running   0          26m
php-demo-85849d58df-sl5ss   2/2     Running   0          26m

配置索引： 

nginx-access-*
查看日志即可；

mark

5.2、前端java-demo的例子：

# cat tomcat-deployment.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: tomcat-java-demo
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      project: www
      app: java-demo
  template:
    metadata:
      labels:
        project: www
        app: java-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: tomcat
        image: zhdya/java-demo:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
        volumeMounts:
        - name: tomcat-logs 
          mountPath: /usr/local/tomcat/logs

      - name: filebeat
        image: elastic/filebeat:7.3.1 
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        - name: tomcat-logs 
          mountPath: /usr/local/tomcat/logs
      volumes:
      - name: tomcat-logs
        emptyDir: {}
      - name: filebeat-config
        configMap:
          name: filebeat-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: test

data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /usr/local/tomcat/logs/catalina.*

      fields:
        app: www
        type: tomcat-catalina
      fields_under_root: true
      multiline:
        pattern: '^\['
        negate: true
        match: after

    setup.ilm.enabled: false
    setup.template.name: "tomcat-catalina"
    setup.template.pattern: "tomcat-catalina-*"

    output.elasticsearch:
      hosts: ['elasticsearch.kube-system:9200']
      index: "tomcat-catalina-%{+yyyy.MM.dd}"
[root@k8s-master elk]# kubectl get pod -n test
NAME                                READY   STATUS    RESTARTS   AGE
tomcat-java-demo-7ffd4dc7c5-26xjf   2/2     Running   0          5m19s
tomcat-java-demo-7ffd4dc7c5-lwfgr   2/2     Running   0          7m31s
tomcat-java-demo-7ffd4dc7c5-pwj77   2/2     Running   0          8m50s

增加索引： 

tomcat-catalina-*
同理查看应用日志即可；

K8S Kubernetes

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！