Kubernetes Ingress — Kong
1、Kong 主要的概念
Service 服务，也称服务对象，是 Kong 管理的上游 API 和微服务，通过 Kong 转发后根据请求的协议，host，method，path 匹配到实际的服务地址。
Route 路由，作为客户端的入口，可以通过定义一些规则来匹配客户端的请求，每个路由都会关联一个 Service , 并且 Service 可以关联多个 Route，当匹配到客户端的请求时，每个请求都会被代理到其配置的 Service 中。
Upstream 相当于Kong提供了一个负载均衡的功能，基于 Nginx 的虚拟主机的方式做的负载功能，在 Service 中指定 host 的时候，也可以指定 upstream 。
Target 是 upstream 进行负载均衡的终端，当我们的服务是多节点时，那每个节点都是作为一个 target 。可以负载的权重，也可以通过 upstream 对 target 进行健康检查。
Consumer 可以代表一个 Service 的请求消费者，也可以代表对应到实际应用中的一个用户。
Plugin 在请求被代理到上游 API 之前或之后执行相关动作的插件，如限流插件。
Kong-ingress-controller 是对 Kubernetes Ingress API 的实现，通过和 Kubernetes API 交互，动态的去感知 Kubernetes 中 Ingress 规则变化。
整体的链路是 Route > Service > Upstream > Target
2、Kong 安装
Kong 在 1.1 版本后引入了一个新特性 DB-less 模式，即无 DB 运行 Kong，官方推荐在 Kubernetes 采用该模式运行，所有配置都存储在 Kubernetes 控制面板中。简化了 Kong 的操作，而无需再维护一个关系型数据库。
当然也可以根据自己的需求，采用数据库的模式运行 Kong， 官方推荐 Postgres。需要注意的是，如果采用了无 DB 模式，那么在 Konga 的管理页面是无法添加插件和证书等资源。
在 Kubernetes 环境下，Admin API 方式不是很适应 Kubernetes 声明式管理方式。所以 Kong Ingress Controller 定义了五个 CRDs，基本上涵盖了原 Admin API 的各个方面。
[root@k8s-test01 ~]# kubectl get crds | grep konghq.com
kongclusterplugins.configuration.konghq.com 2021-02-04T16:02:43Z
kongconsumers.configuration.konghq.com 2021-02-04T16:02:43Z
kongingresses.configuration.konghq.com 2021-02-04T16:02:43Z
kongplugins.configuration.konghq.com 2021-02-04T16:02:43Z
tcpingresses.configuration.konghq.com 2021-02-04T16:02:43Zkongconsumers：给不同的API用户提供不同的消费者身份
kongcredentials：用户的认证凭证
kongplugins：插件的配置
kongingresses、tcpingresses：定义代理行为规则，是对 Ingress 的补充配置
安装普罗米修斯监控 Kong 网络请求，如果已经安装则跳过。
# 创建 namespace
kubectl create ns monitoring
# 安装 namespace
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm search repo prometheus-community
helm install my-release prometheus-community/prometheus --namespace monitoring --values https://bit.ly/2RgzDtg --version 13.2.1 \
--set alertmanager.persistentVolume.enabled=false \
--set server.persistentVolume.enabled=false
# 安装 grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm search repo grafana
helm install my-grafana grafana/grafana --namespace monitoring --values http://bit.ly/2FuFVfV --version 6.2.1 --set persistence.enabled=false --set service.type=NodePort --set service.nodePort=3000
# 查看 grafana admin用户的密码
kubectl get secret --namespace monitoring my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echoKong 可以使用环境变量的方式调整配置，但是 helm 的安装方式文档有点糟糕，这里使用 yaml 的方式安装，无 DB 模式运行。
wget https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/master/deploy/single/all-in-one-dbless.yaml
kubectl create -f all-in-one-dbless.yamlservice 改成 nodePort 模式
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-type: nlb
name: kong-proxy
namespace: kong
spec:
ports:
- name: proxy
port: 80
protocol: TCP
targetPort: 8000
nodePort: 8000
- name: proxy-ssl
port: 443
protocol: TCP
targetPort: 8443
nodePort: 8443
- name: admin
port: 8444
protocol: TCP
targetPort: 8444
nodePort: 8444
selector:
app: ingress-kong
type: NodePort
externalTrafficPolicy: LocalKong 的一些配置
# Kong 管理地址，这里使用的http，所以 kong-ingress-controller、Konga 只能用 http 连接
- name: KONG_ADMIN_LISTEN
value: 0.0.0.0:8444
# 无 DB 模式
- name: KONG_DATABASE
value: "off"
# nginx 工作进程数
- name: KONG_NGINX_WORKER_PROCESSES
value: "2"
# 开启压缩
- name: KONG_NGINX_HTTP_GZIP
value: "on"
- name: KONG_NGINX_HTTP_GZIP_TYPES
value: "*"
- name: KONG_NGINX_HTTP_GZIP_PROXIED
value: "any"
- name: KONG_NGINX_HTTP_GZIP_MIN_LENGTH
value: "1000"
# 日志级别
- name: KONG_LOG_LEVEL
value: debug
# 获取真实IP
- name: KONG_TRUSTED_IPS
value: "0.0.0.0/0,::/0"
- name: KONG_REAL_IP_HEADER
value: "X-Forwarded-For"Konga 依赖 Postgres 数据库，如果已经有 Postgres 可以不用安装，创建一个用户和数据库即可。
[root@k8s-test01 kong]# cat pg.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
selector:
matchLabels:
app: postgres
serviceName: postgres
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: postgres
spec:
terminationGracePeriodSeconds: 60
containers:
- name: postgres
imagePullPolicy: IfNotPresent
image: docker.io/library/postgres:10-alpine
env:
- name: POSTGRES_PASSWORD
value: "PGadmin"
- name : PGTZ
value: "Asia/Shanghai"
resources:
requests:
cpu: 500m
memory: 500Mi
limits:
cpu: 3000m
memory: 3096Mi
ports:
- containerPort: 5432
name: tcp
livenessProbe:
tcpSocket:
port: 5432
initialDelaySeconds: 60
periodSeconds: 60
volumeMounts:
- mountPath: /var/lib/postgresql/data
readOnly: false
name: data
- mountPath: /etc/localtime
readOnly: false
name: time-data
- mountPath: /etc/timezone
readOnly: false
name: timezone
volumes:
- name: time-data
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
- name: timezone
hostPath:
path: /etc/timezone
- name: data
emptyDir: {} # 注意！！！ 数据没有持久化！
---
apiVersion: v1
kind: Service
metadata:
labels:
app: postgres
name: postgres
spec:
type: ClusterIP
ports:
- port: 5432
targetPort: 5432
name: tcp
selector:
app: postgres进入 Postgres 创建数据库和用户
[root@k8s-test01 kong]# kubectl -n kong exec -it postgres-0 -- bash
bash-5.1# psql -h 127.0.0.1 -p 5432 -U postgres
psql (10.15)
Type "help" for help.
postgres=# CREATE USER konga WITH PASSWORD 'Konga2021';
CREATE ROLE
postgres=# CREATE DATABASE konga OWNER konga ENCODING UTF8;
CREATE DATABASE
postgres=# GRANT ALL PRIVILEGES ON DATABASE konga TO konga;
GRANT
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
konga | konga | UTF8 | en_US.utf8 | en_US.utf8 | =Tc/konga +
| | | | | konga=CTc/konga
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=# \q
bash-5.1# 安装 Konga
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: konga
name: konga
namespace: kong
spec:
replicas: 1
selector:
matchLabels:
app: konga
template:
metadata:
labels:
app: konga
spec:
restartPolicy: Always
containers:
- env:
- name: PORT
value: "1337"
- name: DB_ADAPTER
value: postgres
- name: DB_URI
value: "postgresql://konga:Konga2021@postgres:5432/konga"
image: pantsel/konga:0.14.9
name: konga
ports:
- containerPort: 1337
protocol: TCP
name: http
---
apiVersion: v1
kind: Service
metadata:
name: konga
namespace: kong
spec:
type: NodePort
ports:
- name: http
port: 1337
targetPort: 1337
protocol: TCP
nodePort: 1337
selector:
app: konga浏览器打开 nodeport 端口打开 Konga，配置 Kong 的管理地址即可。
至此，kong 环境已经部署完成，下面我们来简单测试下 Kong 在 Kubernetes 下的应用，使用 Kubernetes 声明式的方式配置。也可以在Konga 页面上配置，但笔者嫌麻烦，在页面配置的时间，yaml 文件早就编排好了....
3、Prometheus插件
创建测试的 pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-svc
spec:
replicas: 1
selector:
matchLabels:
app: http-svc
template:
metadata:
labels:
app: http-svc
spec:
containers:
- name: http-svc
image: docker.io/kennethreitz/httpbin
ports:
- containerPort: 80
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP在服务上创建普罗米修斯插件配置，global 为 true 时，全局网络请求都使用普罗米修斯插件。
[root@k8s-test01 kong]# cat prometheus-plugin.yaml
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: prometheus
annotations:
kubernetes.io/ingress.class: kong
labels:
global: "false"
plugin: prometheus
[root@k8s-test01 kong]# kubectl get KongClusterPlugin
NAME PLUGIN-TYPE AGE
prometheus prometheus 38h
[root@k8s-test01 kong]#创建测试 pod 的 service 和 ingress，通过 annotations 加载普罗米修斯插件。
---
apiVersion: v1
kind: Service
metadata:
name: billing
labels:
app: billing
annotations:
konghq.com/plugins: prometheus
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: http-svc
---
apiVersion: v1
kind: Service
metadata:
name: invoice
labels:
app: invoice
annotations:
konghq.com/plugins: prometheus
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: http-svc
---
apiVersion: v1
kind: Service
metadata:
name: comments
labels:
app: comments
annotations:
konghq.com/plugins: prometheus
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: http-svc
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo
annotations:
konghq.com/strip-path: "true"
kubernetes.io/ingress.class: kong
spec:
rules:
- host: "foo.bar.com"
http:
paths:
- pathType: Prefix
path: /billing
backend:
service:
name: billing
port:
number: 80
- pathType: Prefix
path: /comments
backend:
service:
name: comments
port:
number: 80
- pathType: Prefix
path: /invoice
backend:
service:
name: invoice
port:
number: 80创建完成后，可以在 Konga 查看详情
我们制造一些流量，查看效果。
while true; do \
curl http://foo.bar.com:8000/billing/status/200; \
curl http://foo.bar.com:8000/billing/status/501; \
curl http://foo.bar.com:8000/invoice/status/201; \
curl http://foo.bar.com:8000/invoice/status/404; \
curl http://foo.bar.com:8000/comments/status/200; \
curl http://foo.bar.com:8000/comments/status/200; \
sleep 0.01; \
done打开grafana
官方模板:https://grafana.com/grafana/dashboards/7424
我们可以根据这些网络指标配置普罗米修斯的告警规则。
4、key-auth 认证登录
key-auth 类似 nginx 的 auth_basic 认证登录。
开启前
[root@k8s-test01 kong]# curl -i http://foo.bar.com:8000/billing/status/200
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:00:13 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1开启key-auth 验证
# 在路由上创建 key-auth 插件，配置身份验证
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: httpbin-auth
plugin: key-auth
" | kubectl apply -f -
# 创建一个消费者
echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: harry
annotations:
kubernetes.io/ingress.class: kong
username: harry" | kubectl apply -f -
# 创建带有 API 密钥的 Secret 资源
kubectl create secret generic harry-apikey \
--from-literal=kongCredType=key-auth \
--from-literal=key=my-sooper-secret-key
# API密钥和消费者关联
echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: harry
annotations:
kubernetes.io/ingress.class: kong
username: harry
credentials:
- harry-apikey" | kubectl apply -f -更新 Ingress
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo
annotations:
konghq.com/strip-path: "true"
kubernetes.io/ingress.class: kong
konghq.com/plugins: httpbin-auth
spec:
rules:
- host: "foo.bar.com"
http:
paths:
- pathType: Prefix
path: /billing
backend:
service:
name: billing
port:
number: 80
- pathType: Prefix
path: /comments
backend:
service:
name: comments
port:
number: 80
- pathType: Prefix
path: /invoice
backend:
service:
name: invoice
port:
number: 80测试
# 不带 key 访问，401
[root@k8s-test01 kong]# curl -i http://foo.bar.com:8000/billing/status/200
HTTP/1.1 401 Unauthorized
Date: Sat, 06 Feb 2021 09:16:48 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
WWW-Authenticate: Key realm="kong"
Content-Length: 45
X-Kong-Response-Latency: 0
Server: kong/2.2.1
{
"message":"No API key found in request"
}
# 带 key 访问，200
[root@k8s-test01 kong]# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:17:02 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 1
Via: kong/2.2.15、rate-limiting 速率限制
没有限制之前
[root@k8s-test01 kong]# while true; do curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200; done
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:27:18 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:27:18 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1在服务上创建 rate-limiting 插件限制配置，
echo "
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: http-ratelimit
annotations:
kubernetes.io/ingress.class: kong
labels:
global: \"false\"
config:
policy: local
second: 1
plugin: rate-limiting
" | kubectl apply -f -更新 Service
---
apiVersion: v1
kind: Service
metadata:
name: billing
labels:
app: billing
annotations:
konghq.com/plugins: http-ratelimit, prometheus
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: http-svc测试
[root@k8s-test01 kong]# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200;
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:55:02 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-RateLimit-Limit-Second: 1
X-RateLimit-Remaining-Second: 0
X-RateLimit-Remaining-Minute: 0
RateLimit-Limit: 1
RateLimit-Remaining: 0
X-RateLimit-Limit-Minute: 1
RateLimit-Reset: 58
X-Kong-Upstream-Latency: 4
X-Kong-Proxy-Latency: 1
Via: kong/2.2.1
[root@k8s-test01 kong]# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200;
HTTP/1.1 429 Too Many Requests
Date: Sat, 06 Feb 2021 09:55:04 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Retry-After: 56
Content-Length: 41
X-RateLimit-Limit-Second: 1
X-RateLimit-Remaining-Second: 1
X-RateLimit-Remaining-Minute: 0
RateLimit-Limit: 1
RateLimit-Remaining: 0
X-RateLimit-Limit-Minute: 1
RateLimit-Reset: 56
X-Kong-Response-Latency: 0
Server: kong/2.2.1
{
"message":"API rate limit exceeded"
}06、ip-restriction IP限制
在路由上创建 ip-restriction 插件配置，允许8.210.1.244 访问，拒绝 47.242.91.20 访问。
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: ip-restriction-test
config:
allow:
- 8.210.1.244
deny:
- 47.242.91.20
plugin: ip-restriction
" | kubectl apply -f -更新 Ingress
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo
annotations:
konghq.com/strip-path: "true"
kubernetes.io/ingress.class: kong
konghq.com/plugins: httpbin-auth, ip-restriction-test
spec:
rules:
- host: "foo.bar.com"
http:
paths:
- pathType: Prefix
path: /billing
backend:
service:
name: billing
port:
number: 80
- pathType: Prefix
path: /comments
backend:
service:
name: comments
port:
number: 80
- pathType: Prefix
path: /invoice
backend:
service:
name: invoice
port:
number: 80在 47.242.91.20 上访问拒绝
[root@k8s-test01 kong]# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/invoice/status/200
HTTP/1.1 403 Forbidden
Date: Sat, 06 Feb 2021 10:18:54 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.2.1
{
"message":"Your IP address is not allowed"
}在 8.210.1.244 上正常访问
---
root@k8s-test02:~# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/invoice/status/200
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 10:30:20 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 25
X-Kong-Proxy-Latency: 0
Via: kong/2.2.107、总结
通过上面的示例，我们大概了解了在 Kubernetes 环境下 Kong 与 Ingress 的结合使用，目前对 Consumer 没有太多的使用，后续有机会再跟大家介绍下。官方也有很多的示例，感兴趣的朋友可以去官网查看。
Kong 拥有很强的扩展能力，如果自带的插件不能满足我们的需求，我们可以写插件来实现。笔者的同事也写了一些插件在生产环境中运行。
写好的插件，重新构建镜像，通过下面环境变量启用。
- name: KONG_LUA_PACKAGE_PATH
value: "/go/kong-lua/kong-plugin/?.lua;;"
- name: KONG_PLUGINS
value: "bundled,myplugin,jwt-plus,oauth2-plus,oss-simple"
本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！