k8s sidecar 实现nginx热重载

原理说明

k8s cm在不配置subpath的情况下，当你更新挂载的文件内容的时候，也会同步更新到容器内部的文件（会有时延）类似prometheus等应用可以在文件更新后自动热加载新配置文件，但是诸如nginx\apache等应用并不会自己去加载文件reload，但这类应用往往支持通过信号的形式来控制，这时可以为其配一个sidecar容器以帮助其实现自动更新

参考文档：https://blog.fleeto.us/post/refresh-cm-with-signal/
https://www.jianshu.com/p/57e3819a2e7c
https://jimmysong.io/kubernetes-handbook/usecases/sidecar-pattern.html
https://izsk.me/2020/05/10/Kubernetes-deploy-hot-reload-when-configmap-update/

sidecar：全称sidecar proxy，为某种为了给应用程序提供补充功能而运行的单独的进程，通常以容器的形式和应用容器处于同一pod内，它可以在无需为应用程序添加额外的第三方组件的情况下为应用程序添加功能或者修改程序的代码和配置，目前已有较多成功的使用示例，如prometheus的社区开源高可用方案thaos

原理：使用inotify去监控config的变化，当发生改变的时候，使用信号去通知nginx加载配置文件，对于nginx来说可以使用nginx -s或kill来实现，需要注意两者之间信号相互的映射关系
http://io.upyun.com/2017/08/19/nginx-signals/
http://www.wuditnt.com/775/

-s signal      Send a signal to the master process.  The argument signal can be one of: stop, quit, reopen, reload.
                    The following table shows the corresponding system signals:
                    stop    SIGTERM
                    quit    SIGQUIT
                    reopen  SIGUSR1
                    reload  SIGHUP
SIGNALS
     The master process of nginx can handle the following signals:
     SIGINT, SIGTERM  Shut down quickly.
     SIGHUP           Reload configuration, start the new worker process with a new configuration, and gracefully shut
                      down old worker processes.
     SIGQUIT          Shut down gracefully.
     SIGUSR1          Reopen log files.
     SIGUSR2          Upgrade the nginx executable on the fly.
     SIGWINCH         Shut down worker processes gracefully.

示例

准备监听脚本

#!/bin/sh
while :
do
  # 获取文件名称
  REAL=`readlink -f ${FILE}`
  # 监控指定事件
  inotifywait -e delete_self "${REAL}"
  # 获取特定进程名称的 PID
  PID=`pgrep ${PROCESS} | head -1`
  # 发送信号
  kill "-${SIGNAL}" "${PID}"
  #nginx -s reload
done

准备sidecar

FROM alpine
RUN apk add --update inotify-tools
ENV FILE="/etc/nginx/conf.d/" PROCESS="nginx" SIGNAL="hup"
COPY inotifynginx.sh /usr/local/bin
CMD ["/usr/local/bin/inotifynginx.sh"]

准备yaml

apiVersion: apps/v1 
kind: Deployment
metadata: 
  name: nginx-test
spec: 
  replicas: 1
  selector: 
    matchLabels: 
      app: nginx-test
  template: 
    metadata: 
      labels: 
        app: nginx-test
    spec:
      shareProcessNamespace: true
      containers: 
      - name: "nginx-test"
        image: nginx
        ports: 
        - name: httpd  
          containerPort: 8888
          protocol: TCP 
        volumeMounts: 
          - mountPath: /etc/nginx/conf.d/
            name: nginx-config
            readOnly: true
          - mountPath: /usr/share/nginx/
            name: nginx-file
      - name: "inotify"
        image: inotify:v1
        imagePullPolicy: IfNotPresent
        volumeMounts: 
        - name: nginx-config
          mountPath: /etc/nginx/conf.d/
          readOnly: true
      volumes: 
      - name: nginx-config
        configMap: 
          name: nginx-config
      - name: nginx-file
        hostPath: 
          path: /tmp/nginx       
---

apiVersion: v1
kind: Service
metadata:
  name: nginx-service-test
spec: 
  selector: 
    app: nginx-test
  type: NodePort
  ports: 
    - port: 8888
      targetPort: 8888
      nodePort: 31111 

---
apiVersion: v1
kind: ConfigMap
metadata: 
  name: nginx-config
  labels: 
    app: nginx-test
data:
  test1.conf: | 
    server{
        listen *:8888;
        server_name www.test1.xyz;
        location  /
        {
                root /usr/share/nginx/;
                index test1.html;
        }
    }

pod中的容器间共享进程命名空间https://kubernetes.io/zh-cn/docs/tasks/configure-pod-container/share-process-namespace/，需要注意的是并不是所有的容器都适合这样的方法来实现，具体详情可以参考连接

测试

[root@worker-node01 ~]# curl 10.99.31.106:8888
test1
#修改cm中的index
[root@worker-node01 ~]# kubectl get cm
NAME               DATA   AGE
confnginx          1      4d1h
kube-root-ca.crt   1      24d
nginx-config       1      48s
[root@worker-node01 ~]# kubectl edit nginx-config
error: the server doesn't have a resource type "nginx-config"
[root@worker-node01 ~]# kubectl edit cm nginx-config
configmap/nginx-config edited
#可以发现nginx完成了自动热加载
[root@worker-node01 ~]# curl 10.99.31.106:8888
test2