Increasing the usefulness of your Kubernetes Ingress logging

Like most cloud folks you are probably using Kibana + Elasticsearch as part of your log management solution. But did you know with a little regex-fu you can make that logging more interesting? See the kibana expansion in the image, the URI, host, service, etc are all expanded for your reporting pleasure.

First, lets install our ingress with some annotations. I've made the interesting bits red.

helm install stable/nginx-ingress --name ingress \
  --set controller.service.externalTrafficPolicy=Local \
  --set rbac.create=true \
  --set controller.podAnnotations.fluentbit\\.io/parser=k8s-nginx-ingress

If your ingress is already running you can use this instead:

kubectl annotate pods --overwrite ingress-nginx-####   fluentbit.io/parser=k8s-nginx-ingress

Now, lets install fluent-bit (to feed the Elasticsearch). We will add a custom-regex for the nginx-ingress log format. Its not the same as the nginx default so we can't use the built-in.

image:
  fluent_bit:
    repository: fluent/fluent-bit
    tag: 0.14.1
  pullPolicy: IfNotPresent
metrics:
  enabled: true
  service:
    port: 2020
    type: ClusterIP
trackOffsets: false
backend:
  type: es
  forward:
    host: fluentd
    port: 24284
  es:
    host: elasticsearch
    port: 9200
    index: kubernetes_cluster
    type: flb_type
    logstash_prefix: logstash
    time_key: "@timestamp"
    http_user:
    http_passwd:
    tls: "off"
    tls_verify: "on"
    tls_ca: ""
    tls_debug: 1

parsers:
  enabled: true
  regex:
    - name: k8s-nginx-ingress
      regex:  '^(?<host>[^ ]*) - \[(?<real_ip>)[^ ]*\] - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<last>[^$]*)'

Once this is done you'll have something like below in your logs. See how all the fields are expanded to their own rather than being stuck in log: ?

{
  "_index": "logstash-2018.09.13",
  "_type": "flb_type",
  "_id": "s_0x1GUB6XzNVUp1wNV6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-13T18:29:14.897Z",
    "log": "10.40.0.0 - [10.40.0.0] - - [13/Sep/2018:18:29:14 +0000] \"GET / HTTP/1.1\" 200 9056 \"-\" \"curl/7.58.0\" 75 0.000 [default-front-end-80] 10.40.0.9:8079 9056 0.000 200 a134ebded3504000d63646b647e54585\n",
    "stream": "stdout",
    "time": "2018-09-13T18:29:14.897196588Z",
    "host": "10.40.0.0",
    "real_ip": "",
    "user": "-",
    "method": "GET",
    "path": "/",
    "code": "200",
    "size": "9056",
    "referer": "-",
    "agent": "curl/7.58.0",
    "request_length": "75",
    "request_time": "0.000",
    "proxy_upstream_name": "default-front-end-80",
    "upstream_addr": "10.40.0.9:8079",
    "upstream_response_length": "9056",
    "upstream_response_time": "0.000",
    "upstream_status": "200",
    "last": "a134ebded3504000d63646b647e54585",
    "kubernetes": {
      "pod_name": "ingress-nginx-ingress-controller-6577665f8c-wqg76",
      "namespace_name": "default",
      "pod_id": "0ea2b2c8-b5e8-11e8-bc8c-d237edbf1eb2",
      "labels": {
        "app": "nginx-ingress",
        "component": "controller",
        "pod-template-hash": "2133221947",
        "release": "ingress"
      },
      "annotations": {
        "fluentbit.io/parser": "k8s-nginx-ingress"
      },
      "host": "kube-spawn-flannel-worker-913bw7",
      "container_name": "nginx-ingress-controller",
      "docker_id": "40daa91b8c89a52e44ac1458c90967dab6d8a0e43c46605b0acbf8432f2d9f13"
    }
  },
  "fields": {
    "@timestamp": [
      "2018-09-13T18:29:14.897Z"
    ],
    "time": [
      "2018-09-13T18:29:14.897Z"
    ]
  },
  "highlight": {
    "kubernetes.labels.release.keyword": [
      "@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
    ],
    "kubernetes.labels.app": [
      "nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
    ],
    "kubernetes.annotations.fluentbit.io/parser": [
      "k8s-nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
    ],
    "kubernetes.container_name": [
      "nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@-controller"
    ],
    "kubernetes.pod_name": [
      "@kibana-highlighted-field@ingress@/kibana-highlighted-field@-nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@-controller-6577665f8c-wqg76"
    ],
    "kubernetes.labels.release": [
      "@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1536863354897
  ]
}
Tagged with: , , , , ,
One comment on “Increasing the usefulness of your Kubernetes Ingress logging
  1. db Naseem says:

    Great post, thank you.
    Hopefully this becomes an out-of-the-box parser since fluent-bit and nginx-ingress are often both used in k8s clusters.

Leave a Reply

Your email address will not be published. Required fields are marked *

*