Laughably Loquacious Logging
So you are pretty proud of yourself. You have a full micro-services running in Kubernetes with a service mesh (courtesy of Istio). You have configured your liveness probes to once per second. You are using an EFK stack (Elasticsearch / Fluent-Bit/ Kubernetes). Live is good. You are evaluating turning on either Jaeger or Zipkin. You have Prometheus, Grafana going and regularly go swimming in charts of the highest beauty and speed.
Only one problem. Every month your cloud bill goes up, mostly due to storage, and you have to lay off another person on the team to pay for Jeff Bezo's rocket fetish. What's up with this?
So you finally roll-up your sleeves and dig in. You take a look at one of those kube-probe liveness checks. Its got a unique requestID so tracing it is not hard. Lets just take a snoop in Elasticsearch and see how much 'raw' size (ignoring the index) it uses.
curl -XGET https://elastic/logstash-2018.11.13/_search -d '{ "query": {
"query_string": {
"query": "5159e6fa-07e7-90df-aa69-cef9dd6cb606" } } }' | wc -cOK that was easy. But... the answer might shock you. It turns out for me it was ~32KiB. Yes, more memory than your first computer had, for 1 message. For your viewing pleasure this is below, I wouldn't read all of it 🙂
And it starts to sink in. All those pods, each with sidecars and services and so on, and all the proxy servers... Each probe goes a lot of path, with a lot of logging. And you have a lot of pods. As you scale up, it gets bigger and bigger. And its mostly duplicated data.
Maybe you should look @ Druid.io ? It handles the duplicates differently, not costing you in cloud storage IOPS.
Maybe you should remove the liveness checks? Filter them out on the ingress of the logging?
Its a lot of data being written 'just in case' you read it, most is never viewed. Hmm.
{
"_shards": {
"failed": 0,
"skipped": 0,
"successful": 5,
"total": 5
},
"hits": {
"hits": [
{
"_id": "_Y6nDmcBGvwBJNsJ1ckB",
"_index": "logstash-2018.11.13",
"_score": 43.775322,
"_source": {
"@timestamp": "2018-11-13T19:58:58.785Z",
"flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
"kubernetes": {
"annotations": {
"scheduler_alpha_kubernetes_io/critical-pod": "",
"sidecar_istio_io/inject": "false"
},
"container_name": "mixer",
"host": "aks-nodepool1-19254313-3",
"labels": {
"app": "telemetry",
"istio": "mixer",
"istio-mixer-type": "telemetry",
"pod-template-hash": "2277284263"
},
"namespace_name": "istio-system",
"pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "istio-telemetry-66cc6d86b7-89vpl"
},
"log": "{\"level\":\"info\",\"time\":\"2018-11-13T19:58:57.777798Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"\",\"destinationIp\":\"10.244.5.42\",\"destinationName\":\"carts-5ff9d74d6d-mlwrn\",\"destinationNamespace\":\"sock-shop\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"carts.sock-shop.svc.cluster.local\",\"destinationWorkload\":\"carts\",\"httpAuthority\":\"10.244.5.42:80\",\"latency\":\"5.51559ms\",\"method\":\"GET\",\"protocol\":\"http\",\"receivedBytes\":333,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"5159e6fa-07e7-90df-aa69-cef9dd6cb606\",\"requestSize\":0,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":151,\"responseTimestamp\":\"2018-11-13T19:58:57.783121Z\",\"sentBytes\":435,\"sourceApp\":\"\",\"sourceIp\":\"0.0.0.0\",\"sourceName\":\"unknown\",\"sourceNamespace\":\"default\",\"sourceOwner\":\"unknown\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"unknown\",\"url\":\"/health\",\"userAgent\":\"kube-probe/1.11\",\"xForwardedFor\":\"10.244.5.1\"}\n",
"stream": "stdout",
"values": {
"apiClaims": "",
"apiKey": "",
"clientTraceId": "",
"connection_security_policy": "none",
"destinationApp": "",
"destinationIp": "10.244.5.42",
"destinationName": "carts-5ff9d74d6d-mlwrn",
"destinationNamespace": "sock-shop",
"destinationOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts",
"destinationPrincipal": "",
"destinationServiceHost": "carts.sock-shop.svc.cluster.local",
"destinationWorkload": "carts",
"httpAuthority": "10.244.5.42:80",
"instance": "accesslog.logentry.istio-system",
"latency": "5.51559ms",
"level": "info",
"method": "GET",
"protocol": "http",
"receivedBytes": 333,
"referer": "",
"reporter": "destination",
"requestId": "5159e6fa-07e7-90df-aa69-cef9dd6cb606",
"requestSize": 0,
"requestedServerName": "",
"responseCode": 200,
"responseSize": 151,
"responseTimestamp": "2018-11-13T19:58:57.783121Z",
"sentBytes": 435,
"sourceApp": "",
"sourceIp": "0.0.0.0",
"sourceName": "unknown",
"sourceNamespace": "default",
"sourceOwner": "unknown",
"sourcePrincipal": "",
"sourceWorkload": "unknown",
"time": "2018-11-13T19:58:57.777798Z",
"url": "/health",
"userAgent": "kube-probe/1.11",
"xForwardedFor": "10.244.5.1"
}
},
"_type": "_doc"
},
{
"_id": "Y46nDmcBGvwBJNsJ8sv5",
"_index": "logstash-2018.11.13",
"_score": 38.377754,
"_source": {
"@timestamp": "2018-11-13T19:58:57.777Z",
"flb-key": "kube.sock-shop.carts-5ff9d74d6d-mlwrn.istio-proxy",
"kubernetes": {
"annotations": {
"sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
},
"container_name": "istio-proxy",
"host": "aks-nodepool1-19254313-1",
"labels": {
"name": "carts",
"pod-template-hash": "1995830828"
},
"namespace_name": "sock-shop",
"pod_id": "1689ea16-e77e-11e8-a3a6-0a58ac1f0442",
"pod_name": "carts-5ff9d74d6d-mlwrn"
},
"values": {
"agent": "kube-probe/1.11",
"authority": "10.244.5.42:80",
"bytes_received": "0",
"bytes_sent": "151",
"code": "200",
"duration": "5",
"flags": "-",
"method": "GET",
"path": "/health",
"protocol": "HTTP/1.1",
"real_ip": "10.244.5.1",
"remainder_ip": "",
"request_id": "5159e6fa-07e7-90df-aa69-cef9dd6cb606",
"stream": "stdout",
"upstream": "127.0.0.1:80",
"upstream_service_time": "4"
}
},
"_type": "_doc"
},
{
"_id": "2o6bDmcBGvwBJNsJ5S0N",
"_index": "logstash-2018.11.13",
"_score": 8.98986,
"_source": {
"@timestamp": "2018-11-13T19:45:56.813Z",
"flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
"kubernetes": {
"annotations": {
"scheduler_alpha_kubernetes_io/critical-pod": "",
"sidecar_istio_io/inject": "false"
},
"container_name": "mixer",
"host": "aks-nodepool1-19254313-3",
"labels": {
"app": "telemetry",
"istio": "mixer",
"istio-mixer-type": "telemetry",
"pod-template-hash": "2277284263"
},
"namespace_name": "istio-system",
"pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "istio-telemetry-66cc6d86b7-89vpl"
},
"log": "{\"level\":\"info\",\"time\":\"2018-11-13T19:45:56.234795Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"\",\"destinationIp\":\"10.244.5.38\",\"destinationName\":\"front-end-7fb8c76cc7-brcgf\",\"destinationNamespace\":\"sock-shop\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"front-end.sock-shop.svc.cluster.local\",\"destinationWorkload\":\"front-end\",\"httpAuthority\":\"10.244.5.38:8079\",\"latency\":\"2.859902ms\",\"method\":\"GET\",\"protocol\":\"http\",\"receivedBytes\":308,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"81ad8b69-c186-9e6b-aa69-ea6939186dfa\",\"requestSize\":0,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":9056,\"responseTimestamp\":\"2018-11-13T19:45:56.237563Z\",\"sentBytes\":10309,\"sourceApp\":\"\",\"sourceIp\":\"0.0.0.0\",\"sourceName\":\"unknown\",\"sourceNamespace\":\"default\",\"sourceOwner\":\"unknown\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"unknown\",\"url\":\"/\",\"userAgent\":\"kube-probe/1.11\",\"xForwardedFor\":\"10.244.5.1\"}\n",
"stream": "stdout",
"values": {
"apiClaims": "",
"apiKey": "",
"clientTraceId": "",
"connection_security_policy": "none",
"destinationApp": "",
"destinationIp": "10.244.5.38",
"destinationName": "front-end-7fb8c76cc7-brcgf",
"destinationNamespace": "sock-shop",
"destinationOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end",
"destinationPrincipal": "",
"destinationServiceHost": "front-end.sock-shop.svc.cluster.local",
"destinationWorkload": "front-end",
"httpAuthority": "10.244.5.38:8079",
"instance": "accesslog.logentry.istio-system",
"latency": "2.859902ms",
"level": "info",
"method": "GET",
"protocol": "http",
"receivedBytes": 308,
"referer": "",
"reporter": "destination",
"requestId": "81ad8b69-c186-9e6b-aa69-ea6939186dfa",
"requestSize": 0,
"requestedServerName": "",
"responseCode": 200,
"responseSize": 9056,
"responseTimestamp": "2018-11-13T19:45:56.237563Z",
"sentBytes": 10309,
"sourceApp": "",
"sourceIp": "0.0.0.0",
"sourceName": "unknown",
"sourceNamespace": "default",
"sourceOwner": "unknown",
"sourcePrincipal": "",
"sourceWorkload": "unknown",
"time": "2018-11-13T19:45:56.234795Z",
"url": "/",
"userAgent": "kube-probe/1.11",
"xForwardedFor": "10.244.5.1"
}
},
"_type": "_doc"
},
{
"_id": "BZDMDmcBGvwBJNsJjosW",
"_index": "logstash-2018.11.13",
"_score": 8.986363,
"_source": {
"@timestamp": "2018-11-13T20:39:05.641Z",
"flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
"kubernetes": {
"annotations": {
"scheduler_alpha_kubernetes_io/critical-pod": "",
"sidecar_istio_io/inject": "false"
},
"container_name": "mixer",
"host": "aks-nodepool1-19254313-3",
"labels": {
"app": "telemetry",
"istio": "mixer",
"istio-mixer-type": "telemetry",
"pod-template-hash": "2277284263"
},
"namespace_name": "istio-system",
"pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "istio-telemetry-66cc6d86b7-89vpl"
},
"log": "{\"level\":\"info\",\"time\":\"2018-11-13T20:39:05.256852Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"telemetry\",\"destinationIp\":\"10.244.3.176\",\"destinationName\":\"istio-telemetry-66cc6d86b7-89vpl\",\"destinationNamespace\":\"istio-system\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"istio-telemetry.istio-system.svc.cluster.local\",\"destinationWorkload\":\"istio-telemetry\",\"httpAuthority\":\"mixer\",\"latency\":\"1.275642ms\",\"method\":\"POST\",\"protocol\":\"http\",\"receivedBytes\":856,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"892bcaa5-b3fd-90df-8060-361d993c2054\",\"requestSize\":462,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":5,\"responseTimestamp\":\"2018-11-13T20:39:05.257946Z\",\"sentBytes\":174,\"sourceApp\":\"\",\"sourceIp\":\"10.244.4.19\",\"sourceName\":\"carts-db-f894ff6d8-5hq9l\",\"sourceNamespace\":\"sock-shop\",\"sourceOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts-db\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"carts-db\",\"url\":\"/istio.mixer.v1.Mixer/Report\",\"userAgent\":\"\",\"xForwardedFor\":\"10.244.4.19\"}\n",
"stream": "stdout",
"values": {
"apiClaims": "",
"apiKey": "",
"clientTraceId": "",
"connection_security_policy": "none",
"destinationApp": "telemetry",
"destinationIp": "10.244.3.176",
"destinationName": "istio-telemetry-66cc6d86b7-89vpl",
"destinationNamespace": "istio-system",
"destinationOwner": "kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry",
"destinationPrincipal": "",
"destinationServiceHost": "istio-telemetry.istio-system.svc.cluster.local",
"destinationWorkload": "istio-telemetry",
"httpAuthority": "mixer",
"instance": "accesslog.logentry.istio-system",
"latency": "1.275642ms",
"level": "info",
"method": "POST",
"protocol": "http",
"receivedBytes": 856,
"referer": "",
"reporter": "destination",
"requestId": "892bcaa5-b3fd-90df-8060-361d993c2054",
"requestSize": 462,
"requestedServerName": "",
"responseCode": 200,
"responseSize": 5,
"responseTimestamp": "2018-11-13T20:39:05.257946Z",
"sentBytes": 174,
"sourceApp": "",
"sourceIp": "10.244.4.19",
"sourceName": "carts-db-f894ff6d8-5hq9l",
"sourceNamespace": "sock-shop",
"sourceOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts-db",
"sourcePrincipal": "",
"sourceWorkload": "carts-db",
"time": "2018-11-13T20:39:05.256852Z",
"url": "/istio.mixer.v1.Mixer/Report",
"userAgent": "",
"xForwardedFor": "10.244.4.19"
}
},
"_type": "_doc"
},
{
"_id": "uI6dDmcBGvwBJNsJk0Em",
"_index": "logstash-2018.11.13",
"_score": 8.148528,
"_source": {
"@timestamp": "2018-11-13T19:47:46.515Z",
"flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
"kubernetes": {
"annotations": {
"scheduler_alpha_kubernetes_io/critical-pod": "",
"sidecar_istio_io/inject": "false"
},
"container_name": "mixer",
"host": "aks-nodepool1-19254313-3",
"labels": {
"app": "telemetry",
"istio": "mixer",
"istio-mixer-type": "telemetry",
"pod-template-hash": "2277284263"
},
"namespace_name": "istio-system",
"pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "istio-telemetry-66cc6d86b7-89vpl"
},
"log": "{\"level\":\"info\",\"time\":\"2018-11-13T19:47:45.613613Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"telemetry\",\"destinationIp\":\"10.244.3.176\",\"destinationName\":\"istio-telemetry-66cc6d86b7-89vpl\",\"destinationNamespace\":\"istio-system\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"istio-telemetry.istio-system.svc.cluster.local\",\"destinationWorkload\":\"istio-telemetry\",\"httpAuthority\":\"mixer\",\"latency\":\"1.147538ms\",\"method\":\"POST\",\"protocol\":\"http\",\"receivedBytes\":864,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"6ecdf6bd-3895-974c-aa69-9c00c1a93a06\",\"requestSize\":470,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":5,\"responseTimestamp\":\"2018-11-13T19:47:45.614632Z\",\"sentBytes\":174,\"sourceApp\":\"\",\"sourceIp\":\"10.244.5.36\",\"sourceName\":\"orders-db-db59dffd-bgd57\",\"sourceNamespace\":\"sock-shop\",\"sourceOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders-db\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"orders-db\",\"url\":\"/istio.mixer.v1.Mixer/Report\",\"userAgent\":\"\",\"xForwardedFor\":\"10.244.5.36\"}\n",
"stream": "stdout",
"values": {
"apiClaims": "",
"apiKey": "",
"clientTraceId": "",
"connection_security_policy": "none",
"destinationApp": "telemetry",
"destinationIp": "10.244.3.176",
"destinationName": "istio-telemetry-66cc6d86b7-89vpl",
"destinationNamespace": "istio-system",
"destinationOwner": "kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry",
"destinationPrincipal": "",
"destinationServiceHost": "istio-telemetry.istio-system.svc.cluster.local",
"destinationWorkload": "istio-telemetry",
"httpAuthority": "mixer",
"instance": "accesslog.logentry.istio-system",
"latency": "1.147538ms",
"level": "info",
"method": "POST",
"protocol": "http",
"receivedBytes": 864,
"referer": "",
"reporter": "destination",
"requestId": "6ecdf6bd-3895-974c-aa69-9c00c1a93a06",
"requestSize": 470,
"requestedServerName": "",
"responseCode": 200,
"responseSize": 5,
"responseTimestamp": "2018-11-13T19:47:45.614632Z",
"sentBytes": 174,
"sourceApp": "",
"sourceIp": "10.244.5.36",
"sourceName": "orders-db-db59dffd-bgd57",
"sourceNamespace": "sock-shop",
"sourceOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders-db",
"sourcePrincipal": "",
"sourceWorkload": "orders-db",
"time": "2018-11-13T19:47:45.613613Z",
"url": "/istio.mixer.v1.Mixer/Report",
"userAgent": "",
"xForwardedFor": "10.244.5.36"
}
},
"_type": "_doc"
},
{
"_id": "tY-3DmcBGvwBJNsJ5Y5x",
"_index": "logstash-2018.11.13",
"_score": 8.148528,
"_source": {
"@timestamp": "2018-11-13T20:16:31.812Z",
"flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
"kubernetes": {
"annotations": {
"scheduler_alpha_kubernetes_io/critical-pod": "",
"sidecar_istio_io/inject": "false"
},
"container_name": "mixer",
"host": "aks-nodepool1-19254313-3",
"labels": {
"app": "telemetry",
"istio": "mixer",
"istio-mixer-type": "telemetry",
"pod-template-hash": "2277284263"
},
"namespace_name": "istio-system",
"pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "istio-telemetry-66cc6d86b7-89vpl"
},
"log": "{\"level\":\"info\",\"time\":\"2018-11-13T20:16:31.234709Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"\",\"destinationIp\":\"10.244.5.38\",\"destinationName\":\"front-end-7fb8c76cc7-brcgf\",\"destinationNamespace\":\"sock-shop\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"front-end.sock-shop.svc.cluster.local\",\"destinationWorkload\":\"front-end\",\"httpAuthority\":\"10.244.5.38:8079\",\"latency\":\"2.8308ms\",\"method\":\"GET\",\"protocol\":\"http\",\"receivedBytes\":308,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"58dade92-1f67-9ecc-aa69-ce96c5fa495b\",\"requestSize\":0,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":9056,\"responseTimestamp\":\"2018-11-13T20:16:31.237461Z\",\"sentBytes\":10309,\"sourceApp\":\"\",\"sourceIp\":\"0.0.0.0\",\"sourceName\":\"unknown\",\"sourceNamespace\":\"default\",\"sourceOwner\":\"unknown\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"unknown\",\"url\":\"/\",\"userAgent\":\"kube-probe/1.11\",\"xForwardedFor\":\"10.244.5.1\"}\n",
"stream": "stdout",
"values": {
"apiClaims": "",
"apiKey": "",
"clientTraceId": "",
"connection_security_policy": "none",
"destinationApp": "",
"destinationIp": "10.244.5.38",
"destinationName": "front-end-7fb8c76cc7-brcgf",
"destinationNamespace": "sock-shop",
"destinationOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end",
"destinationPrincipal": "",
"destinationServiceHost": "front-end.sock-shop.svc.cluster.local",
"destinationWorkload": "front-end",
"httpAuthority": "10.244.5.38:8079",
"instance": "accesslog.logentry.istio-system",
"latency": "2.8308ms",
"level": "info",
"method": "GET",
"protocol": "http",
"receivedBytes": 308,
"referer": "",
"reporter": "destination",
"requestId": "58dade92-1f67-9ecc-aa69-ce96c5fa495b",
"requestSize": 0,
"requestedServerName": "",
"responseCode": 200,
"responseSize": 9056,
"responseTimestamp": "2018-11-13T20:16:31.237461Z",
"sentBytes": 10309,
"sourceApp": "",
"sourceIp": "0.0.0.0",
"sourceName": "unknown",
"sourceNamespace": "default",
"sourceOwner": "unknown",
"sourcePrincipal": "",
"sourceWorkload": "unknown",
"time": "2018-11-13T20:16:31.234709Z",
"url": "/",
"userAgent": "kube-probe/1.11",
"xForwardedFor": "10.244.5.1"
}
},
"_type": "_doc"
},
{
"_id": "tJDUDmcBGvwBJNsJBOU_",
"_index": "logstash-2018.11.13",
"_score": 8.148528,
"_source": {
"@timestamp": "2018-11-13T20:47:14.900Z",
"flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
"kubernetes": {
"annotations": {
"scheduler_alpha_kubernetes_io/critical-pod": "",
"sidecar_istio_io/inject": "false"
},
"container_name": "mixer",
"host": "aks-nodepool1-19254313-3",
"labels": {
"app": "telemetry",
"istio": "mixer",
"istio-mixer-type": "telemetry",
"pod-template-hash": "2277284263"
},
"namespace_name": "istio-system",
"pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "istio-telemetry-66cc6d86b7-89vpl"
},
"log": "{\"level\":\"info\",\"time\":\"2018-11-13T20:47:14.700695Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"telemetry\",\"destinationIp\":\"10.244.3.176\",\"destinationName\":\"istio-telemetry-66cc6d86b7-89vpl\",\"destinationNamespace\":\"istio-system\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"istio-telemetry.istio-system.svc.cluster.local\",\"destinationWorkload\":\"istio-telemetry\",\"httpAuthority\":\"mixer\",\"latency\":\"1.123237ms\",\"method\":\"POST\",\"protocol\":\"http\",\"receivedBytes\":864,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"8af48874-f711-94e4-aa69-141586faaca5\",\"requestSize\":470,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":5,\"responseTimestamp\":\"2018-11-13T20:47:14.701706Z\",\"sentBytes\":174,\"sourceApp\":\"\",\"sourceIp\":\"10.244.4.16\",\"sourceName\":\"orders-6587948d7d-dx62n\",\"sourceNamespace\":\"sock-shop\",\"sourceOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"orders\",\"url\":\"/istio.mixer.v1.Mixer/Report\",\"userAgent\":\"\",\"xForwardedFor\":\"10.244.4.16\"}\n",
"stream": "stdout",
"values": {
"apiClaims": "",
"apiKey": "",
"clientTraceId": "",
"connection_security_policy": "none",
"destinationApp": "telemetry",
"destinationIp": "10.244.3.176",
"destinationName": "istio-telemetry-66cc6d86b7-89vpl",
"destinationNamespace": "istio-system",
"destinationOwner": "kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry",
"destinationPrincipal": "",
"destinationServiceHost": "istio-telemetry.istio-system.svc.cluster.local",
"destinationWorkload": "istio-telemetry",
"httpAuthority": "mixer",
"instance": "accesslog.logentry.istio-system",
"latency": "1.123237ms",
"level": "info",
"method": "POST",
"protocol": "http",
"receivedBytes": 864,
"referer": "",
"reporter": "destination",
"requestId": "8af48874-f711-94e4-aa69-141586faaca5",
"requestSize": 470,
"requestedServerName": "",
"responseCode": 200,
"responseSize": 5,
"responseTimestamp": "2018-11-13T20:47:14.701706Z",
"sentBytes": 174,
"sourceApp": "",
"sourceIp": "10.244.4.16",
"sourceName": "orders-6587948d7d-dx62n",
"sourceNamespace": "sock-shop",
"sourceOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders",
"sourcePrincipal": "",
"sourceWorkload": "orders",
"time": "2018-11-13T20:47:14.700695Z",
"url": "/istio.mixer.v1.Mixer/Report",
"userAgent": "",
"xForwardedFor": "10.244.4.16"
}
},
"_type": "_doc"
},
{
"_id": "hZDHDmcBGvwBJNsJXUz2",
"_index": "logstash-2018.11.13",
"_score": 8.0215845,
"_source": {
"@timestamp": "2018-11-13T20:33:25.117Z",
"flb-key": "kube.sock-shop.user-5bbd6dd84-qg6l7.istio-proxy",
"kubernetes": {
"annotations": {
"sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
},
"container_name": "istio-proxy",
"host": "aks-nodepool1-19254313-0",
"labels": {
"name": "user",
"pod-template-hash": "166828840"
},
"namespace_name": "sock-shop",
"pod_id": "6e247a0f-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "user-5bbd6dd84-qg6l7"
},
"values": {
"agent": "kube-probe/1.11",
"authority": "10.244.4.18:80",
"bytes_received": "0",
"bytes_sent": "180",
"code": "200",
"duration": "3",
"flags": "-",
"method": "GET",
"path": "/health",
"protocol": "HTTP/1.1",
"real_ip": "10.244.4.1",
"remainder_ip": "",
"request_id": "06fb5497-da09-90df-a4f3-6742759c7f20",
"stream": "stdout",
"upstream": "127.0.0.1:80",
"upstream_service_time": "1"
}
},
"_type": "_doc"
},
{
"_id": "nI6dDmcBGvwBJNsJZz_F",
"_index": "logstash-2018.11.13",
"_score": 8.014115,
"_source": {
"@timestamp": "2018-11-13T19:47:28.117Z",
"flb-key": "kube.sock-shop.user-5bbd6dd84-qg6l7.istio-proxy",
"kubernetes": {
"annotations": {
"sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
},
"container_name": "istio-proxy",
"host": "aks-nodepool1-19254313-0",
"labels": {
"name": "user",
"pod-template-hash": "166828840"
},
"namespace_name": "sock-shop",
"pod_id": "6e247a0f-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "user-5bbd6dd84-qg6l7"
},
"values": {
"agent": "kube-probe/1.11",
"authority": "10.244.4.18:80",
"bytes_received": "0",
"bytes_sent": "180",
"code": "200",
"duration": "2",
"flags": "-",
"method": "GET",
"path": "/health",
"protocol": "HTTP/1.1",
"real_ip": "10.244.4.1",
"remainder_ip": "",
"request_id": "64c754f6-d57d-90df-b818-d91932d7c66c",
"stream": "stdout",
"upstream": "127.0.0.1:80",
"upstream_service_time": "2"
}
},
"_type": "_doc"
},
{
"_id": "XI6bDmcBGvwBJNsJ9C6s",
"_index": "logstash-2018.11.13",
"_score": 8.013674,
"_source": {
"@timestamp": "2018-11-13T19:45:56.234Z",
"flb-key": "kube.sock-shop.front-end-7fb8c76cc7-brcgf.istio-proxy",
"kubernetes": {
"annotations": {
"fluentbit_io/parser-front-end": "apache2",
"sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
},
"container_name": "istio-proxy",
"host": "aks-nodepool1-19254313-1",
"labels": {
"name": "front-end",
"pod-template-hash": "3964732773"
},
"namespace_name": "sock-shop",
"pod_id": "6e798647-e77a-11e8-a3a6-0a58ac1f0442",
"pod_name": "front-end-7fb8c76cc7-brcgf"
},
"values": {
"agent": "kube-probe/1.11",
"authority": "10.244.5.38:8079",
"bytes_received": "0",
"bytes_sent": "9056",
"code": "200",
"duration": "2",
"flags": "-",
"method": "GET",
"path": "/",
"protocol": "HTTP/1.1",
"real_ip": "10.244.5.1",
"remainder_ip": "",
"request_id": "81ad8b69-c186-9e6b-aa69-ea6939186dfa",
"stream": "stdout",
"upstream": "127.0.0.1:8079",
"upstream_service_time": "2"
}
},
"_type": "_doc"
}
],
"max_score": 43.775322,
"total": 35
},
"timed_out": false,
"took": 376
}