Laughably Loquacious Logging

So you are pretty proud of yourself. You have a full micro-services running in Kubernetes with a service mesh (courtesy of Istio). You have configured your liveness probes to once per second. You are using an EFK stack (Elasticsearch / Fluent-Bit/  Kubernetes). Live is good. You are evaluating turning on either Jaeger or Zipkin. You have Prometheus, Grafana going and regularly go swimming in charts of the highest beauty and speed.

Only one problem. Every month your cloud bill goes up, mostly due to storage, and you have to lay off another person on the team to pay for Jeff Bezo's rocket fetish. What's up with this?

So you finally roll-up your sleeves and dig in. You take a look at one of those kube-probe liveness checks. Its got a unique requestID so tracing it is not hard. Lets just take a snoop in Elasticsearch and see how much 'raw' size (ignoring the index) it uses.

curl -XGET https://elastic/logstash-2018.11.13/_search -d '{ "query": { 
  "query_string": { 
    "query": "5159e6fa-07e7-90df-aa69-cef9dd6cb606" } } }' | wc -c

OK that was easy. But... the answer might shock you. It turns out for me it was ~32KiB. Yes, more memory than your first computer had, for 1 message. For your viewing pleasure this is below, I wouldn't read all of it 🙂

And it starts to sink in. All those pods, each with sidecars and services and so on, and all the proxy servers... Each probe goes a lot of path, with a lot of logging. And you have a lot of pods. As you scale up, it gets bigger and bigger. And its mostly duplicated data.

Maybe you should look @ Druid.io ? It handles the duplicates differently, not costing you in cloud storage IOPS.

Maybe you should remove the liveness checks? Filter them out on the ingress of the logging?

Its a lot of data being written 'just in case' you read it, most is never viewed. Hmm.

{
    "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 5,
        "total": 5
    },
    "hits": {
        "hits": [
            {
                "_id": "_Y6nDmcBGvwBJNsJ1ckB",
                "_index": "logstash-2018.11.13",
                "_score": 43.775322,
                "_source": {
                    "@timestamp": "2018-11-13T19:58:58.785Z",
                    "flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
                    "kubernetes": {
                        "annotations": {
                            "scheduler_alpha_kubernetes_io/critical-pod": "",
                            "sidecar_istio_io/inject": "false"
                        },
                        "container_name": "mixer",
                        "host": "aks-nodepool1-19254313-3",
                        "labels": {
                            "app": "telemetry",
                            "istio": "mixer",
                            "istio-mixer-type": "telemetry",
                            "pod-template-hash": "2277284263"
                        },
                        "namespace_name": "istio-system",
                        "pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "istio-telemetry-66cc6d86b7-89vpl"
                    },
                    "log": "{\"level\":\"info\",\"time\":\"2018-11-13T19:58:57.777798Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"\",\"destinationIp\":\"10.244.5.42\",\"destinationName\":\"carts-5ff9d74d6d-mlwrn\",\"destinationNamespace\":\"sock-shop\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"carts.sock-shop.svc.cluster.local\",\"destinationWorkload\":\"carts\",\"httpAuthority\":\"10.244.5.42:80\",\"latency\":\"5.51559ms\",\"method\":\"GET\",\"protocol\":\"http\",\"receivedBytes\":333,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"5159e6fa-07e7-90df-aa69-cef9dd6cb606\",\"requestSize\":0,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":151,\"responseTimestamp\":\"2018-11-13T19:58:57.783121Z\",\"sentBytes\":435,\"sourceApp\":\"\",\"sourceIp\":\"0.0.0.0\",\"sourceName\":\"unknown\",\"sourceNamespace\":\"default\",\"sourceOwner\":\"unknown\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"unknown\",\"url\":\"/health\",\"userAgent\":\"kube-probe/1.11\",\"xForwardedFor\":\"10.244.5.1\"}\n",
                    "stream": "stdout",
                    "values": {
                        "apiClaims": "",
                        "apiKey": "",
                        "clientTraceId": "",
                        "connection_security_policy": "none",
                        "destinationApp": "",
                        "destinationIp": "10.244.5.42",
                        "destinationName": "carts-5ff9d74d6d-mlwrn",
                        "destinationNamespace": "sock-shop",
                        "destinationOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts",
                        "destinationPrincipal": "",
                        "destinationServiceHost": "carts.sock-shop.svc.cluster.local",
                        "destinationWorkload": "carts",
                        "httpAuthority": "10.244.5.42:80",
                        "instance": "accesslog.logentry.istio-system",
                        "latency": "5.51559ms",
                        "level": "info",
                        "method": "GET",
                        "protocol": "http",
                        "receivedBytes": 333,
                        "referer": "",
                        "reporter": "destination",
                        "requestId": "5159e6fa-07e7-90df-aa69-cef9dd6cb606",
                        "requestSize": 0,
                        "requestedServerName": "",
                        "responseCode": 200,
                        "responseSize": 151,
                        "responseTimestamp": "2018-11-13T19:58:57.783121Z",
                        "sentBytes": 435,
                        "sourceApp": "",
                        "sourceIp": "0.0.0.0",
                        "sourceName": "unknown",
                        "sourceNamespace": "default",
                        "sourceOwner": "unknown",
                        "sourcePrincipal": "",
                        "sourceWorkload": "unknown",
                        "time": "2018-11-13T19:58:57.777798Z",
                        "url": "/health",
                        "userAgent": "kube-probe/1.11",
                        "xForwardedFor": "10.244.5.1"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "Y46nDmcBGvwBJNsJ8sv5",
                "_index": "logstash-2018.11.13",
                "_score": 38.377754,
                "_source": {
                    "@timestamp": "2018-11-13T19:58:57.777Z",
                    "flb-key": "kube.sock-shop.carts-5ff9d74d6d-mlwrn.istio-proxy",
                    "kubernetes": {
                        "annotations": {
                            "sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
                        },
                        "container_name": "istio-proxy",
                        "host": "aks-nodepool1-19254313-1",
                        "labels": {
                            "name": "carts",
                            "pod-template-hash": "1995830828"
                        },
                        "namespace_name": "sock-shop",
                        "pod_id": "1689ea16-e77e-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "carts-5ff9d74d6d-mlwrn"
                    },
                    "values": {
                        "agent": "kube-probe/1.11",
                        "authority": "10.244.5.42:80",
                        "bytes_received": "0",
                        "bytes_sent": "151",
                        "code": "200",
                        "duration": "5",
                        "flags": "-",
                        "method": "GET",
                        "path": "/health",
                        "protocol": "HTTP/1.1",
                        "real_ip": "10.244.5.1",
                        "remainder_ip": "",
                        "request_id": "5159e6fa-07e7-90df-aa69-cef9dd6cb606",
                        "stream": "stdout",
                        "upstream": "127.0.0.1:80",
                        "upstream_service_time": "4"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "2o6bDmcBGvwBJNsJ5S0N",
                "_index": "logstash-2018.11.13",
                "_score": 8.98986,
                "_source": {
                    "@timestamp": "2018-11-13T19:45:56.813Z",
                    "flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
                    "kubernetes": {
                        "annotations": {
                            "scheduler_alpha_kubernetes_io/critical-pod": "",
                            "sidecar_istio_io/inject": "false"
                        },
                        "container_name": "mixer",
                        "host": "aks-nodepool1-19254313-3",
                        "labels": {
                            "app": "telemetry",
                            "istio": "mixer",
                            "istio-mixer-type": "telemetry",
                            "pod-template-hash": "2277284263"
                        },
                        "namespace_name": "istio-system",
                        "pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "istio-telemetry-66cc6d86b7-89vpl"
                    },
                    "log": "{\"level\":\"info\",\"time\":\"2018-11-13T19:45:56.234795Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"\",\"destinationIp\":\"10.244.5.38\",\"destinationName\":\"front-end-7fb8c76cc7-brcgf\",\"destinationNamespace\":\"sock-shop\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"front-end.sock-shop.svc.cluster.local\",\"destinationWorkload\":\"front-end\",\"httpAuthority\":\"10.244.5.38:8079\",\"latency\":\"2.859902ms\",\"method\":\"GET\",\"protocol\":\"http\",\"receivedBytes\":308,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"81ad8b69-c186-9e6b-aa69-ea6939186dfa\",\"requestSize\":0,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":9056,\"responseTimestamp\":\"2018-11-13T19:45:56.237563Z\",\"sentBytes\":10309,\"sourceApp\":\"\",\"sourceIp\":\"0.0.0.0\",\"sourceName\":\"unknown\",\"sourceNamespace\":\"default\",\"sourceOwner\":\"unknown\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"unknown\",\"url\":\"/\",\"userAgent\":\"kube-probe/1.11\",\"xForwardedFor\":\"10.244.5.1\"}\n",
                    "stream": "stdout",
                    "values": {
                        "apiClaims": "",
                        "apiKey": "",
                        "clientTraceId": "",
                        "connection_security_policy": "none",
                        "destinationApp": "",
                        "destinationIp": "10.244.5.38",
                        "destinationName": "front-end-7fb8c76cc7-brcgf",
                        "destinationNamespace": "sock-shop",
                        "destinationOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end",
                        "destinationPrincipal": "",
                        "destinationServiceHost": "front-end.sock-shop.svc.cluster.local",
                        "destinationWorkload": "front-end",
                        "httpAuthority": "10.244.5.38:8079",
                        "instance": "accesslog.logentry.istio-system",
                        "latency": "2.859902ms",
                        "level": "info",
                        "method": "GET",
                        "protocol": "http",
                        "receivedBytes": 308,
                        "referer": "",
                        "reporter": "destination",
                        "requestId": "81ad8b69-c186-9e6b-aa69-ea6939186dfa",
                        "requestSize": 0,
                        "requestedServerName": "",
                        "responseCode": 200,
                        "responseSize": 9056,
                        "responseTimestamp": "2018-11-13T19:45:56.237563Z",
                        "sentBytes": 10309,
                        "sourceApp": "",
                        "sourceIp": "0.0.0.0",
                        "sourceName": "unknown",
                        "sourceNamespace": "default",
                        "sourceOwner": "unknown",
                        "sourcePrincipal": "",
                        "sourceWorkload": "unknown",
                        "time": "2018-11-13T19:45:56.234795Z",
                        "url": "/",
                        "userAgent": "kube-probe/1.11",
                        "xForwardedFor": "10.244.5.1"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "BZDMDmcBGvwBJNsJjosW",
                "_index": "logstash-2018.11.13",
                "_score": 8.986363,
                "_source": {
                    "@timestamp": "2018-11-13T20:39:05.641Z",
                    "flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
                    "kubernetes": {
                        "annotations": {
                            "scheduler_alpha_kubernetes_io/critical-pod": "",
                            "sidecar_istio_io/inject": "false"
                        },
                        "container_name": "mixer",
                        "host": "aks-nodepool1-19254313-3",
                        "labels": {
                            "app": "telemetry",
                            "istio": "mixer",
                            "istio-mixer-type": "telemetry",
                            "pod-template-hash": "2277284263"
                        },
                        "namespace_name": "istio-system",
                        "pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "istio-telemetry-66cc6d86b7-89vpl"
                    },
                    "log": "{\"level\":\"info\",\"time\":\"2018-11-13T20:39:05.256852Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"telemetry\",\"destinationIp\":\"10.244.3.176\",\"destinationName\":\"istio-telemetry-66cc6d86b7-89vpl\",\"destinationNamespace\":\"istio-system\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"istio-telemetry.istio-system.svc.cluster.local\",\"destinationWorkload\":\"istio-telemetry\",\"httpAuthority\":\"mixer\",\"latency\":\"1.275642ms\",\"method\":\"POST\",\"protocol\":\"http\",\"receivedBytes\":856,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"892bcaa5-b3fd-90df-8060-361d993c2054\",\"requestSize\":462,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":5,\"responseTimestamp\":\"2018-11-13T20:39:05.257946Z\",\"sentBytes\":174,\"sourceApp\":\"\",\"sourceIp\":\"10.244.4.19\",\"sourceName\":\"carts-db-f894ff6d8-5hq9l\",\"sourceNamespace\":\"sock-shop\",\"sourceOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts-db\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"carts-db\",\"url\":\"/istio.mixer.v1.Mixer/Report\",\"userAgent\":\"\",\"xForwardedFor\":\"10.244.4.19\"}\n",
                    "stream": "stdout",
                    "values": {
                        "apiClaims": "",
                        "apiKey": "",
                        "clientTraceId": "",
                        "connection_security_policy": "none",
                        "destinationApp": "telemetry",
                        "destinationIp": "10.244.3.176",
                        "destinationName": "istio-telemetry-66cc6d86b7-89vpl",
                        "destinationNamespace": "istio-system",
                        "destinationOwner": "kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry",
                        "destinationPrincipal": "",
                        "destinationServiceHost": "istio-telemetry.istio-system.svc.cluster.local",
                        "destinationWorkload": "istio-telemetry",
                        "httpAuthority": "mixer",
                        "instance": "accesslog.logentry.istio-system",
                        "latency": "1.275642ms",
                        "level": "info",
                        "method": "POST",
                        "protocol": "http",
                        "receivedBytes": 856,
                        "referer": "",
                        "reporter": "destination",
                        "requestId": "892bcaa5-b3fd-90df-8060-361d993c2054",
                        "requestSize": 462,
                        "requestedServerName": "",
                        "responseCode": 200,
                        "responseSize": 5,
                        "responseTimestamp": "2018-11-13T20:39:05.257946Z",
                        "sentBytes": 174,
                        "sourceApp": "",
                        "sourceIp": "10.244.4.19",
                        "sourceName": "carts-db-f894ff6d8-5hq9l",
                        "sourceNamespace": "sock-shop",
                        "sourceOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/carts-db",
                        "sourcePrincipal": "",
                        "sourceWorkload": "carts-db",
                        "time": "2018-11-13T20:39:05.256852Z",
                        "url": "/istio.mixer.v1.Mixer/Report",
                        "userAgent": "",
                        "xForwardedFor": "10.244.4.19"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "uI6dDmcBGvwBJNsJk0Em",
                "_index": "logstash-2018.11.13",
                "_score": 8.148528,
                "_source": {
                    "@timestamp": "2018-11-13T19:47:46.515Z",
                    "flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
                    "kubernetes": {
                        "annotations": {
                            "scheduler_alpha_kubernetes_io/critical-pod": "",
                            "sidecar_istio_io/inject": "false"
                        },
                        "container_name": "mixer",
                        "host": "aks-nodepool1-19254313-3",
                        "labels": {
                            "app": "telemetry",
                            "istio": "mixer",
                            "istio-mixer-type": "telemetry",
                            "pod-template-hash": "2277284263"
                        },
                        "namespace_name": "istio-system",
                        "pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "istio-telemetry-66cc6d86b7-89vpl"
                    },
                    "log": "{\"level\":\"info\",\"time\":\"2018-11-13T19:47:45.613613Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"telemetry\",\"destinationIp\":\"10.244.3.176\",\"destinationName\":\"istio-telemetry-66cc6d86b7-89vpl\",\"destinationNamespace\":\"istio-system\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"istio-telemetry.istio-system.svc.cluster.local\",\"destinationWorkload\":\"istio-telemetry\",\"httpAuthority\":\"mixer\",\"latency\":\"1.147538ms\",\"method\":\"POST\",\"protocol\":\"http\",\"receivedBytes\":864,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"6ecdf6bd-3895-974c-aa69-9c00c1a93a06\",\"requestSize\":470,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":5,\"responseTimestamp\":\"2018-11-13T19:47:45.614632Z\",\"sentBytes\":174,\"sourceApp\":\"\",\"sourceIp\":\"10.244.5.36\",\"sourceName\":\"orders-db-db59dffd-bgd57\",\"sourceNamespace\":\"sock-shop\",\"sourceOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders-db\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"orders-db\",\"url\":\"/istio.mixer.v1.Mixer/Report\",\"userAgent\":\"\",\"xForwardedFor\":\"10.244.5.36\"}\n",
                    "stream": "stdout",
                    "values": {
                        "apiClaims": "",
                        "apiKey": "",
                        "clientTraceId": "",
                        "connection_security_policy": "none",
                        "destinationApp": "telemetry",
                        "destinationIp": "10.244.3.176",
                        "destinationName": "istio-telemetry-66cc6d86b7-89vpl",
                        "destinationNamespace": "istio-system",
                        "destinationOwner": "kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry",
                        "destinationPrincipal": "",
                        "destinationServiceHost": "istio-telemetry.istio-system.svc.cluster.local",
                        "destinationWorkload": "istio-telemetry",
                        "httpAuthority": "mixer",
                        "instance": "accesslog.logentry.istio-system",
                        "latency": "1.147538ms",
                        "level": "info",
                        "method": "POST",
                        "protocol": "http",
                        "receivedBytes": 864,
                        "referer": "",
                        "reporter": "destination",
                        "requestId": "6ecdf6bd-3895-974c-aa69-9c00c1a93a06",
                        "requestSize": 470,
                        "requestedServerName": "",
                        "responseCode": 200,
                        "responseSize": 5,
                        "responseTimestamp": "2018-11-13T19:47:45.614632Z",
                        "sentBytes": 174,
                        "sourceApp": "",
                        "sourceIp": "10.244.5.36",
                        "sourceName": "orders-db-db59dffd-bgd57",
                        "sourceNamespace": "sock-shop",
                        "sourceOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders-db",
                        "sourcePrincipal": "",
                        "sourceWorkload": "orders-db",
                        "time": "2018-11-13T19:47:45.613613Z",
                        "url": "/istio.mixer.v1.Mixer/Report",
                        "userAgent": "",
                        "xForwardedFor": "10.244.5.36"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "tY-3DmcBGvwBJNsJ5Y5x",
                "_index": "logstash-2018.11.13",
                "_score": 8.148528,
                "_source": {
                    "@timestamp": "2018-11-13T20:16:31.812Z",
                    "flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
                    "kubernetes": {
                        "annotations": {
                            "scheduler_alpha_kubernetes_io/critical-pod": "",
                            "sidecar_istio_io/inject": "false"
                        },
                        "container_name": "mixer",
                        "host": "aks-nodepool1-19254313-3",
                        "labels": {
                            "app": "telemetry",
                            "istio": "mixer",
                            "istio-mixer-type": "telemetry",
                            "pod-template-hash": "2277284263"
                        },
                        "namespace_name": "istio-system",
                        "pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "istio-telemetry-66cc6d86b7-89vpl"
                    },
                    "log": "{\"level\":\"info\",\"time\":\"2018-11-13T20:16:31.234709Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"\",\"destinationIp\":\"10.244.5.38\",\"destinationName\":\"front-end-7fb8c76cc7-brcgf\",\"destinationNamespace\":\"sock-shop\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"front-end.sock-shop.svc.cluster.local\",\"destinationWorkload\":\"front-end\",\"httpAuthority\":\"10.244.5.38:8079\",\"latency\":\"2.8308ms\",\"method\":\"GET\",\"protocol\":\"http\",\"receivedBytes\":308,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"58dade92-1f67-9ecc-aa69-ce96c5fa495b\",\"requestSize\":0,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":9056,\"responseTimestamp\":\"2018-11-13T20:16:31.237461Z\",\"sentBytes\":10309,\"sourceApp\":\"\",\"sourceIp\":\"0.0.0.0\",\"sourceName\":\"unknown\",\"sourceNamespace\":\"default\",\"sourceOwner\":\"unknown\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"unknown\",\"url\":\"/\",\"userAgent\":\"kube-probe/1.11\",\"xForwardedFor\":\"10.244.5.1\"}\n",
                    "stream": "stdout",
                    "values": {
                        "apiClaims": "",
                        "apiKey": "",
                        "clientTraceId": "",
                        "connection_security_policy": "none",
                        "destinationApp": "",
                        "destinationIp": "10.244.5.38",
                        "destinationName": "front-end-7fb8c76cc7-brcgf",
                        "destinationNamespace": "sock-shop",
                        "destinationOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/front-end",
                        "destinationPrincipal": "",
                        "destinationServiceHost": "front-end.sock-shop.svc.cluster.local",
                        "destinationWorkload": "front-end",
                        "httpAuthority": "10.244.5.38:8079",
                        "instance": "accesslog.logentry.istio-system",
                        "latency": "2.8308ms",
                        "level": "info",
                        "method": "GET",
                        "protocol": "http",
                        "receivedBytes": 308,
                        "referer": "",
                        "reporter": "destination",
                        "requestId": "58dade92-1f67-9ecc-aa69-ce96c5fa495b",
                        "requestSize": 0,
                        "requestedServerName": "",
                        "responseCode": 200,
                        "responseSize": 9056,
                        "responseTimestamp": "2018-11-13T20:16:31.237461Z",
                        "sentBytes": 10309,
                        "sourceApp": "",
                        "sourceIp": "0.0.0.0",
                        "sourceName": "unknown",
                        "sourceNamespace": "default",
                        "sourceOwner": "unknown",
                        "sourcePrincipal": "",
                        "sourceWorkload": "unknown",
                        "time": "2018-11-13T20:16:31.234709Z",
                        "url": "/",
                        "userAgent": "kube-probe/1.11",
                        "xForwardedFor": "10.244.5.1"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "tJDUDmcBGvwBJNsJBOU_",
                "_index": "logstash-2018.11.13",
                "_score": 8.148528,
                "_source": {
                    "@timestamp": "2018-11-13T20:47:14.900Z",
                    "flb-key": "kube.istio-system.istio-telemetry-66cc6d86b7-89vpl.mixer",
                    "kubernetes": {
                        "annotations": {
                            "scheduler_alpha_kubernetes_io/critical-pod": "",
                            "sidecar_istio_io/inject": "false"
                        },
                        "container_name": "mixer",
                        "host": "aks-nodepool1-19254313-3",
                        "labels": {
                            "app": "telemetry",
                            "istio": "mixer",
                            "istio-mixer-type": "telemetry",
                            "pod-template-hash": "2277284263"
                        },
                        "namespace_name": "istio-system",
                        "pod_id": "39efc4a8-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "istio-telemetry-66cc6d86b7-89vpl"
                    },
                    "log": "{\"level\":\"info\",\"time\":\"2018-11-13T20:47:14.700695Z\",\"instance\":\"accesslog.logentry.istio-system\",\"apiClaims\":\"\",\"apiKey\":\"\",\"clientTraceId\":\"\",\"connection_security_policy\":\"none\",\"destinationApp\":\"telemetry\",\"destinationIp\":\"10.244.3.176\",\"destinationName\":\"istio-telemetry-66cc6d86b7-89vpl\",\"destinationNamespace\":\"istio-system\",\"destinationOwner\":\"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry\",\"destinationPrincipal\":\"\",\"destinationServiceHost\":\"istio-telemetry.istio-system.svc.cluster.local\",\"destinationWorkload\":\"istio-telemetry\",\"httpAuthority\":\"mixer\",\"latency\":\"1.123237ms\",\"method\":\"POST\",\"protocol\":\"http\",\"receivedBytes\":864,\"referer\":\"\",\"reporter\":\"destination\",\"requestId\":\"8af48874-f711-94e4-aa69-141586faaca5\",\"requestSize\":470,\"requestedServerName\":\"\",\"responseCode\":200,\"responseSize\":5,\"responseTimestamp\":\"2018-11-13T20:47:14.701706Z\",\"sentBytes\":174,\"sourceApp\":\"\",\"sourceIp\":\"10.244.4.16\",\"sourceName\":\"orders-6587948d7d-dx62n\",\"sourceNamespace\":\"sock-shop\",\"sourceOwner\":\"kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders\",\"sourcePrincipal\":\"\",\"sourceWorkload\":\"orders\",\"url\":\"/istio.mixer.v1.Mixer/Report\",\"userAgent\":\"\",\"xForwardedFor\":\"10.244.4.16\"}\n",
                    "stream": "stdout",
                    "values": {
                        "apiClaims": "",
                        "apiKey": "",
                        "clientTraceId": "",
                        "connection_security_policy": "none",
                        "destinationApp": "telemetry",
                        "destinationIp": "10.244.3.176",
                        "destinationName": "istio-telemetry-66cc6d86b7-89vpl",
                        "destinationNamespace": "istio-system",
                        "destinationOwner": "kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry",
                        "destinationPrincipal": "",
                        "destinationServiceHost": "istio-telemetry.istio-system.svc.cluster.local",
                        "destinationWorkload": "istio-telemetry",
                        "httpAuthority": "mixer",
                        "instance": "accesslog.logentry.istio-system",
                        "latency": "1.123237ms",
                        "level": "info",
                        "method": "POST",
                        "protocol": "http",
                        "receivedBytes": 864,
                        "referer": "",
                        "reporter": "destination",
                        "requestId": "8af48874-f711-94e4-aa69-141586faaca5",
                        "requestSize": 470,
                        "requestedServerName": "",
                        "responseCode": 200,
                        "responseSize": 5,
                        "responseTimestamp": "2018-11-13T20:47:14.701706Z",
                        "sentBytes": 174,
                        "sourceApp": "",
                        "sourceIp": "10.244.4.16",
                        "sourceName": "orders-6587948d7d-dx62n",
                        "sourceNamespace": "sock-shop",
                        "sourceOwner": "kubernetes://apis/apps/v1/namespaces/sock-shop/deployments/orders",
                        "sourcePrincipal": "",
                        "sourceWorkload": "orders",
                        "time": "2018-11-13T20:47:14.700695Z",
                        "url": "/istio.mixer.v1.Mixer/Report",
                        "userAgent": "",
                        "xForwardedFor": "10.244.4.16"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "hZDHDmcBGvwBJNsJXUz2",
                "_index": "logstash-2018.11.13",
                "_score": 8.0215845,
                "_source": {
                    "@timestamp": "2018-11-13T20:33:25.117Z",
                    "flb-key": "kube.sock-shop.user-5bbd6dd84-qg6l7.istio-proxy",
                    "kubernetes": {
                        "annotations": {
                            "sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
                        },
                        "container_name": "istio-proxy",
                        "host": "aks-nodepool1-19254313-0",
                        "labels": {
                            "name": "user",
                            "pod-template-hash": "166828840"
                        },
                        "namespace_name": "sock-shop",
                        "pod_id": "6e247a0f-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "user-5bbd6dd84-qg6l7"
                    },
                    "values": {
                        "agent": "kube-probe/1.11",
                        "authority": "10.244.4.18:80",
                        "bytes_received": "0",
                        "bytes_sent": "180",
                        "code": "200",
                        "duration": "3",
                        "flags": "-",
                        "method": "GET",
                        "path": "/health",
                        "protocol": "HTTP/1.1",
                        "real_ip": "10.244.4.1",
                        "remainder_ip": "",
                        "request_id": "06fb5497-da09-90df-a4f3-6742759c7f20",
                        "stream": "stdout",
                        "upstream": "127.0.0.1:80",
                        "upstream_service_time": "1"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "nI6dDmcBGvwBJNsJZz_F",
                "_index": "logstash-2018.11.13",
                "_score": 8.014115,
                "_source": {
                    "@timestamp": "2018-11-13T19:47:28.117Z",
                    "flb-key": "kube.sock-shop.user-5bbd6dd84-qg6l7.istio-proxy",
                    "kubernetes": {
                        "annotations": {
                            "sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
                        },
                        "container_name": "istio-proxy",
                        "host": "aks-nodepool1-19254313-0",
                        "labels": {
                            "name": "user",
                            "pod-template-hash": "166828840"
                        },
                        "namespace_name": "sock-shop",
                        "pod_id": "6e247a0f-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "user-5bbd6dd84-qg6l7"
                    },
                    "values": {
                        "agent": "kube-probe/1.11",
                        "authority": "10.244.4.18:80",
                        "bytes_received": "0",
                        "bytes_sent": "180",
                        "code": "200",
                        "duration": "2",
                        "flags": "-",
                        "method": "GET",
                        "path": "/health",
                        "protocol": "HTTP/1.1",
                        "real_ip": "10.244.4.1",
                        "remainder_ip": "",
                        "request_id": "64c754f6-d57d-90df-b818-d91932d7c66c",
                        "stream": "stdout",
                        "upstream": "127.0.0.1:80",
                        "upstream_service_time": "2"
                    }
                },
                "_type": "_doc"
            },
            {
                "_id": "XI6bDmcBGvwBJNsJ9C6s",
                "_index": "logstash-2018.11.13",
                "_score": 8.013674,
                "_source": {
                    "@timestamp": "2018-11-13T19:45:56.234Z",
                    "flb-key": "kube.sock-shop.front-end-7fb8c76cc7-brcgf.istio-proxy",
                    "kubernetes": {
                        "annotations": {
                            "fluentbit_io/parser-front-end": "apache2",
                            "sidecar_istio_io/status": "{\\\"version\\\":\\\"e5b877e0587fff4797e0dc3a6c01514601ea2d562cf5cd7e3f927bcaaea3e7ec\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":[\\\"regcred\\\"]}"
                        },
                        "container_name": "istio-proxy",
                        "host": "aks-nodepool1-19254313-1",
                        "labels": {
                            "name": "front-end",
                            "pod-template-hash": "3964732773"
                        },
                        "namespace_name": "sock-shop",
                        "pod_id": "6e798647-e77a-11e8-a3a6-0a58ac1f0442",
                        "pod_name": "front-end-7fb8c76cc7-brcgf"
                    },
                    "values": {
                        "agent": "kube-probe/1.11",
                        "authority": "10.244.5.38:8079",
                        "bytes_received": "0",
                        "bytes_sent": "9056",
                        "code": "200",
                        "duration": "2",
                        "flags": "-",
                        "method": "GET",
                        "path": "/",
                        "protocol": "HTTP/1.1",
                        "real_ip": "10.244.5.1",
                        "remainder_ip": "",
                        "request_id": "81ad8b69-c186-9e6b-aa69-ea6939186dfa",
                        "stream": "stdout",
                        "upstream": "127.0.0.1:8079",
                        "upstream_service_time": "2"
                    }
                },
                "_type": "_doc"
            }
        ],
        "max_score": 43.775322,
        "total": 35
    },
    "timed_out": false,
    "took": 376
}

Its looking like HTTP/3 will be all encrypted and all UDP, all the time

I started my web-ish life with HTTP 0/9. It was the dialect that 'escaped' from Cern. Soon after HTTP/1.0 came along, and then 1.1. And we stuck on 1.1 for a long time (more than 2 decades). And then HTTP/2 came along, and it was great. 100% encrypted, asynchronous, etc.

In parallel some folks started experimenting with HTTP and TLS-like encryption over UDP. This allowed them to build their own congestion-control algorithms independent of the operating system, to reduce latency of setup, of throughput. And it went quite well but was a bit controversial for some.

And now, after some discussion it looks like the camps are merging. There seems to be a fair bit of support for HTTP/3 to be UDP-based, all-encrypted all the time.

So, if you are still on HTTP/1.1, get going. HTTP/2/SPDY/QUIC/... are all about higher performance, better safety, more security, more privacy. Who doesn't want that?

Smart speaker bearing witness in murder trial

In 2017 an Amazon Alexa 'testified' in a murder trial in Arkansas, and is now scheduled to do so in New Hampshire (a double murder!).

No word on Siri, Bixby, "OK Google" and their kin.

Now, obviously big tech would prefer to keep mum on what they are recording around your house, and a subpoena kinda makes that hard. We've already had some issues with smart TV listening in.

I wonder who would be interested in a 'what goes on in my house when i'm not using the Internet' report?' Have we reached the state where home 'robots' use more bandwidth than people?

What's next to get smart around the home that can be subpoened? I mean, toilets got Internet some time ago.  Definitely we need some sort of kitchen-counter that is a large touch display (that is using some super gorilla glass that can withstand the slap-chop).

I look forward to the day we have a smart central vacuum testify in court. Or run for government.

Today’s adventure: the Chevy Bolt gets a double dashcam

This turned out really well. You can't see a single wire anywhere.

Ingredients. 1 generic 'Car Dash Camera'. Its so generic there is not a model number anywhere on it or the packaging. Nothing. I got it from Aliexpress here. It was described as "
WiFi Car DVRs Recorder Dash Cam Dual Lens Vehicle Rear Camera Built in GPS Camcorder 4K 2160P Night Vision Dashcam Novatek 96663". The 4K is a bit of a hmm. It is a pair of 1080P cameras (front, rear). Not exactly 4K, but we'll go w/ it.

It uses a Sony IMX323 sensor, which is something you need to look for in 'generic chinese cameras'. You can easily end up with a 'gopro 4k like' which sure, it records @ 4K, but the sensor is vaseline and waxed paper, the effective resolution is more like QCIF. But I digress. It has GPS (for speed) and WiFi (so you can install the worlds crappiest app to really do nothing with), and shock-sensor (to lock the recording if you bump). Interestingly it also has a super-capacitor in it for clean power down when you yank the power (and also for parking monitoring... if you are bumped while parked, it will take 60s of capture and snooze again).

Not content with 4G or 5G, they have gone with a 6G lens. Sounds great!

Somehow the 2 x 4MP sensors become 12MP when joined. The marketing specs are... aggressive!

To find out what was inside, first we employed a screwdriver. The only label is T3-NT96663 (Novatek). I've attached the datasheet which comes from ip dashcamtalk.Novatek NT96655 Data Sheet

Now, I was wondering, do I wire it to be 'always on' or to be 'Retained Accessory Power' (e.g. switched). Lets look at the power draw. It seems like its 0.8A @ 5V, so ~4W. Now, you might think, well, the Bolt has a 60KWh battery, run it all the time, how can this matter. But, like all electric cars, there is a small 12V lead-acid battery lurking in there which is used to drive everything except the traction power. Still, that is ~80Ah @ 12V nominal, ~= 960Wh. So at perfect efficiency (hah! physics!) we'd have 240 hours to 'full flat', 10 days. Derate by probably 75% (don't want to run a lead-acid down too far, some innefficiency), we'd be good for a few days. Hmm. I'll revisit later and put it on a RAP circuit.

Wiring was simple. I considered wiring it into the overhead console power (onstar, dome lights), but instead drove it back to the fuse panel where there were quite a few blanks. I used an 'add-a-circuit' micro-fuse tap. I then put it into a small buck converter. The wiring was easy to run (up the kick-panel, up the A-pillar, across the roof) in the lip under the wind-seal. I then wired the rear-camera by wiring from front to rear along the wind-seal and headliner lip. You cannot see the wires at all. The pain bit was getting it through the rubber boots into the hatch-back assembly. For this I used a 'fish stix' to fish it. A few stickers, and we are good to go. You can see the fuse panel at the right, lots of empty space to be used (but the add-a-fuse can share an existing tap if needed).

Now on to the app. Oh dear. Rated 1.8 on the Play Store. This will be wonderful. Well, all I can say is, it kind of works if you are patient. 'Roadcam' is the name that eventually worked for me, there are a lot, and with such a generic device (no model name at all)...

The user manual... well... it gets the job done. I'm not sure the grammar is world-class, but the concepts come through.

Does it work? Lets see some video (from the rear cam firsts, then the front) for the maiden run:

The agony and the ecstasy of the read-only

So earlier today I counselled to run your container filesystem read-only.

  • Its higher security (something can't weasel in as easily)
  • You want to be able to dynamically dispose and restart containers somewhere else, how can you do this if they are stateful
  • The overlay fileystem is not hgh performance

Now, this last one. Lets say you have a container, oh, say, elasticsearch for the sake of argument.  Its already a pain to schedule under Kubernetes since it is Java (it wants all the memory and then allocate it locally, fighting the scheduler), and its stateful (needing a StatefulSet). But you might not also realise that it logs locally to it (naughty naughty). And you only find this when you make it read-only. You get this lovely error:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[0.000s][error][logging] Error opening log file 'logs/gc.log': Read-only file system
Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.

But, now you know. And the solution is simple. Add an 'emptyDir' with 'medium = Memory' and mount it on /usr/share/elasticsearch/logs (and mount one on /tmp while you are at it). Presto.

Top