When your security tools cost more than the thing they protect

Lets say you have a micro-services app. Its got a bunch of containers that you've orchestrated out with Kubernetes. Deployments, Pods, Daemonsets all over the place. Autoscaling. You are happy. Now it comes time to implement that pesky 'security' step. You are a bit nervous, there's no internal firewall, all the services listen on port 80, no encryption. All the passwords are hard-coded and in the global environment. No one would guess your l33t mysql password right? So you google 'how is secur networx'. And you click I'm feeling lucky.

Good thing for you google was watching your previous searches and had the microphone on, so it not only corrected your txt-speak spelling but also selected Istio for you.

But suddenly you need to triple the capacity of your cluster. Lets take a look. Here's kubectl top from my cluster. The lines in red are associated with the securing + auditing. See that last column? Seems we are using 8144MiB for monitoring the thing that is using 2259MiB. And don't get me started on the CPU.

I said it before, the cloud doesn't scale down.

$ kubectl top pod --all-namespaces

NAMESPACE      NAME                                                    CPU(cores)   MEMORY(bytes)   
default        ingress-nginx-ingress-controller-d77f4bd55-xv4h6        4m           146Mi           
default        ingress-nginx-ingress-default-backend-bc76f4cc7-fp6jk   0m           3Mi             
istio-system   istio-citadel-84fb7985bf-bb7cf                          0m           12Mi            
istio-system   istio-egressgateway-bd9fb967d-mxhgr                     2m           35Mi            
istio-system   istio-galley-655c4f9ccd-x9qqw                           13m          39Mi            
istio-system   istio-ingressgateway-7f5fdcf65-nxj9k                    3m           37Mi            
istio-system   istio-pilot-6cd69dc444-t465g                            8m           84Mi            
istio-system   istio-policy-77f6849776-6z8vq                           89m          419Mi           
istio-system   istio-policy-77f6849776-865xv                           97m          521Mi           
istio-system   istio-policy-77f6849776-d6bqn                           99m          492Mi           
istio-system   istio-policy-77f6849776-wc2nj                           62m          345Mi           
istio-system   istio-policy-77f6849776-xhwmw                           63m          351Mi           
istio-system   istio-sidecar-injector-8975849b4-47rx7                  13m          27Mi            
istio-system   istio-statsd-prom-bridge-7f44bb5ddb-zr25c               34m          23Mi            
istio-system   istio-telemetry-77f7949485-24xgl                        76m          440Mi           
istio-system   istio-telemetry-77f7949485-l69tn                        105m         559Mi           
istio-system   istio-telemetry-77f7949485-srjr5                        109m         525Mi           
istio-system   istio-telemetry-77f7949485-vbxdb                        106m         574Mi           
istio-system   istio-telemetry-77f7949485-wg8z5                        79m          437Mi           
istio-system   prometheus-84bd4b9796-fftw8                             51m          689Mi           
kube-system    cert-cert-manager-645675c97c-xxtfh                      2m           22Mi            
kube-system    heapster-6c4947855f-tjnw4                               0m           41Mi            
kube-system    kube-dns-v20-5fd69f859d-bxv6z                           18m          27Mi            
kube-system    kube-dns-v20-5fd69f859d-zmhmr                           18m          28Mi            
kube-system    kube-proxy-5rhch                                        3m           36Mi            
kube-system    kube-proxy-dxk9f                                        3m           42Mi            
kube-system    kube-svc-redirect-dpxvl                                 11m          156Mi           
kube-system    kube-svc-redirect-z5qhw                                 5m           110Mi           
kube-system    kubernetes-dashboard-66bf8db6cf-cwnbz                   0m           15Mi            
kube-system    metrics-server-64f6d6b47-jfc9l                          0m           26Mi            
kube-system    tiller-deploy-895d57dd9-ll66n                           0m           45Mi            
kube-system    tunnelfront-7794f9f645-dnvhx                            21m          16Mi            
logging        elasticsearch-867b4f6769-s4bzp                          567m         1420Mi          
logging        fluent-bit-56d6z                                        21m          11Mi            
logging        fluent-bit-8cbnl                                        17m          12Mi            
logging        logging-fluentd-69fdd5dbd6-mtsmm                        1m           59Mi            
logging        logging-kibana-76849979dd-m2vz9                         1m           152Mi           
logging        sysctl-conf-92l84                                       0m           0Mi             
logging        sysctl-conf-hb2vn                                       0m           0Mi             
monitoring     alertmanager-monitoring-0                               1m           15Mi            
monitoring     monitoring-exporter-kube-state-7c757d986f-75rrq         3m           37Mi            
monitoring     monitoring-exporter-node-57g9n                          1m           14Mi            
monitoring     monitoring-exporter-node-gdczp                          1m           10Mi            
monitoring     monitoring-grafana-59fcbfc89f-5qhzb                     0m           35Mi            
monitoring     monitoring-prometheus-operator-7d99d5f9c9-w82gv         2m           30Mi            
monitoring     prometheus-monitoring-prometheus-0                      7m           176Mi           
socks          carts-6994d7d589-6j5c2                                  5m           340Mi           
socks          carts-db-7dd64bfd7b-cld92                               5m           96Mi            
socks          catalogue-849865789c-k9pqn                              4m           47Mi            
socks          catalogue-db-6d66678d9-tfkzx                            3m           236Mi           
socks          front-end-855684fd8c-fv2qp                              4m           118Mi           
socks          orders-7d9cf5cb46-d2h2z                                 5m           350Mi           
socks          orders-db-6db4678bf6-6fx9j                              5m           93Mi            
socks          payment-6cdc5b656-84fbb                                 4m           48Mi            
socks          queue-master-7b99dbb88c-zrxst                           5m           301Mi           
socks          rabbitmq-7c5fbf778d-f5fbr                               7m           127Mi           
socks          session-db-fdd649d68-cq5sp                              3m           52Mi            
socks          shipping-5b9ffdbdfb-xnsjj                               5m           321Mi           
socks          user-84ccd5fd57-2vpb5                                   4m           47Mi            
socks          user-db-7dcc9649dc-b72rs                                4m           83Mi            


Tagged with: , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *