The case of the nosy neighbour: Azure is poking me!

We’ve all been there. Working to a deadline trying to get our e-commerce site going to make sure cats don’t get cold feet for the winter.

And because its a microservices cloud jwt polyglot kubernetes istio [insert jargon here] world, well, its not as easy to debug. So many moving pieces. Remember when I said the cloud is built for width, not downward-scalability?

You are polishing the demo in Azure AKS, its looking good. When suddenly a wild set of flows occur. You whip out your your trusty tools that show you North-South, East-West, South-North traffic, which conveniently have IP transparency enabled so there is no NAT affects from the 3-levels of address translation in Kubernetes.

You get this chart below. Hmm, I’m seeing about 25k flows @ ~1k/s/new coming in, from 168.63.129.16. Whois tells me this is Microsoft. Oh no! The attack is coming from inside the house!

We look into Kibana, we see all the flows helpfully logged. They each look like this. We then look at the MS Developer page here. Huh. The correlation is, all of our ContainerPort are being **hammered** by this. But we are not responding (because out network policy is to avoid this!). So it tries again. The ports its contacting, some of them are not HTTP, so I don’t know how they would know what service. The article suggests “Bring-your-own IP Virtual Network”, which is not us.

We conclude that, well, yes, Azure has a very high interest in pinging all services with ContainerPort enabled. And that yes, an automatically-responding firewall might consider this an attack. And no, nothing bad happens when counter-measures are deployed. And, nothing better occurs if you whitelist it (as the article suggests). This is likely the ‘health check’ of the LoadBalancer, in my case:

istio-ingressgateway       LoadBalancer   10.0.151.211   40.85.211.205   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:30380/TCP,8060:30376/TCP,853:32481/TCP,15030:32411/TCP,15031:31158/TCP   1h

Perhaps we do need to respond after all? Nothing bad happens if we don’t, but might it declare this ingressgateway down?

Note: its also the ‘upstream DNS’ of your kube-dns, so be careful about a flat-out block. YMMV. Void where prohibited. This is not legal advice.