So another day and another ‘registry out of space’. I wrote earlier about the crappy experience increasing this size (and GKE is still on 1.10 so I can’t use the 1.11+ resize mechanism!!!)

Vowing not to repeat the ‘that can’t be the right method’ that I pioneered in that post, I decided to dig a bit into why it was so large. One of the culprits was a surprise, I’ll share.

So we have a (few) Dockerfiles that do something like:

FROM node:10.8-alpine
LABEL maintainer=""

WORKDIR /usr/src/app
COPY . /usr/src/app/package.json
RUN npm install 

Seems simple enough right? And the current directory really just has about 4kB of data. How could this container be the largest one?

Digging into it… our CI system (gitlab-ci) has a ‘caching’ mechanism between stages, which creates a ‘’ of whatever you choose.

In turn, I’m doing a ‘docker save’ in the build step so that the various security scan stages (which run in parallel) are more efficient… they just used the cached copy.

And in turn, gitlab-ci makes this dir in “.cache” in the repo directory (because its mounted that way in Kubernetes, the gitlab-runner has no other storable space).

So what happens is, the first pipeline runs, does a bunch of work, saves some stuff in the cache. Later, this runs again, and the cache increases. But… each docker build incorporates the cache into the image, which is then saved back into the cache.

So… first run… 10M container
Second run, incorporates the first 10M container (from cache) + itself = 20M
Third run… well… 40M

later this gets quite big. Huh.

This is where the ‘.dockerignore‘ file should be used!

And now I know, and now you do to.

Tagged with: , , , , ,

We’ve all been there. Working to a deadline trying to get our e-commerce site going to make sure cats don’t get cold feet for the winter.

And because its a microservices cloud jwt polyglot kubernetes istio [insert jargon here] world, well, its not as easy to debug. So many moving pieces. Remember when I said the cloud is built for width, not downward-scalability?

You are polishing the demo in Azure AKS, its looking good. When suddenly a wild set of flows occur. You whip out your your trusty tools that show you North-South, East-West, South-North traffic, which conveniently have IP transparency enabled so there is no NAT affects from the 3-levels of address translation in Kubernetes.

You get this chart below. Hmm, I’m seeing about 25k flows @ ~1k/s/new coming in, from Whois tells me this is Microsoft. Oh no! The attack is coming from inside the house!

We look into Kibana, we see all the flows helpfully logged. They each look like this. We then look at the MS Developer page here. Huh. The correlation is, all of our ContainerPort are being **hammered** by this. But we are not responding (because out network policy is to avoid this!). So it tries again. The ports its contacting, some of them are not HTTP, so I don’t know how they would know what service. The article suggests “Bring-your-own IP Virtual Network”, which is not us.

We conclude that, well, yes, Azure has a very high interest in pinging all services with ContainerPort enabled. And that yes, an automatically-responding firewall might consider this an attack. And no, nothing bad happens when counter-measures are deployed. And, nothing better occurs if you whitelist it (as the article suggests). This is likely the ‘health check’ of the LoadBalancer, in my case:

istio-ingressgateway       LoadBalancer   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:30380/TCP,8060:30376/TCP,853:32481/TCP,15030:32411/TCP,15031:31158/TCP   1h

Perhaps we do need to respond after all? Nothing bad happens if we don’t, but might it declare this ingressgateway down?

Note: its also the ‘upstream DNS’ of your kube-dns, so be careful about a flat-out block. YMMV. Void where prohibited. This is not legal advice.



Tagged with: , , , ,

Earlier I wrote about the ‘elastic-prune‘ a simple cron-job that lived in Kubernetes to clean up an Elasticsearch database. When I wrote it, I decided to give ‘distroless‘ a whirl. Why distroless? Some will say its because of size, they are searching for the last byte of free space (and thus speed of launching). But, I think this is about moot. The Ubuntu 18.04 image and the Alpine image are pretty close in size, the last couple of MB doesn’t matter.

‘distroless’ is all the code none of the cruft. No /etc directory. The side affect is its small, but the rationale is its secure. Its (more) secure because there are no extra tools laying around for things to ‘live off the land‘. This limits the ‘blast-radius’. If something wiggles its way into a ‘distroless’ container it has less tools available to go onward and do more damage.

No shell, no awk, no netcat, no busybox. The only executable is yours. And this is what your build looks like. You can see we use a normal ‘fat old alpine’ source to build. We run ‘pip’ in there. Then we create a new container, copying from the ‘build’ only the files we need. We are done.

Doing the below I ended up with a ‘mere’ 3726 files. Yup, that is the list, see if your favourite tool made the cut.

Going ‘distroless’ saved me 33MB (from 86.3MB to 53.3MB). Was this worth it?

FROM python:3-alpine as build
LABEL maintainer=""

COPY . /elastic-prune
WORKDIR /elastic-prune

RUN pip install --target=./ -r requirements.txt

COPY --from=build /elastic-prune /elastic-prune
WORKDIR /elastic-prune
ENTRYPOINT ["/usr/bin/python3", "./"]
Tagged with: , , ,

Let’s say one day you are casually browsing the logs of your giant Kubernetes cluster. You spot this log message: “npm update check failed”. Hmm. Fortunately you have an egress firewall enabled, blocking all outbound traffic other than to your well-known API’s, so you know why it failed. You now worry that maybe some of your projects are able to auto-update because it was difficult or impossible to fence them off.

Why would you want a container updating within itself? You rebuild them once a week in your CI, and deploy those tested, scanned updates. Here you are live-importing the risk we talked about in the ESLint debacle, meaning that someone in the universe might have a password they use on more than one site, it gets compromised, and an attacker pushes new code to their repo. And then boom, you’d install it without knowing.

As a backup for your ‘egress firewall’ you have also made all the rootfs read-only in your containers, mitigating the possibility of new installs like this. But, most containers have to have some writeable volume somewhere (even if only for the environment-variable-to-/etc/foo.conf-dance on startup).

Here the solution is a magic, ‘lightly documented’ environment variable NO_UPDATE_NOTIFIER. But…

The code behind this (for nodejs) is here. You can see it would spawn out to the shell. It doesn’t do this check for a while (a day?) after you startup, so you might be lulled into a false sense of security. And it would be hard to construct a generic test to check for this behaviour. So yeah, the best seat-belts are the egress firewall, strict as strict can be, and a read-only rootfs.

Tagged with: , , ,

The world is getting faster with shorter cycle times. Software releases, once things that celebrated birthdays are now weekly.

Emboldened by the seemingly bullet-proof nature of Kubernetes and Helm, and trying to resolve an issue with an errant log message, I update the nginx-controller. Its easy:

helm upgrade nginx-ingress stable/nginx-ingress

Moments later it is done. And, it seems to be still working. But what’s this? It keeps starting and CrashLoopBackoff two of the Pods? Huh. So I look at the logs.

nginx: [emerg] bind() to failed (2: No such file or directory)

shows up. Hmm. This is the Gitlab ssh forward. It was working before, so I know the configmap is correct. I see that CAP_NET_BIND is enabled, so that should be ok?

After some groping around, I discover its the ‘go-proxy’ which is part of this for some reason. Go doesn’t work with authbind, and thus the securityContext. But nginx is in C, why does this matter? I then happen on this issue. Its fresh.  It implies that it will be fixed in 0.20.0.

I apply the workaround:

kubectl set image deployment/nginx-ingress-controller

In doing so I learn about ‘kubectl set image’. Huh, no more patching for me.

Back in the day there were Release Notes. Now, well, you try and then fail and then Google.

Tagged with: , , ,