When good containers go bad: github issues are the new release notes

The world is getting faster with shorter cycle times. Software releases, once things that celebrated birthdays are now weekly.

Emboldened by the seemingly bullet-proof nature of Kubernetes and Helm, and trying to resolve an issue with an errant log message, I update the nginx-controller. Its easy:

helm upgrade nginx-ingress stable/nginx-ingress

Moments later it is done. And, it seems to be still working. But what’s this? It keeps starting and CrashLoopBackoff two of the Pods? Huh. So I look at the logs.

nginx: [emerg] bind() to 0.0.0.0:22 failed (2: No such file or directory)

shows up. Hmm. This is the Gitlab ssh forward. It was working before, so I know the configmap is correct. I see that CAP_NET_BIND is enabled, so that should be ok?

After some groping around, I discover its the ‘go-proxy’ which is part of this for some reason. Go doesn’t work with authbind, and thus the securityContext. But nginx is in C, why does this matter? I then happen on this issue. Its fresh.  It implies that it will be fixed in 0.20.0.

I apply the workaround:

kubectl set image deployment/nginx-ingress-controller nginx-ingress-controller=quay.io/kubernetes-ingress-controller/nginx-ingress-controller-amd64:fix-tcp-udp

In doing so I learn about ‘kubectl set image’. Huh, no more patching for me.

Back in the day there were Release Notes. Now, well, you try and then fail and then Google.