Increase your CI speed && decrease your cost. The preemptible node

We are running gitlab, self-hosted, in Google Kubernetes Engine (GKE). And we use gitlab runner for our CI. And I have to say, this has been working beyond expectations for me: it works really well.

Now a bit of a puzzle hit our happy landscape about 6 months ago or so. One large project which didn't economically fit into the model. I tried a few things, finally settling on running 2 runners (each in a separate Kubernetes cluster). The one in the GKE was labelled 'small' and the other 'big'. The 'big' one runs in my basement on the 72 thread / 256GB machine which would be uneconomical to leave running in GKE.

Enter the 'pre-emptible' VM. Pricing is here. As you can see, its quite a bit less. In return, you get reset at least once per day. Also, if the neighbours get 'noisy' you get unscheduled for a bit. This is probably acceptable for the CI pipeline.

I added this nodeSelector to the gitlab-runner:

nodeSelector: "true"

I then added a 'taint' (no really that is what it is called) to prevent this nodepool from attracting scheduled Pods that didn't explicitly tolerate:

kubectl taint nodes [NODE_NAME]"true":NoSchedule
And boom, we have a faster 'small' CI, which costs less than what it replaced. I still am going to keep the beast of the basement online for a bit.

Let’s Encrypt Staging. Curl without the -k

Are you lazy and use '-k' to curl all the time when using Let's Encrypt staging? Or worse, use snake-oil? Or even worse, use just http for 'test'?

curl --cacert fakelerootx1.pem https://my-site-issued-with-le-staging

There, how hard was that? Now you can test that the cert was generated properly (even though its not properly signed).

Let’s Encrypt Staging. Safely.

Let's Encrypt. One of the best things done in recent years. It makes it simple and free to have decent TLS security. There's really no excuse not to now.

One minor challenge has been the 'staging' environment. You want to use this when you are debugging your setup,  automatically creating certificates for the first time, etc. They have a generous but not unlimited set of certificates you can create per time and you don't want to hit this limit because your un-debugged script went nuts. So for this they make the staging environment available.

Now the only problem with the staging environment, the intermediate certificate is not in the root store of your browser. And there's a reason. They don't hold it to the same standard (its for debugging after all).

So let's say you have a shiny new .dev domain. Its in the HSTS store of your browser, and you want to use Let's Encrypt staging.

Well, you can simply import the staging intermedate cert into a new browser profile, one that is only used for this testing. Download the Fake LE Intermediate X1. Run a chrome with google-chrome --profile-directory=lets-encrypt-staging-trust. And then in it, import this cert. Use this profile, and only this profile, for your testing.

Import the certificate by opening chrome://settings/certificates?search=certif and then select 'authorities'. This browser has none of your bookmarks, saved passwords, etc. So don't make it sync them 🙂

Have fun using the Let's Encrypt staging environment. When done, don't forget to switch to the live environment tho!

I made a .desktop file and special icon so i could launch it like my regular browser, as below, but this is not required.

$ cat ~/.local/share/applications/chrome-le.desktop 
[Desktop Entry]
Exec=google-chrome-beta "--profile-directory=lets-encrypt-staging-trust"

pause: how to debug your Kubernetes setup

Sometimes you need a debug container hanging around to check something from within your cluster. You cobble something together, make the 'command' be 'sleep 3600' or 'tail -f /dev/null' and call it a day. But they don't terminate gracefully.
kubectl run debug --restart=Never --image=agilicus/pause
The magic is this 'pause.c'. It simply waits for a couple of signals, calls pause(2) and thus waits. It exits immediately if anything happens. This means that it uses near zero resources while sleeping and exits gracefully.

#include <unistd.h>
#include <signal.h>

static void _endme(int sig)
main(int argc, char **argv)
  signal(SIGINT, _endme);
  signal(SIGTERM, _endme);

Now, this seems esoteric, but give it a try. Now, once you have run that run command above, you can simply  kubectl exec -it debug bash and from in there apk add tool.

So you might apk add curl and then curl http://myservice. Simple, right?

Now, I know a lot of you are committing the cardinal sin of having a shell and debug environment in every container just in case. Well, let me tell you, that security attacker is going to love your just in case toolset. Why not let the container run as root with a writeable filesystem and a compiler while we are at it.

You can check out the copious code @

When you throw in the towel on declarative

I've talked a lot recently about the declarative versus imperative viewpoints. Its the Lilliput vs Blefuscu of our time. Its the vi versus emacs saga.

Today I ran into a scenario that I just threw in the towel on. I had a largish yaml file (~300 lines) that is actually a config to a container (e.g. its not Kubernetes yaml).

I'm using kustomize which means i cannot use the tricks I would in helm with 'tpl' and {{ }} (those are imperative or templating!).

I need to change one line in it per environment (a hostname). And I really didn't feel good about copying the file into multiple output directories.

After an hour, I threw in the towel. The declarative police will come and get me, but I present to you my solution. Open sourced for your pleasure.

See my new container 'envsubst'. It auto-builds into dockerhub for you lazy folks. You are one

docker pull agilicus/envsubst

away from this big pile of perfection.

It's simple. This container takes arguments in the form input:output. `input` will be passed through envsubst and redirected to `output`, making directories as needed.

`docker run --rm -it CONTAINER /etc/passwd:/dev/stdout`

So e.g.:

`docker run --rm -it CONTAINER /config/ifile:/etc/config/dir/ofile`

will take `ifile`, run through `envsubst`, `mkdir -p /etc/config/dir`, and place the output in `/etc/config/dir/ofile`

Now, from Kubernetes, I make an 'emptyDir: {}'. I mount it (read-only) in my ultimate container, and read-write in this new one (as an initContainer). I pass 'args' as the list of files above. Presto. All environment variables are expanded. Into that big grotty yaml file that started this problem i place a ${THE_URL}. And I'm done.

Am I proud of this? Well, I don't have a lot of skin in the declarative vs imperative game. So. Um. I'm done with it.