Less is more. Famous saying, perhaps less well known person (Ludwig Mies van der Rohe). Another way to look at this is “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away” (Antoine de Saint-Exupery). Course he also said ‘Dessine-moi un mouton!’ which became a 1999 pop not so hit. Draw that sheep!

We all know that reducing the attack surface is a good thing. And, recently, the concept of ‘living off the land‘ has become popular for cyber-security-attackers, the concept of using what tools are available. Left ‘nc’ and ‘curl’ and ‘awk’ in your image? You’ve left a lot. What about gcc?

OK, got it, memo to self, remove stuff not using. But, what am I not using? Well here’s an interesting technique, using a combination of ‘apk add -virtual’ (which adds but remembers to delete later), and, scanelf. Scanelf? Is that Will Ferrell elf?

No, its scanelf(1), a tool you probably are not using. But, if we look at how they are using it here, all will become clear. Its like a way of scrubbing things not needed, without knowing:

apk add --virtual .build-deps gcc make musl-dev ...
make ... make install
runDeps=$(scanelf --needed --nobanner --format '%n#p' --recursive /usr/local \
 | tr ',' '\n' \
 | sort -u \
 | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }') 
apk add --virtual $runDeps
apk del .build-deps

(I simplified it a bit).

What this does is:

  1. Install the build tools, but tagged with .build-deps so we can later remove
  2. Make the executable
  3. Make a list of all the libraries referenced by the executable, figure out what package they are in
  4. Add those packages (in case they were in the build-deps and might get removed)
  5. Remove the build-deps

Nowhere did I need to specify a list of what files are in the executable that come from system packages. I end up w/ the original image + what system libs my package needs, plus my package. Nothing more. Less is more.

And now, that 2-decode old ‘draw me a sheep’ song. Enjoy.


You know the joke about the crappy horror movie, they trace the IP, its, the killer was in the house (localhost)?

True story, this just happened to me. So settle down and listen to a tale of NAT, Proxy, Kubernetes, and Fail2Ban (AKA Rack Attack in ruby land).

You see, we run a modest set of infrastructure in Kubernetes on GKE. Its about what you would expect, a LoadBalancer (which owns the external IP) feeds an Ingress controller which in turn has a set of routing rules based on vhost. And one of those endpoints is Gitlab (now you see why I mentioned Ruby above). And one of the things you should know about the cloud is… NAT is common, and multiple NAT is usually present.

So here, the chain:

[LoadBalancer]->[Ingress]->[Gitlab nginx]->[Gitlab unicorn]

has 3 NAT steps. Don’t believe me? Lets count

  1. The Load Balancer does a NAT.
  2. The Ingress is a proxy server, so inherently NAT.
  3. The Gitlab nginx (sidecar) is a proxy server, so inherently NAT.

So, what IP will Gitlab unicorn see? Well, that of the gitlab nginx. If REMOTE_IP is used, maybe that of the Ingress.

So, when some $!# tries to hack my gitlab, what will happen? I get blocked!

# redis-cli> keys *rack*attack*
1) "cache:gitlab:rack::attack:allow2ban:ban:"

‘403 forbidden’. Courtesy of this feature ‘rack attack’ which is a type of fail to ban. Now, I’m not dis’ing fail to ban, its a powerful technique. But, well, you gotta know who you are banning.

Recently Google announced Filestore. I was all set to rejoice, after my heartbreaks of recent days. After all, it seemed like NFS might have been the answer for me, but I would have to have it run outside of Kubernetes. So it was with great joy I signed up for the beta program, and even greater excitement I clicked on the ‘ready to try’ link today.

Excitement dashed (delayed?). Its not available to those of us of the northern part of north america persuasion (nor of the eastern part of north america for that matter).

O well, the current solution (the sidecar running restic syncing to my external restic server) is working.


So I’m here at Google Next this week. Cloud cloud cloud, security security security, that sort of thing.

Question 1. Would you buy this ‘spam musibi‘ from this vending machine? Or you’d rather have that tasty-looking turkey? Or that delicious looking hot dog? Keep in mind there is no microwave around, so that cold vended hotdog might be a bit slimy. Choose wisely. Feel free to click to zoom in.

I would buy, eat, and enjoy

View Results

Loading ... Loading ...

Question 2. Who on earth would set out ~10 feeding trays for raccoons on at an overpass of the 101 @ millbrae? I mean, wtf? As I trudged over the highway from the BART station to the hotel, I was surprised, they were surprised. 1 stuck around for a photo, the others bugged out. Um, ???

Question 3 (photo didn’t turn out). So you’re doing a trade show. You’ve spent a big chunk of $$$ on your booth. The centre-piece is a huge LED sign, ~30′ tall. Its setup w/ all kinds of live feed of info. Do you hook it to your twitter anytime someone mentions you? Think a bit. So Intel did that, and, well, a lot of people are unhappy about the recent ME issues. So a bit of a twitter war erupts, most of it not that complimentary of Intel. And its on their big screen. Hmm. Prob would have been better off w/ kittenwar.


Well this is awkward. That Intel Management Engine (AMT) that you can’t disable. That one that runs even when your machine is turned off. The one that runs with highest privilege, no oversight, outside your operating system. The same one that everyone wonders, just what is in there.

Turns out, from the security audit, some not so great flaws. And the upshot is if you are running an older processor (3rd generation into core or later) it would be a good time to discard it.

If its 4th gen Intel core or later you own, you can consider getting a firmware update from the vendor of your motherboard.

You can find out your generation of processor, and whether it has AMT (vPro, ME) from ark.intel.com.

Now, if we apply our secret decoder ring to the CVSSv3 Vectors:


Hmm. Attack Vector: Network, Adjacent. Privilege required: none, user interaction: none.

So this means anyone who can get onto the same subnet as you (or a remote subnet if you route) can own your machine. This includes from itself, meaning unprivileged local code can escalate up.

If I snoop around my house, I still have two systems in operation that are:

  1. vulnerable
  2. no fix

My i7-3770s NAS is the complex one. Its doing what i want and i don’t really want to swap it out. Hmm.