O great SHA, protect me from this hack

So I’m building a container. And, well, it fails. The reason? One of the packages has a bad sha256 checksum. This is important so listen up!

Many people would either disable the check, or just update the checksum to match what you received. Here I decided to dig into it a bit. The scary message I was presented with is below, in its original high-definition technicolour.

Now, what could be going wrong? Clearly somewhere between me and what I’m hoping to get, something has been modified. And the file-(crypto) checksum is the last line of defence, so I don’t want to disable it.

I run the same test from home, and I see no issue. And the host resolves to the same IP. Hmm. So something in the path of my office is modifying the file or feeding me a different one.

On looking in a capture, I see the SYN/ACK comes from very close in time:

17:56:29.902424 IP 10.173.231.51.57292 > 93.184.216.34.80: Flags [S]
17:56:29.902679 IP 93.184.216.34.80 > 10.173.231.51.57292: Flags [S.]
17:56:29.902699 IP 10.173.231.51.57292 > 93.184.216.34.80: Flags [.]

OK, 0.255ms away. Clearly we are not going far. The attack is in the building.

I fetch the good and the bad. I compare. Its not merely truncated, they are different size and different .deb.

I run dpkg-deb -I on both. The ‘bad’ one is actually a good .deb, just the wrong one (its libk5crypto3, not libgmp10). So we are not malicously hacked, instead, we have a mis-behaving cache in the network. Argh.

So workarounds. I could try and force the Debian update to use https. I can try and make it not-cache. I can run a local cache upstream of the bad one, and pre-populate it.

So I first try the ‘not cache’. I set the cache-control max age to 0. No good. same for no-store. This cache doesn’t listen.

I don’t really want to force Debian to use https (since this would mean altering the Dockerfile). So I install apt-cacher-ng and ‘fix’ the broken files.

So the moral of the story. Listen when the SHA checksum fails. Assume middleboxes are evil.
Update edit: its a sophos UTM.

GET /debian-security/pool/updates/main/u/util-linux/libblkid1_2.29.2-1%2bdeb9u1_amd64.deb HTTP/1.1
Host: security.debian.org
User-Agent: Debian APT-HTTP/1.3 (1.4.8)

fails and returns a different package.