So I'm building a container. And, well, it fails. The reason? One of the packages has a bad sha256 checksum. This is important so listen up!
Many people would either disable the check, or just update the checksum to match what you received. Here I decided to dig into it a bit. The scary message I was presented with is below, in its original high-definition technicolour.
Now, what could be going wrong? Clearly somewhere between me and what I'm hoping to get, something has been modified. And the file-(crypto) checksum is the last line of defence, so I don't want to disable it.
I run the same test from home, and I see no issue. And the host resolves to the same IP. Hmm. So something in the path of my office is modifying the file or feeding me a different one.
On looking in a capture, I see the SYN/ACK comes from very close in time:
17:56:29.902424 IP 10.173.231.51.57292 > 18.104.22.168.80: Flags [S] 17:56:29.902679 IP 22.214.171.124.80 > 10.173.231.51.57292: Flags [S.] 17:56:29.902699 IP 10.173.231.51.57292 > 126.96.36.199.80: Flags [.]
OK, 0.255ms away. Clearly we are not going far. The attack is in the building.
I fetch the good and the bad. I compare. Its not merely truncated, they are different size and different .deb.
I run dpkg-deb -I on both. The 'bad' one is actually a good .deb, just the wrong one (its libk5crypto3, not libgmp10). So we are not malicously hacked, instead, we have a mis-behaving cache in the network. Argh.
So workarounds. I could try and force the Debian update to use https. I can try and make it not-cache. I can run a local cache upstream of the bad one, and pre-populate it.
So I first try the 'not cache'. I set the cache-control max age to 0. No good. same for no-store. This cache doesn't listen.
I don't really want to force Debian to use https (since this would mean altering the Dockerfile). So I install apt-cacher-ng and 'fix' the broken files.
So the moral of the story. Listen when the SHA checksum fails. Assume middleboxes are evil.
Update edit: its a sophos UTM.
GET /debian-security/pool/updates/main/u/util-linux/libblkid1_2.29.2-1%2bdeb9u1_amd64.deb HTTP/1.1 Host: security.debian.org User-Agent: Debian APT-HTTP/1.3 (1.4.8)
fails and returns a different package.