ELI5: Bloomberg, China, Supermicro, Apple, Amazon, Facebook et al security

So quite a few people have asked me about this. So let me explain what the technical assertions are, how they would work. Note I’m not taking a position on whether this did/is/didn’t not occur since there seems to be some controversy on this, and I have no information.

First, what is asserted. Bloomberg is reporting that servers manufactured by Supermicro were being used by Elemental had been altered somewhere in the supply chain to add a ‘tiny chip” to the BMC. So what does that mean?

The ‘BMC’ is the Baseband Management Controller. Effectively it allows remote access to the keyboard/monitor/mouse (sometimes called KVM). This remote access can be local (e.g. a second board in the server running its own operating system), or via the network. Its part of a larger set of standards for remote management. This includes Intel’s AMT which I wrote earlier about its vulnerabilities, Dell’s iDRAC (which has its own issue called iDracula). These standards (IPMI) are more than a decade old (dating to 1998), and include a variety of specs and standards (RMCP et al) for remotely controlling/upgrading/managing servers. There have been many security issues against these over the years.

What has made the Intel AMT issues more high profile is that AMT is:

• Opaque. You cannot know what it is or what its doing
• Always-on. You cannot disable it
• Ubiquitous. Its on (nearly) all servers

The closed nature of AMT has made many security researchers concerned and it was probably the #2 security issue recently after the Spectre class of vulnerabilities.

So its relatively obvious to see that ‘remote management’ is something that needs to be very tightly controlled, it is inherently insecure otherwise.

Another concern, the BMC, the Intel AMT… They each run their own small operating systems in custom processors. These are present in the Ethernet chip, in the WiFi chip, in the BMC module, etc. So many companies who are running anti-virus and scanning tools in their x86 space are missing a big part of the picture.

Yet another concern, the Intel System Management Mode (SMM) is a part of the system which is walled off from the main processor code. It runs a type of hypervisor that can affect/interact with your operating system, and yet be invisible to it. Recently some attacks have used this secure enclave itself as a vector.

Stirring the pot, the upgrade path of all of this miscellaneous firmware is incredibly insecure. Many vendors have an FTP server (which is susceptible to man-in-the-middle attacks as well as having the password in the clear) or HTTP (same problems). This means that you might think you are downloading a patch from your vendor, but it really comes from someone malicious. This injection is relatively simple, one can either interfere with DNS (poisoning etc), re-route packets (e.g. ARP poison), use a race condition (e.g. respond in parallel with a 30x redirect), or be inline and inject a 30x redirect or send the malicious content itself. Note this is not just theoretical, there have been articles written on this occurring in practise to inject spyware to journalists in Turkey. Compounding this, the firmware is not signed (e.g. PGP), so you cannot tell if it valid, and the devices will typically load anything presented.

Now, the responses. Amazon has written a fairly firm response to Bloomberg. Apple likewise has. Facebook emailed a response. These seem to confirm the malicious firmware via portal and deny the ‘BMC hack chip’.

Now, what is a ‘BMC hack tiny chip’? Most likely this is an I2C or SPI flash which has new/different firmware. This is the typical device in the form-factor shown, and is a type of flash memory commonly used for this purpose (as well as being a small number of pins to wire in). How would that ‘call home’? Well, it can inject code into the main processor to run (via Direct Memory Access or DMA), or it can usually directly drive the network chip. So this brings up how one protects against those. In the first case, this would be very challenging. That Intel SGX attack above, as the SMM, the main processor cannot see this memory. So no amount of UEFI secure-boot signing will protect. So the best protection would be an ‘egress firewall’. This is a concept that is common yet unknown: a firewall to intercept the traffic that *leaves* your server outbound.

In many cases, the BMC network is a separate physical interface. So it can be either unconnected, or connected to a strictly policed network. However, some servers (including the Supermicro I have at home) there is a type of bridge inside, so the main network ports are both for the server’s use and for the BMC. This can be very tricky to police. You can try to match by MAC address the BMC, but, well, that is changeable. You can try and match by application type, but then you need an exhaustive dictionary of where the server will talk to, and what protocols.

So. If we look at the diagram above. If we wanted to ensure we had some code running on the BMC that the user wouldn’t update by accident, and didn’t want to count on hacking the portal path to entice them to download it, yes, soldering in a new flash in the chain would be doable and hard to detect. And, yes, some companies would have the BMC able to talk outbound to the Internet (in the same way all the devices in your home are able to connect outbound). If code can talk out to the Internet you are not protected by your inbound firewall (think about your Nest thermostat…. you can control the temperature from anywhere and you didn’t alter your firewall… it does this by maintaining a persistent outbound connection to something that is open for inbound connections, something in the cloud).

Now, would someone have caught this on some audit? There are a lot of packets on a network, its hard to see them all. If you block outbound traffic, and you see something trying where you don’t expect, then you would notice that. If you don’t block outbound traffic, I guess some flows to external IP might eventually get noticed. So this is where ‘covert exfiltration’ comes in. Think about when you were a kid. Your mom was making cookies. You need to get in, grab one, and go, before she noticed. If you came running in staring at the cooling rack you got busted. But if you maybe put it in your pocket you could get away with it. The same is kind of true of Internet. There are channels that are allowed that can carry additional data. Maybe DNS. Would someone notice a few DNS lookups to xxxx.FOOBAR.com? The host name itself can carry the data (IP over DNS). Maybe it could look like normal Anti-Virus updates? This field is called ‘steganography’, hiding one thing in another.

So back to the story. From a technical standpoint, yes, remote management would be fertile ground with lots of feasible avenues to get remote access, control, surveillance. From a commercial standpoint, cloud computing is very valuable to get control of, since you then get access to the tenants that run on it. Some will be more interesting than others.

Did it happen? We have strong players (Bloomberg, Amazon) saying opposite things. Hmmm.