Using Azure Container Instances to expand CI?

So Azure has a ‘serverless’ kubelet concept. In a nutshell we follow virtual-kubelet instructions  (except they were missing az provider register –namespace ‘Microsoft.ContainerInstance’, pull request sent).

What this does is schedule Pods (which have a special annotation) to a farm of servers which are willing to accept ‘foreign’ containers (Pods). This means your Kubernetes master delegates work to a shared Node. What this means for security, well, lets chat about that another day. But what it means for horizontal scale is Good Things(tm). We can now share the pool of many servers rather than the small number of virtual machines we have pressed into service as Nodes in our own K8s cluster.

So, when I run this:

kind: Pod
metadata:
  name: u
spec:
  containers:
  - image: ubuntu:18.04
    command: ['sh', '-c', 'sleep 3600']
    imagePullPolicy: Always
    name: u
    resources:
      requests:
        memory: 4G
        cpu: 2
  dnsPolicy: ClusterFirst
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
  - key: azure.com/aci
    effect: NoSchedule

It runs ‘somewhere’, a container floating in the universe with no specific host to call home. Where it lands, nobody knows, who its beside on that server, nobody knows.

Now, is this good enough for me with my Gitlab Runner CI?

In short, not really.

1. Max size is 4 CPU, 14GB ram (https://docs.microsoft.com/en-us/azure/container-instances/container-instances-quotas#region-availability)
2. It is not available in Canada. Closest is US-EAST. So this would be an issue for people needing Data Sovereignty.
3. I would have to change Gitlab Runner to add these tolerations (indulgences?)

But, I think its a step in the right direction. I wonder if we will see a Google ‘raw container pool’ engine, given that Azure has ACI and AWS has Fargate?

A little snooping on the remote container in dmesg. Its interesting, there is the full boot sequence, then a large gap, then a line about my interface coming up:

[ 72.832894] hv_balloon: Received INFO_TYPE_MAX_PAGE_CNT
[ 72.832960] hv_balloon: Data Size is 8
[104939.085094] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[104939.104792] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[104939.105115] device veth32524353 entered promiscuous mode
[104939.105197] cbr0: port 1(veth32524353) entered forwarding state

How is this achieved? We can see that the host is running Ubuntu 16.04, that it has an uptime of 1 day, 5:13 hours. So it is not a VM created just for me.

We can see that the host is itself running on Microsoft HyperV (dmesg rats it out).

We can see we are vulnerable to some spectre + meltdown madness:

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
* Indirect Branch Restricted Speculation (IBRS)
* SPEC_CTRL MSR is available: UNKNOWN (is msr kernel module available?)
* CPU indicates IBRS capability: UNKNOWN (is cpuid kernel module available?)
* Indirect Branch Prediction Barrier (IBPB)
* PRED_CMD MSR is available: UNKNOWN (is msr kernel module available?)
* CPU indicates IBPB capability: UNKNOWN (is cpuid kernel module available?)
* Single Thread Indirect Branch Predictors (STIBP)
* SPEC_CTRL MSR is available: UNKNOWN (is msr kernel module available?)
* CPU indicates STIBP capability: UNKNOWN (is cpuid kernel module available?)
* Speculative Store Bypass Disable (SSBD)
* CPU indicates SSBD capability: NO
* L1 data cache invalidation
* FLUSH_CMD MSR is available: UNKNOWN (is msr kernel module available?)
* Enhanced IBRS (IBRS_ALL)
* CPU indicates ARCH_CAPABILITIES MSR availability: UNKNOWN (is cpuid kernel module available?)
* ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: UNKNOWN
* CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO): UNKNOWN
* CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): UNKNOWN
* Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): UNKNOWN
* CPU microcode is known to cause stability problems: NO (model 0x3f family 0x6 stepping 0x2 ucode 0xffffffff cpuid 0x0)
* CPU microcode is the latest known available version: UNKNOWN (latest microcode version for your CPU model is unknown)
* CPU vulnerability to the speculative execution attack variants
* Vulnerable to Variant 1: YES
* Vulnerable to Variant 2: YES
* Vulnerable to Variant 3: YES
* Vulnerable to Variant 3a: YES
* Vulnerable to Variant 4: YES
* Vulnerable to Variant l1tf: YES

CVE-2017-5753 [bounds check bypass] aka ‘Spectre Variant 1’
* Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec: UNKNOWN (couldn’t check (couldn’t find your kernel image in /boot, if you used netboot, this is normal))
* Kernel has the Red Hat/Ubuntu patch: UNKNOWN (missing ‘strings’ tool, please install it, usually it’s in the binutils package)
* Kernel has mask_nospec64 (arm64): UNKNOWN (couldn’t check (couldn’t find your kernel image in /boot, if you used netboot, this is normal))
* Checking count of LFENCE instructions following a jump in kernel… UNKNOWN (couldn’t check (couldn’t find your kernel image in /boot, if you used netboot, this is normal))
> STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization)

CVE-2017-5715 [branch target injection] aka ‘Spectre Variant 2’
* Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline)
* Mitigation 1
* Kernel is compiled with IBRS support: YES
* IBRS enabled and active: NO
* Kernel is compiled with IBPB support: YES
* IBPB enabled and active: NO
* Mitigation 2
* Kernel has branch predictor hardening (arm): NO
* Kernel compiled with retpoline option: UNKNOWN (couldn’t read your kernel configuration)
> STATUS: VULNERABLE (IBRS+IBPB or retpoline+IBPB is needed to mitigate the vulnerability)

CVE-2017-5754 [rogue data cache load] aka ‘Meltdown’ aka ‘Variant 3’
* Mitigated according to the /sys interface: YES (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI): NO
* PTI enabled and active: YES
* Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU: NO
> STATUS: NOT VULNERABLE (Mitigation: PTI)

CVE-2018-3640 [rogue system register read] aka ‘Variant 3a’
* CPU microcode mitigates the vulnerability: NO
> STATUS: VULNERABLE (an up-to-date CPU microcode is needed to mitigate this vulnerability)

CVE-2018-3639 [speculative store bypass] aka ‘Variant 4’
* Mitigated according to the /sys interface: NO (Vulnerable)
* Kernel supports speculation store bypass: YES (found in /proc/self/status)
> STATUS: VULNERABLE (Your CPU doesn’t support SSBD)

CVE-2018-3615/3620/3646 [L1 terminal fault] aka ‘Foreshadow & Foreshadow-NG’
* Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion)
> STATUS: NOT VULNERABLE (Mitigation: PTE Inversion)