Home servers: free can be expensive, TSC scaling

So last night i built a new machine for @ home. And I learned a few things I thought i would share.

First, there is a motherboard form-factor called EE-ATX. That first E matters. Its huge. I bought the biggest case (Thermaltake X9, about the size of a bar-fridge) on the theory that this should be simple to install the Supermicro X10DRi-LN4+ motherboard. It was not, I had to take a hack-saw to the case, and then drill and tap for the the #6-32 standoffs all the holes. Ugh. But, finished (or is it? read on!) work below.

OK, that done, installed both processors (2699 v3, 2696 v3) and the 8 x 32GB DDR4 ram. Should be done now right? Just leave it run some stress test over night to check temperatures etc.

OK, that was good. Now a quick Ubuntu 16.04 install and we should have more home VM horsepower (72 threads, 256GB ram, should be a good test bench).

Hmm. When it boots it gives this mystery message about CPU consistency. Huh, that’s weird. I’ve checked a bunch of times, the 2696 and 2699 are the same processor (one is the OEM tray SKU, one is the single-package retail SKU). They have the same stepping ID, same # cores, cache, frequency all is good. And it does work, just not w/ kvm-intel.ko (boo, its the whole purpose of having this).

OK, so use the source. Dig a bit, add some debugging to the kernel, and lo and behold one of them (the 2696) has an extended vmcs flag (tsc scaling). TSC scaling is designed so that you can hot-migrate a VM from one physical machine to another, where the 2nd one has a different clock rate. I don’t need that.

So, I introduce this:

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 5382b82..5ace575 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -103,6 +103,9 @@ module_param_named(enable_shadow_vmcs, enable_shadow_vmcs, bool, S_IRUGO);
 static bool __read_mostly nested = 0;
 module_param(nested, bool, S_IRUGO);
 
+static bool __read_mostly enable_tsc_scaling = true;
+module_param(enable_tsc_scaling, bool, S_IRUGO);
+
 static u64 __read_mostly host_xss;
 
 static bool __read_mostly enable_pml = 1;
@@ -3449,6 +3452,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
 vmcs_conf->pin_based_exec_ctrl = _pin_based_exec_control;
 vmcs_conf->cpu_based_exec_ctrl = _cpu_based_exec_control;
 vmcs_conf->cpu_based_2nd_exec_ctrl = _cpu_based_2nd_exec_control;
+ if (!enable_tsc_scaling)
+ vmcs_conf->cpu_based_2nd_exec_ctrl &= ~SECONDARY_EXEC_TSC_SCALING;
 vmcs_conf->vmexit_ctrl = _vmexit_control;
 vmcs_conf->vmentry_ctrl = _vmentry_control;

And boom, we are solved.  Just change the command line to:

GRUB_CMDLINE_LINUX="nomodeset kvm-intel.nested=1 kvm-intel.enable_tsc_scaling=0"

and I’m good to go.

But this got me to thinking. If i had 2 very similar processors, would I prefer to have a non-shared feature just masked off on the better one? or have nothing work? I think they made a mistake here.

Now on to the next problem (the stupid UPS I bought for it has a L5-30P twist-lock connector). So i’ve acquired the L5-30R receptacle and 10m of #10-3 armoured cable and 30A single-pole breaker. So I guess Sunday morning will be spent fishing a new power cable.

Nothing is ever simple!