Running magnum (Kubernetes orchestration) on OpenStack Queens: lxd to the rescue

So the home OpenStack system is running Queens on Ubuntu 18.04, courtesy of Kolla. Great. The all-NVME Ceph I talked about in the previous post is kicking ass and taking names for glance/nova/cinder. Now, lets try some container orchestration and install Kubernetes via Magnum. Make sure to use Fedora-Atomic and not coreos because of this. But also because, well, RedHat has layed out their plans for CoreOS post-acquisition.

So we simply run:

openstack coe cluster create k8s --cluster-template k8s-atomic --node-count 3 --master-count 1

and we are done, right? Not so fast. It seems the

openstack coe cluster config k8s

command, which fetches the kubectl config file has some issues w/ the mandatory RBAC if you are using magnumclient < 2.9.0. And I have 2.8.0. Hmm. Well, that’s ok, we got this. Lets make a lxd image to run sandboxed but transparent.

So the final recipe was:

Step 1. Setup magnum, create a Kubernetes cluster

openstack image create --min-disk 6 --disk-format raw --container-format bare --public --property os_type=linux --property os_distro='fedora-atomic' --file fedora-atomic-latest.raw fedora-atomic
openstack coe cluster template create k8s-atomic --image fedora-atomic --keypair default --external-network public --dns-nameserver 172.16.0.1 --flavor m1.small --docker-storage-driver overlay2 --volume-driver cinder --network-driver flannel --coe kubernetes
openstack coe cluster create k8s --cluster-template k8s-atomic --node-count 3 --master-count 1          

Step 2. Create a lxd image that we can use transparently with all the config in it and a 2.9.0+ magnumclient. For convenience, I change the username in it to mine, but this is not really necessary (that is the 3 sed lines)

lxc launch ubuntu:18.04 os
lxc exec os -- sed -i -e "s?/home/ubuntu?~?g" -e "s?ubuntu?$(id -nu)?" /etc/passwd
lxc exec os -- sed -i -e "s?ubuntu?$(id -nu)?" /etc/group
lxc exec os -- sed -i -e "s?ubuntu?$(id -nu)?" /etc/shadow
lxc config device add os home disk path=~ source=~
lxc config set os raw.idmap "both $(id -u) $(id -u)"
lxc restart os

Now we have an image called ‘os’ which maps our home dir, runs as our user-id otherwise unprivileged. If we were to run:

lxc exec os -- sudo -H --login --user $(id -nu) bash

We would find ourselves in a bare Ubuntu 18.04 image, with our home dir mounted.

Step 3. One-time setup in container. Install the openstack clients (pip), install kubectl (curl > bin), create an env file.

(side note: is anyone else disturbed by this trend of curl | bash? Its the official instructions for lots of things, curl | kubectl -f – , curl > /usr/bin; curl | bash… Installing pip, calico, kubectl, … you name it. Comments?)

lxc exec os bash
apt update
apt -y install python3-pip curl
for i in openstack magnum nova heat glance cinder neutron
do
 pip3 install python-${i}client
done

cd /root
curl -o /usr/local/bin/kubectl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod a+rx /usr/local/bin/kubectl 
curl -o /tmp/helm.tar.gz https://storage.googleapis.com/kubernetes-helm/helm-v2.9.0-linux-amd64.tar.gz
tar zxf /tmp/helm.tar.gz
mv linux-amd64/helm /usr/local/bin
chmod a=rx /usr/local/bin/helm

mkdir -p /k8s
cd /k8s

cat << EOF > env
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=XXXX
export OS_AUTH_URL=http://XXXX:35357/v3
export OS_INTERFACE=internal
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password
export KUBECONFIG=/k8s/config
EOF
. ./env
openstack coe cluster config k8s 

chown -R 1000:1000 /k8s

OK, that was a mouthful, be we are done, honest. Now we can run this anytime we want, and have a bash-shell w/ the env vars loaded, ready to run any openstack command or kubernetes command, with our home dir mounted:

alias os='lxc exec os -- sudo -H --login --user $(id -n) bash --rcfile /k8s/env'

Neat?

Looking for an update on Outdoor Kitty instead?  Well, that is him on the right today. As the temperature has risen, his interest in me has dropped. I’m still allowed to feed him, but we are back to a 2-5m relationship.