Deploy a Production Ready Kubernetes Cluster With lxc Container and Kubespray

I am going to show the workground how you can use lxc container to create a production grade cluster. Though its hard to create kubernetes cluster with lxc container but its possible. So lets see how we can solve all of those challenges step by step

Step 1: Prepare host machine

a) edit following file

nano /etc/sysctl.conf
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

b) disable firewall 

ufw disable

c) disable swap

swapoff -a; sed -i '/swap/d' /etc/fstab

d) update sysctl settings for kubernetes networking

cat >>/etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

Step 2: Create lxc profile

config:
  boot.autostart: "true"
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,br_netfilter,nf_conntrack,xt_conntrack
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw cgroup:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: microk8s
used_by:
- /1.0/instances/node1
- /1.0/instances/node2
- /1.0/instances/node3

Step 3: Create a linux container 

lxc launch -p default -p microk8s ubuntu:21.04 node1

Step 4: Inside container do following 

a) following command should return output

conntrack -L
modinfo overlay

b) if above command output error then its seems that there some karnel related problem. Install and fix karnel issue

sudo apt install linux-generic
sudo apt install --reinstall linux-image-$(uname -r);
sudo apt install --reinstall linux-modules-$(uname -r);
sudo apt install --reinstall linux-modules-extra-$(uname -r);

this should fix karnel related issue.

c) Recent kubernetes versions want to read from /dev/kmsg which is not present in the container. You need to instruct systemd to always create a symlink to /dev/console instead:

echo 'L /dev/kmsg - - - - /dev/null' > /etc/tmpfiles.d/kmsg.conf

if it not working then run following

echo 'L /dev/kmsg - - - - /dev/console' > /etc/tmpfiles.d/kmsg.conf

if it still not work then do following

# Hack required to provision K8s v1.15+ in LXC containers
mknod /dev/kmsg c 1 11
echo 'mknod /dev/kmsg c 1 11' >> /etc/rc.local
chmod +x /etc/rc.local

d) if you need to load any module then you can run following comman

# cmd    module name
------------------------
modprobe br_netfilter

Thats all. Now follow the Kubespray official document.

To access k8s cluster without execing into master node

Download the kubectl command into your local.

which kubectl
# output: /usr/bin/kubectl

Create .kube directory

mkdir ~/.kube

copy config from kmaster into .kube directory

lxc file pull kmaster/etc/kubernetes/admin.conf ~/.kube/config

#check cluster
kubectl get nodes