A small talk with the Kommander about Cattle AWS vs. Cattle EKSThe KommanderI met this lovely captain recently on Kos Island and had a small talk with her/him regarding our Kubernetes projects. I spoke to her/him about a dilemma which we’re facing these days. This write up is a very short abstract of our conversation, which I’d like to share with you.In the meanwhile our lovely captain has joined us as “The Kommander” and has the code name TK8.We’re delighted to welcome “The Kommander” as a Kubernaut to the Kubernauts community together with you!The dilemmaI spoke to the Kommander about a dilemma by deciding on how to run and manage multiple Kubernetes clusters using OpenShift, RKE, EKS or Kubeadm (w/ or w/o ClusterAPI support) on AWS for a wide range of applications and services in the e-Mobility field with the following requirements:We should be able to have AWS spot instances with auto-scaling support or other choices like using reserved instances to minimize our costs for our highly scalable apps with high peaks in productionWe should be able to avoid licensing costs for an enterprise grade k8s subscription in the first phaseWe should be able to have a great community support and contribute our work to the communityWe should be able to buy enterprise support at any timeWe should be able to deploy Kubernetes through a declarative approach using Terraform and a single config.yaml manifest the GitOps wayWe should be able to upgrade our clusters easily without downtimeWe should be able to recover our clusters from disaster in regions within few hoursWe need to manage all of our clusters and users through a single interfaceWe should be able to use SAML for SSOWe need to address day 2 operation needsWe should be able to move our workloads to other cloud providers like Google, Azure, DigitalOcean, Hertzner, etc. within few days with the same Kubernetes versionWith these requirements in mind, we had to decide to go either with EKS or Rancher’s RKE and use Rancher Server to manage all of our clusters and users.OpenShiftThe main reason not to be able to go with OpenShift was the fact that OpenShift does not support any hosted Kubernetes provider like EKS and needs high cost licensing in the first phase and has support only for reserved instances and not spot instances on AWS. And OKD, the open source version of OpenShift is not available at this time of writing and one can’t switch from OKD to OCP seamlessly and get enterprise support later. And the fact that OpenShift is not OS agnostic, is something which we don’t like so much. But what we like about OpenShift is the self-hosted capability with Operator Framework support and it’s CRI-O integration.KubeadmUsing Vanilla Kubeadm with ClusterAPI is something which we are looking into for about 9 months now and we love it very much. We were thinking of going with Kubeadm and use Rancher to manage our clusters, but with Kubeadm itself at this time, there is no enterprise support option available and we’re not sure how spot instances and auto-scaling support does work with Kubeadm at this time. What we like very much about Kubeadm is the fact that it has support for all container runtimes, docker, containerd and cri-o and the self-hosted capability is coming soon.EKSEKS is still one of our favourites on the list and we have managed to automate EKS deployments with TK8 Cattle EKS Provisioner and Terraform Provider Rancher2, but unfortunately at this time of writing without auto-scaling and spot instances support, since Rancher API doesn’t provide spot-instances support at this time and we hope to get it soon by one of the next Rancher 2.3.x releases.EKS itself provides spot-instances support and we were thinking about using the eksctl tool to deploy EKS clusters with spot-instances and auto-scaling support and then import our EKS clusters into Rancher and implement IAM and SSO integration and handle upgrades through eksctl. But with that we have to deal with 2 tools, eksctl and Rancher and we are not sure how EKS upgrades will affect our Rancher Server, since Rancher can’t deal with EKS upgrades if we deploy EKS with eksctl. But I think this should not be a no-go issue for us at this time.The main reason why we’d love to go with EKS is the fact that we’ll get a full managed control plane and we have to deal only with managing and patching our worker nodes.The down side of EKS is, that we have to go often with an older version of Kubernetes and if we want to move our workloads to other cloud providers, this might become an issue. And for sure vendor lock-in is something which we are concerned with.RancherUsing Rancher with Terraform Provider Rancher2 and TK8 Cattle AWS Provisioner to deploy RKE clusters with spot instances support with tk8ctl on AWS is something which we are thinking to go with most probably at this time, despite the fact and dilemma that we have to manage our control plane along with the stacked etcd nodes on our own.But with this last option we get a full range of benefits through Rancher and can move our RKE clusters with the same Kubernetes version to any cloud provider and deal with upgrades and our day2 operation needs with a single tool.Other products like RancherOS, Longhorn, Submariner, k3s and k3sos and the great community traction and support on slack gives us the peace of mind to go with Rancher, either with or without EKS!After these explanations I got this nice feedback from the Kommander which I wanted to share with you and thank you for your time reading this post :-)Thanks God, it’s Rancher!Try itIf you’d like to learn about TK8 and how it can help you to build production ready Kubernetes clusters with Terraform Provider Rancher2, please refer to the following links under the related resources.Questions?Feel free to join us on kubernauts slack and ask any questions in #tk8 channel.Related resourcesTK8: The KommanderTK8 Cattle AWS ProvisionerTK8 Cattle EKS ProvisionerA Buyer’s Guide to Enterprise Kubernetes Management PlatformsCreditsMy special thanks goes to my awesome colleague Shantanu Deshpande who worked in his spare time on TK8 Cattle AWS and EKS development and for sure to the brilliant team by Rancher Labs and the whole Rancher Kommunity!We’re hiring!We are looking for engineers who love to work in Open Source communities like Kubernetes, Rancher, Docker, etc.If you wish to work on such projects please do visit our job offerings page.Thanks God, it’s Rancher! was originally published in Kubernauts on Medium, where people are continuing the conversation by highlighting and responding to this story.
Quelle: blog.kubernauts.io
