In this blog post we are going to introduce Multi-cluster Management patterns with GitOps and how you can implement these patterns on OpenShift.
If you’re interested in diving into an interactive tutorial, try this link.
In the introductory blog post to GitOps we described some of the use cases that we can solve with GitOps on OpenShift. In
today’s blog post we are going to describe how we can leverage GitOps patterns to perform tasks on multiple clusters.
We are going to explore the following use cases:
Deploy an application to multiple clusters
Customize the application by cluster
Perform a canary deployment
During this blog post we are not going to cover advanced GitOps workflows, instead we are going to show you basic capabilities around
the topic. More advanced posts around GitOps workflows will follow.
Environment
Two OpenShift 4.1 clusters, one for preproduction (context: pre) environment and one for production (context: pro) environment.
ArgoCD used as the GitOps tool
Demo files here
Deploy an Application to Multiple Clusters
In this first example, we are going to deploy our base application to both clusters.
As we are using ArgoCD as our GitOps tool an ArgoCD Server is already deployed in our environment as well as the argocd cli tool.
Our application definition can be found here
Ensure we have access to both clusters
$ oc –context pre get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-128-17.ap-southeast-1.compute.internal Ready master 19h v1.13.4+ab8449285
ip-10-0-136-41.ap-southeast-1.compute.internal Ready worker 19h v1.13.4+ab8449285
ip-10-0-151-90.ap-southeast-1.compute.internal Ready worker 19h v1.13.4+ab8449285
$ oc –context pro get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-140-239.ap-southeast-1.compute.internal Ready master 19h v1.13.4+ab8449285
ip-10-0-142-57.ap-southeast-1.compute.internal Ready worker 19h v1.13.4+ab8449285
ip-10-0-170-168.ap-southeast-1.compute.internal Ready worker 19h v1.13.4+ab8449285
Ensure we have our clusters registered in ArgoCD
$ argocd cluster list
SERVER NAME STATUS MESSAGE
https://api.openshift.pre.example.com:6443 pre Successful
https://api.openshift.pro.example.com:6443 pro Successful
https://kubernetes.default.svc Successful
Add our GitOps repository to ArgoCD
$ argocd repo add https://github.com/mvazquezc/gitops-demo.git
repository ‘https://github.com/mvazquezc/gitops-demo.git’ added
Deploy our application to preproduction and production clusters
# Create the application on Preproduction cluster
$ argocd app create –project default –name pre-reversewords –repo https://github.com/mvazquezc/gitops-demo.git –path reversewords_app/base –dest-server https://api.openshift.pre.example.com:6443 –dest-namespace reverse-words –revision pre
application ‘pre-reversewords’ created
# Create the application on Production cluster
$ argocd app create –project default –name pro-reversewords –repo https://github.com/mvazquezc/gitops-demo.git –path reversewords_app/base –dest-server https://api.openshift.pro.example.com:6443 –dest-namespace reverse-words –revision pro
application ‘pro-reversewords’ created
4.1 Above commands create a new ArgoCD Application named pre-reversewords and pro-reversewords that will be deployed on preproduction and production clusters in reverse-words namespace using the code from pre/pro branch located under path reversewords_app/base
As we haven’t defined a sync policy, we need to force ArgoCD to sync the Git repo content on our pre and pro clusters
$ argocd app sync pre-reversewords
$ argocd app sync pro-reversewords
After a few seconds we will see our application deployed on pre and pro clusters
# Get application status on preproduction cluster
$ argocd app get pre-reversewords
Name: pre-reversewords
Project: default
Server: https://api.openshift.pre.example.com:6443
Namespace: reverse-words
URL: https://argocd.apps.example.com/applications/pre-reversewords
Repo: https://github.com/mvazquezc/gitops-demo.git
Target: pre
Path: reversewords_app/base
Sync Policy: <none>
Sync Status: Synced to pre (306ce10)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH
Namespace reverse-words Synced
Service reverse-words reverse-words Synced Healthy
apps Deployment reverse-words reverse-words Synced Healthy
# Get application status on production cluster
$ argocd app get pro-reversewords
Name: pro-reversewords
Project: default
Server: https://api.openshift.pro.example.com:6443
Namespace: reverse-words
URL: https://argocd.apps.example.com/applications/pro-reversewords
Repo: https://github.com/mvazquezc/gitops-demo.git
Target: pro
Path: reversewords_app/base
Sync Policy: <none>
Sync Status: Synced to pro (98bbfb1)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH
Namespace reverse-words Synced
Service reverse-words reverse-words Synced Healthy
apps Deployment reverse-words reverse-words Synced Healthy
Our application defines a service for accessing its API, let’s try to access and get the release name for both clusters
# Get the preproduction cluster LB hostname
$ PRE_LB_HOSTNAME=$(oc –context pre -n reverse-words get svc reverse-words -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’)
# Get the production cluster LB hostname
$ PRO_LB_HOSTNAME=$(oc –context pro -n reverse-words get svc reverse-words -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’)
# Access the preproduccion LB and get the release name
$ curl http://${PRE_LB_HOSTNAME}:8080
Reverse Words Release: Base release. App version: v0.0.2
# Access the production LB and get the release name
$ curl http://${PRO_LB_HOSTNAME}:8080
Reverse Words Release: Base release. App version: v0.0.2
As you have seen, we have been able to deploy to multiple clusters from a single tool (ArgoCD). In the next section we are going to explore how we can override some configurations depending on the destination cluster by using embedded Kustomize on ArgoCD.
Customize the Application by Cluster
In this second example, we are going to modify the application behavior depending on which cluster is deployed.
We want the application to have a release name preproduction or production depending on which environment the application gets deployed on.
ArgoCD leverages Kustomize under the hood to deal with configuration overrides across environments.
The way we organize our application in Git is as follows:
The Git Repository has two branches, pre which has manifests for preproduction env, and pro for production env.
Application overrides can be found in their respective folders and branch:
Preproduction cluster overrides
Production cluster overrides
We placed the application overrides in the Git repository, there is only one override that configures a release name different than the default based on the cluster the application gets deployed
Deploy our Kustomized application to preproduction and production clusters
# Create the application on Preproduction cluster
argocd app create –project default –name pre-kustomize-reversewords –repo https://github.com/mvazquezc/gitops-demo.git –path reversewords_app/overlays/pre –dest-server https://api.openshift.pre.example.com:6443 –dest-namespace reverse-words –revision pre –sync-policy automated
application ‘pre-kustomize-reversewords’ created
# Create the application on Production cluster
argocd app create –project default –name pro-kustomize-reversewords –repo https://github.com/mvazquezc/gitops-demo.git –path reversewords_app/overlays/pro –dest-server https://api.openshift.pro.example.com:6443 –dest-namespace reverse-words –revision pro –sync-policy automated
application ‘pro-kustomize-reversewords’ created
2.1 Above commands create a new ArgoCD Application named pre-kustomize-reversewords and pro-kustomize-reversewords that will be deployed on preproduction and production clusters in reverse-words namespace using the code from pre and pro branch respectively. Each application will get the code from a different folder in our overlays folder, that way the application will be customized depending on which environment it gets deployed on. Note that only the modified values are stored in the overlay folder, the base application is still deployed from the base folder, so we don’t end up having duplicate application files.
As we have defined an automated sync policy we don’t need to force the sync, ArgoCD will start synching our application once it gets created. On top of that, if changes were made to the application repository, ArgoCD would re-deploy the changes for us.
After a few seconds we will see our application deployed on pre cluster
# Get application status on preproduction cluster
$ argocd app get pre-kustomize-reversewords
Name: pre-kustomize-reversewords
Project: default
Server: https://api.openshift.pre.example.com:6443
Namespace: reverse-words
URL: https://argocd.apps.example.com/applications/pre-kustomize-reversewords
Repo: https://github.com/mvazquezc/gitops-demo.git
Target: pre
Path: reversewords_app/overlays/pre
Sync Policy: Automated
Sync Status: Synced to pre (306ce10)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH
Namespace reverse-words Synced
Service reverse-words reverse-words Synced Healthy
apps Deployment reverse-words reverse-words Synced Healthy
# Get application status on production cluster
$ argocd app get pro-kustomize-reversewords
Name: pro-kustomize-reversewords
Project: default
Server: https://api.openshift.pro.example.com:6443
Namespace: reverse-words
URL: https://argocd.apps.example.com/applications/pro-kustomize-reversewords
Repo: https://github.com/mvazquezc/gitops-demo.git
Target: pro
Path: reversewords_app/overlays/pro
Sync Policy: Automated
Sync Status: Synced to pro (98bbfb1)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH
Namespace reverse-words Synced
Service reverse-words reverse-words Synced Healthy
apps Deployment reverse-words reverse-words Synced Healthy
Our application defines a service for accessing its API, let’s try to access and get the release name for both clusters
# Get the preproduction cluster LB hostname
$ PRE_LB_HOSTNAME=$(oc –context pre -n reverse-words get svc reverse-words -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’)
# Get the production cluster LB hostname
$ PRO_LB_HOSTNAME=$(oc –context pro -n reverse-words get svc reverse-words -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’)
# Access the preproduccion LB and get the release name
$ curl http://${PRE_LB_HOSTNAME}:8080
Reverse Words Release: Preproduction release. App version: v0.0.2
# Access the production LB and get the release name
$ curl http://${PRO_LB_HOSTNAME}:8080
Reverse Words Release: Production release. App version: v0.0.2
As you have seen, we have been able to deploy to multiple clusters and use custom configurations depending on which cluster we are using to deploy the application. In the next section we are going to explore how we can use GitOps to perform a basic canary deployment.
Perform a Canary Deployment
A common practice is to deploy a new version of an application to a small subset of the available clusters, and once the application has been proven to work as expected, then it gets promoted to the rest of the clusters.
We are going to use the Kustomized apps that we created before, let’s verify which versions are we running:
# Get the preproduction cluster LB hostname
$ PRE_LB_HOSTNAME=$(oc –context pre -n reverse-words get svc reverse-words -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’)
# Get the production cluster LB hostname
$ PRO_LB_HOSTNAME=$(oc –context pro -n reverse-words get svc reverse-words -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’)
# Access the preproduccion LB and get the release name
$ curl http://${PRE_LB_HOSTNAME}:8080
Reverse Words Release: Preproduction release. App version: v0.0.2
# Access the production LB and get the release name
$ curl http://${PRO_LB_HOSTNAME}:8080
Reverse Words Release: Production release. App version: v0.0.2
As you can see the current deployed version is v0.0.2, let’s perform a canary deployment to v0.0.3.
We need to update the container image that will be used on preproduction cluster, we are going to modify the Deployment overlay as follows:
# reversewords_app/overlays/pre/deployment.yaml in git branch pre
apiVersion: apps/v1
kind: Deployment
metadata:
name: reverse-words
labels:
app: reverse-words
spec:
template:
spec:
containers:
– name: reverse-words
image: quay.io/mavazque/reversewords:v0.0.3
env:
– name: RELEASE
value: “Preproduction release”
– $patch: replace
We send our changes to the git repository
git add reversewords_app/overlays/pre/deployment.yaml
git commit -m “Updated preproduction image version from v0.0.2 to v0.0.3″
git push origin pre
ArgoCD will detect the update in our code and will deploy the new changes, now we should see the version v0.0.3 deployed on pre and the version v0.0.2 deployed on pro.
# Access the preproduccion LB and get the release name
$ curl http://${PRE_LB_HOSTNAME}:8080
Reverse Words Release: Preproduction release. App version: v0.0.3
# Access the production LB and get the release name
$ curl http://${PRO_LB_HOSTNAME}:8080
Reverse Words Release: Production release. App version: v0.0.2
Let’s verify that our application is working as expected
$ curl http://${PRE_LB_HOSTNAME}:8080 -X POST -d ‘{“word”:”PALC”}’
{“reverse_word”:”CLAP”}
The application is working fine, now it’s time to update production to v0.0.3 as well. Let’s update the overlay:
# reversewords_app/overlays/pro/deployment.yaml in git branch pro
apiVersion: apps/v1
kind: Deployment
metadata:
name: reverse-words
labels:
app: reverse-words
spec:
template:
spec:
containers:
– name: reverse-words
image: quay.io/mavazque/reversewords:v0.0.3
env:
– name: RELEASE
value: “Production release”
– $patch: replace
Send the changes to Git
git add reversewords_app/overlays/pro/deployment.yaml
git commit -m “Updated production image version from v0.0.2 to v0.0.3″
git push origin pro
Get versions in use
# Access the preproduccion LB and get the release name
$ curl http://${PRE_LB_HOSTNAME}:8080
Reverse Words Release: Preproduction release. App version: v0.0.3
# Access the production LB and get the release name
$ curl http://${PRO_LB_HOSTNAME}:8080
Reverse Words Release: Production release. App version: v0.0.3
We should now update the base deployment so newer deployments use v0.0.3 version
Final Thoughts
We have updated our application by modifying the application overlays in Git, this is a very basic scenario, advanced scenarios may include CI tests, multiple approvals, etc.
We have pushed our code to the pre/pro branches directly, that is not a good practice, in a real life scenario a more advanced workflow should be used. We will discuss GitOps workflows in future blog posts.
We have ArgoCD Cli, ArgoCD has a WebUI where you can do almost the same operations as with the cli, on top of that you can visualize your applications and its components.
Next Steps
In future blog posts we will talk about multiple topics related to GitOps such as:
GitOps Workflows in Production
Disaster Recovery with GitOps
Moving to GitOps
The post Multi-cluster Management with GitOps appeared first on Red Hat OpenShift Blog.
Quelle: OpenShift
