Kategorie: technology

OpenShift 4.2 Disconnected Install

In a previous blog, it was announced that Red Hat is making the OpenShift nightly builds available to everyone. This gives users a chance to test upcoming features before their general availability. One of the features planned for OpenShift 4.2 is the ability to perform a “disconnected” or “air gapped” install, allowing you to install in an environment without access to the Internet or outside world.
NOTE: that nightly builds are unsupported and are for testing purposes only!
In this blog I will be going over how to perform a disconnected install in a lab environment. I will also give an overview of my environment, how to mirror the needed images, and any other tips and tricks I’ve learned along the way.
Environment Overview
In my environment, I have two networks. One network is completely disconnected and has no access to the Internet. The other network is connected to the Internet and has full access. I will use a bastion host that has access to both networks. This bastion host will perform the following functions.

Registry server (where I will mirror the content)
Apache web server (where I will store installation artifacts)
Installation host (where I will be performing the installation from)

Here is a high-level overview of the environment I’ll be working on.

In my environment, I have already set up DNS, DHCP, and other ancillary services for my network. Also, it’s important to get familiar with the OpenShift 4 prerequisites before attempting an install.
Doing a disconnected install can be challenging, so I recommend trying a fully connected OpenShift 4 install first to familiarize yourself with the install process (as they are quite similar).
Registry Set Up
You can use your own registry or build one from scratch. I used the following steps to build one from scratch. Since I’ll be using a container for my registry, and Apache for my webserver, I will need podman and httpd on my host.
yum -y install podman httpd httpd-tools

Create the directories you’ll need to run the registry. These directories will be mounted in the container running the registry.
mkdir -p /opt/registry/{auth,certs,data}

Next, generate an SSL certificate for the registry.  This can, optionally, be self-signed if you don’t have an existing, trusted, certificate authority. I’ll be using registry.ocp4.example.com as the hostname of my registry. Make sure your hostname is in DNS and resolves to the correct IP.
cd /opt/registry/certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt

Generate a username and password (must use bcrypt formatted passwords), for access to your registry.
htpasswd -bBc /opt/registry/auth/htpasswd dummy dummy

Make sure to open port 5000 on your host, as this is the default port for the registry. Since I am using Apache to stage the files I need for installation, I will open port 80 as well.
firewall-cmd –add-port=5000/tcp –zone=internal –permanent
firewall-cmd –add-port=5000/tcp –zone=public   –permanent
firewall-cmd –add-service=http  –permanent
firewall-cmd –reload

Now you’re ready to run the container. Here I specify the directories I want to mount inside the container. I also specify I want to run on port 5000. I recommend you put this in a shell script for ease of starting.
podman run –name poc-registry -p 5000:5000
-v /opt/registry/data:/var/lib/registry:z
-v /opt/registry/auth:/auth:z
-e “REGISTRY_AUTH=htpasswd”
-e “REGISTRY_AUTH_HTPASSWD_REALM=Registry”
-e “REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry”
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd
-v /opt/registry/certs:/certs:z
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key
docker.io/library/registry:2

Verify connectivity to your registry with curl. Provide it the username and password you created.
curl -u dummy:dummy -k https://registry.ocp4.example.com:5000/v2/_catalog

Note, this should return an “empty” repo

If you have issues connecting try to stop the container.
podman stop poc-registry

Once it’s down, you can start it back up using the same podman run command as before.
Obtaining Artifacts
You will need the preview builds for 4.2 in order to do a disconnected install. Specifically, you will need the client binaries along with the install artifacts. This can be found in the dev preview links provided below.

Client Binaries
Install Artifacts

Download the binaries and any installation artifacts you may need for the installation. The file names will differ depending on when you choose to download the preview builds (they get updated often).
You can inspect the nightly release notes and extract the build number from there. I did this with the curl command.
export BUILDNUMBER=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep ‘Name:’ | awk ‘{print $NF}’)
echo ${BUILDNUMBER}

To download the client binaries to your staging server/area (in my case, it’s the registry server itself) use curl:
curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/openshift-client-linux-${BUILDNUMBER}.tar.gz

curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/openshift-install-linux-${BUILDNUMBER}.tar.gz

You’ll also need these clients on your registry host, so feel free to un-tar them now.
tar -xzf /var/www/html/openshift-client-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/
tar -xzf /var/www/html/openshift-install-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/

Depending on what kind of install you will do, you would need to do one of the following.
PXE Install
If you’re doing a PXE install, you’ll need the BIOS, initramfs, and the kernel files. For example:
curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-${BUILDNUMBER}-metal-bios.raw.gz

curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-${BUILDNUMBER}-installer-initramfs.img

curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-${BUILDNUMBER}-installer-kernel

Once you have staged these, copy them over into your environment. Once they are in your PXE install server and your configuration updated, you can proceed to mirror your images.
ISO Install
If you’re doing an ISO install, you’ll still need the BIOS file but only the ISO for the install.
curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-${BUILDNUMBER}-metal-bios.raw.gz

curl -o /var/www/html/

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/rhcos-${BUILDNUMBER}-installer.iso

Once these are staged, copy them over to where you’ll need them for the installation. The BIOS file will need to be on a web server accessible to the OpenShift nodes. The ISO can be burned onto a disk/usb drive or mounted via your virtualization platform.
Once that’s done, you can proceed to mirror the container images.
Mirroring Images
The installation images will need to be mirrored in order to complete the installation. Before you begin you need to make sure you have the following in place.

An internal registry to mirror the images to (like the one I just built)

You’ll also need the certificate of this registry
The username/password for access

A pullsecret obtained at https://cloud.redhat.com/openshift/install/pre-release

I downloaded mine and saved it as ~/pull-secret.json

The oc and openshift-install CLI tools installed
The jq command is also helpful

First, you will need to get the information to mirror. This information can be obtained via the dev-preview release notes. With this information, I constructed the following environment variables.
export OCP_RELEASE=”4.2.0-0.nightly-2019-08-29-062233″
export AIRGAP_REG=’registry.ocp4.example.com:5000′
export AIRGAP_REPO=’ocp4/openshift4′
export UPSTREAM_REPO=’openshift-release-dev’   ## or ‘openshift’
export AIRGAP_SECRET_JSON=’~/pull-secret-2.json’
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_RELEASE}
export RELEASE_NAME=”ocp-release-nightly”

I will now go over how to construct these environment variables from the release notes

OCP_RELEASE – Can be obtained by the Release Metadata.Version section of the release page.
AIRGAP_REG – This is your registry’s hostname with port
AIRGAP_REPO – This is the name of the repo in your registry (you don’t have to create it beforehand)
UPSTREAM_REPO – Can be obtained from the Pull From section of the release page.
AIRGAP_SECRET_JSON – This is the path to your pull secret  with your registry’s information (which we will create later)
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE – This environment variable is set so the installer knows to use your registry.
RELEASE_NAME – This can be obtained in the Pull From section of the release page.

Before you can mirror the images, you’ll need to add the authentication for your registry to your pull secret file (the one you got from try.openshift.com). This will look something like this:
{
“registry.ocp4.example.com:5000″:
{
“auth”: “ZHVtbXk6ZHVtbXk=”,
“email”: “noemail@localhost”
}
}

The base64 is a construction of the registry’s auth in the username:password format. For example, with the username of dummy and password of dummy; I created the base64 by running:
echo -n ‘dummy:dummy’ | base64 -w0

You can add your registry’s information to your pull secret by using jq and the pull secret you downloaded (thus creating a new pull secret file with your registry’s information).
jq ‘.auths += {“registry.ocp4.example.com:5000″: {“auth”: “ZHVtbXk6ZHVtbXk=”,”email”: “noemail@localhost”}}’ < ~/pull-secret.json > ~/pull-secret-2.json

Also, if needed and you haven’t done so already, make sure you trust the self-signed certificate. This is needed in order for oc to be able to login to your registry during the mirror process.
cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

With this in place, you can mirror the images with the following command.
oc adm release mirror -a ${AIRGAP_SECRET_JSON}
–from=quay.io/${UPSTREAM_REPO}/${RELEASE_NAME}:${OCP_RELEASE}
–to-release-image=${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_RELEASE}
–to=${AIRGAP_REG}/${AIRGAP_REPO}

Part of the output will have an example imageContentSources to put in your install-config.yaml file. It’ll look something like this.
imageContentSources:
– mirrors:
– registry.ocp4.example.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-release-nightly
– mirrors:
– registry.ocp4.example.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

Save this output, as you’ll need it later
Installation
At this point you can proceed with the normal installation procedure, with the main difference being what you specify in the install-config.yaml file when you create the ignition configs.
Please refer to the official documentation for specific installation information. You’re most likely doing a Bare Metal install, so my previous blog would be helpful to look over as well.
When creating an install-config.yaml file, you need to specify additional parameters like the example below.
apiVersion: v1
baseDomain: example.com
compute:
– hyperthreading: Enabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: ocp4
networking:
clusterNetworks:
– cidr: 10.254.0.0/16
hostPrefix: 24
networkType: OpenShiftSDN
serviceNetwork:
– 172.30.0.0/16
platform:
none: {}
pullSecret: ‘{“auths”:{“registry.ocp4.example.com:5000″: {“auth”: “ZHVtbXk6ZHVtbXk=”,”email”: “noemail@localhost”}}}’
sshKey: ‘ssh-rsa …. root@helper’
additionalTrustBundle: |
—–BEGIN CERTIFICATE—–
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
—–END CERTIFICATE—–
imageContentSources:
– mirrors:
– registry.ocp4.example.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-release-nightly
– mirrors:
– registry.ocp4.example.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

Some things to note here:

pullSecret – only the information about your registry is needed.
sshKey – the contents of your id_rsa.pub file (or another ssh public key that you want to use)
additionalTrustBundle – this is your crt file for your registry. (i.e. the output of cat domain.crt)
imageContentSources –  What is the local registry is and the expected original source that should be in the metadata (otherwise they should be considered as tampered)

You will also need to export the OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE environment variable. This tells OpenShift which image to use for bootstrapping. This is in the form of ${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_RELEASE}. It looked like this in my environment:
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.ocp4.example.com:5000/ocp4/openshift4:4.2.0-0.nightly-2019-08-29-062233

I created my install-config.yaml under my ~/ocp4 install directory. At this point you can create your Ignition configs as you would normally.
# openshift-install create ignition-configs –dir=/root/ocp4
INFO Consuming “Install Config” from target directory
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
WARNING Found override for ReleaseImage. Please be warned, this is not advised

Please note that it warns you about overriding the image and that, for the 4.2 dev preview, the masters are schedulable.

At this point, you can proceed with the installation as you would normally.
Troubleshooting
A good thing to do during the bootstrapping process is to login to the bootstrap server and tail the journal logs as the bootstrapping process progresses. Many errors or misconfigurations can be seen immediately when tailing this log.
[core@bootstrap ~]$ journalctl -b -f -u bootkube.service

There are times where you might have to approve the worker/master node’s CSR. You can check pending CSRs with the oc get csr command. This is important to check since the cluster operators won’t finish without any worker nodes added. You can approve all the pending CSRs in one shot with the following command.
[user@bastion ~]$ oc get csr –no-headers | awk ‘{print $1}’ | xargs oc adm certificate approve

After the bootstrap process is done, it’s helpful to see your cluster operators running. You can do this with the oc get co command. It’s helpful to have this in a watch in a separate window.
[user@bastion ~]$ watch oc get co

The two most common issues are that the openshift-install command is waiting for the image-registry and ingress to come up before it considers the install a success. Make sure you’ve approved the CSRs for your machines and you’ve configured storage for your image-registry. The commands I’ve provided should help you navigate any issues you may have.
Conclusion
In this blog, I went over how you can prepare for a disconnected install and how to perform a disconnected install using the nightly developer preview of OpenShift 4. Disconnected installs were a highly popular request for OpenShift 4, and we are excited to bring you a preview build.
Nightly builds are a great way to preview what’s up and coming with OpenShift, so you can test things before the GA release. We are excited to bring you this capability and hope that you find it useful. If you have any questions or comments, feel free to use the comment section below!
The post OpenShift 4.2 Disconnected Install appeared first on Red Hat OpenShift Blog.
Quelle: OpenShift

Simplify modernization and build cloud-native with open source technologies

Cloud-native technologies are the new normal for application development. Cloud-native creates immeasurable business value with increased velocity and reduced operational costs. Together, these support emerging business opportunities.
Advancements in application development have focused on net new applications. We have seen that existing applications that cannot easily move to the cloud have been left on traditional technologies. As a result, less than 20 percent of enterprise workloads are deployed to the cloud according to an IBM-commissioned IBM-commissioned study by McKinsey & Company.
At IBM, we see open source as a foundation for the new hybrid multicloud world, and our recent acquisition of Red Hat underscores our long commitment to open technologies.
Open source allows consistency and choice
Key open source technologies – containers, Kubernetes, Istio, Knative and others – together define the new hybrid multicloud platform, providing consistency and choice across any cloud provider. These technologies allow developers to build applications to support enterprise workloads using a common technology base with flexible vendor choices. They establish freedom for enterprises to deploy applications across public, private and hybrid cloud platforms.
Kubernetes provides a container orchestration layer that consistently manages workloads. Developers have full freedom of choice on languages, runtimes, and frameworks, while Kubernetes maintains a consistent operational platform across diverse technologies. This approach provides a basis for microservices-based container applications as well as existing enterprise applications.
New IBM open source project accelerates the cloud journey
In 2017, IBM began modernizing our software portfolio into containers and Kubernetes, and optimized more than 100 products for Red Hat OpenShift. In addition to our own journey, we’ve learned a lot about modernization from our clients and partners. Together, we’ve migrated or modernized more than 100,000 workloads.
With the new IBM Cloud Pak for Applications, we’ve encoded our experience into a set of technologies to accelerate the journey to cloud. Built on open source technologies, IBM Cloud Pak for Applications delivers tools, technologies and platforms designed to bring WebSphere workloads to any cloud through Kabanero.io and Red Hat Runtimes.
IBM Cloud Pak for Applications provides a rich set of open source technologies and functions that allow enterprises to secure and curate their favorite frameworks and runtimes, including those using Java, Open Liberty, SPRING BOOT® with Tomcat®, JBoss®, Node.js®, Vert.x and more. IBM Cloud Pak for Applications performs vulnerability scanning on all open source frameworks and runtimes to prevent security issues. All IBM Cloud Paks are supported by IBM and contain Docker-certified middleware.
Move WebSphere applications to any cloud
For existing applications, modernization tools in the new IBM Cloud Pak chart a path to modernize WebSphere applications into a fully open source stack. IBM Cloud Pak for Applications analyzes applications and provides a modernization plan specific for each application. Many WebSphere applications can be migrated to containers with automation and without code changes.
In the end, traditional applications are ready to deploy to any cloud — from OpenShift on IBM Cloud, an existing infrastructure, or to your cloud of choice.
IBM Cloud Pak for Applications delivers the open technology platform for the future and enables businesses to address the 80 percent of enterprise workloads that have yet to move to cloud according the report.
Learn more about IBM Cloud Pak for Applications and register to join us for the upcoming IBM Application Modernization Technical Conference 2019, Chicago, IL, United States on 24-25 September 2019. Experience two days of in-depth technical sessions for developers, administrators and architects at the inaugural IBM Application Modernization Technical Conference 2019 and hear from top subject matter from our labs, IBM Business Partners and customers.
The post Simplify modernization and build cloud-native with open source technologies appeared first on Cloud computing news.
Quelle: Thoughts on Cloud

Kubeflow + OpenShift Container Platform + Dell EMC Hardware: A Complete Machine Learning Stack

Kubeflow is an open source machine learning toolkit for Kubernetes. It bundles popular ML/DL frameworks such as TensorFlow, MXNet, Pytorch, and Katib with a single deployment binary. By running Kubeflow on Red Hat OpenShift Container Platform, you can quickly operationalize a robust machine learning pipeline. However, the software stack is only part of the picture. You also need high performance servers, storage, and accelerators to deliver the stack’s full capability. To that end, Dell EMC and Red Hat’s Artificial Intelligence Center of Excellence recently collaborated on two white papers about sizing hardware for Kubeflow on OpenShift.
The first whitepaper is called “Machine Learning Using the Dell EMC Ready Architecture for Red Hat OpenShift Container  Platform.”  It describes how to deploy Kubeflow 0.5 and OpenShift Container Platform 3.11 on Dell PowerEdge servers. The paper builds on Dell’s Ready Architecture for OpenShift Container Platform 3.11 — a prescriptive architecture for running OpenShift Container Platform on Dell hardware. It includes a bill of materials for ordering the exact servers, storage and switches used in the architecture. The machine learning whitepaper extends the ready architecture to include workload-specific recommendations and settings. It also includes instructions for configuring OpenShift and validating Kubeflow with a distributed TensorFlow training job.
Kubeflow is developed on upstream Kubernetes, which lacks many of the security features enabled in OpenShift Container Platform by default. Several of OpenShift Container Platform default security controls are relaxed in this whitepaper to get Kubeflow up and running. Additional steps might be required to meet your organization’s security standards for running Kubeflow on OpenShift Container Platform in production. These steps may include defining cluster roles for the Kubeflow services with appropriate permissions, adding finalizers to Kubeflow resources for reconciliation, and/or creating liveness probes for Kubeflow pods.
The second whitepaper is called “Executing ML/DL Workloads Using Red Hat OpenShift Container Platform v3.11.” It explains how to leverage Nvidia GPUs with Kubeflow for best performance on inferencing and training jobs. The hardware profile used in this whitepaper is similar to the ready architecture used in the first paper except the servers are outfitted with Nvidia Tesla GPUs. The architecture uses two GPU models. The OpenShift worker nodes have Nvidia Tesla T4 GPUs. Based on the Turing architecture, the T4s deliver excellent inference performance in a 70-Watt power profile. The storage nodes have Nvidia Tesla V100 GPUs. The V100 is a state of the art data center GPU. Based on the Volta architecture, the V100s are deep learning workhorses for both training and inference. 

The researchers compared the GPU models when training the Resnet50 TensorFlow benchmark. This is shown in the figure above. Not surprisingly, the Tesla V100s outperformed the T4s when training. They have double the compute capability — both in terms of FP64 and TensorCores — along with higher memory bandwidth due to the HBM2 memory subsystem. But the T4s should give better performance per Watt than the V100s when running less floating-point intensive tasks, particularly inferencing in mixed precision.
These whitepapers make it easier for you to select hardware for running Kubeflow on premises. Dell and Red Hat are continuing to collaborate on updating these documents to the latest version of Kubeflow and OpenShift Container Platform 4.
The post Kubeflow + OpenShift Container Platform + Dell EMC Hardware: A Complete Machine Learning Stack appeared first on Red Hat OpenShift Blog.
Quelle: OpenShift

Red Hat OpenShift Service Mesh is now available: What you should know

Today, Red Hat OpenShift Service Mesh is now available. 
As Kubernetes and Linux-based infrastructure take hold in digitally transforming organizations, modern applications frequently run in a microservices architecture and therefore can have complex route requests from one service to another. With Red Hat OpenShift Service Mesh, we’ve gone beyond routing the requests between services and included tracing and visualization components that make deploying a service mesh more robust. The service mesh layer helps us simplify the connection, observability and ongoing management of every application deployed on Red Hat OpenShift, the industry’s most comprehensive enterprise Kubernetes platform.
Red Hat OpenShift Service Mesh is available through the OpenShift Service Mesh Operator, and we encourage teams to try this out on Red Hat OpenShift 4 here.
Better track, route and optimize application communication
With hardware-based load balancers, bespoke network devices, and more being the norm in modern IT environments, it was complex, if not nearly impossible, to have a consistent, general purpose way to manage and govern service-to-service communications between applications and their services. With a service mesh management layer, containerized applications can better track, route and optimize communications with Kubernetes as the core. Service mesh can help manage hybrid workloads in multiple locations and is more granularly aware of data locality. With the official introduction of OpenShift Service Mesh, we believe this critical layer of the microservices stack has the power to further enable a multi- and hybrid cloud strategy in enterprises.
OpenShift Service Mesh is based on a set of community projects, including Istio, Kiali and Jaeger, providing capability that encodes communication logic for microservices-based application architectures. This helps to free development teams to focus on coding applications and services that can add real business value. 
Making Developers’ Lives Easier
 
Like we said before, until service mesh, much of the burden of managing these complex services interactions has been placed on the application developer. Developers need a set of tools that can help them manage their entire application lifecycle beginning with determining the success of deploying their code all the way to managing the traffic patterns in production. Each service needs to properly interact with other services for the complete application to run. Tracing provides a way that developers can track how each service interacts with these other functions to determine if there are latency bottlenecks as they operate together. 
The ability to visualize the connections between all the services and look at the topology of how they interconnect can also be helpful in understanding these complex service interconnections. By packaging these features together as part of the OpenShift Service Mesh, Red Hat is making it easier for developers to access more of the tools they need to successfully develop and deploy cloud-native microservices. 
To ease the implementation of a service mesh, the Red Hat OpenShift Service Mesh can be added to your current OpenShift instance through the OpenShift Service Mesh Operator. The logic of installation, connecting the components together and the ongoing management of all the bits is built into the Operator, allowing you to get right to managing the service mesh for your application deployments. 
By reducing the overhead of implementing and managing the service mesh, it becomes easier to introduce the concept earlier in you application lifecycle and enjoy the benefits before things start to get out of hand. Why wait until it’s too late and you’re overwhelmed managing the communication layer? OpenShift Service Mesh can enable the scalable features you will need in an easy to implement fashion before you start to scale.
The features that OpenShift customers can benefit from with OpenShift Service Mesh include:

Tracing and Measurement (via Jaeger): While enabling service mesh can come with performance trade-offs for the betterment of management, OpenShift Service Mesh now measures baseline performance. This data can be analyzed and used to drive future performance improvements.

 

Visualization (via Kiali): Observability of the service mesh allows for an easier way to view the topology of the service mesh, and to observe how the services are interacting together.  

 

Service Mesh Operator: A Kubernetes Operator minimizes the administrative burdens of application management, allowing for automation of common tasks such as install, maintenance of the service and lifecycle management. Adding business logic with these applications helps to ease management and bring the latest upstream features to customers sooner. The OpenShift Service Mesh operator deploys Istio, Kiali and Jaeger bundles along with the configuration logic needed to make implementing all the features at once easier. 
Multiple interfaces for networking (via multus): OpenShift Service Mesh takes out manual steps and enables developers to execute code with increased security via Security Context Constraint (SCC). This allows additional lockdown of workloads on the cluster, such as outlining which workloads in a namespace can run as root and not. With this feature, developers get the usability of Istio while cluster admins gain well-defined security controls.
Integrations with Red Hat 3scale API management: For developers or operations teams looking to further secure API access across services, OpenShift Service Mesh ships with the Red Hat 3scale Istio Mixer Adapter. This allows for deeper control over the API layer of service communications beyond the traffic management of the service mesh capabilities.

Looking to the future of service mesh capabilities and interoperability, we introduced the Service Mesh Interface (SMI) earlier this year. Our hope in collaborating across the industry is to continue working to abstract these components in an effort to make it easier for service meshes to run on Kubernetes. This collaboration can help maximize choice and flexibility for our Red Hat OpenShift customers, and once applied, has the potential to bring a “NoOps” environment for developers closer to reality.
Try OpenShift
Service mesh can help ease the journey in operating a microservices based stack across the hybrid cloud. Customers with a growing Kubernetes and containers based environment are invited to try out Red Hat OpenShift Service Mesh.
 
Learn more:

About OpenShift 4 and OpenShift Service Mesh
Attend a virtual event to learn more about building applications on OpenShift

The post Red Hat OpenShift Service Mesh is now available: What you should know appeared first on Red Hat OpenShift Blog.
Quelle: OpenShift

How AIOps helps Nextel Brazil predict and prevent network outages

Mobile smartphones are playing a significant role in the lives and productivity of people around the world. Consider these statistics about smartphone usage from TechJury.

Internet users worldwide who visit the Web on a mobile device: 67%
Percent of emails read on mobile devices: 49.1%
Smartphone users are addicted to their phones: 66%

Clearly, many people today don’t want to (or can’t) be without their smartphones. And like all telecommunications companies, Nextel Brazil is trying to be as customer centric as possible. We strive to make customer service part of the DNA of the company and treat customers as our primary asset, because they are.
I have a great team of 75 people working with me. There are three shifts working morning, noon and night. We do the best we can to satisfy our customer needs because we know our subscribers depend on their mobile phones to work and live their lives. Every second that there’s a network outage and customers don’t have service, we have to be there for them. That’s what being customer centric means to us, especially in operations.
Reducing network outages and meantime to repair with IBM Netcool
Mean time to repair (MTTR) is the key performance indicator for us. We started our partnership with IBM when we began using IBM Netcool Operations Insight software to correlate alarms and get to the root cause of problems faster. We have more than 25,000 established network elements and multiple management systems being monitored by Netcool. The solution has helped us reduce the MTTR to receive an alarm and solve a problem in the field or with some configuration from 30 minutes to less than five minutes.
Still people don’t even want to wait one minute, never mind five minutes, to get their services recovered. As our services and network increase in complexity, so has the amount of data generated.
After approximately three years of maturing this solution, we started to say, “Hey, we can do better. We can be more proactive to treat the problem. Let’s start looking at the data. Let’s start with some analytics into the data.”
We have to be able to predict and to be prepared for network problems, because we know that they will happen. This is our day to day. We wanted to be better prepared for incidents and be able to make adjustments to avoid a network outage.
Moving from reactive to predictive with AIOps
We began working with IBM Watson technology to implement artificial intelligence for IT operations (AIOps). Watson helps us to categorize all the incidents, so we have a better understanding of what is happening in the network, such as if the outage is due to a utilities problem. More than just knowing we have a problem, Watson tells us why we have the problem. Now we can group incidents together and focus on fixing things at the source.
We’re also working with The Weather Company, an IBM Business to predict weather-related incidents and prevent them from impacting service. Our Network Operations team has a high dependence on utility companies because our cell towers are based on electric power. We will have a problem when they have a problem, and they are very dependent on the weather.
With the Weather Company data, we can correlate and look into historical data and know every time that we have a certain threshold of rain, of wind speed, of soil moisture, or whatever set of parameters, that we will have a problem with cell towers in this region.
If we know that one of these conditions is going to happen in the next 72 hours, we can be more prepared to act. As a result, we might send a small generator or extra batteries to the site to keep it up longer. By better knowing the probability and duration of the fault, we can prepare such that we can help avoid an outage for our customers in that region.
AIOps with Watson and The Weather Company data has helped us complete the journey to being predictive in network operations. It’s a great feeling to know that we don’t just have to wait for something terrible to happen and then react to it. We can actually do something about it before it happens. And this means that our customers who depend on their mobile phone are less likely to be without service.
Watch the related video.
The post How AIOps helps Nextel Brazil predict and prevent network outages appeared first on Cloud computing news.
Quelle: Thoughts on Cloud

How Full is my Cluster – Part 5: A Capacity Management Dashboard

Introduction
This is the fifth installment in the series regarding capacity management with OpenShift.

In the first post, we saw the basic notions on how resource management works in OpenShift and a way to visualize node and pod resource consumption using kube-ops-view.
In the second post, we illustrated some best practices on how to protect the nodes from being overcommitted.
In the third post, we presented best practices on how to setup a capacity management process and which metrics to follow.
In the fourth post, we introduced the vertical pod autoscaler operator was introduced along with its potential use as a way to estimate the correct sizing of pods.

In this post, we will introduce a ready to use dashboard for which you can base your capacity management process.
The primary goal of this dashboard is to answer a very specific question: Do I need a new node?
Naturally, this is not the only question for a well conceived capacity management process. But, it is certainly a good starting point. It can be used as a foundation for a more sophisticated dashboard that will fully support your capacity management process.
The Dashboard
This capacity management dashboard works on a group of nodes and will help deciding if you need more or less nodes for that group. You can have many node groups in your cluster, so the user can select the node group they want to work on with a drop down list box.
The following illustrates the contents of the capacity management dashboard:
This primary metrics presented are for memory and cpu. The way metrics are collected is the same for memory and CPU. The metrics always refer to aggregates within the selected node group. The Dashboard displays three metrics:

Quota/Allocatable ratio
Request/Allocatable ratio
Usage/Allocatable ratio

If you may recall from the second post on overcommitment, Allocatable is the amount of resources actually available to pods on a given node. This value is calculated out of the total capacity of a node after reserved resources for the operating system and other basic services, such as the container runtime service and the node service, have been accounted for.
In OpenShift 4.x, Prometheus Alerts are also set up during installation. These alerts trigger when the when the Quota/Allocatable ratio passes 100% and when the Request/Allocatable ratio passes 80%. 
You can find the dashboard and its installation instructions at this repository.  In order to work properly, this dashboard requires the cluster nodes and projects to be organized in a certain way, see at the end of the post more information on this.
In the next section we are going to explain how the collected metrics can be interpreted and used to deduce or forecast whether you need more nodes.
Interpretation of quota/allocatable ratio
This metric is collected as in the formula:

This metric can be interpreted as: what has been promised (the granted quota) vs the actual available amount (the allocatable).
Changes to his metric do not occur frequently and fluctuations  typically only occur when new projects are introduced. This metrics is suitable for making long term projections of the needed cluster capacity. Organizations with OpenShift deployments where the nodes cannot be scaled quickly (non-cloud deployments such as bare metal deployments) should most likely use this metric to make a decision on when to scale up nodes. 
Depending on your tolerance to risk, it can be ok for this metric to be above 100%, which signals that you have overcommitted the cluster. 
Interpretation of request/allocatable ratio
This metric is collected as in the formula:

This metrics can be interpreted as: the value tenants are estimating (if you recall from blog post #3 of this series on developing a capacity management process, resource requests on containers should correspond to the estimated value that will be needed at runtime) they would need versus the  actual available amount.
This metric is more volatile than the previous metric because the value changes when new pods are added/removed and is more suitable for making a scaling decision when a new node can be provisioned quickly. This is likely to happen when you are running on cloud environments. 
The OpenShift 4.x cluster autoscaler uses this metric indirectly. In fact, it will trigger the addition of a new node if a pod is stuck in a pending state if it cannot be allocated due to a lack of resources. This is approximately the same as triggering a scale up when this metric is measuring close to 100%.
With an approach based node groups, however, we can be more flexible than the cluster autoscaler because we can decide at which threshold we want to scale. For example, with this metric, we can be proactive and add a new node when the ratio hits 80%, so that no pods need to wait before it can be provisioned.
Interpretation of Usage/Allocatable ratio
This metric is collected as in the following formula:

This metrics can be interpreted as what is being currently used vs the amount of available resources. This is clearly the most volatile metric, as it depends on an instant by instant load of running pods.
This metric is generally poor for making capacity forecasts because the values fluctuate too often. However, it can be used to, provide a general overview of the cluster. 
The primary function of this metric is to be able to compare the actual (what we are using) with the estimates (the sum of requests from the previous ratio). If these two measures diverge, it means that our tenants are not estimating their resources correctly, and corrective actions are needed.
Another function of this metric is that it allows us to judge whether we have enough resources for the current workload. However, we have to be cognizant that the fact that at the node group level there are enough resources does not guarantee by itself that individual nodes are not resource constrained. For that reason, another set of metrics and alerts needs to be utilized. Fortunately, these node level metrics and alerts are included in the Prometheus setup that comes with OpenShift.
Assumptions
In order for the dashboard to be effective and accurate, there are several  assumptions that must be considered 
Nodes are grouped into non-overlapping groups. Groups are identified by a node label. The default for the label is nodegroup. Each node must belong to exactly one group. The capacity management dashboard will tell us if we need to add a node to a specific group. Groups can be used to manage zones or areas of the cluster that house different workloads and may need to scale independently. For example, there might be a cluster with high-priority workload nodes, normal priority workload nodes, PCI-dedicated nodes, GPU-enabled nodes, and so on. In OpenShift 3.x, there are always at least three groups: master, infranodes and worker nodes (obviously worker nodes can be fragmented further if needed). In OpenShift 4.x there is no such requirement that infranodes must exist, however the same concepts apply.
ClusterResourceQuotas are defined to limit tenants resource consumption. ClusterResourceQuota is an OpenShift-specific API object that allows for a quota to be defined across multiple projects as opposed to ResourceQuota, which is associated to just one project. The ability of a ClusterResourceQuota to be defined across several projects allows the administrator to choose the granularity at which quotas are applied, achieving higher flexibility. For example, an organization can choose to grant quotas at the application level (an application usually exist on multiple projects to support the environments as defined by a SDLC), the business capability level (usually business capability are provided by a set of applications) or finally, at the line of business level (a set of business capabilities). As a result, ClusterResourceQuotas also allow for a flexible showback/chargeback model should a company decide to enact those processes. 
ClusterResourceQuotas must be defined for memory and cpu requests. At the moment, these are the only monitored resources. Other resources can be put under quota, but they would be ignored by the dashboard.
Each ClusterResourceQuota refers to only one node group. For the dashboard to function properly, each ClusterResourceQuota must have the same label as the label used to determine the node group. For example:
apiVersion: quota.openshift.io/v1
kind: ClusterResourceQuota
metadata:
name: quota1
labels:
  nodegroup: group1
spec:
quota:
  hard:
    requests.cpu: “2”
    requests.memory: 4Gi
selector:
  labels:
    matchLabels:
      quota: quota1
Each tenant project refers to a ClusterResourceQuota and deploys resources to the corresponding nodegroup. Each tenant project must be controlled by one and only one ClusterResourceQuota. The project default node selector must be configured to select the nodes belonging to the node group the ClusterResourceQuota refers to. For example:

kind: Namespace
apiVersion: v1
metadata:
 name: p1q1
 labels:
   quota: quota1
 annotations:
   openshift.io/node-selector: nodegroup=group1
Non-tenant projects, such as for example administrative projects, do not have to be under quota. However, the dashboard shows more accurate information if every project deployed on the monitored node groups is under quota.
The recommended approach is to define node labeling at cluster setup and to define the ClusterResourceQuotas and the project during the application onboarding process. The application onboarding process is the set of steps a development team must go through to be able to operate an application on OpenShift. Most organizations have a formalized process detailing these steps.
The below pseudo entity relationship diagram represents the configuration one would attain by the end of this preparation:

Conclusions
In this article, a capacity management dashboard was introduced that can be used as the baseline for an organization’s capacity management process.
With OpenShift 4.x and the introduction of the cluster autoscaler, the urgency of having such a dashboard may be reduced. However, the autoscaler is not always an option and even when it is implemented, it is currently very reactive (it triggers only when a pod cannot be allocated due to lack of capacity). As a result, this dashboard should provide value in understanding capacity management with OpenShift
 
The post How Full is my Cluster – Part 5: A Capacity Management Dashboard appeared first on Red Hat OpenShift Blog.
Quelle: OpenShift