da Agency

How GKE surge upgrades improve operational efficiency

A big part of keeping a Kubernetes environment healthy is performing regular upgrades. At Google Cloud, we automatically upgrade the cluster control plane for Google Kubernetes Engine (GKE) users, but you’re responsible for upgrading the cluster’s individual nodes, as well as any additional software installed on the nodes. And while you can choose to enable node auto-upgrade to perform these updates behind the scenes, we recently introduced a ‘surge nodes upgrade’ feature that gives you fine-grained control over the upgrade process, to minimize the risk of disruption to your GKE environment, as well as to expedite the upgrade process. This is particularly important at a time when external forces are pushing many organizations to transition to a digital-only business model, where availability is key for business continuity. Surge upgrade reduces disruption to existing workloads while keeping clusters up-to-date with the latest version, security patches, and bug fixes.The importance of node upgradesNodes are where your Kubernetes workloads run. Open-source Kubernetes releases a new minor version approximately every three months, and patches more frequently. GKE follows this same release schedule, providing regular security patches and bug fixes, so you can reduce your exposure to security vulnerabilities, bugs, and version skew between control plane and nodes.Enabling node auto-upgrade is a popular choice for performing this important task. The node pool upgrade process recreates every VM in the node pool with a new (upgraded) VM image in a rolling update fashion. To do so, it shuts down all the pods running on the given node. And while most customers run workloads with sufficient redundancy and Kubernetes helps with the process of moving and restarting pods, in practice, the temporarily reduced number of replicas may not be sufficient to serve all your traffic, resulting in production incidents.Simply enabling node auto-upgrade isn’t enough for some GKE users. FACEIT provides an independent online competitive gaming platform that lets players create communities and compete in tournaments, leagues, and matches. With over a million monthly active users, FACEIT relies on GKE, benefitting from the platform’s agility and simple and automated scalability. But to eliminate the chance of downtime, FACEIT wasn’t using the node auto-upgrade feature. Instead, it used the following manual process:Create a new identical node pool, running on the new versionCordon off the nodes in the old node poolStart evicting pods by draining the nodes, causing Kubernetes to reschedule the pods on nodes in the new node poolFinally, remove the old node pool once all the nodes were drainedWith this manual process, FACEIT was able to balance upgrade speed and avoid disruptions. Introducing surge upgradesTo help ensure that all node upgrades complete successfully and in a timely fashion, we’re excited to offer surge upgrades for GKE. Surge upgrades reduces the potential for disruption by starting up new nodes before it drains the old ones, and supports upgrading multiple nodes concurrently. Upgrades are only initiated after all the required resources (VMs) are secured, ensuring that surge upgrades can complete successfully. Surge upgrades will be enabled by default on April 20, 2020, and we will also migrate existing node pools later in the quarter.The new surge upgrades feature helps to reduce workload disruption in two key ways.1. No decreased capacity during node upgradesThanks to surge upgrades, a node pool cannot transition into a state where it has less capacity than it had at the start of the upgrade process (assuming maxUnavailable is set to 0).In contrast, without surge upgrades, the node upgrade happens by recreating the node. This means there is a period during the upgrade process when the node is not available to the cluster. If there is sufficient redundancy in the cluster, this in itself may not cause any disruption to the workloads. However, any other failure with the workloads or with the infrastructure—for example, an unrelated node failure—may result in disruption.2. No evicted pod will remain unscheduled due to a lack of capacityThis feature of surge upgrades is a consequence of the above. Since there is equivalent additional capacity available (i.e., the surge node), it is always possible to schedule the evicted pod.As mentioned before, node upgrades happen by recreating Compute Engine instances with a new instance template, then evicting the pods and rescheduling them—assuming there’s the capacity to schedule them. Whenever one or more pods remain unscheduled, that means one or more workloads are running a lower number of replicas than desired. This may impact workload health (i.e., the system may not be able to tolerate the loss of the replica). Regardless, to ensure high availability, the time a workload is running with a reduced number of replicas should be shortened.The lack of surge nodes during an upgrade does not necessarily result in pods becoming unschedulable; If the node pool has sufficient available capacity, it will use it. In the picture below, the left node pool has enough capacity to be able to reschedule all the evicted pods immediately when the first node is drained. The right node pool has some available capacity, but not enough, so only two of the three evicted pods can be rescheduled immediately; Pod3 will need to wait for the node upgrade to complete, then it will be scheduled again.Let’s see how the presence of a surge node changes the situation. In the case of the less utilized node pool, the surge node does not help with pod scheduling, since the pods already had enough capacity. But in the case of the more utilized node pool, the extra capacity is necessary to be able to schedule the evicted pods right away.It’s worth noting that surge capacity can be useful for more than just upgrades; it can also be ‘spent’ on other demands like scaling up an application faster in case of a spike in load during the upgrade. Control how you upgrade, not just whenGKE’s node auto-upgrade feature helps administrators ensure that their environment stays up-to-date with the latest patches and updates. Now, with surge upgrades, you can know those upgrades will occur successfully, and without impacting production workloads. Early adopters like FACEIT report that this enhanced upgrade process is not only more reliable, but that it’s also faster, as it allows concurrent node upgrades. “Before surge upgrades, an upgrade of one environment required around seven hours to complete, multiplied by the number of environments. We used to spend roughly two weeks upgrading all of FACEIT’s environments,” said Emanuele Massara, VP Engineering, FACEIT. “With surge upgrades, the entire process takes less than a day, freeing up the team to focus on other tasks.” FACEIT has since turned down its manual upgrade process.Using surge upgrades in conjunction with correctly configured PDB (Pod Disruption Budget) can also help ensure the availability of applications during the upgrade process, said Bradley Wilson-Hunt, DevOps & Service Delivery Manager. “Without a PDB in place, Kubernetes can reschedule the pods of a deployment without waiting for the new pod to be ready, which could lead to a service disruption.”To learn more about using surge upgrades, read these guidelines on how to determine the parameters to configure your upgrades. You can also try surge upgrades yourself, using a demo application that follows this tutorial.
Quelle: Google Cloud Platform

da Agency

Protect your running VMs with new OS patch management service

Managing patches effectively is a great way to keep your infrastructure up-to-date and reduce the risk of security vulnerabilities. But without the right tools, patching can be daunting and labor intensive.Today, we are announcing the general availability of Google Cloud’s OS patch management service to protect your running VMs against defects and vulnerabilities. The service works on Google Compute Engine and across OS environments (Windows, Linux).Automate OS security and complianceWith OS patch management, you can apply OS patches across a set of VMs, receive patch compliance data across your environments, and automate installation of OS patches across VMs—all from one centralized location. The OS patch management service has two main components:Compliance reporting, which provides detailed compliance reports and insights on the patch status of your VM instances across Windows and Linux distributions. Patch deployment, which automates the installation of OS patches across your VM fleet, with flexible scheduling and advanced patch configuration controls. For added convenience, you can set up flexible schedules and still keep systems up-to-date by running your patch updates within designated maintenance windows.Managing patches for your applications doesn’t have to be a time-consuming exercise. OS patch management’s automated compliance reporting feature helps your systems stay up-to-date against vulnerabilities, reducing the risk of downtime for your business and the productivity of your internal users. IT administrators now can also focus on other business critical tasks, not on manual patch update processes.Get started todayThe current release of OS patch management is available at no cost from now through December 31, 2020. You can start using OS patch management in the Google Cloud Console today. To learn more about how to set up the service, check out the documentation.
Quelle: Google Cloud Platform

da Agency

Protecting businesses against cyber threats during COVID-19 and beyond

No matter the size of your business, IT teams are facing increased pressure to navigate the challenges of COVID-19. At the same time, some things remain constant: Security is at the top of the priority list, and phishing is still one of the most effective methods that attackers use to compromise accounts and gain access to company data and resources. In fact, bad actors are creating new attacks and scams every day that attempt to take advantage of the fear and uncertainty surrounding the pandemic. It’s our job to constantly stay ahead of these threats to help you protect your organization. In February, we talked about a new generation of document malware scanners that rely on deep learning to improve our detection capabilities across over 300 billion attachments we scan for malware every week. These capabilities help us maintain a high rate of detection even though 63% of the malicious docs blocked by Gmail are different from day to day. To further help you defend against these attacks, today we’re highlighting some examples of COVID-19-related phishing and malware threats we’re blocking in Gmail, sharing steps for admins to effectively deal with them, and detailing best practices for users to avoid threats.The attacks we’re seeing (and blocking)Every day, Gmail blocks more than 100 million phishing emails. During the last week, we saw 18 million daily malware and phishing emails related to COVID-19. This is in addition to more than 240 million COVID-related daily spam messages. Our ML models have evolved to understand and filter these threats, and we continue to block more than 99.9% of spam, phishing, and malware from reaching our users. The phishing attacks and scams we’re seeing use both fear and financial incentives to create urgency to try to prompt users to respond. Here are some examples:Impersonating authoritative government organizations like the World Health Organization (WHO) to solicit fraudulent donations or distribute malware. This includes mechanisms to distribute downloadable files that can install backdoors. In addition to blocking these emails, we worked with the WHO to clarify the importance of an accelerated implementation of DMARC (Domain-based Message Authentication, Reporting, and Conformance) and highlighted the necessity of email authentication to improve security. DMARC makes it harder for bad actors to impersonate the who.int domain, thereby preventing malicious emails from reaching the recipient’s inbox, while making sure legitimate communication gets through.This example shows increased phishing attempts of employees operating in a work-from-home setting.This example attempts to capitalize on government stimulus packages and imitates government institutions to phish small businesses.This attempt targets organizations impacted by stay-at-home orders.Improving security with proactive capabilities We have put proactive monitoring in place for COVID-19-related malware and phishing across our systems and workflows. In many cases, these threats are not new—rather, they’re existing malware campaigns that have simply been updated to exploit the heightened attention on COVID-19. As soon as we identify a threat, we add it to the Safe Browsing API, which protects users in Chrome, Gmail, and all other integrated products. Safe Browsing helps protect over four billion devices every day by showing warnings to users when they attempt to navigate to dangerous sites or download dangerous files. In G Suite, advanced phishing and malware controls are turned on by default, ensuring that all G Suite users automatically have these proactive protections in place.These controls can: Route emails that match phishing and malware controls to a new or existing quarantineIdentify emails with unusual attachment types and choose to automatically display a warning banner, send them to spam, or quarantine the messages Identify unauthenticated emails trying to spoof your domain and automatically display a warning banner, send them to spam, or quarantine the messages Protect against documents that contain malicious scripts that can harm your devices Protect against attachment file types that are uncommon for your domainScan linked images and identify links behind shortened URLsProtect against messages where the sender’s name is a name in your G Suite directory, but the email isn’t from your company domain or domain aliasesBest practices for organizations and usersAdmins can look at Google-recommended defenses on our advanced phishing and malware protection page, and may choose to enable the security sandbox. Users should: Complete a Security Checkup to improve your account securityAvoid downloading files that you don’t recognize; instead, use Gmail’s built-in document previewCheck the integrity of URLs before providing login credentials or clicking a link—fake URLs generally imitate real URLs and include additional words or domainsAvoid and report phishing emails Consider enrolling in Google’s Advanced Protection Program (APP)—we’ve yet to see anyone that participates in the program be successfully phished, even if they’re repeatedly targeted At Google Cloud, we’re committed to protecting our customers from security threats of all types. We’ll keep innovating to make our security tools more helpful for users and admins and more difficult for malicious actors to circumvent.
Quelle: Google Cloud Platform