How spam detection taught us better tech support

Information Technology teams, especially in help desk and support, need a way to track what problems people are having. Ideally they also can know how those problems change over time, especially when technology or policy shifts.Imagine you are in charge of sending a newspaper delivery team to different neighborhoods. Each person has a bicycle, so you give them a route, and they leave the papers at the right doors. But the roads change. Every day they change. It’s chaos.What do you do when routes are changing constantly?  How do you provide the information needed when the context is shifting all the time?In an IT context we run into similar challenges with traditional problem management frameworks such as ITIL 4, which tend to always assume a fixed, well-defined catalog of services. That way every issue that the IT folks solve is tracked and accounted for. That connection back to the catalog allows insight into what’s causing issues, or where outages or incidents may be impacting a large group of employees. At Google we don’t have that. In part because we focus on putting the user first, and so we focus on getting people back to a productive state as job #1. Also because products, services and issues are always shifting–just like the roads, the route is never the same, even if the goal remains consistent. That means users come into our IT service desk with new problem types everyday.Our tech support team, called Techstop, acts as the one-stop shop for all IT issues, and supports people across chat, email and video channels. They need to remain adaptable to new problems Googlers experience and new products they use. In order to track what problems might be on the rise, the Techstop team needs a way to catalog what tools, applications and services are in use at Google. Thinking back to the newspaper delivery routes, we used a rough approximate map, rather than a very detailed one, giving us a taxonomy of services that was “good enough” for most of our use cases. We got some useful data out of it, but it didn’t give us very granular insight.Need for innovationCovid-19 put a new focus on scalable problem understanding, specifically for everyday employee IT issues. With so much of the workforce moved to a work-from-home model, we really needed to know where employees were experiencing technology pain. It’s as if whole new neighborhoods popped into existence overnight, but our newspaper delivery crew was the same. More ground to cover, with totally novel street maps.Furthermore, products used everyday for productivity, such as Google Meet, began to see exponential growth in usage, causing scaling issues and outages. These product teams looked to the Techstop organization to help them prioritize the ever increasing list of feature requests and bugs being filled every day.Ultimately the “good enough” problem taxonomy failed to produce truly helpful insights. We could find out which products were being affected the most, but not what issues people were having with those products. Even worse, new issues that were unique to the work-from-home model were being hidden by the fact that the catalog could not update in time to catch the rapidly changing problem space underneath it.Borrowing spam techTaking a look around other efforts at Google, the Techstop team found examples of solving a similar problem: detecting new patterns quickly in rapidly changing data.Gmail handles spam filtering for over a billion people. Those engineers had thought through “how do we detect a new spam campaign quickly?”  Spammers rapidly send bulk messages with slight variations in content (noise, misspellings, etc.) Most classification attempts would become a game of cat and mouse since it takes classifiers some time to learn about new patterns.Invoking a trend identification engine using unsupervised density clustering on unstructured text unlocked the ability for Gmail to detect ephemeral spam campaigns more quickly.The Techstop problem had a similar shape to it. Issues caused by rapidly changing products caused highly dynamic user journeys for both employees and the IT professionals troubleshooting these issues. The tickets filed — like the spam emails — were similar, with slight differences in spelling and word choice.Density clusteringIn contrast to more rigid approaches, such as centroid-based algorithms like k-means, density-based clustering is better suited to large and highly heterogeneous data sets, which may contain clusters of drastically variant size. This flexibility helps us tackle the task of problem identification across the entire scope of the company, which requires the ability to detect and distinguish small-but-significant perturbations in the presence of large-but-stable patterns.Our implementation uses ClustOn, an in-house technology with a hybrid approach that incorporates density-based clustering. But a more time-tested algorithm such as DBSCAN — an open-source implementation of which is available via scikit-learn’s clustering module — could be leveraged to similar effect.Middle of the road solution using MLPiggy-backing off of what Gmail was able to do using density clustering techniques, the Techstop team built a robust solution to tracking problems in a way that solved the rigid taxonomy problem. With density clustering, the taxonomy buckets are redefined as trending clusters and provide an index of issues happening in real-time within the company. Importantly, these buckets emerge naturally, rather than being defined ahead of time by the Engineering or Tech Support teams.By using the technology built for billions of email accounts, we knew we could handle the scale of Google’s support requests. And the solutions would be more flexible than a tightly defined taxonomy, without compromising on relevance or granularity.  The team took it one step further by modeling cluster behavior using Poisson regression and implemented anomaly detection measures to alert operations teams in real time about ongoing outages, or poorly executed changes. With a lightweight operations team and this new technology, Techstop was able to find granular insights that would have taken an entire dedicated team to manually comb through and aggregate each incident.The combination of ML and Operations transformed Techstop data into a valuable reference for product managers and engineering teams looking to understand the issues users face with their products in an enterprise environment. How it worksTo bring it all together, we built a ML pipeline that we call Support Insights, so we could automatically distill summary data from the many interactions and tickets we received. The Support Insights Pipeline combines machine learning, human validation and probabilistic analysis  together in a single systems dynamics approach. As data moves through this pipeline, they are:Extracted – Uses the BigQuery API to store and extract, train and load support data. To ingest the 1M+ amount of IT related support data.Processed Part-of-speech tagging, PII Redaction and TF-IDF transformations to model support data for our clustering algorithmsClustered Centroid-based clustering runs in timed batches with persistent snaphotting of previous run states to maintain cluster ids and track behavior of clusters over time.Scored Uses Poisson Regression to model both long-term and short-term behavior of cluster trends and calculates the difference between the two to measure deviation. This deviation score is used to detect anomalous behavior within a trend.Operationalized Trends with an anomalous score over a certain threshold trigger an IssueTracker API bug. This bug is then picked up by operations teams for relevant deep dive and incident tracking.Resampled – Uses statistical methods to estimate proportions of customer user journeys (CUJs) within trendsCategorized/mapped – We work with the Operations teams to map trend proportions to User Journey SegmentsIn our next post we’ll detail what technologies and methods we used for these seven steps, and walk through how you could use a similar pipeline yourself. To get started start by loading your data into BigQuery and use BigQuery ML to cluster your support data.Related ArticleHow to provide better search results with AI rankingAutomatic scoring and natural language models can make internal search results much more relevant and timely, especially when new trends …Read Article
Quelle: Google Cloud Platform

Amazon ECS fügt jetzt Informationen zum Zustand von Container-Instances hinzu

Amazon Elastic Container Service (Amazon ECS) bietet Kunden jetzt einen besseren Einblick in den Zustand ihrer Computing-Infrastruktur. Kunden, die containerisierte Workloads mit Amazon ECS auf der Amazon Elastic Compute Cloud (Amazon EC2) oder vor Ort mit Amazon ECS Anywhere ausführen, können jetzt den Zustandsstatus der Container-Laufzeit (z. B. Docker) für ihre Container-Instances direkt über die Amazon-ECS-API abfragen. Dies hilft den Kunden, die Ausfallsicherheit ihrer Anwendungen zu verbessern.
Quelle: aws.amazon.com

Zentrale Verwaltung des Zugriffs für JumpCloud-Benutzer mit AWS Single Sign-On

Kunden können nun ihre JumpCloud-Verzeichnis-Plattform (JumpCloud) einmal mit Single Sign-On (SSO) von Amazon Web Services verbinden, den Zugriff auf AWS zentral in AWS SSO verwalten und Endbenutzern ermöglichen, sich mit JumpCloud anzumelden, um auf alle ihre zugewiesenen AWS-Konten zuzugreifen. Die Integration hilft Kunden, die AWS-Zugriffsverwaltung über mehrere Konten hinweg zu vereinfachen und gleichzeitig das vertraute JumpCloud-Erlebnis für Administratoren, die Identitäten verwalten, und für Endbenutzer bei der Anmeldung beizubehalten. AWS SSO und JumpCloud nutzen standardbasierte Automatisierung zur Bereitstellung von Benutzern und Gruppen in AWS SSO, wodurch Kunden Verwaltungszeit sparen und die Sicherheit erhöhen können.
Quelle: aws.amazon.com

Ankündigung von neuem Integritätsschutz bei der Bereitstellung für Amazon-SageMaker-Inference-Endpunkte

Amazon SageMaker Inference unterstützt jetzt neue Modellbereitstellungsoptionen, um Ihre Modelle für Machine Learning in der Produktion zu aktualisieren. Mit dem neuen Integritätsschutz bei der Bereitstellung können Sie einfach und kontrolliert vom aktuellen Modell in der Produktion zu einem neuen wechseln. Mit dieser Einführung werden die Modi Canary und Linear zur Verlagerung des Datenverkehrs eingeführt, so dass Sie die Verlagerung des Datenverkehrs von Ihrem aktuellen Modell auf das neue Modell im Verlauf der Aktualisierung detailliert steuern können. Mit integrierten Schutzmechanismen wie Auto-Rollbacks können Sie Probleme frühzeitig erkennen und automatisch Korrekturmaßnahmen ergreifen, bevor sie erhebliche Auswirkungen auf die Produktion haben.
Quelle: aws.amazon.com

Amazon EKS auf AWS Fargate unterstützt jetzt den Fluent-Bit-Kubernetes-Filter

Amazon Elastic Kubernetes Service (EKS) auf Fargate unterstützt jetzt die Verwendung von Kubernetes-Fluent-Bit-Filtern, die angereicherte Kubernetes-spezifische Metadaten an Fluent-Bit-Protokolle liefern. Kunden können ihre Anwendungen nun einfacher beobachten und Fehler beheben, indem sie den Namen des Kubernetes-Pods, Containers oder Namespaces sowie andere Kubernetes-Metadaten mit den Protokollen ihrer Anwendungen verknüpfen.
Quelle: aws.amazon.com

Avoiding GCF anti-patterns part 4: How to handle Promises correctly in your Node.js Cloud Function

Editor’s note: Over the next several weeks, you’ll see a series of blog posts focusing on best practices for writing Google Cloud Functions based on common questions or misconceptions as seen by the Support team.  We refer to these as “anti-patterns” and offer you ways to avoid them.  This article is the fourth post in the series.ScenarioYou notice that the data your Function saves to a database is either “undefined” or it is saving a cached value. For example, you have a Cloud Task that every hour invokes a Cloud Function that retrieves data from one database, transforms that data, and then saves the modified data to another database. Yet, you notice that your data is either undefined or a cached value. Most common root issueAn unhandled promise. One common anti-pattern we notice when using the then-able approach in a Cloud Function is that it is easy to overlook an unhandled Promise. For example, can you spot the issue in the following example?You will not know how long the call transformData(response.data) will take. And more likely, this call is probably an async method. The result is that the saveDataToDatabase method is executed before transformData() has completed. Thus, the variable dataNowTransformed is undefined upon saving to the database.  How to investigateAre you using await? If not, we recommend using async and await keywords to help improve the readability of your code.If you cannot convert to await at this time, you’ll need to add a logging layer (see example below) to determine if you have an unhandled promise.How to add a logging layer using then-able functionsThere are two ways to log: sync logging by modifying your callbacksasync logging to avoid modifying your callbacks (i.e. adding a logging layer using then-able functions)Suppose you are okay with modifying your callbacks. You can add a synchronous logging layer just by using console.log(). We recommend creating a synchronous method called logData() function to keep your code clean and avoid numerous console.log() statements throughout your code.where logData()  looks like: Now suppose you do not want to modify the code within your callbacks. We recommend adding an async logging method as follows:  1. Create an asynclogDataAsync() method in your Function.2. Call the logDataAsync method using the then-able() approachPlease see the helpful tips section for a more compact way to apply the async logging approach. How to handle the Promise using the then-able approachWe recommend that you perform one task at a time within a .then() callback. Going back to our original anti-pattern example, let’s update it to use the then-able approach. Here’s an end-to-end working example:But all these back to back .then()  calls (called Promise chaining) make the code difficult to read and maintain. Please see the helpful tips section for a more compact way to write this code, in case you see it elsewhere.If possible, we suggest that you use awaits. Notice how the code in the Function event handler callToSlowRespondingAPI now becomes more succinct and readable. In addition, if anything goes wrong in these async method calls, an exception is thrown, in lieu of returning null or false in the return statement.  Other helpful tipsAre you testing locally using the Functions Framework? You can follow this codelab to learn how to debug Node.js functions locally in Visual Studio Code.Whenever you are logging data, be aware of how much data you are logging (see log size limits) and whether your data has any sensitive information or personally identifiable information.Using the then-able() approach, you might often see code written as follows. This is functionally equivalent to the longer then-able() Promise-chaining version used above. However, we recommend using the async await approach for readability. Related ArticleAvoiding GCF anti-patterns part 3: How to establish outbound connections correctlyThird post in a series on how to avoid anti-patterns in Google Cloud Functions as seen by the Support team. This post explores how to mak…Read Article
Quelle: Google Cloud Platform

Modernizing compliance: Introducing Risk and Compliance as Code

Almost all publicly reported breaches in the cloud stem from misconfigurations, rather than from attacks that compromise underlying cloud infrastructure. Misconfigurations continue to be a source of security risk because most security and compliance practices play catchup – teams are involved later in the CI/CD process and misconfigurations are identified at runtime, instead of during the build process. Reliance on runtime security also creates friction between developers and security professionals because runtime tools, by their nature, are deployed at the end of the CI/CD process, and are therefore often seen as the final gate or blocker to production.To prevent and address the risk of misconfigurations and compliance violations earlier in the development process, security leaders have started to embrace security as code to achieve the speed and agility of DevOps, reduce risk, and more securely create value in the cloud. “Being able to precisely model and then continuously monitor the adoption and correct operation of controls in any environment is essential. In the software defined environment (i.e., cloud-native workloads) this is not only possible but more importantly it’s actually more easily achievable than other environments—and the more you do it the easier it becomes for continued monitoring.” —Phil Venables, Chief Information Security Officer, Google Cloud.Recognizing the need and opportunity to help customers prevent security misconfigurations and  automate cloud compliance, the Google Cybersecurity Action Team is thrilled to announce the launch of our Risk and Compliance as Code (RCaC) Solution. The RCaC solution stack enables compliance and security control automation through a combination of Google Cloud Products, Blueprints, Partner Integrations, workshops and services to simplify and accelerate time to value:Existing products such as Assured Workloads, Security Command Center (SCC), and Risk Manager. Assured Workloads helps you define secure configurations and controls as code in your cloud architecture via APIs which are also expressed in some of our blueprints. SCC allows you to monitor for security misconfigurations and compliance violations on a continuous basis.  Risk Manager gives you tools to leverage cyber insurance to deal with risks in the Google Cloud environment. A core set of blueprints such as Secure Foundations, Anthos Security blueprints, workload specific blueprints such as PCI DSS on GKE, and FedRAMP aligned 3-tier workload that codify infrastructure and policies. Blueprints can help you rapidly configure cloud environments in a secure and compliant manner. Partner integrations (such as Sysdig and others) with SCC to detect drift from blueprinted environments. These integrations expand the coverage beyond Google Cloud’s native controls to help deliver improved multi-cloud compliance and risk reduction.A policy library set mapped to common compliance frameworks such as NIST 800-53, PCI DSS, and ISO 27001 with preventative and detective controls that can be expressed as code. These policies communicate which controls can be codified from the above frameworks. Whitepapersand workshops  for rapid security organization transformation and DevSecOps transformation. Professional services and partner-led accelerator programs that enable organizations to pilot the solution. Operationalizing Risk and Compliance as CodeThrough the RCaC solution, customers can introduce automation via IaC (Infrastructure as Code) and PaC (Policy as Code) in the form of blueprints. This lays the foundation of preventative controls. Additionally, customers can “shift-left” their security and compliance practices by evaluating IaC and PaC templates for security and compliance violations before they are used in a build.The next level of maturity is detection as code which involves monitoring for (security and compliance) drifts and applying remediations when an out-of-compliance infrastructure is identified. This forms a continuous monitoring loop that helps prevent misconfigurations. Cloud-native tooling helps to operate this model at scale.Three key benefits of Risk and Compliance as CodeWith RCaC, our hope is to provide our customers with the necessary components to express security and compliance requirements as code and shift left, leading to reduced risk and impact of misconfigurationsa continued compliance and security monitoring environment that is based on automation and code.Encourage a shift in the culture where there is reduced friction between developers and security and compliance teams Our goal with RCaC is to reduce the audit burden and fatigue that is experienced by GRC professionals as they modernize their infrastructure and at the same time continue to meet their compliance obligations.Implementing the Risk and Compliance as Code (RCaC) approachImplementing RCaC requires a substantial policy, architectural, and cultural change for almost all organizations. It requires a change in mindset from compliance being a reactive or a check-box exercise vs. addressing it proactively. Our solution helps organizations progress through this transition.For this reason, many have found it helpful to use the RCaC framework to classify workloads according to sensitivity and criticality to apply specific preventative controls based on workload risk and deployment type. Once this codification is realized, customers can leverage tools inside Security Command Center to continuously monitor for drift and non-compliance. Finally, customers can also build custom drift correction or leverage our Risk Protection Program with insurance providers to reduce security risk and gain access to an exclusive cyber insurance policy designed exclusively for Google Cloud customers. “Using blueprints to simplify meeting compliance standards is a real win for customers. Automation of the right configurations and controls as code helps reduce risk and accelerate cloud success. With tools like Sysdig Secure, Google Cloud customers can easily monitor for drift to make sure they remain secure and compliant over time, while also gaining runtime visibility,” said Omer Azaria, Sysdig VP of Research and Development.Furthermore, RCaC provides a model for future-state architecture and outlines key decisions necessary for various automation use cases. This approach also paves the path towards self-healing cloud native infrastructure and autonomic cloud security.  To learn more about the solution, review the details at the website. For broader context, read the Google Cybersecurity Action Team paper “Assuring Compliance in the Cloud” and listen to our Google Cloud Security Podcast “Making Compliance Cloud-native” (episode 14). If you need the solution, request a briefing with Google Cloud Sales.Related ArticleModernizing SOC … Introducing Autonomic Security OperationsWe’ve launched the Autonomic Security Operations solution, a new approach to transforming Security Operations to protect against modern-d…Read Article
Quelle: Google Cloud Platform