da Agency

TechEd 2021: What’s new for SAP customers on Google Cloud

SAP’s TechEd 2021 is the premier developer and IT-focused virtual event for any business that runs — or is planning to run — SAP applications, and it gets under way this week. What better time to take stock and share how, together, we’re creating value for SAP customers? For Google Cloud, 2021 has easily been the most productive year in our working relationship with SAP, as we continue our joint focus on helping enterprises complete their journey to the cloud as quickly and successfully as possible. In the last 12 months alone, we’ve added three times the number of SAP customers as the year prior. Here’s what new and what’s just ahead for SAP customers on Google Cloud: Google Cloud rollouts for SAP Business Technology PlatformWe’re excited to announce that a new instance of SAP Business Technology Platform (BTP) is now available on Google Cloud, and we’re kicking off our BTP rollouts in North America and Europe during Q4 of 2021. This will be followed by BTP rollouts for our customers in Asia Pacific Japan (APJ) with a BTP instance available in India during Q1 of 2022.SAP Business Technology Platform enables SAP customers to move quickly to leverage the cloud, even for relatively complex use cases. And it puts an emphasis on freedom of choice by allowing customers to choose the right blend of cloud solutions to address their unique needs — and to pivot quickly with new capabilities as those needs change.Perhaps most important, running BTP (or any other set of SAP technologies) on Google Cloud gives customers the benefits of operating on the industry’s most extensive, fully owned and operated, global network infrastructure. That gives SAP customers some unique and very important network security and latency advantages.HANA Cloud to launch on Google CloudSAP’s fully managed HANA Cloud is an important component of SAP’s cloud portfolio, offering a database that can serve OLTP, OLAP, and xOLTP use cases. Like any business-critical application running in the cloud, it’s important to ensure that HANA Cloud delivers flawless performance — something that can be a major challenge for SAP customers looking to minimize network latency.That makes Google Cloud, with its low-latency global networking capabilities and flexible Compute Engine application platform, an ideal partner for SAP customers running HANA Cloud as part of their SAP portfolio. HANA Cloud also provides data virtualization capabilities with BigQuery, allowing customers flexibility in managing their enterprise data footprint. We’re pleased to announce that HANA Cloud is officially launched on Google Cloud in North America, with expansion to EMEA customers expected later this quarter (Q4 2021). We’re also launching HANA Cloud on Google Cloud for our APAC customers, with a rollout in India planned for early 2022.Even SAP customers who aren’t currently Google Cloud customers can take advantage of our HANA Cloud rollout — giving them the same performance and latency benefits even if they choose to run other SAP applications on other hyperscalers or within on-premises environments.Filestore Enterprise for RISE with SAPAnother key element of our SAP partnership centers on RISE with SAP — a “business transformation as a service” offering that gives enterprises a faster, lower-risk strategy for migrating their SAP environments to the cloud. Now, SAP is integrating our Filestore Enterprise technology as a standard part of RISE with SAP S/4HANA Cloud, private edition solutions. With its 99.99% regional-availability SLA, Filestore Enterprise is a fully-managed, cloud-native NFS solution designed for applications that demand high availability. With a few clicks (or a few lines of gCloud CLI or API calls), you can simply provision NFS shares that are synchronously replicated across three zones within a region. Filestore also lets you take periodic snapshots of the file system, and easily recover an individual file or an entire file system in less than 10 minutes from any prior snapshot recovery points.For HEC/RISE with SAP customers, Filestore Enterprise will be an important capability in terms of ensuring high availability, high performance, high levels of redundancy, and higher SLAs for business-critical SAP applications. It’s another way to give SAP customers a simpler, lower-risk path to the cloud.VPC Network Peering: More value for RISE with SAPFinally, we wanted to call out an existing Google Cloud capability with significant benefits for RISE with SAP customers: VPC Network Peering.When SAP implements a PCE or HEC environment on Google Cloud, it also manages the VPC — a virtualized network that provides the same functionality, performance, and security benefits as a dedicated physical network — for that environment. When you connect your SAP environment with Google Cloud services and applications through VPC Network Peering, the traffic between peered VPC networks remains securely within Google’s network. This is a big advantage for customers concerned with protecting business-critical SAP applications and data while connecting seamlessly to other applications running on Google Cloud.A partnership built to create lasting valueAll of these announcements build on the things that already make Google Cloud a natural partner for SAP customers: security, reliability, flexibility, and freedom of choice. Best of all, as a trusted partner in this journey with SAP and its customers, we know that even bigger successes are coming on the road ahead. Learn more about Google Cloud solutions for SAP customers.
Quelle: Google Cloud Platform

da Agency

Get planet-scale monitoring with Managed Service for Prometheus

Prometheus, the de facto standard for Kubernetes monitoring, works well for many basic deployments, but managing Prometheus infrastructure can become challenging at scale. As Kubernetes deployments continue to play a bigger role in enterprise IT, scaling Prometheus for a large number of metrics across a global footprint has become a pressing need for many organizations. Today, we’re excited to announce the public preview of Google Cloud Managed Service for Prometheus, a new monitoring offering designed for scale and ease of use that maintains compatibility with the open-source Prometheus ecosystem. With Google Cloud Managed Service for Prometheus, you have an alternative to taking on the toil of self-managing a Prometheus or Thanos stack in perpetuity. Instead, you can get global and globally scalable monitoring with Prometheus interfaces, letting you keep open-source ecosystem compatibility and portability.Details about Google Cloud Managed Service for Prometheus, currently in previewManaged Service for Prometheus is Google Cloud’s fully managed collection, storage, and query service for Prometheus metrics. It is built on top of Monarch, the same globally scalable data store that powers all application monitoring at Google. Managed Service for Prometheus lets you monitor and alert on your Kubernetes deployments using Prometheus without having to manually manage and operate Prometheus infrastructure at scale. The service is built as a drop-in replacement for Prometheus and eliminates the need for self-managed solutions like Thanos or Cortex.Google Cloud Managed Service for Prometheus is built on Monarch, which powers all application monitoring at GoogleAn easy to use service with open source interfacesAs a drop-in replacement for Prometheus, Managed Service for Prometheus was designed to have easy onboarding by letting you reuse your existing Prometheus configs. You can also opt to deploy managed collectors for even greater simplicity. It has hybrid- and multi-cloud compatibility, meaning you can monitor any environment where Prometheus can run.Looking beyond data collection, you can keep your existing dashboards in Grafana and PromQL-based rules and alerts without modifying any queries. This means you maintain portability with open source-compatible interfaces that proprietary managed solutions usually do not support.Managed Service for Prometheus is built on the same global and globally scalable backend that Google uses for its own metrics. Collecting over 2 trillion active time series holding 65 quadrillion points, the service can support practically any metric volume that your business produces. The system supports ad-hoc global aggregations at query time over regionally-stored raw metric data. Plus, you get 2-year data retention by default, at no extra cost. Being on the same backend as the rest of Google Cloud’s monitoring services means Managed Service for Prometheus is compatible with Cloud Monitoring. In fact, free Google Cloud platform metrics can be queried alongside your Managed Service for Prometheus metrics within Cloud Monitoring.Customers already seeing success in previewWhen we first announced the preview at Next 2021, we were joined by Bartosz Jakubski from OpenX, who described how Managed Service for Prometheus was supporting the growth of their business by offloading the management of Prometheus and Thanos. Horizon Blockchain Games is another customer that is previewing the service, and they’re already seeing benefits. “We have been running Prometheus ourselves for GKE metrics, but the ongoing maintenance took up too many development hours and we did not want to deal with scaling our metrics infrastructure with our growing business,” said Peter Kieltyka, CEO and Chief Architect. “We started using Google Cloud Managed Service for Prometheus and it just works. It can handle whatever volume we have because it’s built on the same backend that Google uses itself, and we get to keep using the same Grafana dashboards as before while keeping open standards and protocols.”How to get startedConfiguring ingestion for Managed Service for Prometheus is straightforward. Just swap out your existing Prometheus binaries with the Managed Service for Prometheus binary, and set up a new data source so you can configure Grafana to read your metrics. See Managed Service for Prometheus documentation to get started.
Quelle: Google Cloud Platform

da Agency

Cart.com empowers brands with unified ecommerce platform

The ecommerce playing field has been hard to navigate for most retailers, and Cart.com is on a mission to change that. Traditionally, retailers needing to run their online store, order fulfillment, customer service, marketing, and other essential activities have had to cobble together systems to get the capabilities they need – much less having access to analytics across these functions. The result is costly, siloed ecommerce operations that are difficult to manage and scale.It’s clearly not a formula for success, yet that’s the reality facing most retailers. Cart.com, in contrast, has set out to democratize ecommerce by giving brands of all sizes the full capabilities they need to take on the world’s largest online retailers. Our end-to-end environment empowers retailers to keep more of their revenue, set up proven strategies for managing all aspects of their business, and act on valuable insights from customer data every step of the way.Together with our talented team, we’re building a unified ecommerce platform that already provides value to many leading or up and coming brands including Whataburger, GUESS, Dr. Scholl’s, Rowing Blazers, and Howler Bros. We’re excited about the opportunity ahead as we reimagine traditional approaches to online sales, fulfillment, marketing, accessing growth capital, providing a unified view of all ecommerce and marketing analytics, and other activities. Expectations for Cart.com are high, and we are building a company that can scale to $100B in revenue and beyond. Supported by the Startup Program by Google Cloud andGoogle Cloud solutions, we’re establishing a technology platform to transform all aspects of ecommerce for brands worldwide. Partner in disruptionAt Cart.com, we’re currently targeting an underserved market. Our ideal customer is beyond demonstrating product-market-fit and is now at an inflection point seeking a growth opportunity. Typically, those companies are generating between $1M and $100M in annual revenue. We’ve seen an enthusiastic response from brands and retailers as well as investors, with backing from investors in just over a year totaling $143 million in three funding rounds.Our strategy is to build an integrated ecommerce model that combines best-of-breed solutions, many of which we gain through acquisitions and then build upon to provide a streamlined and fully integrated experience for our brands. We’ve made seven acquisitions so far to round out our online store, order fulfillment, marketing services, customer service, and we have launched some integral partnerships including easy access to growth capital through our relationship with Clearco and product protection for customers on every purchase with Extend. Instead of acquiring a data company, we’re building our data platform onGoogle Cloud, across each operating function for a single-view for brands to harness actionable data. We see Google Cloud as the leader for data management, analytics, machine learning (ML) and artificial intelligence (AI).Other reasons why we’re building our business on Google Cloud include scalability, excellence, security, reach, and data analytics that are far superior to other environments.We also feel a cultural and mission alignment with Google Cloud and envision leaning into a long-term partnership of marketing, selling, and disrupting the disruptors together. Equally important to us are the investments Google Cloud is willing to make in early-stage companies like ours. The support through the Google Cloud for Startups program has been outstanding.Built on Google CloudA wide range of Google Cloud solutions provide the foundation for our platform. For instance, Cloud Pub/Sub keeps our services communicating with one another. We rely on fully managed relational databases, like Cloud SQL and Cloud Spanner, to securely handle the huge volume of brand and shopper data generated every day.Cloud Run allowed us to develop inside of containers before our Kubernetes infrastructure was ready to go. Now, we are taking advantage of all the capabilities inGoogle Kubernetes Engine.BigQuery integrates with all Google Cloud solutions and offers true data streaming natively out of the box, along withDataflow for advanced analytics. We also useContainer Registry to store and manage our Docker container images. Right now, we’re testingCloud Composer to evaluate using it for data workflow orchestration instead ofApache Airflow.The openness of the Google Cloud environment is further enabled byAnthos, which we may deploy soon to perform data integrations quickly as we acquire more companies over the next year. For example, if we acquire a company using Azure, we can easily align it with our Google Cloud ecosystem.Enabling ecommerce 2.0Recently, our team has been experimenting with Google CloudVertex AI and the fully managed services of AI deployment and ML operations. The capabilities would save us substantial time in the management of the ML lifecycle which allows us to focus more on developing proprietary AI that will transform commerce at scale.Because Google Cloud is so far ahead in data science, our teams benefit from deep Google Cloud expertise as we look to provide brands with unmatched insights into customers to improve services and revenue. We’re also planning to testRecommendations AI among other tools to deploy customer product recommendations and personalization as turnkey productized offerings. Moving forward, we will likely useBigtable to aid in serving machine learning to hundreds of thousands of brands due to its low latency and scalability.Fanatical about brand successWe know that our work with Google Cloud for Startups and use of Google Cloud solutions for best-in-class data management, analytics, ML, and AI will enable us to offer even more transformative services to brands.We also see the opportunity to use our platform and customer insights to break down barriers between brands, enabling retailers to share information and work better together when it’s in their best interests. What we’re building today on Google Cloud is fundamentally changing what’s possible for retailers of any size everywhere. As a startup, when recruiting talent or working with prospective customers, it helps to share our success with Google Cloud. We view them as an extension of the Cart.com team. It also validates our business as we continue building a more integrated, holistic approach to commerce that opens new opportunities and drives growth for brands worldwide.For more details about Cart.com’s vision for unified ecommerce, check out ourvideo.If you want to learn more about how Google Cloud can help your startup, visit our pagehere to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.
Quelle: Google Cloud Platform

da Agency

Get 6X read performance with Memorystore for Redis Read Replicas

Modern applications need to process large-scale data at millisecond latency to provide experiences like instant gaming leaderboards, fast analysis of streaming data from millions of IoT sensors, or real-time threat detection of malicious websites. In-memory datastores are a critical component to deliver the scale, performance, and availability required by these modern applications. Memorystore makes it easy for developers building applications on Google Cloud to leverage the speed and powerful capabilities of the most loved in-memory store: Redis. Memorystore for Redis standard tier instances are a popular choice for applications requiring a highly available Redis instance. Standard tier provides a failover replica across zones for redundancy and provides fast failover with a 99.9% SLA. However, in some cases, your applications will require more read throughput from a standard tier instance. One of the common patterns customers use to scale read queries in Redis is leveraging read replicas.Introducing Memorystore for Redis Read ReplicasToday we are excited to announce the public preview of Memorystore for Redis Read Replicas, which allows you to seamlessly scale your application’s read requests by 6X with the click of a button.With read replicas, you can easily add up to five replicas and leverage the read endpoint to automatically load balance read queries across all the available replicas, increasing read performance linearly with each replica added. Additionally, Memorystore’s support for Redis 6 introduced multi-thread I/O, increasing performance significantly for M3 and higher configurations. Combined, you can achieve read requests of more than a million requests per second. You will benefit from this new functionality in several ways. You will be able to scale on demand with up to five read replicas and use read endpoint with any redis client to easily load balance read queries across multiple replicas. This new functionality will also improve availability with automatic distribution of replicas across multiple zones. With read replicas, you will also be able to minimize application downtime with fast failover to the replica with the least replication lag. In the future, Memorystore will also easily enable read replicas on existing standard tier instances to increase read throughput. You can learn more about how to configure and use read replicas in the Read Replicas Overview. Improving performance with Read Replicas and Redis 6With the launch of read replicas, you can easily increase the read throughput of a Memorystore instance. You can further enhance the read performance by leveraging Redis version 6 along with read replicas.To understand why combining read replicas and Redis 6 can significantly improve your application’s read performance, let’s look at the various Memorystore configurations that you can use with your applications today. Memorystore Basic and Standard offerings  provide different capacity tiers. The capacity tier determines the single node performance of a basic and standard tier instance. The table below outlines the configuration of the various capacity tiers:Up until version 5, Redis processed commands using a single thread. The processing involved reading the request, parsing the request, processing the request, and writing the response back to the socket. This approach means that all of the processing, which includes writing the response, was sequentially processed by a single vCPU regardless of the number of vCPUs available in the instance.Redis 6 introduced IO threading which allows writing the response using parallel threads. This functionality enables Redis 6 to more effectively leverage available vCPUs, thereby increasing the overall throughput compared to lower versions. Memorystore for Redis version 6 leverages IO threading and automatically configures the optimal number of IO threads to achieve the best possible performance for the various capacity tiers. We have also improved the overall network throughput for all redis versions by leveraging improvements in the Google Cloud infrastructure. Together these improvements deliver significant performance improvements and come at no additional cost to you. So what can you expect from using Redis 6? As outlined in the table, Redis version 5 and lower uses a single thread to process the write requests for all tiers while Redis 6 uses a larger number of IO threads at higher capacity tiers which provides substantially incremental throughput for higher capacity tiers.For example, using a Redis 6 standard tier instance with a capacity of 101 GB (M5), we have observed up to a 200% improvement in read/write performance compared to version 5, though the actual value you’ll see is dependent on your workload. You can get the benefits of Redis 6 by upgrading an existing instance or by deploying a new instance.By enabling read replicas on an instance using Redis 6, you can further improve the read performance. Read throughput scales linearly with the number of replicas so you can increase your read queries on an instance by up to 500% using five read replicas. By combining Redis 6 and read replicas, you can see a 10X increase in read performance compared to a version 5 standard tier instance with no read replicas.We are excited about the launch of read replicas but this is just one step in our journey to deliver the scale you need at the best price-performance. You can learn more about read replicas pricing on our Memorystore pricing page and get started using the Read Replicas Overview.
Quelle: Google Cloud Platform

da Agency

How to create and safeguard your admin accounts

Setting up your new cloud infrastructure is scary. Extra scary when you realize that someone (is it gonna be you?) gets to have phenomenal cosmic power over the whole thing. Yes, I’m talking about the admin account, and today we’ll dig into why they are important, dangerous and different.When the team at pistach.io got their nuts-as-a-service business growing fruitfully, they knew they needed to think carefully about admin accounts. These people would have tremendous control over their use of Google Cloud, and they could potentially cause very big problems if any were compromised. Definitely resources that require a protective shell.Early in pistach.io’s development, an employee named Walter Nutt wanted to play a prank by changing his co-worker’s profile photo from an almond to a peanut (pretty devious, since a peanut is actually a legume). He didn’t have access to his friend’s computer, but he did have access to the company’s Cloud Storage bucket. While searching for the profile photo in question, Wally inadvertently deleted the entire contents of the bucket! As he searched for ways to restore its contents, Wally modified access to two other pistach.io buckets.  The company was ground to a halt for a week while teams worked to crack through the permissions issues.Time to rethink permissions a bit, so this couldn’t spoil their buttery smooth operations in the future.Following the resource manager guide, the team made a super admin email address that wasn’t tied to a particular individual or Workspace account, and secured it with strong multi-factor authentication. This would be their backup in case an admin account were to be compromised, so they could recover and repair.The team already uses Google Workspace, so they have an organization set up already. That creation process established initial super administrators, allowing them to create and modify all other resources inside the organization. As they looked toward using Google Cloud, the super administrators could:Give the admin role to people, for CloudAct as a point of contact for account recoveryModify or delete the organization if neededMaking admin users for the organization allows other people to then flesh out the resources and policies for pistach.io, before they go nuts and give everyone all the permissions. While that would speed things up, it would make it easy for an attacker to crack through the security shell because any account compromise could give ousize access. Yikes!Instead, the IT leads specified certain people to act as organization admins, and then gave them permissions to:Define Identity and Access Management policiesStructure the Resource HierarchyDelegate control of specific Cloud elements to others on the teamOnce those organization admins were set up, they could give management and oversight of Compute, Storage, Networking and other resource types to the relevant leads, making sure each person had just the right amount of permission for the role they needed to perform. The organization admins don’t have permissions themselves to make these resources. They just delegate. Now each person can accomplish the job they’re responsible for, but doesn’t have overly permissive access. Delegating like this keeps the entire organization safer, and limits the blast radius if someone does manage to break in.You can go through these steps yourself with this tutorial.By default the creation of an organization resource for the domain gives everyone the ability to create projects and billing accounts. Once they set up their Organization Admin at pistach.io they decided to remove some of these wide permissions and, in a nutshell, bring everything down to a much finer control. So people could get permissions for a folder or a project, but not the entire organization!Remember to take care of your admin roles, as they have the power, and responsibility, to cause serious harm if not used safely. Be safe with your Identity and Access Management. And keep your data yours! Next time we join you we’ll take a crack at creating and provisioning an app to run inside the policies and resource management frameworks created today.Related ArticleYou make the rules with authentication controls for Cloud StorageOnce you’ve got your data into Cloud Storage, it’s time for an important conversation about authentication. In this post, we’ll review so…Read Article
Quelle: Google Cloud Platform

da Agency

How spam detection taught us better tech support

Information Technology teams, especially in help desk and support, need a way to track what problems people are having. Ideally they also can know how those problems change over time, especially when technology or policy shifts.Imagine you are in charge of sending a newspaper delivery team to different neighborhoods. Each person has a bicycle, so you give them a route, and they leave the papers at the right doors. But the roads change. Every day they change. It’s chaos.What do you do when routes are changing constantly?  How do you provide the information needed when the context is shifting all the time?In an IT context we run into similar challenges with traditional problem management frameworks such as ITIL 4, which tend to always assume a fixed, well-defined catalog of services. That way every issue that the IT folks solve is tracked and accounted for. That connection back to the catalog allows insight into what’s causing issues, or where outages or incidents may be impacting a large group of employees. At Google we don’t have that. In part because we focus on putting the user first, and so we focus on getting people back to a productive state as job #1. Also because products, services and issues are always shifting–just like the roads, the route is never the same, even if the goal remains consistent. That means users come into our IT service desk with new problem types everyday.Our tech support team, called Techstop, acts as the one-stop shop for all IT issues, and supports people across chat, email and video channels. They need to remain adaptable to new problems Googlers experience and new products they use. In order to track what problems might be on the rise, the Techstop team needs a way to catalog what tools, applications and services are in use at Google. Thinking back to the newspaper delivery routes, we used a rough approximate map, rather than a very detailed one, giving us a taxonomy of services that was “good enough” for most of our use cases. We got some useful data out of it, but it didn’t give us very granular insight.Need for innovationCovid-19 put a new focus on scalable problem understanding, specifically for everyday employee IT issues. With so much of the workforce moved to a work-from-home model, we really needed to know where employees were experiencing technology pain. It’s as if whole new neighborhoods popped into existence overnight, but our newspaper delivery crew was the same. More ground to cover, with totally novel street maps.Furthermore, products used everyday for productivity, such as Google Meet, began to see exponential growth in usage, causing scaling issues and outages. These product teams looked to the Techstop organization to help them prioritize the ever increasing list of feature requests and bugs being filled every day.Ultimately the “good enough” problem taxonomy failed to produce truly helpful insights. We could find out which products were being affected the most, but not what issues people were having with those products. Even worse, new issues that were unique to the work-from-home model were being hidden by the fact that the catalog could not update in time to catch the rapidly changing problem space underneath it.Borrowing spam techTaking a look around other efforts at Google, the Techstop team found examples of solving a similar problem: detecting new patterns quickly in rapidly changing data.Gmail handles spam filtering for over a billion people. Those engineers had thought through “how do we detect a new spam campaign quickly?”  Spammers rapidly send bulk messages with slight variations in content (noise, misspellings, etc.) Most classification attempts would become a game of cat and mouse since it takes classifiers some time to learn about new patterns.Invoking a trend identification engine using unsupervised density clustering on unstructured text unlocked the ability for Gmail to detect ephemeral spam campaigns more quickly.The Techstop problem had a similar shape to it. Issues caused by rapidly changing products caused highly dynamic user journeys for both employees and the IT professionals troubleshooting these issues. The tickets filed — like the spam emails — were similar, with slight differences in spelling and word choice.Density clusteringIn contrast to more rigid approaches, such as centroid-based algorithms like k-means, density-based clustering is better suited to large and highly heterogeneous data sets, which may contain clusters of drastically variant size. This flexibility helps us tackle the task of problem identification across the entire scope of the company, which requires the ability to detect and distinguish small-but-significant perturbations in the presence of large-but-stable patterns.Our implementation uses ClustOn, an in-house technology with a hybrid approach that incorporates density-based clustering. But a more time-tested algorithm such as DBSCAN — an open-source implementation of which is available via scikit-learn’s clustering module — could be leveraged to similar effect.Middle of the road solution using MLPiggy-backing off of what Gmail was able to do using density clustering techniques, the Techstop team built a robust solution to tracking problems in a way that solved the rigid taxonomy problem. With density clustering, the taxonomy buckets are redefined as trending clusters and provide an index of issues happening in real-time within the company. Importantly, these buckets emerge naturally, rather than being defined ahead of time by the Engineering or Tech Support teams.By using the technology built for billions of email accounts, we knew we could handle the scale of Google’s support requests. And the solutions would be more flexible than a tightly defined taxonomy, without compromising on relevance or granularity.  The team took it one step further by modeling cluster behavior using Poisson regression and implemented anomaly detection measures to alert operations teams in real time about ongoing outages, or poorly executed changes. With a lightweight operations team and this new technology, Techstop was able to find granular insights that would have taken an entire dedicated team to manually comb through and aggregate each incident.The combination of ML and Operations transformed Techstop data into a valuable reference for product managers and engineering teams looking to understand the issues users face with their products in an enterprise environment. How it worksTo bring it all together, we built a ML pipeline that we call Support Insights, so we could automatically distill summary data from the many interactions and tickets we received. The Support Insights Pipeline combines machine learning, human validation and probabilistic analysis  together in a single systems dynamics approach. As data moves through this pipeline, they are:Extracted – Uses the BigQuery API to store and extract, train and load support data. To ingest the 1M+ amount of IT related support data.Processed Part-of-speech tagging, PII Redaction and TF-IDF transformations to model support data for our clustering algorithmsClustered Centroid-based clustering runs in timed batches with persistent snaphotting of previous run states to maintain cluster ids and track behavior of clusters over time.Scored Uses Poisson Regression to model both long-term and short-term behavior of cluster trends and calculates the difference between the two to measure deviation. This deviation score is used to detect anomalous behavior within a trend.Operationalized Trends with an anomalous score over a certain threshold trigger an IssueTracker API bug. This bug is then picked up by operations teams for relevant deep dive and incident tracking.Resampled – Uses statistical methods to estimate proportions of customer user journeys (CUJs) within trendsCategorized/mapped – We work with the Operations teams to map trend proportions to User Journey SegmentsIn our next post we’ll detail what technologies and methods we used for these seven steps, and walk through how you could use a similar pipeline yourself. To get started start by loading your data into BigQuery and use BigQuery ML to cluster your support data.Related ArticleHow to provide better search results with AI rankingAutomatic scoring and natural language models can make internal search results much more relevant and timely, especially when new trends …Read Article
Quelle: Google Cloud Platform

da Agency

Avoiding GCF anti-patterns part 4: How to handle Promises correctly in your Node.js Cloud Function

Editor’s note: Over the next several weeks, you’ll see a series of blog posts focusing on best practices for writing Google Cloud Functions based on common questions or misconceptions as seen by the Support team.  We refer to these as “anti-patterns” and offer you ways to avoid them.  This article is the fourth post in the series.ScenarioYou notice that the data your Function saves to a database is either “undefined” or it is saving a cached value. For example, you have a Cloud Task that every hour invokes a Cloud Function that retrieves data from one database, transforms that data, and then saves the modified data to another database. Yet, you notice that your data is either undefined or a cached value. Most common root issueAn unhandled promise. One common anti-pattern we notice when using the then-able approach in a Cloud Function is that it is easy to overlook an unhandled Promise. For example, can you spot the issue in the following example?You will not know how long the call transformData(response.data) will take. And more likely, this call is probably an async method. The result is that the saveDataToDatabase method is executed before transformData() has completed. Thus, the variable dataNowTransformed is undefined upon saving to the database.  How to investigateAre you using await? If not, we recommend using async and await keywords to help improve the readability of your code.If you cannot convert to await at this time, you’ll need to add a logging layer (see example below) to determine if you have an unhandled promise.How to add a logging layer using then-able functionsThere are two ways to log: sync logging by modifying your callbacksasync logging to avoid modifying your callbacks (i.e. adding a logging layer using then-able functions)Suppose you are okay with modifying your callbacks. You can add a synchronous logging layer just by using console.log(). We recommend creating a synchronous method called logData() function to keep your code clean and avoid numerous console.log() statements throughout your code.where logData()  looks like: Now suppose you do not want to modify the code within your callbacks. We recommend adding an async logging method as follows:  1. Create an asynclogDataAsync() method in your Function.2. Call the logDataAsync method using the then-able() approachPlease see the helpful tips section for a more compact way to apply the async logging approach. How to handle the Promise using the then-able approachWe recommend that you perform one task at a time within a .then() callback. Going back to our original anti-pattern example, let’s update it to use the then-able approach. Here’s an end-to-end working example:But all these back to back .then()  calls (called Promise chaining) make the code difficult to read and maintain. Please see the helpful tips section for a more compact way to write this code, in case you see it elsewhere.If possible, we suggest that you use awaits. Notice how the code in the Function event handler callToSlowRespondingAPI now becomes more succinct and readable. In addition, if anything goes wrong in these async method calls, an exception is thrown, in lieu of returning null or false in the return statement.  Other helpful tipsAre you testing locally using the Functions Framework? You can follow this codelab to learn how to debug Node.js functions locally in Visual Studio Code.Whenever you are logging data, be aware of how much data you are logging (see log size limits) and whether your data has any sensitive information or personally identifiable information.Using the then-able() approach, you might often see code written as follows. This is functionally equivalent to the longer then-able() Promise-chaining version used above. However, we recommend using the async await approach for readability. Related ArticleAvoiding GCF anti-patterns part 3: How to establish outbound connections correctlyThird post in a series on how to avoid anti-patterns in Google Cloud Functions as seen by the Support team. This post explores how to mak…Read Article
Quelle: Google Cloud Platform

da Agency

Modernizing compliance: Introducing Risk and Compliance as Code

Almost all publicly reported breaches in the cloud stem from misconfigurations, rather than from attacks that compromise underlying cloud infrastructure. Misconfigurations continue to be a source of security risk because most security and compliance practices play catchup – teams are involved later in the CI/CD process and misconfigurations are identified at runtime, instead of during the build process. Reliance on runtime security also creates friction between developers and security professionals because runtime tools, by their nature, are deployed at the end of the CI/CD process, and are therefore often seen as the final gate or blocker to production.To prevent and address the risk of misconfigurations and compliance violations earlier in the development process, security leaders have started to embrace security as code to achieve the speed and agility of DevOps, reduce risk, and more securely create value in the cloud. “Being able to precisely model and then continuously monitor the adoption and correct operation of controls in any environment is essential. In the software defined environment (i.e., cloud-native workloads) this is not only possible but more importantly it’s actually more easily achievable than other environments—and the more you do it the easier it becomes for continued monitoring.” —Phil Venables, Chief Information Security Officer, Google Cloud.Recognizing the need and opportunity to help customers prevent security misconfigurations and  automate cloud compliance, the Google Cybersecurity Action Team is thrilled to announce the launch of our Risk and Compliance as Code (RCaC) Solution. The RCaC solution stack enables compliance and security control automation through a combination of Google Cloud Products, Blueprints, Partner Integrations, workshops and services to simplify and accelerate time to value:Existing products such as Assured Workloads, Security Command Center (SCC), and Risk Manager. Assured Workloads helps you define secure configurations and controls as code in your cloud architecture via APIs which are also expressed in some of our blueprints. SCC allows you to monitor for security misconfigurations and compliance violations on a continuous basis.  Risk Manager gives you tools to leverage cyber insurance to deal with risks in the Google Cloud environment. A core set of blueprints such as Secure Foundations, Anthos Security blueprints, workload specific blueprints such as PCI DSS on GKE, and FedRAMP aligned 3-tier workload that codify infrastructure and policies. Blueprints can help you rapidly configure cloud environments in a secure and compliant manner. Partner integrations (such as Sysdig and others) with SCC to detect drift from blueprinted environments. These integrations expand the coverage beyond Google Cloud’s native controls to help deliver improved multi-cloud compliance and risk reduction.A policy library set mapped to common compliance frameworks such as NIST 800-53, PCI DSS, and ISO 27001 with preventative and detective controls that can be expressed as code. These policies communicate which controls can be codified from the above frameworks. Whitepapersand workshops  for rapid security organization transformation and DevSecOps transformation. Professional services and partner-led accelerator programs that enable organizations to pilot the solution. Operationalizing Risk and Compliance as CodeThrough the RCaC solution, customers can introduce automation via IaC (Infrastructure as Code) and PaC (Policy as Code) in the form of blueprints. This lays the foundation of preventative controls. Additionally, customers can “shift-left” their security and compliance practices by evaluating IaC and PaC templates for security and compliance violations before they are used in a build.The next level of maturity is detection as code which involves monitoring for (security and compliance) drifts and applying remediations when an out-of-compliance infrastructure is identified. This forms a continuous monitoring loop that helps prevent misconfigurations. Cloud-native tooling helps to operate this model at scale.Three key benefits of Risk and Compliance as CodeWith RCaC, our hope is to provide our customers with the necessary components to express security and compliance requirements as code and shift left, leading to reduced risk and impact of misconfigurationsa continued compliance and security monitoring environment that is based on automation and code.Encourage a shift in the culture where there is reduced friction between developers and security and compliance teams Our goal with RCaC is to reduce the audit burden and fatigue that is experienced by GRC professionals as they modernize their infrastructure and at the same time continue to meet their compliance obligations.Implementing the Risk and Compliance as Code (RCaC) approachImplementing RCaC requires a substantial policy, architectural, and cultural change for almost all organizations. It requires a change in mindset from compliance being a reactive or a check-box exercise vs. addressing it proactively. Our solution helps organizations progress through this transition.For this reason, many have found it helpful to use the RCaC framework to classify workloads according to sensitivity and criticality to apply specific preventative controls based on workload risk and deployment type. Once this codification is realized, customers can leverage tools inside Security Command Center to continuously monitor for drift and non-compliance. Finally, customers can also build custom drift correction or leverage our Risk Protection Program with insurance providers to reduce security risk and gain access to an exclusive cyber insurance policy designed exclusively for Google Cloud customers. “Using blueprints to simplify meeting compliance standards is a real win for customers. Automation of the right configurations and controls as code helps reduce risk and accelerate cloud success. With tools like Sysdig Secure, Google Cloud customers can easily monitor for drift to make sure they remain secure and compliant over time, while also gaining runtime visibility,” said Omer Azaria, Sysdig VP of Research and Development.Furthermore, RCaC provides a model for future-state architecture and outlines key decisions necessary for various automation use cases. This approach also paves the path towards self-healing cloud native infrastructure and autonomic cloud security.  To learn more about the solution, review the details at the website. For broader context, read the Google Cybersecurity Action Team paper “Assuring Compliance in the Cloud” and listen to our Google Cloud Security Podcast “Making Compliance Cloud-native” (episode 14). If you need the solution, request a briefing with Google Cloud Sales.Related ArticleModernizing SOC … Introducing Autonomic Security OperationsWe’ve launched the Autonomic Security Operations solution, a new approach to transforming Security Operations to protect against modern-d…Read Article
Quelle: Google Cloud Platform

da Agency

How Google Cloud BigQuery enables big DevOps at JFrog

Editor’s note: Today we’re hearing from Mitali Bisht, Software Engineer at JFrog on how BigQuery and Data Studio powers operational analytics on the JFrog DevOps Platform. At JFrog, we know that keeping DevOps running smoothly requires knowing as much as you can about those operations. It’s a key principle of Artifactory, our artifact repository manager that powers the JFrog DevOps Platform. Information – in Artifactory’s case, artifact and build metadata – provides traceable paths through the complex systems we build every day. Data and the power to analyze it enables smart decisions by people and machines.So to better serve our JFrog Cloud customers running their SaaS subscriptions on Google Cloud Platform (GCP), we needed to be able to collect and analyze operational data on hundreds of their deployments.We wanted to gather statistics to serve metadata to users to make better decisions such as:Who is actively using their JFrog accounts, by IP address?Is there activity that suggests an attempted cyberattack?Which modules or packages people use the most?How efficiently are those resources being used?On a single-user scale, we’ve provided some facilities for our self-hosted customers through our JFrog Platform Log Analytics integrations, to configure for themselves and view their high-availability deployment’s activity through analytics programs like Splunk and DataDog.To monitor our SaaS operations on GCP, however, we needed to craft a solution that could extract and analyze this kind of data from multiple deployments on a much more massive scale.Among the many GCP services available, we were able to make particular use of Cloud Logging, BigQuery, and Data Studio to collect, analyze, and visualize such vast quantities of operations data in real time. Let’s dive into the architecture we used for this project at JFrog.Stage 1: Ingesting Data from LogsWe had two sources of logs to ingest data from:The nginx server serving our Artifactory SaaS instancesLogs streamed in from external cloud storageNGINX Access LogsFor the first, we already had the google-fluentd logging agent pod setup by default  while setting up our Kubernetes cluster on GKE. The logging agent google-fluentd is a modified version of the fluentd log data collector. In its default configuration, the logging agent streams logs, as included in the list of default logs, to Cloud Logging. This default setup for nginx-access was sufficient; there was no need to customize the agent configuration to stream any additional logs.In Cloud Logging, all logs, including audit logs, platform logs, and user logs, are sent to the Cloud Logging API where they pass through the Logs Router. The Logs Router checks each log entry against existing rules to determine which log entries to discard, which log entries to ingest (store) in Cloud Logging, and which log entries to route to supported destinations using log sinks. Here we created the log sinks to export the logs into the BigQuery partitioned table. The object `Sink` holds the inclusion/exclusion filter and destination. You can create/view sinks under the Logging–>Logs router section of your GCP project. For example, our inclusion filter reads:External Cloud Storage LogsIn our external cloud storage, logs for multiple projects accumulate in the same bucket. To select only the logs related to our project, we created a custom Python script and scheduled it to run daily to perform these tasks:Authenticate, read and select the data related to our project.Process the data.Loadthe processed data into BigQuery.We used the BigQuery stream ingestion API to stream our log data directly into BigQuery. There is also  BigQuery Data Transfer Service (DTS) which is a fully managed service to ingest data from Google SaaS apps such as Google Ads, external cloud storage providers such as Amazon S3 and transferring data from data warehouse technologies such as Teradata and Amazon Redshift. DTS automates data movement into BigQuery on a scheduled and managed basis. Stage 2: Storage in BigQueryBigQuery organizes data tables into units called datasets. These datasets are scoped to a  GCP project. These multiple scopes — project, dataset, and table — help structure information logically. In order to refer to a table from the command line, in SQL queries, or in code, we refer to it by using the following construct: `project.dataset.table`.BigQuery leverages the columnar storage format and compression algorithm to store data in Colossus, optimized for reading large amounts of structured data. Colossus also handles replication, recovery (when disks crash) and distributed management (so there is no single point of failure). Colossus enables BigQuery users to scale to dozens of petabytes of data stored seamlessly, without paying the penalty of attaching much more expensive compute resources as in traditional data warehouses.Keeping data in BigQuery is a best practice if you’re looking to optimize both cost and performance. Another best practice is using BigQuery’s table partitioning and clustering features to structure the data to match common data access patterns.When a table is clustered in BigQuery, the table data is automatically organized based on the contents of one or more columns in the table’s schema. The columns you specify are used to collocate related data. When new data is added to a table or a specific partition, BigQuery performs automatic re-clustering in the background to restore the sort property of the table or partition. Automatic reclustering is completely free and autonomous for users.A partitioned table is a special table that is divided into segments, called partitions, that make it easier to manage and query your data. You can typically split large tables into many smaller partitions using data ingestion time or TIMESTAMP/DATE column or an INTEGER column. BigQuery supports the following ways of creating partitioned tables :Ingestion time partitioned tablesDATE/TIMESTAMP column partitioned tablesINTEGER range partitioned tablesWe used ingestion time partitioned BigQuery tables as our data storage. Ingestion time partitioned tables are:Partitioned on the data’s ingestion time or arrival time.BigQuery automatically loads data into daily, date based partitions reflecting the data’s ingestion or arrival time.Partition management is key to fully maximizing BigQuery performance and cost when querying over a specific range — it results in scanning less data per query, and pruning is determined before query start time. While partitioning reduces cost and improves performance, it also prevents cost explosion due to users accidentally querying really large tables in entirety.Ingested log data in BigQueryStage 3: Parse and Process DataBefore we can analyze the raw log data we’ve stored in BigQuery, we need to process it so that it can be more easily queried.Parsing the DataWe used a Python script to massage the raw log data. Our script reads the raw logs we stored in BigQuery partitioned tables, parses them to break down the data, and then stores those refined results in a new BigQuery partitioned table store with more defined columns. We also integrated with MaxMind IP geolocation services to perform IP reverse lookup, and better visualize usage by organization. There are client libraries available for most of the popular languages in order to make API calls to BigQuery.Our Python script runs daily to process the ingested data and return it to BigQuery:Parsed Data in BigQueryAnalyzing the DataBigQuery is highly efficient running multiple concurrent complex queries in very large datasets. The BigQuery compute engine is Dremel, a large multi-tenant cluster that executes SQL queries. Dremel dynamically apportions slots to queries on an as-needed basis, maintaining fairness for concurrent queries from multiple users. A single user can get thousands of slots to run their queries. In between storage and compute is ‘shuffle’, which takes advantage of Google’s Jupiter network to move data extremely rapidly from one place to another.When we run queries in BigQuery, the result sets can be materialized to create new tables rather than storing in temp tables. In this way, we can join data from multiple tables and store in new ones with just one click and hand it over to anybody who does not have access to all those datasets tables by exporting it to GCS or exploring with Google Sheets or Data Studio.Stage 4: Visualize To visualize this processed data, we used GCP Data Studio, a free service that has petabyte-scale processing power and end-to-end integration with the rest of Google Cloud Platform.Data Studio supports 14 Google ecosystem connectors, including BigQuery. One of the unique and beneficial features of Google Data Studio is that it promotes collaboration with other Google Workspace apps. This made it a perfect choice for our BI tool. We created a datasource by selecting the project, dataset, and table we want to visualize. Clicking Explore with Data Studio creates a new report page with options to add charts, filters, and metrics.Big Data for Big DevOpsWith the ability to collect, process, and visualize this huge amount of log data, our JFrog operations teams are better able to understand the evolving needs of our SaaS customers, and to keep their accounts running smoothly on Google Cloud Platform.Through this, we’re even more keenly aware of how JFrog’s enterprise customers rely on the 100s of terabytes of artifact data that collectively pass through their artifact repositories each day to practice DevOps in the cloud. BigQuery’s ability to process this big data helps keep us — and our customers — running at peak efficiency.To accelerate your releases and delivery at enterprise scale through cloud DevOps, check out an Enterprise+ Cloud subscription to the JFrog Platform on Google Cloud Marketplace.
Quelle: Google Cloud Platform

da Agency

A learning journey for members transitioning out of the military

Each year, about 200,000 U.S. veterans transition out of military service. However, despite being well-equipped to work in the tech sector, many of these veterans are unable to identify a clear career path. In fact, a2019 survey found that many veterans feel unprepared for the job market after their service and are unaware of how GI benefits can be used for learning and training. That’s where Google Cloud skills training comes in. This August, 50 service members began a 12-week, Google Cloud-sponsored Certification Learning Journey toward achieving the Google Cloud Associate Cloud Engineer certification. Participants had access to online, on-demand learning assets, bi-weekly technical review sessions taught by Google engineers, and mentoring through Google Groups. Upon completion of the online training, participants will now attempt to pass the Google Cloud Associate Cloud Engineer certification exam. A passing grade will grant these military members a new cloud certification, which is a great way to demonstrate cloud skills to the larger IT market. Getting certified can open up internships, job opportunities and help career progression for our veterans.Why get Google Cloud certified? Cloud computing is one of the fastest-growingareas in IT. As cloud adoption grows rapidly, so do the ways that cloud technologies can solve key business problems in the real world. Cloud certifications are a great way to demonstrate technical skills to the broader market beyond the military. Cloud skills are also in demand. More than 90% of IT leaders say they’re looking to grow their cloud environments in the next several years, yet more than 80% of those same leaders identified a lack of skills and knowledge within their employees as a barrier to this growth. Unfortunately, according to a 2020 survey report, the IT talent shortage continues to be a leading corporate concern, with 86% of respondents believing it will continue to slow down cloud projects. A shrinking pool of qualified candidates poses a top business risk for global executives as they struggle to find and retain talent to meet their strategic objectives.Want to learn more?As we wrap up this first Certification Learning Journey for Service Members, plans are underway to expand to new cohorts in the coming months. The classes are completely virtual, and all training is on-demand so that participants can access their coursework anytime, anywhere via the web or mobile device. To determine whether you (or someone you know) would be a great fit for this certification journey:Watch the Certification Prep: Associate Cloud Engineer webinarComplete the Skill IQ Assessment via PluralsightReview the Associate Cloud Engineer Certification Exam Guide Take the Associate Cloud Engineer sample questionWho’s eligible? U.S. military members transitioning out of service and veterans with CS/CIS related education or relevant work experience (IT, CyberSecurity, Networking, Security,  Information Systems) are eligible for the program. Although the ability to code is not required, familiarity with the following IT concepts is highly recommended: virtual machines, operating systems, storage and file systems, networking, databases, programming, and working with Linux at the command line.At Google Cloud, we are committed to creating training and certification opportunities for transitioning service members, veterans, and military spouses to help them thrive in a cloud-first world. Stay tuned for updates early next year!Related ArticleCloud Career Jump Start: our virtual certification readiness programCloud Career Jump Start is Google Cloud’s first virtual Certification Journey Learning program for underrepresented communities.Read Article
Quelle: Google Cloud Platform