How APIs expand reach and drive ROI for content producers

Over the course of just a few years, the ways in which we consume content have changed dramatically. In order to compete in this new landscape and to adapt to the technological change that underpins it, media studios and other content producers should consider providing relatively open access to their proprietary content. This necessitates a cultural change across the industry. Cable television cancellation, or “cord cutting,” has increased significantly since 2010, and with the pandemic accelerating the trend, there are now more than 30 million cord-cutter U.S. households. The American digital content subscriber now watches streaming content across an average of more than three paid services. For several years, more video content has been uploaded to streaming services every 30 days than the major U.S. television networks have created in 30 years. With an abundance of content readily available across a growing number of platforms, each accessible from a plethora of different devices, media providers should invest in making it easier for consumers to find the video content they want to watch. If a viewer can’t access and stream something with minimal effort, they’ll likely move on to one of the countless alternatives readily at their disposal. Think about voice-based assistants and search services. When prompted to find a piece of content, these services sift through a multitude of third-party libraries, where access is permitted, and remove friction from the user experience. It’s important for media companies to evolve from siloed, closed-off content libraries to participation in digital ecosystems, where a host of partnership opportunities can precipitate wider reach and revenue opportunities. Ultimately, joining these communities facilitates the delivery of the right experience on the right device at the right time to the right consumer.Navigating a streaming jungle Legacy silos prevalent in the media and entertainment industry must be broken down to make way for richer viewing experiences. It’s critical that studios roll out content faster, distribute it more securely, and better understand their audiences so they can provide customers the content they want in the contexts they want. In order to achieve these goals, publishers must leverage technology that’s purpose-built for the demands of a more dynamic, competitive landscape.Publishers should consider embracing application programming interfaces, or APIs, to better connect with viewers and maximize return on content production. APIs, which facilitate interoperability between applications, allow publishers’ content to be consumed by more developers and publishing partners, who subsequently create more intelligent, connected experiences surrounding that content for the viewers.  This new content value chain should leverage an API management tool that resides on top of cloud infrastructure to manage the partnerships that ultimately ensure media can easily make its way to the consumer on their ideal viewing platform. APIs let content owners and distributors interact with partner technologies to drive value from social interactions and attract a wider audience via insights derived from data and analytics. Perhaps most important is the ability for APIs to allow content to follow users as they start watching on one device, stop, and transfer to another. Content is increasingly separated from the device. APIs enable experiential continuity to be maintained when devices are changed, facilitating more seamless experiences across devices of different form factors and screen sizes. Consumers expect content to follow them wherever they go. How APIs improve content creation and distribution Last year, streaming services produced more original content than the entire television industry did in 2005—so for many media producers, adjusting to consumers’ new media consumption habits involves not only making content available on more devices but also producing more content, faster. Studios should explore solutions that help them collaborate globally and produce great content more securely and efficiently. In the content value chain, APIs are used to seamlessly connect artists and production crews to necessary resources and assets across multiple production technologies and locations. For example, via APIs, a film crew in one country can record, encode, and collaborate and share content with another studio in another country. These cloud-based production environments can offer a single destination for all contributors to access the assets they need while also keeping those assets accessible only to the right people in the right contexts. In addition, creating and distributing content requires a complex supply chain. APIs let multiple parties, each responsible for a different core function (such as content purchasing, storage, payments, physical media delivery, customer service, etc.), meld into a seamless experience for the customer.  Rather than reimagining their strategy when it comes to these backend tasks, studios can leverage third-party APIs to expedite getting content in front of the right people and ultimately execute each of those functions more efficiently than they could on their own. Besides tapping into partner APIs, savvy media and entertainment companies can accelerate consumption of content by developing their own public APIs to securely provide access to their asset libraries, pricing, and other relevant information. This is important, as it lets media creators use the same API to serve content to a variety of services and device types, thus helping them scale content distribution without simultaneously having to scale security resources. Media companies’ APIs can also be implemented  to deliver better customer experiences. Because APIs are involved each time a customer streams a video and every time a developer integrates a media asset into a new app or digital experience, API usage analytics can provide powerful insights into where, when, by whom, and on what devices different types of media—from traditional movies to augmented reality and other interactive content—are being accessed. Bringing it all together with an API management tool In order for studios to quickly adapt to a content value chain and distribute their content across multiple platforms, it’s important that they implement an API management tool on top of the cloud environment that powers content creation and distribution. For instance, Google Cloud offers Apigee, which sits on top of its public cloud. This added layer facilitates the integration between a studio’s proprietary environment and the strategic partnerships that APIs make possible. The API lifecycle can be rather complex, especially when multiple APIs are leveraged. It can include:Planning, design, implementation, testing, publication, operation, consumption, maintenance, versioning, and retirement of APIsLaunch of a developer portal to target, market to, and govern communities of developers who leverage APIsRuntime managementEstimation of APIs’ valueAnalytics to understand patterns of API usageUsing a management layer such as Apigee increases the likelihood that media and entertainment companies can combine the ability offered by public clouds and APIs to adapt to the requirements of new devices and protocols. It brings next-generation technology together to ensure studios can scale, secure, monitor, and analyze digital content creation and distribution.Related ArticleHelping media companies navigate the new streaming normalAs media and entertainment companies evolve their future plans as a result of COVID-19, they should keep new audience behaviors top of mi…Read Article
Quelle: Google Cloud Platform

Filestore Backups eases migration of file-based apps to cloud

When you’re considering migrating mission-critical workloads to the cloud, it’s important to make it as easy as possible. We think maintaining your established, on-prem best practices can make a cloud migration a lot easier. Using established best practices reduces the need to rearchitect applications, and also helps ensure continuity as you migrate your infrastructure. We’re announcing the availability of Filestore Backups preview in all regions, making it easier to migrate your business continuity, disaster recovery and backup strategy for your file systems in Google Cloud.File system backups serve use cases such as disaster recovery, protection against accidental data changes, file system cloning and migration of data, all of which benefit from the ability to efficiently store a copy of data and metadata in a separate storage system. File system backups form a central component of any solid backup strategy, as they let you safely retain point-in-time copies of data shared across VMs and applications. While resiliency is an essential aspect of nearly every industry today, websites and ecommerce are one example where 24×7 uptime and reliability is critical. Downtime or lost data can mean a direct and immediate impact to a business. Google Cloud customer Liferay, one of the leading platforms for websites and ecommerce businesses, relies on Filestore backups to make sure they are supporting their customers with best-in-class reliability. “On Liferay DXP Cloud, we work with customers from all sizes and backgrounds with different storage and compliance needs,” says Eduardo Lundgren, CTO of Liferay Cloud. “Using Filestore allowed us to support all these needs while focusing on delivering new features instead of managing storage. Filestore Backups enable us to create daily snapshots of each customer, and if needed, restore their data quickly and safely.” Filestore backups can be used to restore a file system back to a previous state or to create a new Filestore instance whose data is identical to the original at the point in time the backup was taken. Filestore Backups features were designed to answer the requirements of enterprise file users. Here are a few of the benefits that Filestore Backups offers:Regional availability – Filestore backups are regional resources, which enables them to protect users against the rare case of inaccessibility of a given zone. If data in a zone is inaccessible, users can restore the data using the regional backup and continue working in any available zone. Cross-region creation – Filestore backups can also be created and stored to a region different from that of the origin file system. This enables users to protect their workloads against the inaccessibility of an entire region or to efficiently migrate file system data between regions. Compression and incremental data retention – To reduce costs, backups within a region are created incrementally based on previous backups and are automatically compressed. This means that the first backup you create is a compressed copy of the file share, and subsequent backups include only the new or modified data that is not contained in the previous backup.‘Create’ and ‘Restore’ functionality across Filestore Basic HDD and Filestore Basic SSD – Filestore backups can be created for either Filestore Basic HDD or Filestore Basic SSD instances and can be restored to either tier. This means that Filestore backups can be used to migrate data from Filestore Basic HDD to Basic SSD to increase performance or from Filestore Basic SSD to Basic HDD to reduce costs. Customers may use the backup feature to optimize cost and performance based on actual workload needs.Independent lifecycle of backup and filestore instance – Once a backup of a file system is created, the original instance may be safely deleted. As backups are stored on separate storage, the backup will be retained until it is deleted by the user. If access to the data in the backup is required a new filesystem can be created from the backup recreating the data and metadata of the deleted file system.These features let you use Filestore backups across multiple use cases, including:Backing up data for disaster recovery – Use Cloud Scheduler to regularly back up instances to a remote region. In the event of a disaster you can create a new instance in another location from any previous backup.Backing up data to protect against accidental changes – To protect your data against accidental deletions or changes due to human or software errors, back up your file system regularly and before major changes or upgrades. In the event of inadvertent loss or change to a file, you can identify the backup with the version of the file needed, create a new Filestore instance, and copy the original file over. Alternatively, you can do an in-place restore where the backup data is directly restored to the original Filestore instance.Creating clones for development and testing – If your Filestore instance serves production traffic and you want to run a test with the data in the file system as an input, simply create a clone Filestore instance from a backup to enable testing and development usage without interfering with production files systems.Migrating data – If you need to migrate a Filestore instance to another region, you can simply create a backup and restore the Filestore instance in the new region. This same method may also be used to create multiple copies of a file system across a collection of regions.Creating a backup or restoring from a backup is easy from the Google Cloud Console:The list of existing Filestore backups with more details is available in a separate “Backups” tab.Creation of a new instance from a backup details page is just a single click away:To get started, check out the Filestore Backup documentation or create a backup in the Google Cloud Console.This blog is dedicated to Allon Cohen, our colleague and friend who passed away this month. We want to thank Allon for his contributions to Filestore and Google Cloud.
Quelle: Google Cloud Platform

I do declare! Infrastructure automation with Configuration as Data

Over the years there’s been an explosion in infrastructure platforms and application frameworks that form the foundation of “cloud native.” Modern infrastructure platforms range from container orchestrators such as Kubernetes to serverless platforms aimed at rapid application development. In parallel, shell scripts that administrators used to deploy, configure, and manage these platforms evolved into what is now called Infrastructure as Code (IaC), which formalizes the use of higher level programming languages such as Python or Ruby or purpose-built languages such as HashiCorp’s HCL (through Terraform). Though IaC has been broadly adopted, it suffers from a major flaw: code does not provide a contract between the developer’s intent and runtime operation. Contracts are the foundation of a consistent, secure and high-velocity IT environment. But every time you modify or refactor code, you need to run validation tools to determine its intent.Which begs the question, why are admins using programming languages in the first place? Why is all of this so complicated? In many ways it’s an attempt to automate the unknown, the unpredictable. But by nature, most infrastructure is loosely defined and requires baling wire and duct tape to stick things together in ways that mimic what a system administrator would do when logged into a server.Furthermore, while provisioning infrastructure is important, IT practitioners also need to deploy and manage both infrastructure and applications from day two onwards in order to maintain proper operations. Ideally, you could use the same configuration management tools to deploy and manage both your infrastructure and applications holistically. The Kubernetes wayThings are different with Kubernetes…Instead of taking an imperative or procedural approach, Kubernetes relies on the notion of Configuration as Data, taking a declarative approach to deploying and managing cloud infrastructure as well as applications. You declare your desired state without specifying the precise actions or steps for how to achieve it. Every Kubernetes resource instance is defined by Configuration as Data expressed in YAML and JSON files. Creating a Deployment? Defining a Service? Setting a policy? It’s all Configuration as Data, and Kubernetes users have been in on the secret for the past six years. Want to see what we mean? Here’s a simple Kubernetes example…In just 10 lines of YAML, you can define a Service with a unique version of your application, set up the network to create a route, ingress, Service, and load balancer, and automatically scale up and down based on traffic. How does Configuration as Data work? Within the Kubernetes API Server are a set of controllers that are responsible for ensuring the live infrastructure state matches the declarative state that you express. For example, the Kubernetes service controller might ensure that a load balancer and Service proxy are created, that the corresponding Pods are connected to the proxy, and all necessary configuration is set up and maintained to achieve your declared intent. The controller maintains that configured state forever, until you explicitly update or delete that desired state.What’s less well known is that the Kubernetes Resource Model (KRM) that powers containerized applications can manage non-Kubernetes resources including other infrastructure, platform, and application services. For example, you can use the Kubernetes Resource Model to deploy and manage cloud databases, storage buckets, networks, and much more. Some Google Cloud customers also manage their applications and services using Kubernetes controllers that they developed in-house with open-source tools.How do you start leveraging the KRM for managing Google Cloud resources? Last year, Google Cloud released Config Connector, which provides built-in controllers for Google Cloud resources. Config Connector lets you manage your Google Cloud infrastructure the same way you manage your Kubernetes applications—by defining your infrastructure configurations as data—reducing the complexity and cognitive load for your entire team.Following our service example above, let’s say we want to deploy a Google Cloud Redis instance as a backing memory store for our service. We can use KRM by creating a simple YAML representation that is consistent with the rest of our application:We can create the Redis instance via KRM and Config Connector:Where CaD meets IaCDoes that mean you no longer need traditional IaC tools like Terraform? Not necessarily. There will always be a need to orchestrate configuration between systems, for example, collecting service IPs and updating external DNS sources. That’s where those tools come in. The benefit when managing Google Cloud resources with Config Connector is that the contract will be much stronger. This model also offers a better integration story and cleanly separates the responsibility for configuring a resource and managing it. Here’s an example with Terraform:Terraform is used to provision a Google Cloud network named “demo_network” via the Terraform provider for Google and to create a Google Cloud Redis instance connected to it via the Terraform Kubernetes provider and KRM. On the surface, the contract between Terraform and the two providers looks the same, but beneath the surface lies a different story.The Terraform provider for Google calls the Google Cloud APIs directly to create the networking resources. If you wanted to use another configuration tool you would need to create a new set of Google Cloud API integrations. Furthermore, you will jump back and forth between Kubernetes and Terraform to view resources created separately in each interface.On the other hand, the Kubernetes provider is backed by a controller running in Kubernetes that presents a KRM interface for configuring Redis instances. Once Terraform submits configuration in the form of data to a Kubernetes API server, the resource is created and is actively managed by Kubernetes. Configuration as Data establishes a strong contract between tools and interfaces for consistent results. You’re able to remain in the Kubernetes interface to manage resources and applications together. The Kubernetes API server continuously reconciles the live Google Cloud state with the desired state you established in Terraform with KRM. Configuration as Data complements Terraform with consistency between Terraform executions that may be hours, days, or weeks apart.To make a long story short, Configuration as Data is an exciting approach to infrastructure and app management that enables fluid interaction between native resources and configuration tools like IaC and command lines.  It’s also an area that’s moving quickly. Stay tuned for more about Configuration as Data coming soon. In the meantime, try Config Connector with your Google Cloud projects and share your feedback about what you did, what worked, and what new features you’d like to see.Related ArticleUnify Kubernetes and GCP resources for simpler and faster deploymentsThe new Config Connector lets you manage GCP resources as if they were in Kubernetes.Read Article
Quelle: Google Cloud Platform

Use real-time anomaly detection reference patterns to combat fraud

Businesses of every size and shape have a need to better understand their customers, their systems, and the impact of external factors on their business. How rapidly businesses mitigate risks and capitalize on opportunities can set apart successful businesses from businesses that can’t keep up. Anomaly detection—or in broader terms, outlier detection—allows businesses to identify and take action on changing user needs, detect and mitigate malignant actors and behaviors, and take preventive actions to reduce costly repairs.The speed at which businesses identify anomalies can have a big impact on response times, and in turn, associated costs. For example, detecting a fraudulent financial transaction in hours or days after it happens often results in writing off the financial loss. The ability to find the anomalous transaction in seconds allows for the invalidation of the transaction and corrective actions to prevent future fraud. Similarly, by detecting anomalies in industrial equipment, manufacturers can predict and prevent catastrophic failures that could cause capital and human loss by initiating proactive equipment shutdowns and preventative maintenance. Likewise, detecting anomalous user behavior (for example, sign-in into multiple accounts from the same location/device) can prevent malignant abuse, data breaches, and intellectual property theft.In essence, anomalous events have an immediate value. If you don’t seize that value, it vanishes into irrelevance until there’s a large enough collection of events to perform retrospective analysis. (See image below for an illustration of that concept.) To avoid falling off this “value cliff,” many organizations are looking to stream analytics to provide a real-time anomaly detection advantage.At Google Cloud, our customer success teams have been working with an increasing number of customers to help them implement streaming anomaly detection. In working with such organizations to help them build anomaly detection systems, we realized that providing these reference patterns can significantly reduce the time to solution for those and future customers.Reference patterns for streaming anomaly detectionReference patterns are technical reference guides that offer step-by-step implementation and deployment instructions and sample code. Reference patterns mean you don’t have to reinvent the wheel to create an efficient architecture. While some of the specifics (e.g., what constitutes an anomaly, desired sensitivity level, alert a human vs. display in a dashboard) depend on the use case, most anomaly detection systems are architecturally similar and leverage a number of common building blocks. Based on that learning, we have now released a set of repeatable reference patterns for streaming anomaly detection to the reference patterns catalog (see the anomaly detection section).These patterns implement the following step-by-step process:Stream events in real time Process the events, extract useful data points, train the detection algorithm of choiceApply the detection algorithm in near-real time to the events to detect anomaliesUpdate dashboards and/or send alertsHere’s an overview of the key patterns that let you implement this broader anomaly detection architecture:Detecting network intrusion using K-means clusteringWe recently worked with a telecommunications customer to implement streaming anomaly detection for Netflow logs. In the past, we’ve seen that customers have typically implemented signature-based intrusion detection systems. Although this technique works well for known threats, it is difficult to detect new attacks because no pattern or signature is available. This is a significant limitation in times like now, when security threats are ever-present and the cost of a security breach is significant. To address that limitation, we built an unsupervised learning-based anomaly detection system. We also published a detailed writeup: Anomaly detection using streaming analytics and AI. The following video gives a step-by-step overview of implementing the anomaly detection system. Keep in mind that the architecture and steps in the video can be applied to other problem domains as well, not just network logs. Detecting fraudulent financial transactions using Boosted TreesWhile the previous pattern used a clustering algorithm (trained in BigQuery ML), the finding anomalies in financial transactions in real time using Boosted Trees pattern uses a different ML technique called BoostedTrees. BoostedTrees is an ensemble technique that makes predictions by combining output from a series of base models. This pattern follows the same high-level architecture and uses Google Cloud AI Platform to perform predictions. One of the neat things in the reference pattern is the use of micro-batching to group together the API calls to the CAIP Prediction API. This ensures that a high volume of streaming data does not necessarily result in API quota issues. Here’s what the architecture looks like:Time series outlier detection using LSTM autoencoderMany anomaly detection scenarios involve time series data (a series of data points ordered by time, typically evenly spaced in time domain). One of the key challenges with time series data is that it needs to be preprocessed to fill any gaps (either due to source or transmission problems) in data. Another common requirement is the need to aggregate metrics (e.g., Last, First, Min, Max, Count values) from the previous processing window when applying transforms to the current time window. We created a Github library to solve these problems for streaming data and jump-starts your implementation for working with time series data. These patterns are driven by needs we’ve seen in partnering with customers to solve problems. The challenge of finding the important insight or deviation in a sea of data is not unique to any one business or industry; it applies to all. Regardless of where you are starting, we look forward to helping you on the journey to streaming anomaly detection. To get started, head to the anomaly detection section in our catalog of reference patterns. If you have implemented a smart analytics reference pattern, we want to hear from you. Complete this short survey to let us know about your experience.
Quelle: Google Cloud Platform

Introducing Voucher, a service to help secure the container supply chain

Kubernetes helps developers build modern software that scales, but to do so securely, they also need a software supply chain with strong governance. From managed secure base images,  Container Registry vulnerability scanning to Binary Authorization, Google Cloud helps secure that pipeline, giving you the support and flexibility you need to build great software without being locked into a particular provider.Today, we are excited to announce a great open-source addition to the secure software supply chain tool box: Voucher. Developed by the Software Supply Chain Security team at Shopify to work with Google Cloud tools, Voucher evaluates container images created by  CI/CD pipelines and signs those images if they meet certain predefined security criteria. Binary Authorization then validates these signatures at deploy time, ensuring that only explicitly authorized code that meets your organizational policy and compliance requirements can be deployed to production.Voucher is open source from the get-go, following the Grafeas specification. The signatures it generates, or ‘attestations,’ can be enforced by either Binary Authorization or the open-source Kritis admission controller. Out of the box, Voucher lets infrastructure engineers use Binary Authorization policies to enforce security requirements, such as provenance (e.g., a signature that is only added when images are built from a secure source branch) and block vulnerable images (e.g., require a signature that is only applied to images that don’t have any known vulnerabilities above the ‘medium’ level). And because it’s open source, you can also easily extend Voucher to support additional security and compliance checks or integrate it with your CI/CD tool of choice. “At Shopify, we ship more than 8,000 builds a day and maintain a registry with over 330,000 container images. We designed Voucher in collaboration with the Google Cloud team to give us a comprehensive way to validate the containers we ship to production,” said Cat Jones, Senior Infrastructure Security Engineer at Shopify. “Voucher, along with the vulnerability scanning functionality from Google’s Container Registry and Binary Authorization, provides us a way to secure our production systems using layered security policies, with a minimum impact to our unprecedented development velocity. We are donating Voucher to the Grafeas open-source project so more organizations can better protect their software supply chains. Together, Voucher, Grafeas and Kritis help infrastructure teams achieve better security while letting developers focus on their code.”How Voucher simplifies a secure supply chain setupIn the past, if you wanted to gate deployments based on build or vulnerability findings, you needed to write, host and run your own evaluation logic (step 2a and 3a), as shown in the following process:Code is pushed to a repositoryA continuous integration (CI) pipeline tool, such as Cloud Build, builds and tests the container.Write custom code to sign images based on their build provenance (e.g. only sign images built from the production branch)The newly built container image is checked into Google Container Registry and undergoes vulnerability scanning.Write custom code to sign images based on vulnerability findingsBinAuthz verifies the image signatures as part of being deployed to GKE. To avoid privilege escalation, the signing steps should be hosted outside of the CI/CD pipeline (developers who can execute arbitrary code in a build step cannot gain access to the signing key or alter the signing logic). This puts a significant burden on DevOps teams to create and set up these kinds of signing tools. Voucher, however, automates a large portion of this setup — it comes  with a pre-supplied set of security checks, and all you have to do is specify your signing policies in Binary Authorization. Once started, it automates the attestation generation as shown below:Try it out!We’re honored that Shopify used Google Cloud tools to power Voucher, and we’re excited that they’ve decided to share it with developers at large. If you want  to try Voucher, you can find it on GitHub, or a click-to-deploy version on Google Cloud Marketplace. We’ve also created a step-by-step tutorial to help you launch Voucher on Google Cloud with Binary Authorization.
Quelle: Google Cloud Platform

Google Cloud AI digitizes StoryCorps archive: largest collection of human voices on planet

For many of us the holiday season will look different this year, separated from the people we love. If you’re in this boat too—mitigating the spread of the coronavirus—thank you and we hope the following story might offer an alternative, but helpful way to connect with friends and family. While we know virtual get-togethers can never fully match the intimacy of in-person conversations, they can keep us connected and maybe even preserve some special moments for future generations. In this spirit, we are sharing our collaboration with StoryCorps, a national non-profit organization dedicated to preserving humanity’s stories through 1:1 interviews. Over the past 17 years, StoryCorps has recorded with more than 600,000 people and sent those recordings to the U.S. Library of Congress where they are preserved for generations to come at the American Folklife Center. This is the world’s largest collection of human voices on the planet, but, it’s been relatively inaccessible. That’s when StoryCorps approached us to help make its rich archive of first-person history universally accessible and useful. StoryCorps + Google Cloud AIIn 2019, StoryCorps and Google Cloud partnered to unlock this amazing archive using artificial intelligence (AI) and create an open, searchable and accessible audio database for everyone to find and listen to first-hand perspectives from humanity’s most important moments. Diving into how this works: for an audio recording to be searchable, the audio file and “moments” or keywords within that file—needed to be tagged with terms for which you would search. First we used Speech-to-Text API to transcribe the audio file.Then Natural Language API identified keywords and their salience from the transcription.The transcript and keywords were loaded to an Elastic Search index.Resulting in a searchable transcript on the StoryCorps Archive.Here is an example of how these Cloud AI technologies work using an actual StoryCorps interview.Building empathy and understanding through connection StoryCorps’ mission is impressive. Not only is it preserving humanity’s stories, its aim is to “build connections between people and create a more just and compassionate world” by sharing those stories as widely as possible. This is where our path with StoryCorps crosses on a deeper level. Our mission for AI technology is one where everyone is accounted for, extending well beyond the training data in computer science departments. This deeper understanding could allow organizations in every sector to unlock new possibilities of what they have to offer while being inclusive, equitable and socially beneficial. But that’s our story to figure out and we’re working hard at it. Whatever you decide to do this holiday season, please stay safe. In the meantime, perhaps your family would like to use the StoryCorps platform or app to connect, preserve and share a story of your own.Related ArticlePicture what the cloud can do: How the New York Times is using Google Cloud to find untold stories in millions of archived photosThe New York Times is building a pipeline on Google Cloud Platform to preserve its extensive photo archive, store it in the cloud, and le…Read Article
Quelle: Google Cloud Platform

How we're advancing intelligent automation in network security

We’re always looking to make advanced security easier for enterprises so they can stay focused on their core business. Already this year, we’ve worked to strengthen DDoS protection, talked about some of the largest attacks we have stopped and made firewall defences more effective. We continue to push our pace of security innovation, and today we’re announcing enhancements to existing protections, as well as new capabilities to help customers protect their users, data, and applications in the cloud. 1. Using machine learning to detect and block DDoS Attacks with Adaptive ProtectionWe recently talked about how our infrastructure absorbed a 2.54 Tbps DDoS attack, the culmination of a six-month campaign that utilized multiple methods of attack. Despite simultaneously targeting thousands of our IPs, presumably in hopes of slipping past automated defenses, the attack had no impact.We recognize the scale of potential DDoS attacks can be daunting. By deploying Google Cloud Armor integrated into our Cloud Load Balancing service—which can scale to absorb massive DDoS attacks—you can protect services deployed in Google Cloud, other clouds, or on-premise from attacks. Cloud Armor, our DDoS and WAF-as-a-service, is built using the same technology and infrastructure that powers Google services.Today, we are excited to announce Cloud Armor Adaptive Protection—a unique technology that leverages years of experience using machine learning to solve security challenges plus deep experience protecting our own user properties against Layer 7 DDoS attacks. We use multiple machine learning models within Adaptive Protection to analyze security signals for each web service to detect potential attacks against web apps and services. This system can detect high volume application layer DDoS attacks against your web apps and services and dramatically accelerate time to mitigation. For example, attackers frequently target a high volume of requests against dynamic pages like search results or reports in web apps in order to exhaust server resources to generate the page. When enabled, we learn from a large number of factors and attributes about the traffic arriving at your services so we know what “normal” looks like. We’ll generate an alert if we believe there is a potential attack, taking into account all of the relevant context for your workload. In other words, where traditional threshold based detection mechanisms could generate a great deal of lower confidence alerts that would require investigation and triage only once an attack has accelerated to the detection threshold, Adaptive Protection produces high confidence signals about a potential attack much earlier, while the attack is still ramping up. Adaptive Protection won’t just surface the attack, but will actually provide context on why the system felt it was malicious and then provide a rule to mitigate the attack as well. This protection is woven into our cloud fabric and only alerts the operator for more serious issues with context, an attack signature, and a Cloud Armor rule that they can then deploy in preview or blocking mode. Rather than spending hours analysing traffic logs to triage the ongoing attack, application owners and incident responders will have all of the context they need to make a decision on whether and how to stop the potentially malicious traffic. Cloud Armor Adaptive Protection is going to simplify protection in a big way, and will be rolling out to the public in preview soon.Adaptive Protection suggested rule2. Better firewall rule management with Firewall Insights We have been making a number of investments into our network firewall to provide insights and simplify control that allow easier management of more complex environments. Firewall insights helps you optimize your firewall configurations with a number of detection capabilities, including shadowed rule detection to identify firewall rules that have been accidentally shadowed by conflicting rules with higher priorities. In other words, you can automatically detect rules that can’t be reached during firewall rule evaluation due to overlapping rules with higher priorities. This helps detect redundant firewall rules, open ports, and IP ranges and help operators to tighten the security boundary. It will also help surface to admins a sudden hit increases on firewall rules and drill down to the source of the traffic to catch an emerging attack.Within firewall insights you’ll also see metrics reports showing how often your firewall rules are active, including the last time they were hit. This allows security admins to verify that firewall rules are being used in the intended way, ensuring that firewall rules allow or block their intended connections. These insights can operate at massive volume and help remove human errors around firewall rule configuration or simply highlight rules that are no longer needed as an environment changes over time. Firewall insights will be generally available soon.Firewall Insights3. Flexible and scalable controls with Hierarchical Firewall PoliciesFirewalls are an integral part of almost any IT security plan. With our native, fully distributed firewall technology, Google Cloud aims to provide the highest performance and scalability for all your enterprise workloads.  Google Cloud’s hierarchical firewall policies, provide new, flexible levels of control so that you can benefit from centralized control at the organization and folder level, while safely delegating more granular control within a project to the project owner. Hierarchical firewalls provide a means to enforce firewall rules at the organization and folder levels in the GCP Resource Hierarchy.  This allows security administrators at different levels in the hierarchy to define and deploy consistent firewall rules across a number of projects so that they are applied to all VMs in currently existing and yet-to-be-created projects. Hierarchical firewall policies allow configuring rules at the Organization and Folder levels, in addition to firewall rules at the VPC level. Since leveraging Hierarchical Firewalls  requires fewer firewall rules, managing multiple environments becomes simpler and more effective. Further, being able to manage the most critical firewall rules in one place can help free up project level administrators from having to keep up with changing organization wide policies. Hierarchical firewall policies will be generally available soon.Hierarchical firewall policies4. New controls for Packet Mirroring Google Cloud Packet Mirroring allows you to mirror network traffic from your existing Virtual Private Clouds (VPCs) to third party network inspection services. With this service, you can use those third-party tools to collect and inspect network traffic at scale, providing intrusion detection, application performance monitoring, and better security visibility, helping you with the security and compliance of workloads running in Compute Engine and Google Kubernetes Engine (GKE). We are adding new filters to mirror packets that will be generally available soon. With traffic direction control, you can now mirror either the ingress or egress traffic, helping users better manage their traffic volume and reduce costs.Traffic Direction: New Ingress & Egress controls for Packet MirroringWith these enhancements, we are helping Google Cloud customers stay safe when using our network security products. For a hands-on experience on our Network Security portfolio, you can enroll in our network security labs here. You can also learn more about Google Cloud security in the latest installment of Google Cloud Security Talks, live today.Related ArticleExponential growth in DDoS attack volumesHow Google prepares for and protects against the largest volumetric DDoS attacks.Read Article
Quelle: Google Cloud Platform

The need for speed: Using C2 machines for your HPC workloads

Cloud opens many new possibilities for High Performance Computing (HPC). But while the cloud offers the latest technologies and a wide variety of machine types (VMs), not every VM is suited to the demands of HPC workloads. Google Cloud’s Compute-optimized (C2) machines are specifically designed to meet the needs of the most compute-intensive workloads, such as HPC applications in fields like scientific computing, Computer-aided Engineering (CAE), biosciences, and Electronic Design Automation (EDA), among many others.The C2 is based on the second generation Intel® Xeon® Scalable Processor and provides up to 60 virtual cores (vCPUs) and 240GB of system memory. C2s can run at a sustained frequency of 3.8GHz and offer more than 40% improvement compared to previous generation VMs for general applications. Compared to previous generation VMs, total memory bandwidth improves by 1.21X and memory bandwidth/vCPU improves by 1.94X.1 Here we take a deeper look at using C2 VMs for your HPC workloads on Google Cloud.Resource isolationTightly-coupled HPC workloads rely on resource isolation for predictable performance. C2 is built for isolation and consistent mapping of shared physical resources (e.g., CPU caches, and memory bandwidth). The result is reduced variability and more consistent performance. C2 also exposes and enables explicit user control of CPU power states (“C-States”) on larger VM sizes, enabling higher effective frequencies and performance.NUMA nodesIn addition to hardware improvements, Google Cloud has enabled a number of HPC-specific optimizations on C2 instances. In many cases, tightly-coupled HPC applications require careful mapping of processes or threads to physical cores, along with care to ensure processes access memory that is closest to their physical cores. C2s provide explicit visibility and control of NUMA domains to the guest operating system (OS), enabling maximum performance.AVX-512 supportSecond generation Xeon processors support Intel Advanced Vector Extension 512 (Intel AVX-512) for data parallelism. AVX-512 instructions are SIMD (Single Instruction Multiple Data) instructions, and along with additional and wider registers enable packing of 64 single-precision (or 32 double-precision) floating point operations into one instruction. This means that more can be done in every clock cycle, reducing overall execution time. The latest generation of AVX-512 instructions in the 2nd generation Xeon processor include DL Boost instructions that significantly improve performance for AI inferencing by combining three INT8 instructions into one—thereby maximizing the use of compute resources, utilizing the cache better, and avoiding potential bandwidth bottlenecks.Low- latencyHPC workloads often scale out to multiple nodes in order to accelerate time to completion. Google Cloud has enabled “Compact Placement Policy” on the C2, which allocates up to 1320 vCPUs placed in close physical proximity, minimizing cross-node latencies. Compact placements, in conjunction with Intel MPI library, optimizes multi-node scalability of HPC applications. You can learn more about best practices for ensuring low latency on multi-node workloads here.Development toolsAlong with the hardware optimizations, Intel offers a comprehensive suite of development tools (including performance libraries, Intel Compilers, and performance monitoring and tuning tools) to make it simpler to build and modernize code with the latest techniques in vectorization, multithreading, multi-node parallelization, and memory optimization. Learn more about Intel’s Parallel Studio XE here.Bringing it all together Combining all the improvements in hardware and optimizations done in Google Cloud stack, C2 VMs perform up to 2.10X better compared to previous generation N1 for HPC workloads for roughly the same size VM.2In many cases HPC applications can scale up to the full node. A single C2 node (60 vCPUs and 240GB) offers up to 2.49X better performance/price compared to a single N1 node (96 vCPUs and 360GB).3C2s are offered in predefined shapes intended to deliver the most appropriate vCPU and memory configurations for typical HPC workloads. In some cases, it is possible to further optimize performance or performance/price via a custom VM shape. For example, if a certain workload is known to require less than the default 240GB of a C2 standard 60vCPU VM, a custom N2 machine with less memory can deliver roughly the same performance at a lower cost. We were able to achieve up to 1.09X better performance/price by tuning the VM shape to the needs of several common HPC workloads.4Get started todayAs more HPC workloads start to benefit from the agility and flexibility of cloud, Google Cloud and Intel are joining forces to create optimized solutions for specific needs of these workloads. With the latest optimizations in Intel 2nd generation Xeon processors and Google Cloud, C2 VMs deliver the best solution for running HPC applications in Google Cloud, while giving you the freedom to build and evolve around your unique business needs. Many of our customers with need for high performance have moved their workloads to C2 VMs and confirmed our expectations.To learn more about C2 and the second generation of Intel Xeon Scalable Processor, contact your sales representative or reach out to us here. And if you’re participating in SC20 this week, be sure to check out our virtual booth, where you can watch sessions, access resources, and chat with our HPC experts.1. Based on internal analysis of our c2-standard-60 and n1-standard-96 machine types, using the STREAM Triad Best Rate benchmark.2. Based on internal analysis of our c2-standard-60 and n1-standard-96 machine types, using our Weather Research Forecasting (WRF) benchmark.3. Based on the High Performance Conjugate Gradients (HPCG) benchmark, analyzing Google Cloud VM Instance pricing on C2-standard-60 ($3.1321/hour) and N1-standard-96 ($4.559976) as of 10/15/20204. Based on GROMACS and NAMD benchmarks, analyzing Google Cloud VM Instance pricing on N2-custom-80 with 160GB ($3.36528) and C2-standard-60 ($3.1321/hour) as of 10/15/2020Related ArticleIntroducing Compute- and Memory-Optimized VMs for Google Compute EngineGoogle Cloud is the first public cloud provider to offer Compute-Optimized VMs and Memory-Optimized VMs based on Intel 2nd Generation Xeo…Read Article
Quelle: Google Cloud Platform

Empowering customers and the ecosystem with an open cloud

Every organization that moves to the cloud has a unique journey driven by many factors including evolving operating environments and regulatory requirements. For all organizations, including those experiencing growth across regions and dynamic market circumstances, we recommend an open cloud approach that ensures operational and technical consistency across public clouds or private data centers and effective management of infrastructure, applications, and data across the organization.We believe that an open cloud can meet the needs of diverse companies, providing choice, flexibility and openness. Our open cloud philosophy is grounded in the belief that customers need autonomy and control over their infrastructure. Giving customers options to build, migrate and deploy their applications across multiple environments both in the cloud and on-premises allows them to avoid vendor lock-in and innovate across environments faster. We are proud of our leadership in advancing an open cloud, and this commitment underpins and drives our contributions to the open source and open data communities, as well as our approach to building technology solutions. Advancing computing through open source Open source plays a critical role in an open cloud. Many companies have mission-critical workloads or sensitive data that have “survivability requirements” in the event that a provider is forced to suspend or terminate cloud services due to country or region policy changes. To move workloads to other clouds, it’s important to develop them using open source and open standards.At Google Cloud, we don’t think it’s possible to fully address survivability requirements with a proprietary solution. Instead, solutions based on open source tools and open standards are the route to addressing customer and policymaker concerns. More importantly, open source gives customers the flexibility to deploy—and, if necessary, migrate—critical workloads across or off public cloud platforms.Google has a long history of sharing technology through open source—from projects like Kubernetes, which is now the industry standard in container portability and interoperability in the cloud, to TensorFlow, a platform to help everyone develop and train machine learning models. As Google’s Chief Economist Hal Varian said, “Open data and open source are good not only for us and our industry, but also benefit the world at large.” Our belief in customer choice is fundamental to how we develop our technology and rooted in leveraging open source APIs and interoperable solutions. In addition, we partner with the leading organizations in the fields of data management and analytics to build products that combine the benefits of open source with managed cloud solutions.Another way we provide flexibility is hybrid and multi-cloud environments. Anthos, our hybrid and multi-cloud platform, is built on open technologies like Kubernetes, Istio, and Knative, enabling an ecosystem that fosters competition and that unlocks new partnerships. In this spirit, last week OVHcloud and Google Cloud announced a strategic partnership to jointly build a trusted cloud solution in Europe. This partnership will focus on delivering the best of Google Cloud technology innovation and value in the most agile way, and help European customers accelerate their business transformation in the cloud while meeting addressing their strict data security and privacy requirements.Break down data silos and uncover new insights with public datasetsCustomers rely on Google Cloud to get better insights from their data. Our data analytics solutions such as BigQuery help them harness the potential of that data. One of our newest analytics solutions, BigQuery Omni, allows customers to cost-effectively access and securely analyze data across multi-cloud environments. Those tools enable customers to make their own data more open—both in and out of their organization. As we help enable data accessibility and portability, our highest priority is to do so securely and responsibly. At the same time, through the Google Cloud Public Datasets program, we work with data providers to host 100+ high-demand public datasets to allow customers and the research community to discover unique insights for solving real business and societal problems. For example, earlier this year, we added critical COVID-19 public datasets to support the global response to the novel coronavirus.We also share our discoveries and tools with the community to help everyone share data safely and in a manner that advances the important work of researchers, developers and journalists. Teams at Google have released over 80 open datasets through our research site, and share other aggregated, anonymized product insights. Take for example YouTube-8M, a large-scale, labeled video dataset used by researchers to further computer vision and video understanding. In addition, with more than 31 million datasets, Dataset Search allows anyone to discover and filter relevant data sets in accordance with usage rights, formats, and other key parameters. And our Kaggle community of nearly 5 million users hosts 50,000 public datasets and 400,000 public notebooks to support machine learning and artificial intelligence research. Paving the way to an open cloud through continued collaborationGoogle Cloud will continue to build towards an open cloud and work with partners and policymakers to support our customers, open-source communities, and society at large. We are excited to see the important work organizations achieve through openness, and remain committed to supporting them through our continued contributions to open source and open data.
Quelle: Google Cloud Platform

Helping media companies create consumer streaming formats with Transcoder API

Media and entertainment companies across the world are in the midst of a transformational shift to direct-to-consumer (D2C) streaming experiences. With audiences sheltering in place in 2020, this shift has been accelerated as audiences have adopted new streaming services more readily in 2020 than ever before. With more content being streamed in higher resolutions and operations becoming more distributed, media companies today require a cost efficient, high speed, and scalable way to process and deliver video content to an ever increasing number of end devices and platforms.Google Cloud is committed to building industry products and solutions that help media companies simplify operations and distribute their content to the widest possible audience. We’re building a set of video-focused APIs that empower developers to easily build and deploy flexible high quality video experiences. Today we’re announcing the first of these products, the preview availability of the Transcoder API. The Transcoder API is an easy-to-use API for creating consumer streaming formats, including MPEG-4 (MP4), Dynamic Adaptive Streaming over HTTP (DASH, also known as MPEG-DASH), and HTTP Live Streaming (HLS). Many D2C streaming platforms use a multi-codec strategy, and the Transcoder API supports popular codecs, including H.264, VP9, and HEVC. This strategy allows providers to offer a better, high definition experience to more viewers. The API also supports full partitioning for fast encoding of large video files, meaning that entire hours-long movies can be prepared in minutes.Developers can get started quickly by submitting transcoding jobs through REST API, transcoding files in Google Cloud Storage, and using Google Cloud CDN or third party CDNs to effectively distribute content to audiences across the globe. To learn more about the Transcoder API, please visit the documentation and pricing pages.Related ArticleHelping media companies navigate the new streaming normalAs media and entertainment companies evolve their future plans as a result of COVID-19, they should keep new audience behaviors top of mi…Read Article
Quelle: Google Cloud Platform