New Dataproc optional components support Apache Flink and Docker

Google Cloud’s Dataproc lets you run native Apache Spark and Hadoop clusters on Google Cloud in a simpler, more cost-effective way. In this blog, we will talk about our newest optional components available in Dataproc’s Component Exchange: Docker and Apache Flink.Docker container on DataprocDocker is a widely used container technology. Since it’s now a Dataproc optional component, Docker daemons can now be installed on every node of the Dataproc cluster. This will give you the ability to install containerized applications and interact with Hadoop clusters easily on the cluster. In addition, Docker is also critical to supporting these features:Running containers with YARNPortable Apache Beam jobRunning containers on YARN allows you to manage dependencies of your YARN application separately, and also allows you to create containerized services on YARN. Get more details here. Portable Apache Beam packages jobs into Docker containers and submits them the Flink cluster. Find more detail about Beam portability. Docker optional component is also configured to use Google Container Registry, in addition to the default Docker registry. This lets you use container images managed by your organization.Here is how to create a Dataproc cluster with the Docker optional component:gcloud beta dataproc clusters create <cluster-name>   –optional-components=DOCKER   –image-version=1.5When you run the Docker application, the log will be streamed to Cloud Logging, using gcplogs driver.If your application does not depend on any Hadoop services, check out Kubernetes and Google Kubernetes Engine to run containers natively. For more on using Dataproc, check out our documentation.Apache Flink on DataprocAmong streaming analytics technologies, Apache Beam and Apache Flink stand out. Apache Flink is a distributed processing engine using stateful computation. Apache Beam is a unified model for defining batch and steaming processing pipelines. Using Apache Flink as an execution engine, you can also run Apache Beam jobs on Dataproc, in addition to Google’s Cloud Dataflow service.Flink and running Beam on Flink are suitable for large-scale, continuous jobs, and provide:A streaming-first runtime that supports both batch processing and data streaming programsA runtime that supports very high throughput and low event latency at the same timeFault-tolerance with exactly-once processing guaranteesNatural back-pressure in streaming programsCustom memory management for efficient and robust switching between in-memory and out-of-core data processing algorithmsIntegration with YARN and other components of the Apache Hadoop ecosystemOur Dataproc team here at Google Cloud recently announced that Flink Operator on Kubernetes is now available. It allows you to run Apache Flink jobs in Kubernetes, bringing the benefits of reducing platform dependency and producing better hardware efficiency. Basic Flink ConceptsA Flink cluster consists of a Flink JobManager and a set of Flink TaskManagers. Like similar roles in other distributed systems such as YARN, JobManager has responsibilities such as accepting jobs, managing resources and supervising jobs. TaskManagers are responsible for running the actual tasks. When running Flink on Dataproc, we use YARN as resource manager for Flink. You can run Flink jobs in 2 ways: job cluster and session cluster. For the job cluster, YARN will create JobManager and TaskManagers for the job and will destroy the cluster once the job is finished. For session clusters, YARN will create JobManager and a few TaskManagers.The cluster can serve multiple jobs until being shut down by the user.How to create a cluster with FlinkUse this command to get started:gcloud beta dataproc clusters create <cluster-name>   –optional-components=FLINK   –image-version=1.5How to run a Flink jobAfter a Dataproc cluster with Flink starts, you can submit your Flink jobs to YARN directly using the Flink job cluster. After accepting the job, Flink will start a JobManager and slots for this job in YARN. The Flink job will be run in the YARN cluster until finished. The JobManager created will then be shut down. Job logs will be available in regular YARN logs. Try this command to run a word-counting example:The Dataproc cluster will not start a Flink Session cluster by default. Instead, Dataproc will create the script “/usr/bin/flink-yarn-daemon,” which will start a Flink session. If you want to start a Flink session when Dataproc is created, use the metadata key to allow it:If you want to start the Flink session after Dataproc is created, you can run the following command on master node:Submit jobs to that session cluster. You’ll need to get the Flink JobManager URL:How to run a Java Beam jobIt is very easy to run an Apache Beam job written in Java. There is no extra configuration needed. As long as you package your Beam jobs into a JAR file, you do not need to configure anything to run Beam on Flink. This is the command you can use:How to run a Python Beam job written in PythonBeam jobs written in Python use a different execution model. To run them in Flink on Dataproc, you will also need to enable the Docker optional component. Here’s how to create a cluster:You will also need to install necessary Python libraries needed by Beam, such as apache_beam and apache_beam[gcp]. You can pass in a Flink master URL to let it run in a session cluster. If you leave the URL out, you need to use the job cluster mode to run this job:After you’ve written your Python job, simply run it to submit:Learn more about Dataproc.
Quelle: Google Cloud Platform

Prevent planned downtime during the holiday shopping season with Cloud SQL

Routine database maintenance is a way of life. Updates keep your business running smoothly and securely. And with a managed service, like Cloud SQL, your databases automatically receive the latest patches and updates, with significantly less downtime. But we get it: Nobody likes downtime, no matter how brief. That’s why we’re pleased to announce that Cloud SQL, our fully managed database service for MySQL, PostgreSQL, and SQL Server, now gives you more control over when your instances undergo routine maintenance.Cloud SQL is introducing maintenance deny period controls. With maintenance deny periods, you can prevent automatic maintenance from occurring during a 90-day time period. This can be especially useful for the Cloud SQL retail customers about to kick off their busiest time of year, with Black Friday and Cyber Monday just around the corner. This holiday shopping season is a time of peak load that requires heightened focus on infrastructure stability, and any upgrades can put that at risk.By setting a maintenance deny period from mid-October to mid-January, these businesses can prevent planned upgrades from Cloud SQL during this critical time.Understanding Cloud SQL maintenanceBefore describing these new controls, let’s answer a few questions we often hear about the automatic maintenance that Cloud SQL performs.What is automatic maintenance?To keep your databases stable and secure, Cloud SQL automatically patches and updates your database instance (MySQL, Postgres, and SQL Server), including the underlying operating system. To perform maintenance, Cloud SQL must temporarily take your instances offline.What is a maintenance window?Maintenance windows allow you to control when maintenance occurs. Cloud SQL offers maintenance windows to minimize the impact of planned maintenance downtime to your applications and your business. Defining the maintenance window lets you set the hour and day when an update occurs, such as only when database activity is low (for example, on Saturday at midnight). Additionally, you can control the order of updates for your instance relative to other instances in the same project (“Earlier” or “Later”). Earlier timing is useful for test instances, allowing you to see the effects of an update before it reaches your production instances. What are the new maintenance deny period controls?You can now set a single deny period, configurable from 1 to 90 days, each year. During the deny period, Cloud SQL will not perform maintenance that causes downtime on your database instance.Deny periods can be set to reduce the likelihood of downtime during the busy holiday season, your next product launch, end of quarter financial reporting, or any other important time for your business.Paired with Cloud SQL’s existing maintenance notification and rescheduling functionality, deny periods give you even more flexibility and control. After receiving a notification of upcoming maintenance, you can reschedule ad hoc, or if you want to prevent maintenance longer, set a deny period. Getting started with Cloud SQL’s new maintenance controlReview our documentation to learn more about maintenance deny periods and, when you’re ready, start configuring them for your database instances. What’s next for Cloud SQLSupport for additional maintenance controls continues to be a top request from users. These new deny periods are an addition to the list of existing maintenance controls for Cloud SQL. Have more ideas? Let us know what other features and capabilities you need with our Issue Tracker and by joining the Cloud SQL discussion group. We’re glad you’re along for the ride, and we look forward to your feedback!
Quelle: Google Cloud Platform

Making your new normal safer with reCAPTCHA Enterprise

Traffic from both humans and bots are at record highs. Since March 2020, reCAPTCHA has seen a 40% increase in usage – businesses and services that previously saw most of their users in person have shifted to online-first or online-only. This increased demand for online services and transactions can expose businesses to various forms of online fraud and abuse, and without dedicated teams familiar with these attacks and how to stop them, we’ve seen hundreds of thousands of new websites come to reCAPTCHA for visibility and protection.During COVID-19, reCAPTCHA is playing a critical role helping global public sector agencies to distribute masks and other supplies, provide up-to-date information to constituents, and secure user accounts from distributed attacks. The majority of these agencies are using the score-based detection that comes from reCAPTCHA v3 or reCAPTCHA Enterprise instead of showing the visual or audio challenges found in reCAPTCHA v2. This reduces friction for users and also gives teams flexibility on how to take action on bot requests and fraudulent activity.reCAPTCHA Enterprise can also help protect your business. Whether you’re moving operations online for the first time or have your own team of security engineers, reCAPTCHA can help you detect new web attacks, understand the threats, and take action to keep your users safe. Many enterprises lack visibility in parts of their site, and adding reCAPTCHA helps to expose costly attacks before they happen. The console shows the risk associated with each action to help your business stay ahead.Unlike many other abuse and fraud fighting platforms, reCAPTCHA doesn’t rely on invasive fingerprinting. These techniques can often penalize privacy-conscious users who try to keep themselves safe with tools such as private networks, and are in conflict with browsers’ pushes for privacy-by-default. Instead, we’ve shifted our focus to in-session behavioral risk analysis, detecting fraudulent behavior rather than caring about who or what is behind the network connection. We’ve found this to be extremely effective in detecting attacks in a world where adversaries have control of millions of IP addresses and compromised devices, and regularly pay real humans to manually bypass detections.Since we released reCAPTCHA Enterprise last year, we’ve been able to work closer with existing and new customers, collaborating on abuse problems and determining best practices in specific use cases, such as account takeovers, carding, and scraping. The more granular score distribution that comes with reCAPTCHA Enterprise gives customers more fine-tuned control over when and how to take action. reCAPTCHA Enterprise learns how to score requests specific to the use case, but the score is also best used in a context-specific way. Our most successful customers use features to delay feedback to adversaries, such as limiting capabilities of suspicious accounts, requiring additional verification for sensitive purchases, and manually moderating content likely generated by a bot. We also recently released a report by ESG where they evaluated the effectiveness of reCAPTCHA Enterprise as deployed in a real-world hyperscale website to protect against automated credential stuffing and account takeover attacks. ESG noted: “Approximately two months after reCAPTCHA Enterprise deployment, login attempts dropped by approximately 90% while the registered user base grew organically.”We’re continually developing new types of signals to detect abuse at scale. Across the four million sites with reCAPTCHA protections enabled, we defend everything from accounts, to e-commerce transactions, to food distribution after disasters, to voting for your favorite celebrity. Now more than ever, we’re proud to be protecting our customers and their users.To see reCAPTCHA Enterprise in action, check out our latest video.To get started with reCAPTCHA Enterprise, contact our sales team.Related ArticleProtect your organization from account takeovers with reCAPTCHA EnterpriseHow reCAPTCHA Enterprise helps protect your websites from fraudulent activity like account takeovers and hijackingRead Article
Quelle: Google Cloud Platform