da Agency

Introducing Python 3, Python streaming support from Cloud Dataflow

Streaming analytics is becoming an essential part of data platforms, helping businesses collect and analyze data in real time. At Cloud Dataflow, we’ve noticed a few trends in the data engineering industry. First, Python is emerging as one of the most popular choices for data analysts, and second, a growing number of apps are powered by streaming analytics. With these trends in mind, we’re excited to announce the general availability of Python 3 and streaming support for the Python SDK. Stack Overflow’s 2019 developer survey found that Python is the fastest growing major programming language, while a JetBrains survey observed that data analysis has become more popular than web development among Python users. In order to better serve this rapidly growing community, the developers of the Python language announced that Python 2 would be sunset in 2020. A consortium of open-source projects (including Apache Beam, the open-source SDK supported by Cloud Dataflow) followed suit by pledging to drop support for Python 2 no later than 2020.Separately, streaming analytics is becoming the standard for data analytics and ML. The most innovative companies are augmenting their data-driven analytics, letting teams make better strategic decisions with event-driven analytics, which enhances operational agility. Streaming analytics lets you take advantage of use cases such as personalization, fraud detection, IoT applications, and countless more. At Google Cloud Platform (GCP), our fully managed, no-ops streaming analytics stack comes in the form of Cloud Pub/Sub for data ingestion and Cloud Dataflow for data processing. (This pair was recently named a leader in the Forrester Wave for Streaming Analytics, Q3 2019.)We’re pleased to add support for Python 3 to help those of you still running in environments that will be imminently unsupported. This support means you can take advantage of the latest and greatest that Python has to offer. Python streaming will give data engineers the entire suite of streaming features that are offered by Cloud Dataflow, which include:Update: The ability to update your streaming pipeline (such as to improve or fix bugs in your pipeline code, or handle changes in data format)Drain: The ability to drain your data, which prevents data loss when finishing your streaming jobAutoscaling: Cloud Dataflow’s unique capability lets you choose the appropriate number of worker instances to complete your job and dynamically re-allocate more or fewer workers to account for changing traffic patternsStreaming Engine: Our next-generation streaming architecture will allow for a reduction in worker resources and more responsive autoscalingGetting started with Python 3 and Python streamingFollow the instructions in the quickstart to get up and running with your first pipeline. When installing the Apache Beam SDK, make sure to install version 2.16(or above). Command Line: pip install apache-beam[gcp]You can use the Apache Beam SDK with Python versions 3.5, 3.6, or 3.7 if you are keen to migrate from Python 2.x environments. After installation, you are ready to author your first streaming pipeline!In keeping with big data tradition, let’s look at a word count example, except we’ll analyze a stream of data. Here, you can see a snippet of code that consumes a stream of text data from Cloud Pub/Sub, defines a fixed window of 15 seconds, computes the count of distinct words within each 15-second window, and then writes the results to Cloud Pub/Sub:Navigate to the Cloud Dataflow section in the Cloud Console to see the job graph and associated metrics:Take Python streaming on Cloud Dataflow for a spin. Learn more in this handy Python Quickstart.
Quelle: Google Cloud Platform

da Agency

Opening the door to more dev tools for Cloud Spanner

We recently released a new native open-source JDBC driver built by Google Cloud Platform (GCP) to enable the easy integration of Cloud Spanner into graphical database development tools. This can be useful for software engineers designing application database schemas in Cloud Spanner, database architects accessing Cloud Spanner, or database administrators monitoring and maintaining Cloud Spanner.In this blog, we’ll walk through how you can integrate and access your Cloud Spanner databases from DBeaver, a popular open source graphical database development tool. DBeaver lets you browse and edit databases, create and execute SQL scripts, export data, transaction management and ER diagrams. Here’s a look at the tool:In order to access Cloud Spanner from DBeaver, you have to go through the following short process. (This assumes that you have DBeaver installed and at least one Cloud Spanner database created in a GCP project that contains some data.)Create a self-contained JAR file that includes the open-source JDBC driver and all its dependencies.Create a database driver in DBeaver.Create a database connection to your Cloud Spanner instance.Run Cloud Spanner queries.1. Create self-contained JAR fileThe easiest setup of a database driver in DBeaver is based on a self-contained JAR file that contains the JDBC driver and all its dependencies. One way to create a self-contained JAR file is to create a Maven project with only one dependency:and execute it withThis Maven command creates a single JAR file that has all dependencies of the JDBC driver resolved and included, like this: spannerjdbc-1.0-SNAPSHOT-jar-with-dependencies.jar.2. Create a database driverThe next step is to create a database driver in DBeaver. Start DBeaver, select the tab Database, select the drop-down item Driver Manager, and in the window that appears, select New. You’ll see this:Give the driver a name—for example, CloudSpanner—select Add File in the Libraries tab, and open the JAR file you created in the previous step. Then select the Find Class button that automatically determines the driver class name, and click OK twice. You’ve now defined a Cloud Spanner driver that can be used by database connections.3. Create a database connectionThe last step before executing queries is creating a database connection to a Cloud Spanner database. As before, select the tab Database and select the drop-down menu item New Database Connection. Find and select Cloud Spanner (the name of the database driver you created earlier), and select next>. Enter the JDBC URL (this JDBC documentation shows how the URL is constructed) and test it by selecting Test Connection. If all works fine, you’ll see this:Select OK, then Finish, and you are ready to go execute queries.4. Run Cloud Spanner queriesYou can execute queries by typing them into the query window and pressing the small orange triangle on the side of the window. The result appears underneath in a result window:All the functionality of the graphical development tool is now available for you to use with Cloud Spanner. Happy querying!You can also follow the above steps with other graphical database development tools to get access to your Cloud Spanner databases and issue queries. These steps are similar for the various tools and will serve you as a guidance.If you already have a database development tool, chances are that it works with the JDBC drivers, and a similar process to the one above should give you access with your development tool of choice: Try it out!If you do not have a database development tool yet, download and configure one and connect it to your Cloud Spanner database.
Quelle: Google Cloud Platform

da Agency

Updating App Engine with more new runtimes: Nodejs 12, Go 1.13, PHP 7.3 and Python 3.8

Last week, we announced that the Java 11 runtime for App Engine standard environment is now generally available. But that’s not all the App Engine news we have to share. In addition to Java 11, developers can also now use Nodejs 12, Go 1.13, PHP 7.3 and Python 3.8. These latest additions to App Engine mean that you can build applications with your favorite tools, libraries and frameworks with today’s most popular languages.Nodejs 12 (beta)Nodejs 12 entered into long-term support (LTS) on October 22, 2019, and it’s now available in beta on App Engine. Node 12 boasts a big upgrade to the V8 Javascript engine to version 7.4, helping your applications run faster than they did on prior Node versions. The most anticipated feature of the release, however, is dedicated support for async stack traces—making it easier to debug code with asynchronous call frames. In addition, worker threads move out of experimental, TLS 1.3 is now supported, and there’s a new default http parser. Check out this post from the Node Foundation for the full list of features. And if you’re ready to get started, give it a try on App Engine.Go 1.13 (beta)Try out the latest improvements to Go with the launch of Go 1.13 on App Engine, now in beta. This release makes Go’s new “modules” the default method of managing dependencies in Go, reducing the complexity and the amount of code previously required to vendor imports. Go 1.13 also improves memory management dramatically. In production workloads, users are broadly reporting decreases of 20% – 50% in memory footprint. PHP 7.3 (GA)PHP 7.3 is also now generally available. This latest version of PHP adds full support for LDAP controls, improves logging for the Fast CGI Process Manager, and simplifies working with multibyte strings, among other changes. All of these new features are available today on App Engine. Just upload your PHP source code, and we’ll run it for you without having to manage a single VM or server.Python 3.8 (beta)Finally, we’re announcing the availability of Python 3.8 in beta. Python 3.8 introduces a new assignment operator (walrus), positional-only arguments in function definitions, better debugging support for f-strings via the ‘=’ specifier, multiprocess shared memory, and many more improvements. It also features performance enhancements such as speeding up many built-in methods by eliminating unnecessary argument conversions, and utilizing a new parallel filesystem cache for compiled bytecode files.With App Engine, you get the best of both worlds: the modern, idiomatic runtimes and frameworks you love, coupled with the fully managed, pay-as-you-go simplicity of a serverless platform. You write the code; let us worry about running your infrastructure. All of these new runtimes are now available on App Engine. Give them a try today.
Quelle: Google Cloud Platform

da Agency

Cloud TPU breaks scalability records for AI Inference

MLPerf is the industry standard for measuring ML performance, and results from the new MLPerf Inference benchmarks are now available. These benchmarks represent performance across a variety of machine learning prediction scenarios. Our submission demonstrates that Google’s Cloud TPU platform addresses the critical needs of machine learning customers: developer velocity, scalability, and elasticity. MLPerf Inference v0.5 defines three datacenter-class benchmarks: ResNet-50 v1.5 for image classification, SSD-ResNet-34 for object detection, and GNMT for language translation. Google submitted results for all three of these benchmarks using Cloud TPU v3 devices and demonstrated near-linear scalability all the way up to a record 1 million images processed per second on ResNet-50 v1.5 using 32 Cloud TPU v3 devices1.Peak demonstrated scaling for select MLPerf v0.5 Closed offline submission normalized to the highest entry. *2Cloud TPUs are publicly available to Google Cloud customers in beta. These same TPUs are also being used throughout numerous large-scale Google products, including Google Search. Developer velocity: Serve what you train The Cloud TPU architecture is designed from the ground up to more seamlessly move ML workloads from training to serving. Cloud TPUs offer bfloat16 floating-point numerics, which allow for greater accuracy compared to integer numerics. Training and serving on the same hardware platform helps prevent potential accuracy losses at inference time and does not require quantization, recalibration, or retraining. In contrast, serving with low precision (e.g., 8-bit) numerics can create major complexities that require significant developer investment to overcome. For example, quantizing a model can add weeks of effort and risk to a project, and it is not always possible for a quantized model to achieve the same accuracy as the original. Inference hardware is lower-cost relative to ML developer effort, so increasing development velocity by serving ML models in higher precision can help save money and improve application quality.For example, using the TPU v3 platform for both training and inference allows Google Translate to push new models to production within hours of model validation. This enables the team to deploy new advances from machine translation research into production environments faster by eliminating the engineering time required to develop custom inference graphs. This same technology is available to Google Cloud customers to increase the productivity of their machine learning teams, accelerating the development of popular use cases such as call center solutions, document classification, industrial inspection, and visual product search.Inference at scaleMachine learning inference is highly parallel, with no dependency between one input and the next. MLPerf Inference v0.5 defines two different datacenter inference scenarios: “offline” (e.g. processing a large batch of data overnight) and “online” (e.g. responding to user queries in real-time). Our offline submissions leverage large-scale parallelism to demonstrate high scalability across all three datacenter-class benchmarks. In the case of ResNet-50 v1.5, we show near linear scalability going from 1 to 32 Cloud TPU devices. Google Cloud customers can use these MLPerf results to assess their own needs for inference and choose the Cloud TPU hardware configuration that fits their inference demand appropriately.Google Cloud TPU v3 speed-ups as demonstrated by Google’s MLPerf Inference 0.5 Closed submission. Results in this figure are drawn from the offline scenario.*3Cloud elasticity: On-demand provisioningEnterprise inference workloads have time-varying levels of demand for accelerator resources. Google Cloud offers the elasticity needed to adapt to fluctuating demand by provisioning and de-provisioning resources automatically while minimizing cost. Whether customers serve intermittent queries for internal teams, thousands of globally distributed queries every second, or run a giant batch inference job every night, Google Cloud allows them to have just the right amount of hardware to match their demand, minimizing waste due to underutilization of resources.For example, the Cloud TPU ResNet-50 v1.5 offline submission to MLPerf Inference v0.5 Closed demonstrates that just 32 Cloud TPU v3 devices can collectively process more than one million images per second. To understand that scale and speed, if all 7.7 billion people on Earth uploaded a single photo, you could classify this entire global photo collection in under 2.5 hours and do so for less than $600. With this performance, elasticity and affordability, Google Cloud is uniquely positioned to serve the machine learning needs of enterprise customers.Get started todayCloud TPUs have now set records for both training and inference. Google Cloud offers a range of inference solutions for the enterprise, allowing customers to choose among a wide variety of GPUs and Cloud TPUs. For example, we also offer exceptional price/performance with the NVIDIA T4 GPU for inference with quantized models. Google Cloud customers can get started with accelerated ML inference right away instead of waiting months to build an on-premise ML hardware cluster. If cutting-edge deep learning workloads are a core part of your business, we recommend following the Quickstart guides for Cloud TPUs or GPUs to get familiar with our ML accelerator platforms.1. MLPerf v0.5 Inference Closed offline; Retrieved from www.mlperf.org 06 November 2019, entry Inf-0.5-20, respectively. MLPerf name and logo are trademarks. See www.mlperf.org for more information.*2. MLPerf v0.5 Inference Closed offline; Retrieved from www.mlperf.org 06 November 2019, entries Inf-0.5-19, Inf-0.5-20, Inf-0.5-26, Inf-0.5-2, Inf-0.5-1, Inf-0.5-21, Inf-0.5-31, respectively. MLPerf name and logo are trademarks. See www.mlperf.org for more information.*3.  MLPerf v0.5 Inference Closed offline; Retrieved from www.mlperf.org 06 November 2019, entries Inf-0.5-15, Inf-0.5-16, Inf-0.5-17, Inf-0.5-18, Inf-0.5-19, Inf-0.5-20, respectively. MLPerf name and logo are trademarks. See www.mlperf.org for more information.
Quelle: Google Cloud Platform

da Agency

Exploring the machine learning models behind Cloud IAM Recommender

To help you fine-tune your Google Cloud environment, we offer a family of ‘recommenders’ that suggest ways to optimize how you configure your infrastructure and security settings. But unlike many other recommendation engines, which use policy-based rules, some Google Cloud recommenders use machine learning (ML) to generate their suggestions. In this blog post, we’ll take a look at one of our recommendation engines, the Cloud Identity and Access Management (IAM) Recommender, and take you on a behind-the-scenes look at the ML that powers its functionality. IAM Recommender in actionIAM Recommender helps security professionals enforce the principle of least privilege by identifying and removing unwanted access to GCP resources. It does this by using machine learning to help determine what users actually need by analyzing their permission usage over a 90 day period. For example, a user Anita might have been given the Project Editor role when a new Google Cloud Platform (GCP) project was spun up, which gives her more than two thousand permissions. Elisa the Cloud Admin might have granted her a lot more access than required, simply because she did not understand Anita’s needs.Here’s how Cloud IAM Recommender helps. Elisa can now use IAM Recommender to analyze Anita’s permissions usage, and determine that she only needs day-to-day access to the Compute Engine service, and occasional access to Cloud Storage services. Using ML, IAM Recommender predicts what Anita will need in the long-term, and recommends the Compute Engine Admin and the Storage Object Viewer roles. Elisa can choose to apply the recommendations, removing thousands of unneeded permissions in the process. This minimizes the potential attack surface and helps her organization stay compliant with governance best practices.As simple as the idea might sound, it can be challenging to fully capture a given user’s intent and permission needs. On the one hand, we want to make timely recommendations after processing a reasonable amount of usage history (e.g., 90 days). On the other hand, there could be some permission usage missing from our observation window—for example, some operations could be interrupted when a user goes on vacation, or, like cron jobs, only happen very infrequently. This is one of the ways that we leverage ML: using inference to fill those small but crucial gaps and improve the accuracy of our recommendations.Training the modelOnce we have normalized the logs, we run an ML model to answer the question: “Given that a user used permission A, B, and C in the last 90 days, what are the other permissions they might need in order to do their job?” We train our model to answer this question from two sets of signals:Common co-occurrence patterns in the observed history. The fact that a user used permission A, B, and C in the past provides a hint that A, B, and C might be related in some way and that they are needed together to carry out a task on GCP. If our ML model observes this pattern frequently enough across a large user base, the next time if a different user used permission A and B, the model would suggest that the user might need permission C as well.Domain knowledge as encoded in the role definitions. Cloud IAM provides hundreds of different predefined roles that are service-specific. If a set of permissions co-occur in the same predefined role, it is a strong signal that the role creators determined certain permissions should be granted together.Our ML model uses both of these signals as input attributes, and each attribute is an IAM permission name, such as iam.serviceAccounts.get, or bigquery.tables.list. To further capture the semantics encoded in the permission name, which can be easily understood by a human but not by a machine, we employ word embedding, a technique that is widely used in Natural Language Processing applications. The key idea is to project a large number of words (in our case thousands of permission names) to a lower-dimensional vector space where we can calculate the similarity between a pair of words, which is a reflection of the actual semantics of these two words. For example, bigquery.datasets.get and bigquery.tables.list will become very “close” to each other after embedding.Google Cloud takes precautionary measures to maintain the privacy of our users; for example no data from one customer is being shared with another. In addition, we deployed an anonymization scheme to achieve k-anonymity before feeding the usage history data into our training pipeline. First, we drop all personally identifiable information (PII) such as user ID related to each permission usage pattern. Then we drop all usage patterns that do not show up frequently enough across GCP. The global model trained on the anonymized data can be further customized for each organization using Federated Learning.Right-size your permissionsNow we’re ready to make some recommendations! For a given user, we retrieve their usage history for the trailing 90 days and their current role definition, and feed those into the trained model. The model then makes a prediction on what unobserved permissions this user is likely to need. We combine the inferred permissions with the observed usage and rank roles to recommend the least permissive role(s) that can cover them all, thereby helping to ensure our recommendations are safe and any access that has been previously used is not removed.Better yet, unlike deterministic rules-based recommendations that become outdated over time, the ML model will adapt to changes over time. So as your footprint in Google Cloud grows, or as Google Cloud adds more services and permissions, the ML model evolves with these changes to ensure it provides relevant recommendations. To learn more about IAM Recommender, check out the documentation.
Quelle: Google Cloud Platform