Firebase Unity Solutions: Update game behavior without deploying with Remote Config

Last June we announced Firebase Unity Solutions, an open-source Github repository with sample projects and scripts to help you add cloud-based features to your games being built on Unity. Our debut project,Firebase_Leaderboard, utilized Firebase Realtime Database to create and manage a cross-platform high score leaderboard. Today, we’re introducing the second solution incorporating Firebase services into Unity: Auto-Sync with Remote Config.Auto-Sync uses Firebase’s Remote Config to allow Unity developers to modify their game’s behavior in real time, without requiring a fresh build and release cycle. This means you can tweak the behavior of entities in your game without waiting for builds to be processed through the app stores. In days past, making incremental changes to game behavior in real time required a good bit of setup. Firebase Remote Config is an outstanding tool for updating behavior without requiring a new publish cycle, but the Unity library requires hard-coding every single variable a developer wants to control.With this package, you can pop the provided Component onto any GameObject or Prefab, tweak how to identify the fields you want remote access to, and click “Upload to Remote Config.” When the game is live, you can modify those values right in a custom Unity Editor built using Remote Config Management REST API, click “Upload” again, and voilà: the live game updates right before your eyes!Think your platformer hero’s jump should be a little higher? Want your patrolling enemies to move a bit faster? How about modifying the density of powerups, ammo, or obstacles? Or disabling features in a marketplace until translations are ready, or toggling a promotion during a very specific time window? The possibilities are limitless; with Auto-Sync, this can all be done right from the Editor at a moment’s notice. No code push required.To learn more, visit theFirebase Unity Solutions repository on Github. You can incorporate the scripts into your project directly, or import the unitypackage file from the current-builds subdirectory to get started.
Quelle: Google Cloud Platform

Got microservices? Service mesh management might not be enough

A lot of enterprises are evolving their monolithic applications into microservices architectures. In this pattern, applications are composed of fine-grained services that communicate via APIs. Microservices promise, faster development, innovation, cloud scaling, better infrastructure optimization—and happier developers. No wonder this architecture gets so much attention. But that doesn’t mean implementing a microservices strategy is easy (If you’re in the middle of this process, you know it’s complicated). You’ve got lots of different departments working on lots of different projects, and all of them are at different places. How do companies get to a point where they can reap the benefits of microservices?In this post, we’ll explain why a successful microservices approach requires dedicated infrastructure for building and managing those services, how sharing access to services happens through APIs, and why APIs shared outside the domain of ownership need to be managed. We’ll also walk through how Istio, an open-source project that can help with microservices management, helps to control the potential chaos as microservices adoption spreads through an organization.APIs as communication contractCooperating services intercommunicate via APIs. Simply put, APIs are how software talks to software. The API defines the communication contract between a service provider and a service consumer. Whether you think of the provider and the consumer as “services,” or as “applications” is immaterial; the API defines how they make requests and receive responses. Sending and receiving JSON RESTfully over HTTP/1 seems to be the most common technical basis for an API, but APIs can also employ HTTP/2 or TCP, and may use gRPC, GraphQL, jsonRPC or other data and message representations. They’re all just APIs, and they may be more or less formally specified.When an application is “decomposed” into a set of services that intercommunicate via APIs, a new set of problems arises: how to manage all of those interdependent services and the communication between them. As the set of services or the number of instances grows, the service management problem grows, too.For example, one of the very first things to consider when building with microservices is the ability to secure traffic between the microservices. A common method for securing this communication is mutual transport layer security (mTLS), which enables both peers in an exchange to authenticate one another. Once authentication happens, this can be used to perform authorization decisions at the service that receives a request, based on the identity of the caller that’s been asserted with the TLS certificate. This important function is pretty basic and simple to do when you have two services, but it gets more and more difficult when the number of services grows. One might attempt to mitigate this with client libraries.But then there’s the reality that services are developed in various languages: Java, C#, Python, Golang, or Node.js. It gets very difficult to apply a variety of different policies if it requires independent implementations in a set of five distinct languages. The complications multiply, and it becomes obvious that we need a better model: some kind of management infrastructure to control the potential chaos.Enter the service mesh While the term “microservices architecture” refers to a general pattern, a service mesh is a particular realization of that pattern. A service mesh provides a transparent and language-independent way to flexibly and easily automate application network functions. (For more on service mesh, check out this blog series.)Simply put, services meshes were developed to solve the problems of connecting, securing, controlling, and observing a mesh of services. Service meshes handle service-to-service interactions including load balancing, service-to-service authentication, service discovery, routing, and policy enforcement. Istio is an open-source project that delivers a service mesh; it’s backed by Google, IBM, Red Hat, Lyft, Cisco, and others, and is being used in production by companies like eBay, Autotrader, Trulia, Continental, and HP.Istio aims to help connect, secure, control, and observe the services in the mesh.Connect Istio helps control the flow of traffic and API calls between services intelligently; services connect to their dependent services via names, and load automatically gets balanced across all of the available runtime instances of a target service. Retries, circuit breakers, canary releases—all are handled automatically and configured for the mesh.Secure Istio automatically secures communications between services through managed authentication, authorization, and encryption. Each service has an identity asserted by an X.509 certificate that is automatically provisioned and used to implement two-way (mutual) Transport Level Security (TLS) for authorization and encryption of all API exchanges. Control Istio applies policies (for example, routing, rate limits, quotas) and enforces them across services. Inbound and outbound communications are controlled—even requests that go to external systems.Observe Istio ensures visibility with automatic tracing and operational logging of services.The goal of using a service mesh like Istio with your microservices system is better security, more reliability, lower cost, scale, and better resiliency within a set of closely intercommunicating systems.  A look at services in a meshSuppose an application is a custom inventory management system for a retailer, composed of several cooperating, related services:Policies defined in the service mesh might dictate that the pricing service can make outbound calls only to its data store, while the product service can call the location and inventory and pricing services, but not anything else.  If the team uses Kubernetes as the underlying platform for these services, Kubernetes ensures that unhealthy instances of these services get stopped and new instances get started. The service mesh ensures that the new instances are governed by the same set of policies.Sharing APIs with consumers outside the meshService meshes generally focus on the problem of managing and controlling the intercommunication among all the disparate services that comprise the application. Surely, it’s possible to define an Istio gateway to accept inbound requests. But that’s not enough; there’s a need to manage the requests into the mesh from service consumers, and to manage outbound requests—perhaps to a SaaS CRM system or another externally managed service.Back to our example: the product service needs to accept inbound requests from clients, such as an app running on hundreds of thousands of mobile phones. But the product service might want to modify its behavior based on the identity of the user making the call. The app on the mobile phone uses the “Product API”—the communication contract exposed by the product service—to send in requests. The product service might also need to connect to a Salesforce system.Regarding inbound requests arriving from systems that are significantly separated from the services in the mesh, how should those requests be secured, controlled, and managed? (Examples of “significantly separated” could include requests from third-party apps or even from different business units or teams within the same company. The requirements for these separated systems are quite different from those for inter-service communication). For an external or mobile client, we cannot rely solely on a TLS certificate to assert the identity of the inbound request. While app producers can provision client certificates into mobile apps, in many cases the client has no certificate to use for transport-level security. Clients may use a different form of identity assertion, relying on message-level security and signing, such as a self-signed JWT.Often the system would also like to authenticate the identity of a human user. Within the service mesh, the service identity is the only identity, but requests arriving from a mobile app, kiosk, or web app should carry user identity as well. This generally isn’t done with an X.509 certificate, but rather with a token (think OAuth) that asserts identity information about the user.Rate limits for external consumers will be different, and may depend on the status of the developer of the consumer app (or “client”). For example, client apps built by partners might get greater rights and higher transaction allowances. These limits are often used for business purposes or may be in place to protect an entire system rather than an individual service.It may be desirable to modify or filter the original API requests and responses depending on the client’s use case, or the user’s identity. For example, a lightweight client might want a compressed data format in lieu of JSON. Or a client might want a filtered view of the JSON, with some of the fields returned and some omitted.These different requirements apply to externally-developed apps, and in general to any client or service consumer that is significantly separated from the target services. Verification and throttling of requests coming from a client built by a developer on the service-development team is less useful; if the client app (or consuming service) is misbehaving, the API publishing team can notify their internal partners, and that client app development team just gets to work and fixes it. But calls arriving from a client built by an external partner need more verification. It’s not practical to engage on a 1:1 basis with developers of external apps. More strict enforcement is necessary here; that’s where API management come in.API management enables the sharing of APIsSharing a microservice means exposing it as an API for use by a developer outside the team that built the service. Microservice APIs that are shared outside of a small team need to be managed. An API management infrastructure, such as Google Cloud’s Apigee API management platform, helps address the different requirements for requests that are sent from external systems. Apigee also supports Istio as an API gateway or enforcement point. API management enables you to:Share APIs by publishing them and making them available to developers outside the core app team. These external developers need to gain authorization to access the APIs and understand how to use the APIs to build their own apps and clients. The core app team wants a way to distinguish clients built by different developers, and even distinguish between clients built by the same developer. This is all enabled by a developer portal, and the ability for developers to self-register and self-provision credentials for the APIs.Productize by treating APIs as “digital products.” This means: collecting related APIs into a coherent, consumable unit, to enable publishing and sharing. grouping complementary APIs that may have disparate origins into a single unit and normalizing security requirements on those APIs (rate limits, security credentials, message signing, and so on). modernizing APIs by transforming SOAP to JSON, implementing caching, or wrapping validation logic around “bare” or naive internal APIs. potentially monetizing APIs directly, charging developers for inbound traffic.Report on usage trends, traffic splits, latency and user experience. This enables the API product team to feed insights back into the API’s design, to iterate on the API-as-digital-product—and maximize business value. This capability is built on an analytics subsystem that scales.Comparing API management to service managementServices management and API management are different. A typical large, information-driven enterprise will deliver thousands of services, and will share hundreds of those via APIs to outsiders. Service management and API management satisfy the needs in these different spheres:A mature enterprise might aspire to the following goals:All services will be managed; they’ll have consolidated logging, tracing, mTLS, and retry policies applied and enforced. Access to services is via loosely governed APIs.All APIs shared outside their original ownership domain will be managed. Developers external to the team can view them, request access to them, and gain credentials. And access from apps built by external developers will be more strictly governed and throttled. Analytics data will help inform modifications to shared APIs.  But there are some logical parallels between the capabilities in service meshes like Istio and API management platforms like Apigee. Both Istio and Apigee:Enable policy enforcement for requests (rate limiting, quotas, and token verification)Can perform request routing based on data in the requestCollect logging information for observabilityUse a communication proxy to implement these controlsHowever, these two systems are targeted to different needs. Services management is intended for services built by a more or less closely related development team. This includes some aspects of managing the communication among those services, for example, mutual-TLS enforcement and automatic certificate provisioning, or rate limiting, or routing.On the other hand, API management is intended primarily to manage sharing of APIs outside of a core team. An outsider might be a member of the team across the hall, a developer in a different division of your company, or a developer at a different company. In any case there is significant separation between the API consumer and the API provider, which demands a greater degree of management of the API. While technically there are parallels, the requirements are different enough that, especially as the number of services under management grows, and as the number of APIs shared beyond their publishing teams grows, companies will want dedicated infrastructure to manage each.How HP manages microservicesHP Inc., which sells a wide range of printing solutions to consumers and enterprises, builds a variety of core services that are shared across business units at the company, including identity management and content management. The decision to move to a microservices architecture was driven in large part by the need to move faster, says Galo Gimenez-Palop, distinguished technologist and platform architect at HP. Large teams working on applications from different functional areas created the need for lots of synchronization—and lots of meetings. “We had these continuous integration pipelines that would take hours. And because there were so many dependencies, there were manual gates to decide if something was moving or not moving forward,” Gimenez-Palop tells us.So it’s no surprise that HP was attracted to the increased development velocity promised by a microservices architecture. Adopting microservices (alongside Kubernetes container orchestration, which accelerated teams building and deploying applications) would enable smaller development teams working on smaller code bases, with services going into production independently to reduce reliance on other teams—“you build it, you run it,” as Gimenez-Palop puts it. Yet moving from monolithic applications to a more distributed microservices architecture posed several challenges—especially as the number of microservices grew. The stumbling blocks mounted, Gimenez-Palop says: from orchestration difficulties, to challenges in discovering services, to breakages resulting from modifying services that depend on other services, to “policy sprawl,” to issues with ensuring that new versions of services integrate with other services.“How do you know which is the other version of the other service and when the other team changes the version of the service?” Gimenez-Palop asks. “Integration testing became really, really difficult because now, when you release one of the services, you need to do the integration testing with a bunch of other testing other services.”“As soon as you start to do microservices, you will find that it’s very easy to get in a complicated mess,” he adds.Istio proved to be the solution for HP. It simplified the complexities of microservices communications by providing a standardized way to connect, secure, monitor, and manage microservices. As a vital plane for service-to-service control and reliability, Istio handles application-layer load balancing, routing, and service authentication.Sharing microservices-based core services with other business units within HP is done by exposing them as APIs with other teams in the organization or with external partners and developers.But when microservices are exposed as APIs, they require API management, which makes it easier to extend the value of microservices both within the enterprise and to external developers. HP uses the Apigee platform for this purpose, gaining security, visibility, and control along the way.“We can have written contracts with the consumers of our APIs, we can have visibility into how those consumers are using the APIs of different services,” Gimenez-Palop says. “We can have policies like authorization, authentication, and payload inspection, all centralized in a single location.”Learn more by watching the Google Cloud NEXT ‘19 session:Can I use services management alone?As services become more prevalent and more fine-grained within an enterprise, formal service management via dedicated service mesh infrastructure will become a requirement.But is a service mesh alone enough? Sometimes. In the case where all of the inter-cooperating services are built and managed under the same ownership domain (a company’s director of engineering, for example), and access to the services is rarely shared with outsiders (other teams) via exposed APIs, a service mesh such as Istio will likely satisfy the requirements. Clients to the services are all in-house and don’t have significant separation from the services themselves.If services expose an externally consumable API that outsiders can see and use, then API management (as Gimenez-Palop says above) becomes a natural complement to service management. Better togetherMicroservices continues to be a big idea. A successful microservices approach requires tooling and infrastructure for building and managing those services. Sharing access to services happens through APIs, and APIs shared outside the domain of ownership need to be managed. Drawing the line between what is inside the domain of ownership and what is outside, and therefore which API calls need less management and which need more, is a judgment call.  Service meshes and API management are complementary and are used to solve distinct problems around services and APIs. While they both use communication proxies, and while there are parallels in function, the differences in domain set them apart, and most companies will see significant benefits from using them together.For more, watch Dino and Greg’s popular presentation at Google Cloud NEXT ‘19, “APIs, Microservices, and the Service Mesh.”
Quelle: Google Cloud Platform

Analyzing GCP costs using folders and BigQuery Billing export

Effectively managing and reporting on costs is a critical part of financial stewardship, whether you’re running a multi-billion-dollar enterprise business or small household budget. But making data-driven decisions on how to cut costs or double down on a business venture all starts with collecting the data you’ll need to inform those decisions. At Google Cloud Platform (GCP), one of our goals is to help customers make data-driven decisions about their cloud costs and usage. You can do this through multiple native capabilities available in the Cloud Console, including Billing reports, budgets and alerts, configurable quotas, resource hierarchies, and Billing export. While costs per project, product, and SKU are available via our native Billing reports, we often hear that enterprises face a challenge when reporting comprehensive spend, KPIs, and cost of goods sold (COGS) per team or department, which often involves more comprehensive data. Fortunately, detailed GCP Billing data is easily exportable into BigQuery and visualized with a BI tool like Data Studio so you can do more detailed analysis. Right now, GCP folder hierarchies (denoted as project.ancestry_numbers in the Billing export) only reflect their numerical and immutable ID form rather than their folder name (i.e., a GCP project under folder /acme/production/app1 is reflected by something like /720879011932/792354814932/1068832318756).To make it easier to allocate costs across departments, teams, cost centers, etc., we’ve put together this solution. Read on to see how to convert IDs into folder names so you can more easily track and analyze costs.Generating folder hierarchy namesTo convert your Billing export’s project.ancestry_numbers into something human-readable, you’ll first need a key that will map ancestry numbers like 720879011932 into the folder name “acme,” and a way to link parent/child folders.Using our native tools to view your folder structure is simple for one-off requests via the Cloud Console or gcloud CLI, or by API, but it can be complex if your enterprise organization is built from dozens of departments, teams, and hierarchical structures. To make this process even simpler, consider using a GitHub tool like folder-lookup, which uses Cloud Functions to programmatically crawl your organization’s folder structure and create a folder-structure key directly in BigQuery. After running this tool, you’ll see a simple folder-structure key to join against your Billing export, like so:Enhancing your BigQuery Billing exportThe next step is to convert Folder IDs into Folder Names within your Billing export. In order to cut down costs (i.e., the total amount of data queried), we’ll run this query only once per day via a scheduled BigQuery query. This query joins the previous day’s Billing export to this folder-structure key and appends a running Billing export with a folder data table. If data freshness is a high priority for you, you can easily modify the scheduled query to pull data on a more regular cadence (you’ll also have to alter the line ”WHERE _PARTITIONDATE = DATE_SUB(current_date(), INTERVAL 1 DAY)”).The query runs against the previous day’s data, so here we’ll run the scheduled query at 7 a.m. every day to get a full picture of the previous day’s spend. Note that the below query assumes a maximum folder depth of four, so it may have to be slightly augmented to meet your requirements.After running this query, you’ll see there’s now more detailed information on spend per department, and it’s easier to aggregate costs by environment or application.And thanks to Data Studio, visualizing this data is a piece of cake too:Monitoring your company’s cloud spend is crucial for quickly catching issues, prioritizing resources, and making better financial decisions. GCP provides tools to help you conduct a comprehensive analysis of your costs with Billing reports, Billing export, and Data Studio. And thanks to BigQuery’s powerful capabilities, crunching big data and using these insights to make data-driven decisions has never been easier. To see more options for BigQuery Billing export queries, check out our billing query example guide.
Quelle: Google Cloud Platform

DR for cloud: Architecting Microsoft SQL Server with GCP

Database disaster recovery (DR) planning is an important component of a bigger DR plan, and for enterprises using Microsoft SQL Server on Compute Engine, it often involves critical data. When you’re architecting a disaster recovery solution with Microsoft SQL Server running on Google Cloud Platform (GCP), you have some decisions to make to build an effective, comprehensive plan. Microsoft SQL Server includes a variety of disaster recovery strategies and features, such as Always On availability groups or Failover Cluster Instances. And Google Cloud is designed from the start for resilience and availability. There are several types of data centers available within GCP where you can map SQL Server’s availability features based on your specific requirements: zones and regions. Zones are autonomous data centers co-located within a GCP region. These regions are available in different geographies such as North America or APAC. However, there is no single disaster recovery strategy to map Microsoft SQL Server DR features to Google Cloud’s data center topology that satisfies every possible combination of disaster recovery requirements. As a database architect, you have to design a custom disaster recovery strategy based on your specific use cases and requirements.Our new Disaster Recovery for Microsoft SQL Server solution provides information on Microsoft’s SQL Server disaster recovery strategies, and shows how you can map them to zones and regions in GCP based on your business’s particular criteria and requirements. One example is deploying an availability group within a region across three zones (shown in the diagram below). For successful DR planning, you should have a clear conceptual model and established terminology in place. In this solution, you’ll find a base set of concepts and terms in context of Google Cloud DR. This includes defining terms like primary database, secondary database, failover, switchover, and fallback.You’ll also find details on recovery point objective, recovery time objective and single point of failure domain, since those are key drivers for developing a specific disaster recovery solution.Building a DR solution with Microsoft SQL Server in GCP regionsTo get started with implementing the availability features of Microsoft SQL Server in the context of Google Cloud, take a look at this diagram, which shows the implementation of an Always On availability group in a GCP region, using several zones:In the new solution, you’ll see other availability features, like log shipping, along with how they map to GCP. In addition, features in Microsoft SQL Server that are not deemed availability features—like server replication and backup file shipping—can actually be used for disaster recovery, so those are included as well. Disaster recovery features of Microsoft SQL Server do not have to be used in isolation and can be combined for more complex and demanding use cases. For example, you can set up availability groups in two regions with log shipping as the transfer mechanism between the regions.Disaster Recovery for Microsoft SQL Server also describes the disaster recovery process itself, how to test and verify a defined disaster recovery solution, and outlines a basic approach, step-by-step. Learn more about SQL Server on GCP and check out all of our solutions.
Quelle: Google Cloud Platform

Cloud Identity and Atlassian Access: User lifecycle management across your organization

User lifecycle management—the ability to automatically add and remove users to applications—is one of the key features of Cloud Identity, Google Cloud’s identity, access, and device management solution, which is available as a standalone solution and also to our G Suite Enterprise customers. Many of our G Suite customers use a variety of additional SaaS and on-premises applications, including Atlassian products like Jira, Confluence, Bitbucket, and more. Today, we’re pleased to announce that we’re partnering with Atlassian to enable user lifecycle management for these products.This partnership will allow you to provision and de-provision Atlassian users directly from the Cloud Identity or G Suite admin console,meaning admins will automatically have the right accounts created for all of their Atlassian apps. This saves your IT department from the headache of constantly updating user accounts and permissions across apps—now, when you add, modify, or delete a user in the admin console, changes are automatically reflected in your Atlassian apps, making it easier to manage access as employees change roles, join or leave the company. This update will also make life easier for your end users, enabling them to easily move from working in G Suite to their Atlassian apps.We do this by integrating with Atlassian Access, which provides enterprise-wide cloud governance and security across multiple Atlassian cloud products.Enabling SSO for Atlassian is required to set up user provisioningEnabling user provisioning for Atlassian in the admin consoleUnderpinning this partnership is SCIM 2.0, an industry-standard protocol for user lifecycle management that Cloud Identity has supported since 2016. In addition to Atlassian, Google Cloud partners with dozens of other apps and service providers for automated user provisioning and deprovisioning from within Cloud Identity and G Suite, including with Box, Dropbox, Salesforce, Slack, Zendesk, and more. See the full list of supported apps here and sign up today for a free trial to try it out yourself.
Quelle: Google Cloud Platform

5 reasons your legacy data warehouse won’t cut it

As we engage with enterprises across the globe, one thing is becoming clear: Today’s businesses are solving complex business problems that are data-intensive. But often, their data platform infrastructure is holding them back. Data platform architectures that were designed in the 1990s are not ready to solve business problems for 2020. We don’t have to tell you about the explosive data growth that’s going on for businesses around the world. If you’re managing data infrastructure today, you already know plenty about that data growth. Ever faster and larger data streams, global business needs, and tech-savvy users are all putting the pressure on IT teams to move faster, with more agility.Despite all these changes, it is often the legacy, traditional data warehouse where most of the data analytics tasks take place, and they’re underprepared for those demands. When we talk to people working in IT today, we hear a lot about the constraints that come with operating legacy technology while trying to build a modern data strategy. Those legacy data warehouses likely aren’t cutting it anymore. Here’s why—and here’s what you can do about it.  1. Business agility is hard to achieve with legacy tools. Business agility is the main goal as organizations move toward completely digital operations. Think of online banking, or retailers staying ahead of always-on e-commerce needs in a competitive environment. All these great, cutting-edge innovations reflect cultural and technical change, where flexibility is essential. A business has to be able to manage and analyze data quickly to understand how to better serve customers, and allow its internal teams to do their best work with the best data available.We hear that lots of data warehouses running today are operating at 95% or 100%, maxing out what they can provide to the business. Whether it’s on-premises or an existing data warehouse infrastructure moved wholesale to cloud, those warehouses aren’t keeping up with all the data requests users have. Managing and preventing these issues can take up a lot of IT time, and the problems often compound over time. Hitting capacity limits slows down users and ties up database administrators too.From a data infrastructure perspective, separating the compute and storage layers is essential to achieve business agility. When a data warehouse can handle your scalability needs and self-manage performance, that’s when you can really start being proactive. 2. Legacy data warehouses require a disproportionate degree of management. Most of the reports and queries your business runs are probably time-sensitive, and that sense of urgency is only increasing as users and teams see the possibilities of data analytics. In our engagements with customers, we often observe that they are spending a majority of the time on systems engineering, so that only about 15% of the time is spent analyzing data. That’s a lot of time spent on maintenance work. Because legacy infrastructure is complex, we often hear that businesses continue to invest in hiring people to manage those outdated systems, even though they’re not advancing data strategy or agility.To cut time on managing a data warehouse, it helps to automate the system engineering work away from the analytics work, like BigQuery enables. Once those functions are separated, the analytics work can take center stage and let users become less dependent on administrators. BigQuery also helps remove the user access issues that are common with legacy data warehouses. Once that happens, users can focus on building reports, exploring datasets, and sharing trusted results easily. 3. Legacy data warehouse costs make it harder to invest in strategy. Like other on-prem systems, data warehouses adhere to the old-school model of paying for technology, with the associated hardware and licensing costs and ongoing systems engineering. This kind of inefficient architecture drives more inefficiency. When the business is moving toward becoming data-driven, they’ll continue to ask your team for more data. But responding to those needs means you’ll run out of money pretty quickly. Cloud offers much more cost flexibility, meaning you’re not paying for, or managing, the entire underlying infrastructure stack. Of course, it’s possible to simply port an inefficient legacy architecture into the public cloud. To avoid that, we like to talk about total cost of ownership (TCO) for data warehouses, because it captures the full picture of how legacy technology costs and business agility aren’t matching up. Moving to BigQuery isn’t just moving to cloud—it’s moving to a new cost model, where you’re cutting out that underlying infrastructure and systems engineering. You can get more detail on cloud data warehouse TCO comparisons from ESG.4. A legacy data warehouse can’t flexibly meet business needs. While overnight data operations used to be the norm, the global opportunities for businesses mean that a data warehouse now has to load streaming and batch data while also supporting simultaneous queries. Hardware is the main constraint for legacy systems as they struggle to keep up.Moving your existing architecture into the cloud usually means moving your existing issues into the cloud, and we hear from businesses that doing so still doesn’t allow for real-time streaming. That’s a key component for data analysts and users. Using a platform like BigQuery means you’re essentially moving your computational capabilities into the data warehouse itself, so it scales as more and more users are accessing analytics. Unlimited compute is a pretty good way to help your business become digital. Instead of playing catch-up with user requests, you can focus on developing new features. Cloud brings added security, too, with cloud data warehouses able to do things like automatically replicate, restore and back up data, and offer ways to classify and redact sensitive data. 5. Legacy data warehouses lack built-in, mature predictive analytics solutions. Legacy data warehouses are usually struggling to keep up with daily data needs, like providing reports to departments like finance or sales. It can be hard to imagine having the time and resources to start doing predictive analytics when provisioning and compute limits are holding your teams back.We hear from customers that many of them are tasked with simplifying infrastructure and adding modern capabilities like AI, ML and self-service analytics for business users. The best stories about digital transformation are those where the technology changes and business or cultural changes happen at the same time. One customer told us that because BigQuery uses a familiar SQL interface, they were actually able to shift the work of data analytics away from a small, overworked group of data scientists into the hands of many more workers. Doing so also eliminated a lot of the siloed data lakes that had sprung up as data scientists extracted data one project at a time into various repositories to train ML models.These large-scale computational possibilities save time and overhead, but also let businesses explore new avenues of growth. AI and ML are already changing the face of industries like retail, where predictive analytics can provide forecasting and other tasks to help the business make better decisions. BigQuery lets you take on sophisticated machine learning tasks without moving data or using a third-party tool. We designed BigQuery so that our engineers deploy the resources needed for you to scale. It means your focus can change entirely toward meeting the needs the business has put forth, and bringing a lot more flexibility. BigQuery is fully serverless and runs on underlying Google infrastructure, so it integrates with our ecosystem of data and analytics partner tools. This architecture means you’re continually getting the most up-to-date software stack—analytics that scale, real-time insights, and cutting-edge functionality that includes geospatial and machine learning right from the SQL interface. Streamline your path to data warehouse modernization with BigQuery by learning about Google Cloud’s proven migration methodology and get started with your data warehouse by applying for our migration offer.
Quelle: Google Cloud Platform

Introducing Cloud Run Button: Click-to-deploy your git repos to Google Cloud

As developers, we love that Cloud Run allows you to deploy containerized serverless applications without having to worry about operations, scalability and security. You can package your app in a container, deploy to Cloud Run, and get a fully managed HTTPS endpoint.Today, we are introducing the Cloud Run Button, an image and link you can add to the README of your source code repositories to allow others to deploy your application to Google Cloud Platform using Cloud Run. Check out a demo:Cloud Run Button works with any repository that has a Dockerfile, or that can be built using Cloud Native Buildpacks. When you click the Cloud Run Button to deploy an application, it packages the application source code as a container image, pushes it to Google Container Registry, and deploys it on Cloud Run.For example, you can deploy our cloud-run-hello sample application (which has a Dockerfile) to Cloud Run on your GCP project by clicking this button:Once deployed, this application is hosted in your GCP project, and with Cloud Run’s pay-per-use model, you are only charged when the application handles requests.You can also make your Git repositories easily deployable to Cloud Run by following the instructions on the Cloud Run Button repository. Give it a try and let us know what you think!
Quelle: Google Cloud Platform

Introducing the BigQuery Terraform module

It’s no secret software developers love to automate their work away, and cloud development is no different. Since the release of the Cloud Foundation Toolkit (CFT), we’ve offered automation templates with Deployment Manager and Terraform to help engineers get set up with Google Cloud Platform (GCP) quickly. But as useful as the Terraform offering was, it was missing a critical module for a critical piece of GCP: BigQuery.Fortunately, those days are over. With the BigQuery module for Terraform, you can now automate the instantiation and deployment of your BigQuery datasets and tables. This means you have an open-source option to start using BigQuery for data analytics.In building the module, we applied the flexibility and extensibility of Terraform throughout and adhered to the following principles:Referenceable templatesModular, loosely coupled design for reusabilityProvisioning and association for both datasets and tablesSupport for full unit testing (via Kitchen-Terraform)Access control (coming soon)By including the BigQuery Terraform module in your larger CFT scripts, it’s possible for you to go effectively from zero to ML in minutes, with significantly reduced barriers to implementation. Let’s walk through how to set this up.Building blocks: GCP and Terraform prerequisitesTo use the BigQuery Terraform module, you’ll need—you guessed it—to have BigQuery and Terraform ready to go.Note: The steps outlined below are applicable for Unix- and Linux-based devices, and have not been optimized for CI/CD systems or production use.1. Download the Terraform binary that matches your system type and Terraform installation process.2. Install Google Cloud SDK on your local machine.3. Start by creating a GCP project in your organization’s folder and project. Try something via Terraform like the following:4. Let’s set up some environment variables to use. Ensure you updated the values to accurately reflect your environment.5. Go ahead and enable the BigQuery API (or use the helpers directory in the module instead)6. Establish an identity with the IAM permissions required7. Browse through the examples directory to get a full list of examples that are possible within the module.What’s in the box: Get to know the Terraform moduleThe BigQuery module is packaged in a self-contained GitHub repository for you to easily download (or reference) and deploy. Included in the repo is a central module that supports both Terraform v0.12.X and v0.11.X, allowing users (both human and GCP service accounts) to dynamically deploy datasets with any number of tables attached to the dataset. (By the way, the BigQuery module has you covered in case you’re planning to partition your tables using a TIMESTAMP OR DATE column to optimize for faster retrieval and lower query costs.) To enforce naming standardization, the BigQuery module creates a single dataset that is referenced in the multiple tables that are created, which streamlines the creation of multiple instances and generates individual Terraform state files per BigQuery dataset. This is especially useful for customers with hundreds of tables in dozens of datasets, who don’t want to get stuck with manual creation. That said, the module is fundamentally an opinionated method for setting up your datasets and table schemas; you’ll still need to handle your data ingestion or upload via any of the methods outlined here, as that’s not currently not supported by Terraform.In addition, the repo is packaged with a rich set of test scripts that use Kitchen-Terraform plugins, robust examples on how to use the module in your deployments, major version upgrade guides, and helper files to get users started quickly.Putting them together: Deploying the moduleNow that you have BigQuery and Terraform set up, it’s time to plug them together. 1. Start by cloning the repository:2. If you didn’t enable the BigQuery API earlier and create the service account with permissions, run the setup-sa.sh quickstart script in the helpers directory of the repo. This will set up the service account and permissions, and enable the BigQuery API.3. Define your BigQuery table schema, or try out an example schema here.4. Create a deployment (module) directory.5. Create the deployment files: main.tf, variables.tf, outputs.tf, and optionally a terraform.tfvars (in case you want to override default vars in the variables.tf file):6. Populate the files as detailed below.Main.tfOutputs.tfTerraform.tfvarsvariables.tf7. Navigate to the correct directory.8. Initialize the directory and plan.9. Apply the changes.What’s next?That’s it! You’ve used the BigQuery Terraform module to deploy your dataset and tables, and you’re now ready to load in your data for querying. We think this fills a critical gap in our Cloud Foundations Toolkit so you can easily stand up BigQuery with an open-source, extensible solution. Set it and forget it, or update it anytime you need to change your schema or modify your table structure. Once you’ve given it a shot, if you have any questions, give us feedback by opening an issue. Watch or star the module to stay on top of future releases and enjoy all your newfound free time (we hear BQML is pretty fun).
Quelle: Google Cloud Platform

Skip the heavy lifting: Moving Redshift to BigQuery easily

Enterprise data warehouses are getting more expensive to maintain. Traditional data warehouses are hard to scale and often involve lots of data silos. Business teams need data insights quickly, but technology teams have to grapple with managing and providing that data using old tools that aren’t keeping up with demand. Increasingly, enterprises are migrating their data warehouses to the cloud to take advantage of the speed, scalability, and access to advanced analytics it offers. With this in mind, we introduced the BigQuery Data Transfer Service to automate data movement to BigQuery, so you can lay the foundation for a cloud data warehouse without writing a single line of code. Earlier this year, we added the capability to move data and schema from Teradata and S3 to BigQuery via the BigQuery Data Transfer Service. To help you take advantage of the scalability of BigQuery, we’ve now added a service to transfer data from Amazon Redshift, in beta, to that list. Data and schema migration from Redshift to BigQuery is provided by a combination of the BigQuery Data Transfer Service and a special migration agent running on Google Kubernetes Engine (GKE), and can be performed via UI, CLI or API. In the UI, Redshift to BigQuery migration can be initiated from BigQuery Data Transfer Service by choosing Redshift as a source. The migration process has three steps: UNLOAD from Redshift to S3—The GKE agent initiates an UNLOAD operation from Redshift to S3. The agent extracts Redshift data as a compressed file, which helps customers minimize the egress costs. Transfer from S3 to Cloud Storage—The agent then moves data from Amazon S3 to a Cloud Storage bucket using Cloud Storage Transfer Service. Load from Cloud Storage to BigQuery—Cloud Storage data is loaded into BigQuery (up to 10 million files).The BigQuery Data Transfer Service, showing Redshift as a source.You can see more here about how customers are using the BigQuery Data Transfer Service to move database instances easily.To get started, follow our step-by-step guide, or read our article on migrating data to BigQuery using Informatica Intelligent Cloud Services. Qualifying customers can also take advantage of our data warehouse migration offer, which provides architecture and design guidance from Google Cloud engineers, proof-of-concept funding, free training, and usage credits to help speed up your modernization process. Learn more here.
Quelle: Google Cloud Platform

Itaú Unibanco: How we built a CI/CD Pipeline for machine learning with online training in Kubeflow

Itaú Unibanco is the largest private sector bank in Brazil, with a mission to put its customers at the center of everything they do as a key driver of success. As a result, one of its projects is AVI (Itaú Virtual Assistant), a digital customer service tool that uses natural language processing, built with machine learning, to understand customer questions and respond in real time.AVI helps about a million customers per month. It answers all but 2% of customer questions, and answers those questions correctly 85% of the time. In instances where AVI is not best suited to help a customer, it transitions to a live agent transparently, and in the same channel.To help continually improve and evolve AVI, as well as Itaú’s other projects that use machine learning, they needed an efficient strategy for machine learning model deployment. However, they quickly found that building a robust tool that helps their data scientists deploy, manage and govern ML models in production proved challenging. As a result, the team began working with Google Cloud to create a CI/CD pipeline based on the open source project Kubeflow, for online machine learning training and deployment. Here’s how they did it.How Itaú built their pipelineA machine learning project lifecycle mainly comprises four major stages, executed iteratively:Once a data scientist has a set of well-performing machine learning models, they need to operationalize them for other applications to consume. Depending on the business requirements, predictions are produced either in real time or on a batch basis. For the AVI project, two business requirements were essential: (1) the ability to have multiple models in production (whether using different techniques or models trained using distinct data), and (2) the ability to retrain the production model with new data.Although the data science and machine learning features are well cared for by the AVI multidisciplinary team, both model training and deployment are still not fully automated at Itaú. Additionally, internal change management procedures can mean it takes up to one week to retrain and deploy new models. This has made ML initiatives hard to scale for Itaú. Once the CI/CD pipeline is integrated with the AVI platform, the bank hopes that training and deployment will take hours instead of days, or even faster by using GPU or TPU hardware.Some of the main requirements for this deployment pipeline and serving infrastructure include:The Itaú team may work with several ML model architectures in parallel. Each of these models is called a “technique” in the team’s internal jargon.Promoting a new technique to production should be an automated process, triggered by commits to specific branches.It should be possible to re-train each model on new data in the production environment, triggered by the front-end used by agent managers.Several versions of the same or different models could be served simultaneously, for A/B test purposes or to serve different channels.ArchitectureItaú has a hybrid and multi-cloud IT strategy based on open source software and open standards to guarantee maximum portability and flexibility. This created a natural alignment with Google Cloud, which is also committed to open source and hybrid/multi-cloud. Therefore, the architecture was planned around open source platforms, tools and protocols, including Kubeflow, Kubernetes, Seldon Core, Docker, and Git. The goal was to have a single overall solution that could be deployed on GCP or on-premises, according to the needs and restrictions of each team inside the company.This is the high-level, conceptual view of the architecture:Models start their lives as code in the source repository, and data in object storage. A build is triggered in the CI server, producing new container images with the model code packaged for training. The CI process also compiles and uploads a pipeline definition to the training platform, and triggers a new training run with the latest data. At the end of the training pipeline, if everything runs well, a new trained model is written to object storage, and a new serving endpoint is started. The front-end server of the customer service application will use these API endpoints to obtain model predictions from a given input. Service administrators use the same application to manage training example data and classes. These users can trigger the training of a new model version with a new dataset. This is accomplished by triggering a new run of the training pipeline, with no need to reload or re-compile source code.For this project, the concrete architecture was instantiated with the following components:Itaú’s centralized infrastructure teams have selected Jenkins and GitLab as their standard tools for integration and source control, respectively, so these tools were used to build the integrated pipeline. For the container registry and object storage, the cloud-native solutions Container Registry and Cloud Storage were used, since they should be easy to replace with on-premises equivalents without many changes. The core of the system is Kubeflow, the open source platform for ML training and serving that runs on Kubernetes, the industry standard open source container orchestrator. Itaú tested the platform with two flavors of Kubernetes: Origin, the open source version of RedHat OpenShift, used by Itaú in its private cloud, and Google Kubernetes Engine (GKE), for easier integration and faster development. Kubeflow runs well on both.The centerpiece of the pipeline is Kubeflow Pipelines (KFP), which provides an optimized environment to run ML-centric pipelines, with a graphical user interface to manage and analyze experiments. Kubeflow Pipelines are used to coordinate the training and deployment of all ML models.ImplementationIn the simplest case, each pipeline should train a model and deploy an endpoint for prediction. This is what such a pipeline looks like in Kubeflow Pipelines:Since this platform will potentially manage several ML models, Itaú agreed on a convention of repository structure that must be followed for each model:The root of each directory should contain a Dockerfile, to build the image that will train the model, and an optional shell script to issue the docker build and push commands. The src subdirectory contains all source code, including a script called trainer.sh that will initiate the training process. This script should receive three parameters, in the following order: path to the training data set, path to the evaluation data set, and output path where the trained model should be stored.The pipeline directory contains pipeline.py, the definition of the Kubeflow Pipeline that will perform the training and deployment of the model. We’ll take a better look at this definition later.Container ImagesEach step in a KFP pipeline is implemented as a container image. For our minimum viable product (MVP), Itaú created three container images:Model trainer (sklearn_spacy_text_trainer)Model deployment script (serving_deployer)Model serving with Seldon Core (pkl_server)The model trainer image is built from the model source code tree, with the Dockerfile shown in the file structure above. The other two images are more generic, and can be reused for multiple models, receiving the specific model code as runtime parameters.The model trainer and deployer containers are built by simple shell scripts from their respective Dockerfiles. The model serving container is built with the s2i utility, which automatically assembles a container from the source code tree, using the Seldon Python 3.6 base image. The shell script below shows how that’s accomplished:Pipeline definitionA pipeline in Kubeflow Pipelines is defined with a Python-based domain specific language (DSL), which is then compiled into a yaml configuration file. There are two main sections to a pipeline definition: (1) definition of operators and (2) instantiation and sequencing of those operators.For this sample pipeline, an operator was defined for the trainer container and one for the deployer. They are parameterized to receive relevant dynamic values such as input data path and model endpoint name:The pipeline itself declares the parameters that will be customizable by the user in the KFP UI, then instantiates the operations with relevant parameters. Note that there is no explicit dependency between the train and deploy operations, but since the deploy operation relies on the output of the training as an input parameter, the DSL compiler is able to infer that dependency.Pipeline build and deployA commit to the main branch will trigger a build in Jenkins. The build script will execute the following steps:Build the containersCompile the KFP pipeline definitionUpload the new pipeline to KFPTrigger a run of the new pipeline to train the model (this step is optional, depending on what makes sense for each model and the team’s workflow)The sample script below executes steps 2 and 3, receiving a descriptive pipeline name as an argument:Pipeline runWhenever the training dataset is changed, a user can trigger a model training from the administration UI. Training a model is simply a matter of placing the new data in the right location and starting a new run of the pipeline that is deployed to Kubeflow. If successful, the pipeline will train the model and start a new serving endpoint to be called by the front-end.This REST call will return a run ID parameter, which can be used by the UI back end to poll for the run status and update the user when it’s done, or there is an error.Model prediction servingThe final step of the pipeline is, of course, serving model predictions. Since most of our models are created with Scikit Learn, Itaú leveraged Seldon Core, a bundled component of Kubeflow, to implement the serving endpoints. Seldon Core lets you implement just a simple predict method and takes care of all the plumbing for exposing a REST endpoint, with optional advanced orchestration features.Since the serving API tends to change infrequently, we opted to implement a generic class that can serve any model serialized to a PKL file. The deployment definition parameterizes a storage location with the PKL file and bundled model source code, which is then unpacked and used for serving by the container. The Python code that achieves this is listed below:This serving code is deployed for each endpoint by a shell script in the deployer container. The script takes in the location of the trained model, name and version for the endpoint, generates the necessary configuration and deploys it to Kubernetes:ConclusionWith this relatively simple architecture and very little custom development, Itaú was able to build a CI/CD pipeline for machine learning that can accelerate the pace of innovation while simplifying production maintenance for AVI and other teams. It should be fairly easy to replicate and adapt it to many organizations and requirements, thanks to the openness and flexibility of tools like Kubeflow and Kubeflow Pipelines.AcknowledgmentsThis work was created by a joint team between Google Cloud and Itaú Unibanco:Cristiano Breuel (Strategic Cloud Engineer, Google Cloud)Eduardo Marreto (Cloud Consultant, Google Cloud)Rogers Cristo (Data Scientist, Itaú Unibanco)Vinicius Caridá (Advanced Analytics Manager, Itaú Unibanco)
Quelle: Google Cloud Platform