Kategorie: Google Cloud Platform

Building the data analyst driven organization from the first principles

In this blog series, a companion to our white paper, we’re exploring different types of data-driven organizations. In our previous blogs of this series, a data scientist driven organization seeks to maximize value derived from data by making it highly accessible and discoverable, while also applying robust governance and operational rigor to rapidly deploy ML models. A data engineering driven organization typically provides 3 categories of data workers, with data engineers acting as the stewards of data that is used to generate analyses by an analytics team, for consumption by business users. Many of the same design decisions and technologies come into play between these organization types, but the social and organizational aspects are different. Regardless of the composition of your organization’s data workers and their exact roles, you’re probably facing a lot of the same challenges. Some of these may be familiar to you:Your data is stale, noisy, or otherwise untrustworthyYou need reliable data quickly in order to make rapid business decisions, but integrating new data sources is time consuming and costlyYou struggle to find a balance among reducing risk, increasing profitability, and innovationA lot of your time is spent on pulling reports for regulatory compliance instead of generating insights for the businessSome of these challenges are more profound for companies in a highly regulated industry, but data freshness, time to insights, reduction of risk, and innovation are key to any company. The common thread is the tremendous pressure to transform insights into business value, as fast as possible. Your customers are demanding accurate and faster interactions driven by data. As a result, your organization needs to sharpen your data analytics capabilities to stay competitive.At the same time, technology is evolving around you, creating a skill gap with the introduction of new technologies such as data lakes or data processing frameworks such as Spark. These technologies are powerful but require programming skills in languages such as Java or Scala. They present a radically different paradigm to the classic SQL declarative approach. There is a delicate balance of data workers within a company, and more traditional data architectures require very specific technical skills. Any new technology stack that disrupts this balance requires a redistribution of technical skills or a different ratio of engineering resources to other data workers. It’s often easier for a department head to justify an additional person on the team with new skills than it is to make broad sweeping changes to a central IT department, and as a result, evolution and new skill sets only occur in pockets of the org chart.So, why doesn’t technology adapt to your needs?The rise and fall of technologies such as Hadoop has revealed the elephant in the room (pun intended). Technology needs to fit into your culture and needs to build on your capabilities. This allows you to be more productive, reflect business needs, and preserve your subject matter expertise. You don’t need to become an engineering driven organization to leverage new technology!We’re going to explore how a platform like BigQuery, a pioneer in the concept of a cloud structured data lake, can provide a scalable processing engine and storage layer that can deal with the new and diverse data sources, via a familiar, SQL-based user interface.Figure 1 – Data analysts. skill set gap on a data warehouse + data lake architecture vs Structured Data Lake architectureHow do you build the “data-driven” agenda for data analyst driven organizations?Before discussing the main levers to pull for the transformation, let’s define what we mean by a data analyst driven organization. It should be noted that whether an organization is analyst driven is not a binary concept, but instead presents a wide range of overlapping characteristics:Mature industry. At the macro level, these organizations are red-brick established names with legacy systems. Generally, the industry in which they operate can be considered mature and stable.Competition from emerging digital natives. From a competitive standpoint, in addition to other similar organizations, there are also emerging digital organizations (for instance. fintech) that aim to capture the fastest growing digital areas and customer segments that have the highest potential.EDW + Batch ETL. Technically speaking, the central information piece comes in the form of an enterprise data warehouse (EDW) built over the years with a high level of technical debt and legacy technology. The transformation of the data within the data warehouse is carried out through scheduled ETL (Extract Transform Load) processes such as nightly batches. This batch process adds to the latency of serving the data. Business Intelligence. Most data workers in the organization are used to answering business questions by launching SQL queries against a centralized data warehouse, creating reports and dashboard using BI tools. In addition, spreadsheets are used to access similar data. Thus, the internal talent pool is most comfortable with SQL, BI tools, and spreadsheets.Narrowing the focus to the data department, the main personas and processes in these types of organizations can be generalized as follows:Data analysts, focused on receiving, understanding, and serving the requests coming from the business and making sense of the relevant data.Business analysts put the information into a context and act upon the analytical insights. Data Engineers, focused on the downstream data pipeline and the first phases of data transformation, such as, loading and integration of new sources. In addition, managing the data governance and data quality processes.Finally and given its relevance, it is also worth digging deeper on what we understand by a data analyst. As a data analyst, your goal is to meet the information needs of your organization. You are responsible for the logical design and maintenance of the data itself. Some of the tasks may include creating layout and design of tables to meet the business processes, reorganization, and transformation of sources. In addition, you’re also responsible for the generation of reports and insights that effectively communicate trends, patterns, or predictions that the business asks for. Going back to our original question of how we can build the mission for the data analyst driven organizations, the answer is: using, and expanding  the experience and skill-set of the data analyst community.Figure 3 – Data analysts domain expansions for the development of a data-driven strategyOn one hand, we promote the trend of data analysts making steps into the business side. As discussed earlier, data analysts bring in valuable knowledge with a deep knowledge of business domains and with sufficient technical skills to analyze data regardless of its volume or size. Cloud-based data warehouses and serverless technologies such as BigQuery contribute to this expansion of responsibilities toward right (as highlighted in Figure 3). In a way, allowing data analysts to focus on adding value rather than wasting time and effort in administrative / technical management tasks. Furthermore, you can now invest that extra time going deeper into the business without being limited by the volume or type of data that the storage system supports.On the other hand, new data processing paradigms enable a movement in the opposite direction for the data analysts area of ​​responsibility. You can use SQL as the fundamental query tool for data analysis, but now you can also use it for data processing/transformation. In the process, data analysts are able to take on some of the data engineering’s work: data integration and enrichment.Figure 4 – ELT paradigm – a SQL-first approach to data engineeringData analyst driven organizations embrace the concept of ELT (Extract-Load-Transform) rather than the traditional ETL (Extract-Transform-Load). The main difference is the common data processing tasks are handled after the data is loaded to the data warehouse. ELT makes extensive use of SQL logic to enhance, cleanse, normalize, refine, and integrate data and make it ready for analysis. There are several benefits of such an approach: it reduces time to act, data is loaded immediately, and it is made available to multiple users concurrently. A robust, transformational, actionable architecture for data analyst driven organizationsSo far we have talked briefly about the technological innovations that enable the data transformation, in this section we are going to focus on a more detailed description of these building blocks.To define a high-level architecture, we are going to start by defining the first principles from which we derive the components and interrelationships. It goes without saying that a real organization must adapt these principles and therefore the architecture decisions to its reality and existing investments.Principle #1: SQL as the analytics “lingua franca”Technology should adapt to the current organizational culture. Prioritize components that offer a SQL interface, no matter where they are in the data processing pipeline.Principle #2: Rise of the Structured Data LakeInformation systems infrastructure and its data should converge, to help expand the possibilities of analytical processing on new and diverse data sources. This may mean merging a traditional data warehouse with a data lake to eliminate silos. Principle #3: Assume and plan for “data/schema liquidity”Storage is cheap, so your organization no longer needs to impose rigid rules regarding data structures before data arrives. Moving away from a schema-on-write to schema-on-read model enables real-time access to data.  Data can be kept in its raw form and then transformed into the schema that will be useful. In addition, the data platform can manage the process of keeping these copies in sync (for instance using materialized views, CDC, etc.). So do not be afraid to maintain several copies of the same data asset,  Combining these principles we can define a high-level architecture like the one shown in the following diagram.Figure 5 – A high-level informational architecture for the data analyst driven organizationsWhat components do we observe in the informational architecture of this type of organization?First of all, a modern data platform should support an increasing number of data analysis patterns: the “classic” Business Intelligence workloads with tools such as Looker,a SQL based ad hoc analytics interface allowing management of data pipelines through ELTEnabling data science use cases with machine learning techniques real-time event processing Although the first two patterns are quite close to the traditional SQL data warehousing world, the last two present innovations in the form of SQL abstractions to more advanced analytical patterns. In the realm of machine learning, for example, we have BigQuery ML, which lets us execute machine learning models in BigQuery using standard SQL queries. And Dataflow SQL streaming extensions enable aggregating data streams with the underlying Dataflow sources like Pub/Sub or Kafka. Think for a moment the world of possibilities that technology enables without the need to invest in new profiles and/or roles.For a data analysts driven organization, the data preparation and transformation challenge is a clear and loud message in choice between ELT vs ETL. Use ELT wherever possible; the significant difference with this new paradigm is where the data is transformed – inside the Structured Data Lake and by using SQL.It is possible to transform data with SQL without sacrificing functionalities offered by extensive data integration suites. But how do you handle scheduling, dependency management, data quality, or operations monitoring? Products such as dbt or BigQuery Dataform bring a software engineering approach to data modeling and building data workflows. At the same time, they allow non-programmers to carry out robust data transformations. Modelling techniques such as Data Vault 2.0 are making a comeback due to the power of ELT in the new Cloud driven data warehouses. Therefore, It is important to note that the logical distribution of the data remains unaltered following the classical patterns such as the Immon or Kimball reference architectures. [1] [2]In data analyst driven organizations, data engineering teams generally control extraction of data from source systems. While it can be made easier through the use of SQL-based tools, enabling data analysts to do some of that work, there is still a need for a strong data engineering team. There are batch jobs that would still require creating data pipelines that would be more suitable for ETL. For example, bringing data from a mainframe to a data warehouse would require additional processing steps: data types need to be mapped, COBOL books need to be converted, and so on. In addition, for use cases like real time analytics, the data engineering teams will configure the streaming data sources such as Pub/Sub or Kafka topics. The way that you deal with generic tasks is still the same — they can be written as generic ETL pipelines and then reconfigured by the analysts. For example, applying data quality validation checks from various source datasets to the target environment.  The main point is that with the power of cloud data warehouses, it is now possible to use ELT instead of traditional ETL tasks. However, as described above there are use cases such as data quality applications that we need ETL. In summary In this article we have identified the data analyst driven organization and reviewed the challenges faced by them. We have seen how it is possible to build a transformation plan around one of your most valuable assets: data analysts. We have also reviewed the main components that appear in a modern, scalable informational architecture needed to efficiently use such an organization. Data analysts’ responsibilities are expanding to advanced data engineering tasks such as automatic learning or real-time event processing. All of these are still possible through our familiar and beloved favorite interface: SQL. To get started with the Google Cloud data ecosystem, please feel free to contact us or start a free trial.
Quelle: Google Cloud Platform

Empowering non-profit New Incentives with no-code

As people everywhere continue to adapt to new ways of working and collaborating, many trends that might have seemed a decade away in 2019, have cemented themselves as today’s blueprints for success.  We’ve seen firsthand the acceleration in demand for technology that lends itself to agile development without compromising power, ease or security and governance. Businesses cannot afford for solutions to be locked within excessively bureaucratic processes, with line-of-business workers unable to affect real progress. But they also need the solutions developed within individual teams to be manageable and secure, without all the “shadow IT” risks that often arise when teams create their own apps and services. To help businesses balance this dynamic, we’ve provided no-code app creators combating the pandemic special access to AppSheet, Google Cloud’s no-code development platform, and we will continue to do so through December 31, 2021. Nigeria-focused NGO New Incentives is a superlative example of the organizations that have used AppSheet to achieve meaningful results.A non-profit working to improve infant mortality rates through immunization efforts, New Incentives began their no-code journey in 2015. Balancing the needs of a global team with operations that often take place in rural areas, their IT team needed solutions to be both flexible and responsive. Projects such as their expense app helped pave their no-code journey early on, demonstrating that important applications could be developed more quickly than in the past via a mixture of technical talent and non-technical talent, as well as via tools that let IT maintain governance requirements while individual teams experimented.  “It’s like building an entire ecosystem for our needs, with multi-tiered approvals, and the ability to bring in a lot of interesting, contextual information to be able to identify fraud,” said Svetha Janumpalli. “When someone’s reviewing an expense, being able to see their route in Google Maps is very important to know where we’re going and operating in these very hard-to-reach, hard-to-travel areas. Like, are people actually doing what was intended?”Pre-pandemic, most of the non-profit’s no-code app needs were focused on internal processes such as tracking of supplies, expenses, and data intake. When COVID began to emerge, they weren’t certain if their immunization program would be able to continue. “We were thinking early on, this is going to become a problem, and what are we going to do? We basically create crowds [by organizing ways for children to be immunized against disease],” said Janumpalli. “So, we knew early on that we wouldn’t be able to continue unless we got ahead of having good mitigation measures, and unless we made the case to government stakeholders—to our Community Partners—that we take them very seriously. Because AppSheet gives us speed, we were able to introduce these apps, iterate them, and then share that information back.” As the pandemic progressed, New Incentives leaders soon realized that AppSheet apps could be used to address external needs in addition to internal needs as communication with the external community was becoming increasingly important. “The apps are like having a portable office,” Janumpalli stated. To help prevent the spread of COVID-19 while maintaining their ongoing mission, the organization needed to engage in an open dialogue with the community to understand and meet each location’s unique needs. By building apps that let team members collect data from communities, quickly assess the data, and make changes, the New Incentives team knew they could stay aligned with those they served, even as conditions changed.“Sometimes, there’s no internet connectivity. Sometimes even cell phone communications are difficult. AppSheet helps the app user retrieve that information properly. And then it allows us to program the feeding of that data: who needs it and when they need it to make good decisions.”Taking a community-first approach, New Incentives began collecting information on the ground from local members that could then be trickled upwards through both their communities and the organization. Not only did this help build trust in a difficult time, but it allowed for faster app iteration and faster implementation of on-the-ground changes that needed to take place. “We could take that community member feedback and instead of waiting a year, literally within a week, we could review that information and then make improvements.”One of the solutions that came from these community-driven dialogues was their bi-hourly app. “We’re a program that increases demand for immunization, but we couldn’t risk increased crowding,” Janumpalli explained. “So, we had to put in a bunch of new protocols around sanitization, hand-washing, and social distancing. We introduced this app called bi-hourly monitoring, and it was so useful that irrespective of COVID in the future, we’ll keep that as part of our processes.”While COVID is still active, the resilience that New Incentives has built into their problem-solving has provided them an opportunity to continue their mission with minimal disruption. This non-profit has helped to double rates of full immunization (increased from 25% to 54%) in the region, and COVID placed these efforts in jeopardy. “We’re a health organization. We don’t want to introduce measures superficially,” Janumpalli stated. “I would imagine that if we didn’t have this type of real-time information to have the confidence that it’s okay to continue to operate, we would have had to stop operations more often out of a lack of information because it just would have exceeded our risk tolerance” “What I hope other people understand about AppSheet is that it helps with the full spectrum,” she added. “It’s not just data collection, but also how you manage that data, how you integrate that data into your team processes, how you review that data, how you gain actionable insights from that data. So for us it’s data that’s actually influencing people to change and empowering managers to be able to have all the information they need to provide the right guidance and leadership to their teams.”New Incentives is continuing their immunization efforts, scaling from an average of 5,800 new infants added to its program per month in 2020 to 27,000 for the month of June this year alone. The real-time flexibility of AppSheet combined with their innovative culture means the same small team can meet the expanding and diverse demands that will continue to emerge through COVID and beyond. Start innovating with no-code for free.
Quelle: Google Cloud Platform

Service orchestration on Google Cloud

Going from a monolithic architecture to microservices has clear benefits, including reusability, scalability, and ease of change. Most of the time, business problems are solved by coordinating multiple microservices. This coordination is based on event-driven architectures, which can be implemented via two approaches: choreography and orchestration.Click to enlargeService choreography and service orchestrationService Choreography –  With service choreography, each service works independently and interacts with other services in a loosely coupled way through events. Loosely coupled events can be changed and scaled independently, which means there is no single point of failure. But, so many events flying around between services makes it quite hard to monitor. Business logic is distributed and spans across multiple services, so there is no single, central place to go for troubleshooting. There’s no central source of truth to understand the system. Understanding, updating and troubleshooting are all distributedService Orchestration – To handle the monitoring challenges of choreography, developers need to bring structure to the flow of events, while retaining the loosely coupled nature of event-driven services. Using service orchestration, the services interact with each other via a central orchestrator that controls all interactions between the services. This orchestrator provides a high-level view of the business processes to track execution and troubleshoot issues. In Google Cloud,  Workflows  handles service orchestration. Once you have decided between the two approaches for your application, design questions are largely about the characteristics of the services and the use case. You should prefer orchestration within the bounded context of a microservice, but prefer choreography between bounded contexts. That is, you’ll likely have choreography at a higher level, with orchestration at lower levels, both in the same system.Google Cloud provides services supporting both orchestration and choreography approaches. Pub/Sub and Eventarc are both suited for choreography of event-driven services, whereas Workflows is suited for centrally orchestrated services. Google Cloud support for service orchestration Workflows Workflows is a service to orchestrate and automate Google Cloud and HTTP-based API services with serverless workflows. It is a fully managed, scalable, and observable way to define a business process and orchestrate calls to several services. Workflow calls those services as simple web APIs. Using Workflows you can define the flow of your business logic in a YAML-based workflow definition language and use the UI or API to trigger the workflow. You can use Workflows to automate complex processes including event-driven and batch jobs, error handling logic, sequences of operations, and more.  Workflows is particularly helpful with Google Cloud services that perform long-running operations, as Workflows will wait for them to complete, even if they take hours. With callbacks, Workflows can wait for external events for days or months.Google Cloud support for service choreographyPub/SubPub/Sub enables services to communicate asynchronously, with latencies on the order of 100 milliseconds. Pub/Sub is used for messaging-oriented middleware for service integration or as a queue to parallelize tasks. Publishers send events to the Pub/Sub service, without regard to how or when these events will be processed. Pub/Sub then delivers events to all services that need to react to them (Subscribers). Pub/Sub is also used for streaming analytics and data integration pipelines to ingest and distribute data (as covered in the Data Analytics post). Eventarc Eventarc enables you to build event-driven architectures without having to implement, customize, or maintain the underlying infrastructure. It offers a standardized solution to manage the flow of state changes, also known as events, between decoupled microservices. Eventarc routes these events to Cloud Run while managing delivery, security, authorization, observability, and error-handling for you. Eventarc provides an easy way to receive events not only from Pub/Sub topics but from a number of Google Cloud sources with its Audit Log and Pub/Sub integration. Any service with Audit Log integration or any application that can send a message to a Pub/Sub topic can be event sources for Eventarc. Additional services that help with both choreography and orchestrationCloud TasksCloud Tasks lets you separate out pieces of work that can be performed independently, outside of your main application flow, and send them off to be processed asynchronously using handlers that you create. These independent pieces of work are called tasks. Cloud Tasks helps speed user response times by delegating potentially slow background operations like database updates to a worker. It can also help smooth traffic spikes by removing non-user-facing tasks from the main user flow.  Difference between Pub/Sub and Cloud Tasks. Pub/Sub supports implicit invocation: a publisher implicitly causes the subscribers to execute by publishing an event. Cloud Tasks is aimed at explicit invocation where the publisher retains full control of execution including specifying an endpoint where each message is to be delivered. Unlike Pub/Sub, Cloud Tasks provides tools for queue and task management including scheduling specific delivery times, rate controls, retries, and deduplication.Cloud SchedulerWith Cloud Scheduler, you set up scheduled units of work to be executed at defined times or regular intervals, commonly known as cron jobs. Cloud Scheduler can trigger a workflow (orchestration) or generate a Pub/Sub message (choreography). Typical use cases include sending out a report email on a daily basis, updating some cached data every x minutes, or updating summary information once an hour. For a more in-depth look into the services covered in this post, check out the documentation.  For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.devRelated ArticleChoosing the right orchestrator in Google CloudThere are a few tools available for orchestration in Google Cloud—some better suited for microservices and API calls, others for ETL work…Read Article
Quelle: Google Cloud Platform

Standing out to customers through the Partner Differentiation journey

Differentiation brings great customer experiences and repeatable businessCustomers are looking for best-in-class, highly skilled experts that can serve as a trusted advisor through their transformation.  Finding the right partner, at the right time, in the right location, with proven skills and experience is the secret to a high impact innovation journey.  The Google Cloud Partner Advantage team has worked closely with our partners to help customers find that trusted partner advisor through what we refer to as Partner Differentiation.Our goal in creating a Differentiation journey is to help our partners leverage their existing strengths, while building even deeper Google Cloud specific depth in those areas.  In turn, Google’s endorsement of these milestones allows customers to tap into the best in our broad community.  Investing in the Differentiation journey helps partners stand out to customers, enables repeatable customer successes, and signals that they are the right choice to help customers realize their transformation vision with Google Cloud.And now, with two years of work behind us, we have hard data that demonstrates why Partner Differentiation matters.  Across several thousand partners from around the world, the companies that have completed their Differentiation journey with us are, on average, driving larger deals at greater scale, delivering more customer value, and bringing in more revenue.Now, let’s dive deeper into each of the areas of the Differentiation journey where our trusted Google partners drive great customer experience and ensure customers are really extracting the full value of their investment with Google..CertificationsBy maintaining active certifications, partners demonstrate their commitment to Google Cloud skills and convey to customers confidence in these fresh skills.  Partners who have focused on training and enablement of their teams through certifications tackle customer projects with confidence in their mastery of cloud skills. Based on an independent 3rd party study, 87 percent of certified survey participants were more confident about their cloud skills and 71 percent reported that certification enabled or will enable increased work with existing customers and helped to scale their business. Partners can take advantage of the latest Google Cloud certification preparation offerings, such as Partner Certification Kickstart, which provide partners with unique ways to continue their commitment to learning through on-demand and hands-on training with Google Courses, and overall cloud skill development, all while addressing the demand for Google Cloud certified individuals.Customer SuccessPartners continue to share their customer success through the customer success story tool on the Partner Advantage portal.  These industry and solution focused case studies position partners for the next phase of the Differentiation Journey – achieving Expertise, and ultimately, Specialization.  These stories are also shared with the market, customers, and analysts to demonstrate the impact they are having on real businesses.In addition to sharing their customer success through stories, Partner Advantage now offers partners an on-demand, two-way learning opportunity to amplify success and gain new insights through Net Promoter Score (NPS).  As partners encourage their customers to complete the NPS survey, additional opportunities are unlocked to earn new advocacy from Google.  While you can find a wealth of partner customer showcases on the Google Cloud Partner Directory, here are a few of the many great examples of partners leading with Google Cloud to modernize infrastructure in healthcare:Cloudbakers and Comanche County Memorial Hospital transitioned 2,000 employees to Google Workspace to improve collaboration, reduce costs, and increase security.Google Cloud and Quantiphi supported advances in cloud-based machine learning services for John Hopkins University BIOS Division to reduce infrastructure costs, unlock new paths of treatment, and dramatically reduce the amount of time it takes to evaluate scanned imagery following a stroke.With the help of MediaAgility, TRIARQ migrated to Google Cloud to modernize their platform, build new applications, and expand their global footprint.ExpertiseEarning Expertise allows partners to showcase their deeper Google Cloud knowledge through validated skills in select products, horizontal solutions and key industries, – also reflected in the Partner Directory. Customers are increasingly looking for very specific expertise to complete targeted cloud-based projects, in a particular industry even, making specialization more important than ever. Google Cloud now offers 100 Expertise designations that provide partners the platform to achieve Specialization, as well as the visibility among customers of the Specialization.New Expertise categories include:Cloud Foundry MigrationData Protection and PrivacyTelecommunicationsMigrate Openshift to Anthos / GKECompliance ModernizationModernize .NET ApplicationsGoogle Cloud Onboarding  Here are a few partners who had the most Industry and Solution expertise achievements in H1 2021 and continue to demonstrate their capabilities through Expertise attainment:NORTHAM: Quantiphi., Cloudbakers, MediaAgility, SlalomEMEA: TCS, DotModus, Netpremacy, CloudfreshAPAC: Cloud Ace, Transcloud Labs, Loxley Orbit, TPCGLATAM: Reeducation, Gentrop, Wingu Networks, Lineout Servicios Informáticos S.A., JAPAN: Cloud Ace Inc, SCSKSpecializationCongratulations again to our Google Cloud 2020 Partner Awards Specialization winners.Specializations signal the highest achievement within the partner Differentiation journey.  Specialization designations give the customer confidence in the partner’s ability to handle a specific project, and can also open the doors to even more work. A customer greets a specialized partner as an expert in that field with multiple successful implementations and impact value – they are truly trusted and endorsed by Google.Through 14 different strategic solution categories, partners can choose to apply based on sustained success and bench strength of their Google Cloud practice.  Congratulations to all of our partners who have achieved this milestone or renewed in 1H 2021:Application DevelopmentWipro Limited | ATOS | Tata Consultancy Services | Appsbroker Limited | INJENIA SRLCloud Migration Wipro Limited | DoiT International | Cloud Ace | Qarik Group LLC | Rackspace Technology | Accenture |  IPNET | Ackstorm | Bespin Global Inc. | Innext S.r.l. | TaosSela Software Labs | Cloud Mile | WEBEYE CLOUD TECHNOLOGY LIMITEDData AnalyticsUbilabs GmbH | Searce | Slalom | Xertica Labs, Inc | CloudCover | Pluto7 Consulting Inc | Cloudreach | Deloitte | TCS | Myers-Holum| Quantiphi Inc. MINDTREE | SFEIR | Datametica Solutions Inc.Data Management SpringMLEducationEdTechTeam | Educational Collaborators, LLC | BEDU TECNOLOGIA LTDA | PT. Reformasi Generasi Indonesia | EdTechTeacher, Inc | GETECH LIMITED | nivelA | Reeducation | Digital Family | Center for Creative Training Ltd. | Canopy | Tierney | Devoteam | Eduscape | ​Five Star Technology Solutions | EduTech | AMPLIFICA | Safetec Informática | 鴻綸科技 Hunglun Technology | MobileMind | Gestion del conocimiento digital ieducando S.L.InfrastructureATOS | Ackstorm | Cloud Ace | Revolgy Business Solutions | NubosoftSantoDigital | Searce | STACK LABS | Devoteam | Quantiphi Inc. | Cloudwürdig GmbH | Rackspace Technology | Deloitte | PT. InfoFabrica Consulting Indonesia| Cloudreach | SFEIR | TCSWEBEYE CLOUD TECHNOLOGY LIMITED | Engineering do BrasilLocation-Based ServicesUbilabs GmbH | Localyse | Atos | 28East (Pty) Ltd | Web Geo ServicesMachine LearningSoftServe | Dataflix Inc. | Datatonic | Emergya | Servinformacion | DeloitteMarketing AnalyticsQuantiphi Inc. | ATOS | 55 SASTrainingJellyfish | ROI Training | TOPGATE | NEC Management Partner, Ltd | Compendium Centrum Edukacyjne Sp. z o.o.SAP on Google CloudATOSSecurityCloud Ace | SpringML | CloudreachWork Transformation//SEIBERT/MEDIA GmbH | Orión Chile | Qi Network | IPNETWork Transformation EnterpriseATOS | Tigabytes | SADALearn more about how partners are creating positive customer experiences and standing out in the ecosystem through this short Partner Differentiation video. Not yet a Google Cloud partner? Visit Partner Advantage and learn how to become one today! Partners can get started roadmapping your Partner Differentiation Journey in just a few short steps.Looking for a solution focused partner in your region who has achieved Expertise and/or Specialization in your industry? Search our Global Partner Directory.Related ArticleUpdates to our Partner Advantage program help partners differentiate and grow their businessesWe’re showcasing our partners’ achievements and providing updates on our expanding ecosystem.Read Article
Quelle: Google Cloud Platform

How much can you save by migrating legacy apps to Google Cloud? Forrester does the math

Migrating expensive operating systems and traditional workloads to Google Cloud can help organizations reduce data center costs while improving performance and uptime. In addition, organizations can work with Google Cloud and its partners to further modernize those workloads, resulting in additional licensing savings, and avoiding cumbersome licensing audits by legacy enterprise software vendors.To better explore these ideas, Google commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study to examine the potential return on investment (ROI) enterprises may realize by migrating to Google Cloud. The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of migrating to Google Cloud for their organizations.What Forrester found was that the interviewees experienced significant improvements in performance and uptime, reduced management costs, and recognized significant cost savings, especially around licensing. The interviewees were able to vastly reduce licensing costs and the pressure of time-consuming licensing audits. The interviewees worked with Google to optimize and modernize their environment, allowing them to reduce their compute, management, and overpriced licensing costs further.Among the most compelling findings in the study include:Avoiding on-premises hardware, software, and licensing costs can save millions of dollars annually.Optimized cloud infrastructure can reduce cloud spending by 45%.Organizations are completing their cloud migration in half the time they projected.To learn more, download the entire complimentary study, The Total Economic Impact™ Of Migrating Expensive Operating Systems and Traditional Software to Google Cloud. Or, if you’re ready to dive into your cloud migration options, sign up for a free discovery and assessment of your IT landscape so we can help you map out a plan to Google Cloud.Related ArticleSimulate your Microsoft and Windows modernization journey with these demosSee for yourself how Google Cloud supports Windows virtual machines and containers at our online Microsoft and Windows on Google Cloud De…Read Article
Quelle: Google Cloud Platform

Find your GKE cost optimization opportunities right in the console

It doesn’t matter whether you have one Google Kubernetes Engine (GKE) cluster or a thousand, whether they’re all GKE standard clusters or autopilot, whether you’re the only person working on infrastructure, or are part of a larger team—knowing how to optimize your clusters for high performance and low cost is the key to success on GKE. We’ve written extensively about GKE best practices to lessen over-provisioning, and published a guide on monitoring your GKE clusters for cost-optimization. Today, we’re introducing the preview of GKE cost optimization insights, which brings some of that wisdom directly into the GKE interface, helping you strike the right balance between cost and performance for your GKE clusters and workloads.With this release, you’ll see firsthand how effectively your clusters and workloads use the compute resources that you pay for. A new cost optimization tab on both the GKE cluster and workload pages lets you view, filter, and learn more about the amount of CPU and memory that each of your clusters and workloads are using and requesting. On the cluster page, you can also see the total amount of allocatable resources, and on the workloads page, any resource limits you’ve set for specific compute resources. Data visualizations help you quickly identify any areas of slack between the amount of resources you’re allocating or requesting and the amount you’re actually using. With this information, you can rightsize your resource allocations or requests to control costs while maintaining performance.    Understanding resource utilization and GKE costs Working closely with our users has taught us that there are four important things to consider when it comes to running GKE workloads efficiently: the cultural shift toward FinOps, proper bin packing, app right-sizing, and demand-based downscaling.Cultural shift – Many teams that embrace the public cloud haven’t worked with a pay-as-you-go platform like GKE, so they’re unfamiliar with how resource allocation and app deployment processes can affect their costs. New GKE cost optimization insights can help teams evaluate the business trade-offs between cost and performance. Bin packing – The ability to pack apps into GKE nodes. The more efficiently you pack apps into nodes, the more you save. You can pack apps into nodes efficiently by ensuring you’re allocating and requesting the right amount of resources based on your actual usage.App right-sizing – The ability to configure the appropriate resource requests and workload autoscale targets for objects that are deployed in the cluster. The more precise you are in setting accurate resource amounts for your pods, the more reliably your apps will run and, in the majority of cases, the more space you will open in the cluster.Demand-based downscaling – To save money during low-demand periods such as nighttime, your clusters should be able to scale down with demand. However, in some cases, you can’t downscale because there are workloads that cannot be evicted or because a cluster has been misconfigured You can read more about these four considerations in the GKE best practices to lessen over provisioning.Using GKE cost optimization insightsGKE cost optimization insights make it easier to address some or all of those considerations directly in the GKE console.For example, with the cluster page view, you get a complete overview of your resource utilization, helping you visualize the relationship between your allocatable resources, your requested resources, and used resources, as well as the actual CPU and memory hours that you’ve used. The used and requested CPU as well as memory metrics tell you which of your clusters could potentially benefit from app right-sizing. Comparing requested and allocatable resource amounts can help you identify opportunities for better bin-packing. You can also see how GKE autopilot clusters automate bin-packing for your clusters, freeing you and your team up to focus on app right-sizing. With that benefit in mind, adopting autopilot clusters is proving to be the single fastest way to cost-optimize clusters with inefficient bin-packing.  Being able to see total CPU and memory hours for each cluster also helps you better understand any bin-packing or right-sizing problems you might have, while the ability to filter data by specific time periods can help you focus on specific use cases or visualize information within different time horizons.GKE cost-optimization insights are available not just for your clusters, but for your workloads as well.  Seeing your used and requested resource amounts for each of your individual workloads can help you identify and fix any workloads that are requesting or using resources inefficiently. As is the case for the cluster-level view, the workload-level view also aggregates CPU and memory usage to give you a clear understanding of your busiest workloads.Requirements and supported metricsUnder the hood, GKE cost optimization insights use Cloud Monitoring system metrics. Hence, if your clusters have system metrics collection enabled, you will see results out of the box.At the same time, while terms such as CPU and memory metrics are generally well understood, it’s important to understand all the metrics that GKE cost optimization insights uses.The following table lists the kubernetes metrics used to calculate insight’s metrics aggregations.If you need to do a quick analysis of absolute values, you can enable the optional columns that show all numbers behind the charts.Finally, the time picker available in GKE’s UI allows you to pick the time horizon for your calculations. Utilization metrics will be averaged over the specified time period, while total core and memory hours will be added up. Learn more about the cost optimization metrics here.Bringing it all togetherWhile it’s easy to see how these insights can help you discover bin-packing and app-right sizing opportunities, they can also help you with another challenge: establishing a cost-saving culture. Once you have visibility into your GKE resources, your team has access to this important information from right within the product, without having to build a solution yourself. Insights can also help you better understand GKE’s pay-as-you-go model and how unused resources can impact your infrastructure costs. Check your GKE console today for the new cost optimization insights. Ready to turn them into action? Learn more about best practices for running cost-optimized Kubernetes applications on GKE!
Quelle: Google Cloud Platform

Unlocking the hidden value of data: Launching Dataflow Templates for Elastic Cloud

A key strategic priority for many customers that we work with is to unlock the value of the data they already have. Up to 73%of all data within an enterprise goes unused for analytics. “Dark data” is a commonly used term to refer to this and industry observers believe that most enterprises are only leveraging a small percentage of their data for analytics. While there are many dimensions to unlocking data for analytics, one fundamental component is the need to make data available across systems so that users are not limited by organization and technological silos. To that end, we launched Dataflow Templates to address the challenge of making data seamlessly available across various systems that a typical enterprise needs to deal with. Dataflow Templates, built on the rich, scalable and fault tolerant data processing capabilities of Dataflow and Apache Beam, provides a turnkey, utility-like experience for common data ingestion and replication tasks.  Dataflow Templates are prepackaged data ingestion and replication pipelines for many of the common needs that users have to make data available to all their users and systems so that they can make the most of the data. We provide a number of Dataflow Templates including some of the most popular ones such as Pub/Sub to BigQuery, Apache Kafka to BigQuery, Cloud Spanner to Cloud Storage and Data Masking/Tokenization (using Cloud DLP). We continue to make more Templates available and recently launched Templates for Datastream to create up-to-date replicated tables of your data in Database into BigQuery for analytics.Today, we announce a new set of Templates  we  developed in partnership with Elastic.  These Templates allow data engineers, developers, site reliability engineers (SREs) and security analysts to ingest data from BigQuery, Pub/Sub and Cloud Storage to the Elastic Stack with just a few clicks in the Google Cloud Console. Once created, Dataflow Templates run in a serverless fashion: there is no infrastructure to size and setup, no servers to manage and more importantly, no distributed system expertise is required to setup and operate these Templates. Once the data is in Elastic, users can easily search and visualize their data with Elasticsearch and Kibana to complement the analytics they are able to perform on Google Cloud.“It’s never been easier for our customers to explore and analyze their data with Elastic in Google Cloud. By leveraging Google Dataflow templates, customers can ingest data from Google Cloud services, such as BigQuery, Cloud Storage and Pub/Sub without having to install and manage agents,” said Uri Cohen, Product Lead,Elastic Cloud. Getting started with these Templates is very easy, first head to Dataflow in the Google Cloud Console, select Create Job From Template and then select one of the Elastic Templates from the dropdown.The team at Elastic has also published a series of blog posts with detailed information on using these Templates. Ingest data directly from Google Pub/Sub into Elastic using Google DataflowIngest data directly from Google BigQuery into Elastic using Google DataflowIngest data directly from Google Cloud Storage into Elastic using Google DataflowAdditionally, you can  find more information, including other Templates available at Getting started with Dataflow Templates.
Quelle: Google Cloud Platform