How Revionics brings advanced analytics to retailers with help from Google Cloud

Editor’s note: Today we’re hearing from Revionics, a lifecycle pricing platform that gives retailers the confidence to ask the “what ifs” and “why nots” that result in profitable pricing strategies. In this blog, the Revionics team shares their cloud data warehouse migration journey and the lessons learned along the way. At Revionics, we’ve been in the data analytics business for nearly two decades, serving retail customers who wanted to leverage their data to drive their pricing strategies. At the company’s founding in 2002, our advanced AI models were probably the most complex in commercial use. Out of necessity, Revionics built the advanced technology infrastructure to handle the data volume and processing for our AI models. Fast forward to today, and cloud infrastructure has finally caught up. The state of current cloud computing infrastructure has opened up new avenues of data exploration for our teams and customers.Researching cloud optionsFrom the beginning, we didn’t just want to solve known problems. We wanted to improve our overall infrastructure to become more nimble to support our customers’ needs, today and in the future. Just like our AI models help retailers focus on what is possible, we need infrastructure that enables our data scientists to create that vision.We had been using on-premises Teradata appliances, and the parallelism and performance had worked well for us. But appliances have hardware and software restrictions that made it hard for us to share data and seamlessly scale. They’re not elastic, and couldn’t offer us the space we needed. We had maxed out the physical storage capacity of the appliances themselves, and we were rapidly approaching the time where we would need to renew the hardware to upgrade and expand performance and capacity. The space required for processing resulted in access constraints to the latest analytical results, impacting decision making based on current information. Replacing appliances with newer ones couldn’t alone solve our elasticity and data access problems, so we started exploring cloud options. In addition, many of our tech-savvy customers have in-house data analytics experts, and we wanted to offer them plug-and-play capabilities on our data warehouse infrastructure. That way, they could dig into their data and respond to market shifts accordingly. Once we learned more about current cloud options, we could visualize many ways to drive our business forward. For example, the elasticity and amount of storage available could let us really accelerate our product development and overall customer success.Cloud migration decisionsPartnering with Google Cloud, we adopted BigQuery. For each client, we built a data lake and a complete structured data warehouse, so every client’s data is isolated and securely accessible to meet client queries. The scale that BigQuery brought became essential when the COVID-19 pandemic wrought huge, overnight changes for our retail clients, who needed to meet unexpected demand while keeping their employees and customers safe. We were able to rapidly update our AI pricing models with our customers’ most current data and give them clarity to make the right pricing decisions. We migrated Teradata SQL to BigQuery SQL, which facilitated a fast migration. Converting Teradata DDL to BigQuery DDL was straightforward—we encountered a few challenges in our view queries due to differences in SQL, but this also gave us an opportunity to learn about how BigQuery works. We’re a lean company, so we needed to move data efficiently without a lot of manual work required. Our DevOps team helped us build tools so we could create script templates for different projects and customer datasets. For us, it was faster to redeploy our own tools than learn new tools. We had a lot of practical conversations discussing our options, and ultimately we did what was least disruptive to our customers and provided the greatest continuity for operational management, while leveraging select services available in Google Cloud.Revionics puts customers first in all that we do. Ensuring they were not impacted during the migration was a top priority, and we worked with our customers closely throughout the process. We migrated following retail’s busiest season, and we carefully performed our due diligence to internally sync and enable our customer go-live in January 2020. Google Cloud helped support our no-downtime migration and ensure business continuity.Google Cloud helped support our no-downtime migration and ensure business continuity.Migration learnings and lessonsMigrating our reports to run on BigQuery required some planning. We primarily did a lift-and-shift migration to minimize the total changes involved; all we had to do was point our existing metadata model to BigQuery. We made adjustments to some data types where necessary, but most of the reports did not need to be updated. However, some queries generate differently in BigQuery, so we took the opportunity to improve the design and performance of selected reports. As an example, our reports sit on views in a separate dataset, and we were able to simply move logic upstream, leveraging a few of the unique capabilities of BigQuery to improve runtime performance.Click to enlargeFor a successful migration, it helps to establish standards and adopt production tools early. We migrated chunks of the data model at a time, allowing us to test and improve through the project. We also took immediate advantage of several BigQuery database capabilities, such as implementing partitioning and clustering. For us, it was the right move and enabled a fast transition. Now, our next step is how to improve our data model. There are changes we can make under the covers that are impactful, such as views and the interface between BigQuery and our reporting platform. Composer and Airflow both came in handy for our data load process. We built extract pipelines to move data from our SQL Servers to Cloud Storage to load into BigQuery, all executed through Composer. We also take full advantage of built-in monitoring and logging tools (formerly known as Stackdriver). On the other side of migrationToday, the infrastructure we’ve created with Google Cloud has helped address the immediate needs we had, and provides the foundation for Revionics to solve new and interesting problems. We’re opening new doors, and Google Cloud has helped improve how we operate our infrastructure, forecast growth, and manage costs. Data access: For example, moving data to construct new analytics had at times been a slow, unwieldy process that could require days of copying and processing. Now, all of a client’s data is securely co-located in BigQuery, enabling immediate access to data for customer-specific analysis without impacting production operations in any way. Data processing happens in seconds and minutes rather than hours and days, whereas to analyze any one customer’s data before our cloud migration, end users often encountered roadblocks to move data off the SQL Server instance. Now, they don’t have to move data at all.Security: We’d been used to focusing on security, at times sacrificing usability out of necessity. With Google Cloud, we use Google’s built-in encryption at rest and in transit without any impact to usability, and with zero configuration requirements or management needed. We’ve improved our security footprint, lowered our management overhead, and improved performance significantly. Additionally, BigQuery makes it simple to triage issues, and we’ve gained significant efficiency in the way we find and solve any issues. The time spent triaging customer questions has been reduced dramatically. Seeing the business impactWhen we first started almost twenty years ago, retailers would set prices for all their products once a year. As retailers adopted our AI-based pricing models, Revionics introduced the ability to automatically model prices every week, and optimize on demand. We have now set the foundation to enable even more advanced modeling and optimization techniques, and are able to model at a deeper and more granular level than ever before, for orders of magnitude greater data volumes, all while improving our processing times. With this new functionality, we will enable retailers to update prices at the speed of their business, providing the ability to test “what ifs” and run pricing scenarios in minutes. Our data scientists can access so much more data than they could before, at speed, and perform data modeling at the transactional level. We’re able to create models now that we’ve been wanting to create for years that are broad and go deeper into the details. This capability is a pillar for Revionics and has helped speed up our product development and unleashed our data scientists. For our customers, this means that we can continually stay ahead of the complexity of modern retail environments, and this new scalability means we can respond to them immediately and help them adapt quickly.   Earlier tools didn’t allow many of our teams to self-serve their analytics needs, but with access to BigQuery, they’re able to do analytics work on their own. From a support angle, that’s been beneficial. Custom reporting requests that used to take hours now are available immediately and securely for the end user through BigQuery.   Combined, this has opened up exciting new roadmap possibilities. We’re looking at improving how we give our customers access to their data, exploring new and intriguing visualizations, all while leveraging the built-in global access and security, giving us a lot of capabilities and potential products. When it comes to the cloud, there’s a lot to learn. If you’re just getting started, we recommend that you master what you can and don’t try to learn everything all at once. To help, allow your teams to explore, then identify the most critical functional and non-functional requirements and stay focused on those to prevent scope creep and help drive success in your own cloud adoption journey.Learn more about Revionics. Thanks for additional contributions from Clinton Pilgrim and Kausik Kannan.
Quelle: Google Cloud Platform

Getting hands-on: Start building on Google Cloud for free

We’re pleased to share updates to our Google Cloud free programs so developers can get started even faster in solving real-world business challenges. With more and more developers building with the Google Cloud Platform every day, we’ve invested in a broad range of new hands-on resources, videos, tutorials, and comprehensive documentation to help you get started and grow and maintain momentum with Google Cloud. These include:The Free Trial program. Last year more than 750,000 new developers trained on GCP, and many started with our hands-on free trial experience that provides a great on-ramp to use our 200+ cloud products and services. We’ve found that one of the highest predictors of developer success is developers completing a proof of concept during the first three months of our free trial program. As a result, we’ve updated the program so that, starting August 17th, users now have 90 days to use their $300 USD in Google Cloud credits. In addition to the free trial, users can also leverage our free products, resources, and training. This lets us focus on helping developers and organizations maximize the impact of their free trial in those critical first months. There will be no changes for customers already in a free trial. For more information, check out all of the details on our trial program here.The Always Free Tier program. More than 20 of our most popular products and services are available for freeabove and beyond your $300 USD free trial credits, even after your free trial is complete. You can try everything from App Engine and Compute Engine, to industry-leading AI, data analytics, storage and security tools.Better training—and more resources to help you get started. All Google Cloud free trial and free tier users can access: 30 days of unlimited Qwiklabs training as well as more than 60 on-demand webinars, hundreds of how-to videos, playlists, and training courses and in-depth learning paths on platforms including Pluralsight and Coursera.User and expert resources in Google Cloud communities, on our Slack channel and Reddit. You can also check out GitHub or Stack Exchange boards to ask technical questions, discuss projects, and find targeted support.Regularly updated documentation, from detailed cloud basics to enterprise guides, which gives you foundational concepts, quickstarts, top use cases, and advanced hands-on guides. Support resources available right in your Google Cloud console.Like all Google Cloud services, we continue to improve our free trial programs based on your feedback. Please leave a note in any of the above channels letting us know what you think.
Quelle: Google Cloud Platform

How Procter & Gamble uses Google Cloud to improve consumer experience

Editor’s note: In its more than 180 years of history, Procter & Gamble has been at the forefront of consumer insights and innovation. From soap operas to smart toothbrushes, P&G has specialized in consumer experience since the start. Here’s how they’re applying cutting-edge technology to the next century of consumer goods. At P&G, our aspiration is to serve the world’s consumers with goods from baby and feminine care, grooming, home hygiene, and healthcare to beauty products and more. Consumer understanding and insights have always been essential for us, and in today’s highly connected world, we’re using technology to meet consumer needs through more personalized experiences, whether that’s through diapers or toothbrushes. Advanced data analytics lets us offer consumers the best selection of products at their local stores and reach them on their preferred channels. We want to understand and serve our consumers better than anybody else, and data helps us do that. Beyond using data for descriptive and diagnostic purposes to understand what happens in our business and why, we’re using analytics to make predictions, such as the success of a promotion or the best product assortment by store clusters. During the pandemic, this capability helped manage spikes in demand across our supply chain.How technology fuels innovation at P&GUsing cloud infrastructure offers a few essential capabilities: the capacity to store big amounts of data on demand, the tools and the machine learning libraries to work on that data, the means to ensure operational reliability, and the protection of data privacy. At P&G, we believe in a multi-cloud environment as the way forward in modern technology. We maintain data across functional areas to serve many business use cases. About a year ago, we partnered with Google Cloud to store and analyze our brand and marketing information. Since then, we’ve been migrating our consumer information into a data lake. The data lake gives us a consistent, unified view of the consumer, and lets us create omni-channel consumer journeys. That means we can serve the right audiences at the right time with the right content on the right channels.We use many of Google Cloud’s analytics products, and we’ve been especially impressed by BigQuery, Google Cloud’s enterprise data warehouse. We’re using it for data science and to serve data to larger audiences, thanks to its compatibility with other visualization tools. Making our tools work better together is a great benefit and helps our teams meet business and analytics goals.How consumers benefit from back-end technologyWith this rich data lake environment, our data scientists and business analysts can create algorithms to solve some of our toughest questions. Increasingly, we turn those predictions into a prescription—so we automate the result of the algorithm and inject them into our transactional and planning systems, so mainstream decisions are automated, freeing up our team to invest time in more complex and unique challenges. Within the CPG space, you’ll see more examples of connected products, where consumers opt in to share information and get personalized services and advice in return. One of our products, Lumi by Pampers, helps parents track day-to-day developments and monitor their babies’ diapers, sleep habits and more. The Oral-B iO toothbrush has a unique and effective brush head design, but also helps users improve their cleaning routines. All this is made possible with the data we safely store on Google Cloud. Multi-cloud will power the futureWith a powerful cloud infrastructure, we can meet goals and reach new ones—and continue to evolve products and put consumer experience first. We’ve appreciated how Google Cloud’s teams work as an extension of ours, helping us develop our own open ecosystem. We’re looking forward to what working with Google Cloud will help us achieve for our customers.Learn more about P&G, and explore more about data analytics in this week’s Google Cloud Next ‘20: OnAir keynote.
Quelle: Google Cloud Platform

What’s new and what’s next with data analytics — Next OnAir

Data is the foundation for modern enterprises and it goes beyond just collecting and storing it. In a world where things are constantly changing, you need to be able to access and analyze data for insights to drive impactful business decisions. Big data is in our DNA: Google has been at the forefront of data-powered innovation for years, with the mission of organizing the world’s information and making it universally accessible and useful. Google was built to process and analyze large data sets, and the same technology that makes this happen is available for all businesses through our smart analytics platform.If there’s one lesson from this unpredictable year, it’s that we always need to be prepared for anything. Our customers trust us with their data, which is why our vision centers on offering an analytics platform with proven dependability and security for mission-critical workloads. At Google Cloud Next ‘20: OnAir this week, you’ll see our vision come to life across three core principles of our analytics platform: open, intelligent and flexible. Let’s take a deep dive into what’s new in data analytics, and what you can explore this week to help move your data strategy forward. Open platform that provides choice and portabilityWe know that you want options when you’re choosing analytics solutions. That includes choice of deployment across hybrid and multi-cloud , choice to leverage open source services (OSS) and the choice to leverage open APIs to ease migration and data access. BigQuery and its ecosystem of data products bring you an open platform, so you get maximum flexibility for managing analytical applications and data-driven solutions. That’s why we recently announced BigQuery Omni, a flexible, multi-cloud analytics solution that lets you analyze data in Google Cloud, AWS and Azure (coming soon) without moving data. Check out Analytics in a Multi-Cloud World with BigQuery Omni session to learn more.We also welcomed Looker into Google Cloud earlier this year. Looker is a data and analytics platform that runs on Google Cloud or the cloud of your choice, and can take advantage of the benefits of BigQuery. Looker allows data teams to unleash powerful data experiences and go beyond traditional reports and dashboards to deliver modern BI, integrated insights, data-driven workflows, and custom applications. Looker just announced Looker blocks for the complete Google Marketing Platform, and significant upgrades for application builders are now available in the Looker Marketplace. To learn more about these announcements, check out Looker’s Roadmap: 2020 & Beyond session or their post: New Looker feature enhancements for the data-driven workforce. We are also showing our first integration with Looker and BigQuery BI Engine in this session: Always Fast and Fresh Dashboards: Inside BigQuery BI Engine.Finally, we’re also providing a mix of managed services like Dataproc – What’s new in open source data processing and the ability to use open source languages like Spark and Presto with the BigQuery Storage API, allowing customers to continue using familiar tools across their data in Google Cloud. These are great examples of how we’re continuing to provide an open platform for our customers.Built-in intelligent services to do more with what you haveGoogle Cloud’s smart analytics platform provides intelligent services embedded into our tools and processes, so users don’t have to learn new things to take advantage of it. Businesses want to augment current solutions with AI and ML and optimize business outcomes with real-time intelligence. Google Cloud offers industry-leading AI services to improve those decisions and customer experiences.With BigQuery ML, for example, users can build custom ML models using standard SQL without moving data from the warehouse. We’ve recently added new models, including boosted trees using XGBoost, DNNs using TensorFlow, K-means clustering, matrix factorization, and more. We’re taking it further by leveraging these models to build real-time AI solutions like anomaly detection, pattern recognition, and predictive forecasting that can be used across multiple industries. These detailed and prescriptive design patterns help you build solutions that can find anomalies in log transactions, like with this telco, detect objects in video clips, or predict customer lifetime value (LTV). Demandbase is currently using BigQuery ML and will hone in on their use cases in What’s New in BigQuery ML, Featuring Demandbase.Flexibility so you can scale at the speed of businessFinally, we build data analytics tools that provide flexibility. We know that our customers span different industries and multiple use cases. For example, flexibility in BigQuery pricing models lets you uniquely mix and match pricing tiers across environments in order to meet demands and have direct control over cost and performance. Over the past year, we’ve announced Flex Slots, short-term analytics capacity bursts for as little as 60 seconds at a time; a 95% discounted BigQuery promotional offer for new customers with Trial Slots; and now, we’re announcing that you can purchase a minimum of 100 slots at a time in increments of 100 slots. This change applies to all commitment types—flex, monthly, and annual. We’ve heard from SMB and digital-native organizations that they wanted this new pricing tier, and it truly democratizes flat-rate pricing across these different segments. Learn more here.In addition, we’re catering to different customer needs across our portfolio. Data analysts prefer the familiar SQL interface, which they can access in the BigQuery UI, while data scientists prefer notebook environments, made possible using the Storage API and Dataproc Hub, and business analysts prefer an easy-to-consume BI interface like Looker and Tableau. In addition, you can use a familiar spreadsheet interface with Connected Sheets and natural language interface with the new Data QnA service to draw insights out of your data. For any of these interfaces, all the data lives in BigQuery, so you don’t need to worry about data silos or multiple copies of data. This is true democratization of analytics for everyone.Proven dependability for mission-critical workloadsFor all these things to work, you need a dependable platform that offers security, reliability, governance, and compliance for mission-critical applications. That is why we just announced 4 9s of availability for BigQuery—up to 99.99% availability with guaranteed SLAs, providing peace of mind that the platform will be available to handle all of your needs. We’re seeing adoption of our platform across retail, media and entertainment, telecommunications, financial services, transportation and more. You can check out sessions from customers including BlackRock, Iron Mountain, Twitter, MLB, Veolia, Verizon Media, Telus, Carto, Refinitiv, Geotab, Bluecore, and Demandbase. Tune in and watch all sessions across our 40+ breakout sessions to learn more from Googlers, customers, and partners, and find them all on-demand afterward. To get started with your data-driven transformation, download this HBR report to find out how data-to-value leaders succeed in driving results from their enterprise data strategy.
Quelle: Google Cloud Platform

Better BigQuery pricing flexibility with 100 slots

BigQuery is used by organizations of all sizes, and to meet the diverse needs of our users, BigQuery offers highly flexible pricing options. For enterprise customers, BigQuery’s flat-rate billing model is predictable and gives businesses direct control over cost and performance. We’re now making the flat-rate billing model even more accessible by lowering the minimum size to 100 slots, so you can get started faster and quicker.Starting now, you no longer need to purchase a minimum of 500 slots to take advantage of the slots billing model:Purchase as few as 100 slots at a timePurchase in increments of 100 slotsThis change applies to all commitment types—flex, monthly, and annual.Your price per slot remains the same, no matter how many slots you buy. These new billing changes can significantly lower the entry point for slots-based pricing. Here is an example of how this works:100 slots on an annual contract cost $1,700 per month.100 slots on a monthly commitment type cost $2,000 per month.100 flex slots cost just $4 per hour, or less than $0.07 per minute.We’ve heard that a lower-priced offering is especially relevant for organizations in emerging markets. “We have been using BigQuery as our single data warehouse, and using the 1,000 slots flat-rate option effectively,” said Yuan Gao, chief data officer in The Learnings Co., Ltd. “The 100-slot flat-rate offer will give us finer granularity to control the cost. When we have relatively small workloads, we can run them in one project with a 100-slot reservation, without affecting other workloads, with small and predictable costs.” About BigQuery slotsA slot is a virtual CPU used by BigQuery for data processing. When you purchase slots, BigQuery provisions dedicated capacity for your queries to use, and you pay for the seconds your slots were provisioned. For example, when you buy 1,000 slots with an annual commitment, you pay for 1,000 slots for 365 days, and all query costs are included in the price. The more slots you have, the better your throughput and performance. BigQuery’s architecture is serverless and VM-less. Because we don’t keep state in our compute nodes, BigQuery users get unique benefits, like no need to warm up to achieve maximum performance, and no performance cliffs associated with local disk limits.If you are considering switching from the bytes processed billing model to the slots billing model, consider that the bytes processed model gives users up to 2,000 slots or more. therefore, your query performance may not be as good with 100 slots. We recommend you monitor your slot usage and performance for the optimal number of slots to purchase.You can take advantage of the new 100 slots option by enabling BigQuery Reservations, an enterprise-grade workload and capacity management platform. With BigQuery Reservations, you can provision capacity in seconds and departmentalize it into isolated workloads and teams. To get started, head over to Introduction to BigQuery Reservations.To learn more, check out our Google Cloud Next ‘20: OnAir digital conference. This week is focused on data and analytics. My session, DA300: Awesome New Features to Help You Manage BigQuery, discusses in depth how to best utilize BigQuery slots and BigQuery Reservations.
Quelle: Google Cloud Platform

MLB's fan data team hits it out of the park with data warehouse modernization

Editor’s note: In this blog post, VP of Data Engineering at MLB Robert Goretsky provides a deeper dive on MLB’s data warehouse modernization journey. Check out the full Next OnAir session: MLB’s Data Warehouse Modernization.At Major League Baseball (MLB), the data generated by our fans’ digital and in-stadium transactions and interactions allows us to quickly iterate on product features, personalize content and offers, and ensure that fans are connected with our sport. The fan data engineering team at MLB is responsible for managing 350+ data pipelines to ingest data from third-party and internal sources and centralize it in an enterprise data warehouse (EDW). Our EDW is central to driving data-related initiatives across the internal product, marketing, finance, ticketing, shop, analytics, and data science departments, and from all 30 MLB Clubs. Examples of these initiatives include:Personalizing the news articles shown to fans on MLB.com based on their favorite teams.Communicating pertinent information to fans prior to games they’ll be attending.  Generating revenue projections and churn rate analyses for our MLB.tv subscribers. Building ML models to predict future fan purchase behavior.Sharing fan transaction and engagement data from central MLB to the 30 MLB Clubs to allow the Clubs to make informed local decisions.  After a technical evaluation in 2018, we decided to migrate our EDW from Teradata to Google Cloud’s BigQuery. We successfully completed a proof of concept in early 2019, and ran a project to fully migrate from Teradata to BigQuery from May 2019 through November 2019. (Yes, we completed the migration in seven months!) With the migration complete, MLB has realized numerous benefits in migrating to a modern, cloud-first data warehouse platform. Here’s how we did it.How MLB migrated to BigQueryWe ran several workstreams in parallel to migrate from Teradata to BigQuery:Replication: For each of the ~1,000 regularly updated tables in Teradata, we deployed a data replication job using Apache Airflow to copy data from Teradata to BigQuery. Each job was configured to trigger replication only after the data was populated from the corresponding upstream source into Teradata. This meant that data was always as fresh in BigQuery as it was in Teradata. Having fresh data in BigQuery allowed all downstream consumers of the data (including members of the business intelligence, analytics, data science, and marketing teams) to start building all new processes/analyses/reports on BigQuery early on in the project, before most ETL conversion was completed. ETL conversion: We had over 350 ETL jobs running in Airflow and Informatica, each populating data to Teradata or extracting data from Teradata, each of which needed to be converted to interact with BigQuery. To determine the order in which to convert and migrate these jobs, we built a dependency map to determine which tables and ETL jobs were upstream and downstream from others. Jobs that were less entangled, with fewer downstream dependencies, could be migrated first. A SQL Transpiler tool from CompilerWorks was helpful, as it dealt with the rote translation of SQL from one dialect to another. Data engineers needed to individually examine output from this tool, validate results, and, if necessary, adjust query logic accordingly. To assist with validation, we built a table comparison tool that ran on BigQuery and compared output data from ETL jobs. Report conversion: We use two tools to produce reporting for end users—Business Objects and Looker. For each of the Business Objects reports and Looker Dashboards, our business intelligence team converted SQL logic and reviewed report output to ensure accuracy. This workstream was able to run independently of the ETL translation workstream, since the BI team could rely on the data replicated directly from Teradata into BigQuery.End-user training: Users within the marketing, data science, and analytics teams were onboarded to BigQuery early in the project, leveraging the data being replicated from Teradata. This allowed ample time for teams to learn BigQuery syntax and connect their tools to BigQuery.  Security configuration: Leveraging MLB’s existing SSO setup via Okta and G Suite, users were provisioned with access to BigQuery using the same credentials they used for access to their desktop/email. There was no need to set up separate sets of credentials, and users who left the organization were immediately disconnected from data access. The benefits that migration brought MLB Pricing: With BigQuery’s on-demand pricing model, we were able to run side-by-side performance tests with minimal cost and no commitment. These tests involved taking copies of some of our largest and most diverse datasets and running real-world SQL queries to compare execution time. As MLB underwent the migration effort, BigQuery cost increased linearly with the number of workloads migrated. By switching from on-demand to flat-rate pricing using BigQuery Reservations, we are able to fix our costs and avoid surprise overages (there’s always that one user who accidentally runs a ‘SELECT * FROM’ the largest table), and share unused capacity with other departments in our organization, including our data science and analytics teams.Data democratization: Providing direct user access to Teradata was often cumbersome, if not impossible, due to network connectivity restrictions and client software setup issues. By contrast, BigQuery made it trivial to securely share datasets with any G Suite user or group with the click of a button. Users can access BigQuery’s web console to immediately review and run SQL queries on data that is shared with them. They can also use Connected Sheets to analyze large data sets with pivot tables in a familiar interface. In addition, they can import data from files and other databases, and join those private datasets with data shared centrally by the data engineering team.MLB’s central office handles the ingestion and processing of data from a variety of data sources and shares club-specific data with each Club in the initiative known internally as “Wheelhouse.” The previous Wheelhouse data-share infrastructure involved daily data dumps from Teradata to S3, one per Club per dataset, which introduced latency and synchronization issues. The new Wheelhouse infrastructure leverages BigQuery’s authorized views to provide Clubs with real-time access to the specific rows of data relevant to them. For example, MLB receives StubHub sales data for all 30 Clubs, and has set up an authorized view per Club so that each Club can view only the sales for their own team. Due to BigQuery’s serverless infrastructure, there is no concern about one user inadvertently affecting performance for all other users. This simplification in architecture can be seen in the diagrams below:Seeing the IT results from migrating to BigQueryImproved performance: Queries generally complete 50% faster on BigQuery compared with Teradata. In many cases, queries that would simply time out or fail on Teradata (and impact the entire system in the process), or that were not feasible to even consider loading into Teradata, run without issue on BigQuery. This was especially true for our largest data set, which is 150TB+ in size per year and consists of hit-level clickstream data from our websites and apps. This data set previously needed to be stored outside our data warehouse and processed with separate tools from the Hadoop ecosystem, which led to friction for analysts who often wanted to join this data with other transactional data sets.Richer insights via integrations: The BigQuery Data Transfer Service makes it easy to set up integrations with several services MLB currently uses, including Google Ads, Google Campaign Manager, and Firebase. Previously, setting up these kinds of integrations involved hand-coded, time-consuming ETL processes. Looker, our BI tool, seamlessly integrates with BigQuery and provides a clean and highly performing interface for business users to access and drill into data. Third-party vendor support for BigQuery is strong as well. As an example, our marketing analytics team is able to use the data ingested from Google Ads to inform advertising spend and placement decisions. Reduced operational overhead: With Teradata, MLB needed a full-time DBA team on staff to handle 24×7 operational support for database issues, including bad queries, backup issues, space allocation, and user permissioning. With BigQuery, MLB has found no need for this role. Google Cloud’s support covers any major service issues, and former administrative tasks such as restoring a table backup can now be done easily by end users, letting our IT teams focus on more strategic work.Increased developer happiness: Our data engineering, data science, and analytics staff were often frustrated by the lack of documentation and subtle gotchas present within Teradata. Given that Teradata was mainly used within larger enterprise deployments, online documentation on sites such as Stack Overflow was limited. In contrast, BigQuery is well-documented, and given the lack of barriers to entry (any Google Cloud user can try it for free), there already seem to be more resources available online to troubleshoot issues, get answers to questions, and learn about product features.Accelerated time to value: No downtime or upgrade planning is needed to immediately take advantage of new useful features that are added on a regular basis by the BigQuery engineering team.Seeing the business impact from BigQueryWith our migration to BigQuery complete, we’re now able to take a more comprehensive and frictionless approach to leveraging our fan data to serve our fans and the league. A few projects that have already been facilitated by our move to BigQuery are:OneView: This is a new initiative launched by our data platform product team to compile over 30 pertinent data sources into a single table, with one row per fan, to facilitate downstream personalization and segmentation initiatives. Previously, we would have spent a long time developing and troubleshooting an incremental load process to populate this table. Instead, with the power of BigQuery, we’re able to run full rebuilds of this table on a regular basis that complete quickly and do not adversely affect performance of other data workloads. We’re also able to leverage BigQuery’s Array and Struct data types to nest repeated data elements within single columns in this table to allow users to drill into more specific data without needing any lookups or joins. This OneView table is already being used to power news article personalization.Real-time form submission reporting: By using the Google-provided Dataflow template to stream data from Pub/Sub in real time to BigQuery, we are able to create Looker dashboards with real-time reporting on form submissions for initiatives such as our “Opening Day Pick ‘Em” contest. This allows our editorial team to create up-to-the-minute analyses of results.With our modern data warehouse up and running, we’re able to serve data stakeholders better than ever before. We’re excited about the data-driven capabilities we’ve unlocked to continue creating a better online and in-person experience for our fans.
Quelle: Google Cloud Platform

From recovery to reinvention: A Q&A with Equifax CTO Bryson Koehler

With hundreds of millions of people and organizations relying on financial services organizations each day to keep their information safe and secure, trust is at the heart of everything these organizations do.Consumer credit reporting agency Equifax knows this all too well. In 2017, it experienced a historic data breach. As CTO, Bryson Koehler, recognized that rebuilding trust would mean fundamentally rethinking how Equifax managed technology and data security—and their business—as a whole.  Today, Equifax is in a very different place. The company is transforming the majority of their IT operations on the strong foundation of Google Cloud, and in doing so, transforming many aspects of their entire business. The company recently issued a paper outlining their transformation efforts to leverage the cloud while also building world-class data security measures. We sat down with Bryson to get a deeper understanding of what they learned from their successful three year digital transformation journey.Glen Tillman: Where does Equifax see its mission today?Bryson Koehler: It’s about finding opportunities where we can leverage technology and data to help people make smarter, more informed decisions, and improve the process and experience for them. For example, we have mobile devices in our pockets and on our wrists that help us with health coaching, but do we have financial coaching? We get lots of feedback about how to make good health and entertainment decisions, but many of us don’t get the information to make good financial decisions. And I think Equifax is uniquely positioned to help people live their financial best.Glen: After the security incident in 2017, Equifax went through a level of transformation that is unprecedented, especially in your business. Can you tell us what drove that decision?Bryson: When you have a data breach, that’s a kind of existential crisis moment for any company. You try to understand the impact, how to protect people impacted by it, and how you can make sure it doesn’t happen again. We realized we needed to think radically differently about how we could create a sustainable security posture and who we are from a technological culture perspective.The cloud makes world-class security possible by providing consistency in the way we build and deploy technology. On-premises or private cloud installations often create a mixed-generation infrastructure that breaks automation, requiring manual fixes that lead to human error. By removing opportunities for mistakes, the cloud keeps us focused on good hygiene and discipline. It also offers the flexibility that lets you make changes, building reliability into the infrastructure and code without creating vulnerabilities or reducing productivity.Glen: How did you establish what you needed from the cloud and why did that lead you to Google Cloud?Bryson: When I looked at what we needed at Equifax, I realized that Google’s tighter focus around data, security, AI, and machine learning were a better fit. Security is something that has to be built in and not bolted on, and I think the engineering culture at Google—broadly speaking, not just the cloud—has good reason to take pride in their achievements in this regard. This ends up creating a culture that I feel more comfortable placing a bet for five years in the future than I would with any others.I felt that Google Cloud offers a differentiated approach to data that was completely unique and did not seem to be available at other companies. And Google has really been there for us with a commitment of lining us up with the right people to help us achieve success not just in usage but also the outcome.Glen: Many companies are still cautious about going to the cloud—especially in financial services—so what you did was a huge leap.Bryson: It is a big leap but it’s one that’s worth taking. Our job is about ensuring that our data and systems are secure and protected, not just from hackers, but also from misuse. Making sure that data is being used in the right ways, places, and times with the right use cases and intent is just as important—and in time, more important.Things are going to evolve, and you’re going to need to be able to adapt as those rules change. People stick to what they know because they don’t have a deep enough understanding how this all works. To achieve the change we’ve seen requires a technological shift, but more importantly, it demands a cultural shift that functionally changes who you are as a company because when you get back to business, you can’t go back to business as usual.Glen: One thing that moving to the cloud helps with is removing data silos. Has that had a major impact on the culture?Bryson: That will be the next part of our cultural shift. If you went back years ago, Equifax would have needed to get data from multiple sources within our own company to get a full picture. By bringing these data sources into a cloud based environment, we can organize our data into a single, seamless structure—while still keeping all critical governing and separation measures in place—creating differentiated products to help our customers make smarter decisions and to help consumers live their financial best.  Glen: How do you see the cloud evolving the financial services industry moving forward?Bryson: If you think about this strategically, financial services companies are working hard to make better, more confident decisions and matchmake the right customers to the right products and services. When you think about the fraud intelligence exchange we’re doing at Equifax, for example, we all have that incentive to help reduce fraud, so sharing and partnering in the industry helps all boats rise. Fraud adds costs, slows down processes, and encumbers good customer experiences in many cases, so the more we can do as an industry to realize that this is not a competitive asset and work together to reduce it, the better we all will be. There are great dialogues around that, and many companies are recognizing that the cloud is a great place to do this. From a data and analytics perspective, having the infinite horizontal scalability of cloud infrastructure provides much better capabilities to achieve that without increasing the risk footprint through replication and consistency, increasing security. And because so much of our industry moves so fast that transactions are measured in milliseconds, being able to share data in real time with the cloud improves the ability to help people in the future right when they need it.Learn more about Google Cloud solutions for financial services.
Quelle: Google Cloud Platform

12 smart analytics sessions to check out at Next OnAir

Business leaders are looking to generate unmatched value from their data to drive competitive advantage and accelerate digital transformation. Digital innovators are tapping into new customer opportunities, using data to deliver new experiences and disrupt industries. At Google Cloud, we’re building a smart analytics platform that is open, intelligent, and flexible and offers a simple path to modernization. As part of Google Cloud Next ‘20: OnAir, we’ll spend this week exploring how you can incorporate analytics into your business. Take a look at these can’t-miss sessions (available starting at 9am PT tomorrow, August 11, and available on-demand afterward), or explore the full schedule. This year, we’re also highlighting our recent acquisition of Looker, a leading business intelligence and analytics platform. Looker serves as an example of our ongoing investment to support our customers’ success with data modernization and maximizing data’s potential–no matter where it resides. What’s new and what’s next in smart analytics: Join Debanjan Saha, GM and VP of Engineering for Data Analytics at Google Cloud to hear about the latest innovations in Google Cloud’s smart analytics platform. Many data platform architectures in use today were designed 20 years ago, and they’re struggling to solve modern business problems. You’ll get details on our data analytics solutions, and learn about some of our latest innovations, plus hear from current customers like Procter & Gamble on how they’re using analytics for product development and consumer experience transformation.Smart analytics: deep dive on roadmap: Take a comprehensive tour through the technical value of Google Cloud’s smart analytics platform. You’ll hear from Director of Product Management Sudhir Hasbe on what’s new and what’s next in Google Cloud’s smart analytics portfolio across products like BigQuery, Dataflow, Dataproc, Data Fusion, Pub/Sub, Data Catalog, Dataprep, and Looker. What’s new in BigQuery, Google Cloud’s modern data warehouse: There’s a lot of pressure on businesses to access fresh information to make real-time decisions—and start to go beyond real time to make predictions based on data. In this session, you’ll hear about new features in BigQuery, our cloud data warehouse designed to help you adapt and succeed in quickly changing environments. Analytics in a multi-cloud world with BigQuery Omni: Using multi-cloud is a necessity for many businesses, and can help speed up digital transformation journeys. Join this session to learn about our recently announced BigQuery Omni, a flexible, multi-cloud analytics solution, powered by Anthos, that helps organizations analyze data in AWS and Azure (coming soon).Data QnA: how Veolia democratizes access to BigQuery: In this session, you’ll get details on the newly announced Data QnA, a service that empowers business users to simply ask questions of their data in BigQuery, using natural language, and get an answer immediately. See how Veolia, a global leader in water, waste and energy resource management solutions, is using Data QnA to democratize access to analytics across their teams.Looker’s roadmap: 2020 and beyond: Data and analytics platform Looker, a recent addition to Google Cloud, brings data experiences that fit the way people work. In this session, you’ll get a look at Looker’s product roadmap, including upcoming features and new projects.Building data lakes on Google Cloud: Data lakes let you aggregate data and analyze it efficiently using cloud-native or open source tools, no matter where data is managed. Join this session to hear from one company that built a data lake using Google Cloud technologies and learn more about how you can simplify data lake deployment and management within your organization.Easy access to stream analytics with Google Cloud: Real-time is growing at an exponential rate, and streaming analytics for app and user events keep growing in importance. Customers expect businesses to be 0aware and prepared in the moment. Join this session to learn about cloud developments that can help you create and manage real-time, data-driven experiences. Best practices from experts to maximize BigQuery performance (featuring Twitter): BigQuery lets you deploy complex workloads and run fast analytics on your data. In this session, learn tips and tricks from Google Cloud and Twitter engineers on how you can maximize the query performance of your data warehouse and speed up analytics within your environment. Looker and BigQuery: enterprise digital transformation at Sunrun: See how solar electricity provider Sunrun used Looker and BigQuery to modernize its legacy systems. Sunrun’s cloud migration helped teams reduce the complexity of ETL processes, improve database performance, and adapt quickly to changes in data infrastructure.   Awesome new features to help you manage BigQuery: Our serverless analytics platform BigQuery brings scalability and performance to all kinds of users. In this session, you’ll learn about details related to pricing, administration, and monitoring in BigQuery, all designed to provide flexibility that works best for your business and users.BigQuery ML: what’s new: BigQuery lets you build and operationalize machine learning models in BigQuery using standard SQL. Join this session to hear about new features and model types in BigQuery ML, and hear how Demandbase has used BigQuery ML successfully for predictive analytics.In addition to these, you’ll find 30 additional data analytics sessions from Googlers, customers, and partners exploring what’s new in data-driven initiatives. And check out study jams, talks by developer advocates, and other ways to learn more.
Quelle: Google Cloud Platform

Google Cloud Partner opportunity to more than triple by 2025, according to IDC study

Thanks to my role at Google Cloud, I’m privileged with a front-row seat to all of the amazing work our partners do on behalf of customers. Our partners play a critical role in delivering Google Cloud technology and solutions to organizations all over the world, so I am delighted to share the findings of a new study from IDC that shows Google Cloud is thriving, growing, and driving significant economic benefit for partners. To summarize IDC’s study in a few words: Demand for cloud technology and services is growing rapidly, as businesses embark on digital transformations, and Google Cloud partners are particularly well-positioned to help customers plan and execute their digital transformation strategies.According to IDC, “growth mode” continues for the IT market overall, and importantly, demand for cloud infrastructure, and cloud capabilities in areas like data analytics, artificial intelligence, IoT, and security is growing at an even faster pace. I’ve seen firsthand how the global pandemic has further increased businesses’ needs for these capabilities, as they seek to quickly adapt or even to take this opportunity to accelerate their digital transformations.As IDC reports, “This is good news for Google Cloud partners, who by their nature are engaged across many of these technologies.” In fact, the analyst firm’s study forecasts that partners’ revenue from Google Cloud-related opportunities will more than triple by 2025, representing a tremendous opportunity for the ecosystem overall in building out their Google Cloud practices.IDC’s research also found that Google Cloud partners’ businesses are growing at a fast clip. On average, partners are growing their Google Cloud businesses at a rate of 35% year-over-year, with a significant group of partners—one in five—growing even faster, at 75%-plus year-over-year.All of this translates to a thriving ecosystem around Google Cloud technologies, and it’s encouraging to see how much value is passed through to our partners. Globally, for every $1 of Google Cloud technology sold in 2020, partners will generate $5.32 through their own offerings, services, and IP. IDC expects this number to grow, reaching $7.54 by 2025. IDC’s study also provides insights into several other benefits that partners are seeing from working with Google Cloud:Google Cloud partners are leading the way on digital transformation. IDC places a full half (50%) of Google Cloud partners in the “late stage of digital maturity,” and more than a third of Google Cloud partners have “fully integrated digital into their strategies and businesses,” meaning customers can trust Google Cloud partners to bring very strong expertise with modern cloud technologies and solutions.At Google Cloud, we are focused on delivering end-to-end, best-in-class solutions. Many of our partners are extending these solutions, and IDC’s study found that these partners, who are creating their own unique IP around Google Cloud, are seeing very strong margins associated with these products. Partners are benefitting from our goal of 100% partner attach on all customer sales, and partners are seeing strong margins across multiple activities, including resale, IaaS, PaaS, and SaaS add-ons, IT services, business services, and support for hardware and networking.We have a tremendous opportunity together to help customers across industries and around the world transform their organizations with the cloud. I’m proud of the amazing work that our partners are doing, and we’re committed to continuing to work closely with partners to expand the Google Cloud economy together. To read IDC’s findings in full, you can download the report here.
Quelle: Google Cloud Platform

The future of cloud as supply chain for new telco services

If there’s anything we’ve learned so far in 2020, it’s that no one can predict day to day what is going to happen next. Therefore, we believe technology should be there to help developers and operators build for agility and manage for change. This is especially true for our customers in the telecommunications industry. Communications service providers (CSPs) have seen their network resiliency and service delivery put to the test as students, enterprise users, and consumers have all shifted rapidly to digital-only engagements to learn, conduct business, and stay connected. The trick for CSPs, however, is enabling agility and accelerating innovation across a globally distributed network without having to manage underlying complexity.This topic and more are explored by Jennifer Lin, VP of Product Management and Chen Goldberg, Senior Director of Engineering, in “The Future of Tech” podcast hosted by Avishai Sharlin, Division President of Amdocs Technology, a Google Cloud partner and a leading provider of software and services to communications and media companies. They discussed the rise of 5G/edge data-driven services, the evolution of open-source technologies, and the role Anthos can play in driving application modernization and speed of innovation amongst Telcos.Cloud’s role in the supply chain for new 5G/edge servicesOne of the hot topics was 5G/edge computing and the role they can play in helping CSPs deliver more personalized, data-driven services to customers. However, telco networks have grown rather complex over the years. In order to unlock the potential of these technologies, CSPs may need to build more intelligent automation into their networks and remove some of that complexity.“This move from IT monolithic systems from single vendors to 5G/edge data-driven services is about delivering a customer experience,” said Lin. “The pace at which we can move in [the] cloud as the supply chain for new services is phenomenal.” Evolution of open-source and how Anthos speeds Telco innovationThere’s also been an enormous amount of change and development in the application layer over the past few years. Open-source technologies like Docker and Kubernetes helped developers achieve faster speed in innovation by making systems more composable and portable. However, according to Goldberg, there were still some things missing, and that was what drove the creation of Anthos.“We went from building a product like Google Kubernetes Engine, which was just the container orchestration manager experience to something like Anthos [because] we have seen that just the portability of workload[s] is not enough,“ said Goldberg. “Our customers actually want us to take control and give them a managed experience wherever they build. That really gives them that engineering velocity.”Furthermore, a few years ago, the industry was still missing a platform that would enable developers and IT to not only build but also manage applications consistently across on-premises data centers, cloud environments, and at the edge. A solution like this would be key because many CSPs still have a large percentage of their data residing on premises and they will likely continue to live on-premises. Therefore, Anthos for Telecom was also developed to help CSPs more easily manage day two operations for applications that run across mixed deployment environments. To learn more, we invite you to tune in for the full conversation on “The Future of Tech” podcast. And for additional information on how Google Cloud is working with strategic partners like Amdocs to deliver solutions to help CSPs modernize core OSS/BSS systems, harness data and analytics, and monetize on 5G/Edge, you can also check out the Google Cloud Next ‘20 OnAir session, “Accelerating telecommunications growth.”
Quelle: Google Cloud Platform