The 10 most popular sessions from Google Cloud Next ‘20: OnAir

Google Cloud Next ‘20: OnAir looked a little different this year. Instead of a three-day conference, we launched a nine-week digital event series that brought together our global cloud community to discuss and collaborate on the most significant cloud technology challenges facing companies today. We may have gone virtual, but some things stayed the same—all of our sessions (over 200 to be exact) are now available on YouTube to watch and learn from. That’s a lot of content to sift through, so here’s a quick breakdown of our top 10 sessions from Next OnAir: 1. Supercharge Productivity With No-Code Apps Using AppSheetThe road to building new applications in the digital age comes with a tough choice for business and technology leaders: buy or build. Packaged software is often too rigid to meet unique requirements, and building custom apps takes up too much time and resources. But is there a third option? In this session, Santiago Uribe Montoya, Google Senior Product Manager, and Richard Glass, Director of Information Technology at KLB Construction, discuss how AppSheet makes it possible to automate processes while leveraging existing Google Workspace data to build mobile and desktop apps—without coding. 2. How PwC Migrated 275,000+ Users to Google WorkspaceEver wondered what it’s like to migrate to Google Workspace (formerly G Suite)? What about migrating over 275,000 users at a 150-year-old professional services firm operating in over 158 countries? PwC Global Change Director Adrienne Schutte, along with Google Technical Account Manager Regina Houston, shares the challenges, key lessons, management strategies, and long-term impact of PwC’s journey with Google Workspace.3. The Future of Meetings in Google Workspace: Vision and RoadmapThe new normal is here—and so is the new work normal. In this session, Smita Hashim, Director of Product Management for Meeting Solutions and Time Management, and Greg Funk, Group Product Manager, share Google Workspace’s vision for the future of meetings as teams navigate an increasingly video-first world. You’ll also get a sneak peek into how Google Workspace is transforming the lifecycle of a meeting and reimagining teamwork so that people can stay connected no matter where they are working from.4. Do it live! Fitbit’s Zero-Downtime Migration to GCPMoving a monolith without downtime is impossible, right? Think again. In this session, Fitbit’s Principal Software Engineer Sean Michael-Lewis explains how Fitbit migrated its production operations from managed hosting to Google Cloud Platform without impacting its real users. You’ll learn what made Fitbit’s migration challenge unique, how they created a user-centric migration plan, the technology and processes they used, and the key takeaways that have provided a foundation for their new multi-region architecture. 5. What’s New in BigQuery, Google Cloud’s Modern Data WarehouseData is at the heart of many business transformations today. Organizations want to make real-time decisions and future predictions that keep them competitive, but traditional data warehousing wasn’t designed to scale fast or process emerging data processing patterns. In this session, Sudhir Hasbe, Google Cloud Director of Product for Data Analytics, and Tino Tereshko, Google Product Manager, talk about how Google BigQuery addresses the needs of data-driven enterprises and share demos of the latest feature innovations.6. Communication in Google Workspace: The Future of Gmail, Chat, Meet, and MoreHow is communication changing as work goes remote and becomes more flexible? See how new improvements to Gmail, Chat, and Meet are making it easier for modern workers to communicate and collaborate anywhere they are working from—on the web or from an Android or iOS device. In this session, Tom Holman, Google Senior Product Manager, and Dave Loxton, Google Product Manager, share the latest updates, what’s up next, and why Google is more excited than ever about the future of these products in Google Workspace.7. Building Data Lakes on Google CloudTraditional approaches to building data lakes often land organizations with data swamps.   In this session, Google Product Manager Nitin Motgi discusses how Google Cloud makes it easy for enterprises to create and maintain data lakes, allowing customers to aggregate their data and analyze it using cloud-native and open source tools. He’ll also share the most common use cases for how companies use data lakes on Google Cloud.8. Data Catalog for Data Discovery and Metadata ManagementWouldn’t it be great to be able to easily search through your enterprise data assets with the same search technology that powers Gmail and Drive? Many enterprises struggle with data discovery and metadata management across disparate systems and silos. Shekhar Bapat, Google Product Manager, discusses how Data Catalog helps accelerate time to insight by providing discoverability, context, and governance for all of your data assets. He is joined by Shruti Thaker, Head of Alt Vendor Data & Alpha Capture at Blackrock, who shares how Data Catalog helped BlackRock create an effective metadata solution for its data assets. 9. Analytics in a Multi-Cloud World with BigQuery OmniWhile data is a critical component of decision making across organizations, for many, this data is scattered across multiple public clouds. So, how do you help analysts and data scientists handle data from all the tools, systems, and silos? Meet BigQuery Omni, a flexible, fully-managed, multi-cloud analytics solution that lets you analyze data across public clouds without ever leaving the familiar BigQuery user interface. In this session, Google Product Manager, Emily Rapp, shows you how to break down data silos right in your environment and run analytics in a multi-cloud world.10. Master Security and Compliance in the Public CloudUnlocking the promise of the public cloud often brings security and compliance challenges—especially if you’re a leading market-infrastructure provider under the highest level of supervisory scrutiny in Europe. In this session, Christian Tüffers, Senior Cloud Architect at Deutsche Boerse Group, and Grace Mollison, Google Head Cloud Solution Architect, discuss ways customers can work with Google Cloud to create security blueprints to help them deploy workloads that meet regulatory and compliance requirements. Do you want to watch more sessions? You can browse the full session roster from Google Cloud Next ‘20: OnAir here.Related ArticleGoogle Cloud Next ‘20: OnAir—Accelerating digital transformation in the cloudGoogle Cloud CEO Thomas Kurian shares his vision for the future of cloud as we kick off Google Cloud Next ‘20: OnAir.Read Article
Quelle: Google Cloud Platform

How BigQuery helped Theta Labs and NASA bring science and hope to streaming

Editor’s note: We’re hearing today from Theta Labs, a leading decentralized video streaming platform that is powered by users and decentralized on a new blockchain. With their peer-to-peer bandwidth sharing distributed ledger technology, Theta Labs has been able to revolutionize the livestream experience. By adopting Google Cloud, Theta Labs has been able to scalably meet a growing active user base on their blockchain platform which in turn, has helped them expand their strategic partnership with NASA, including hosting the latest SpaceX rocket launch.When we established Theta Labs back in 2016, the goal was to set up a streaming video service with an emphasis on rendering popular PC video games like League of Legends, CS:GO and Dota2 into immersive 360° virtual reality experiences. And yet, thanks to our unique approaches to streaming, video rendering, and patented blockchain video technology, we’ve grown into something so much bigger that we’ve even caught the attention of NASA. All of this was possible thanks to Google Cloud and their databases and analytics products such as BigQuery, Dataflow, Pub/Sub and Firestore. Reaching the heights of video streamingBack when we first launched Sliver.tv—now Theta.tv—we decided to differentiate ourselves from the competition by creating unique live streaming video experiences, especially for streamers and viewers in regions with little or no access to high-speed internet. Our blockchain-based peer-to-peer video delivery technology lets users share their bandwidth with others, letting our streamers reach audiences they never could before.It was this ability to reach more unique and remote viewers and to give larger audiences the opportunity to discover new things that caught NASA’s attention. NASA saw the potential in our service to spread interest in science and technology to an audience of mostly younger viewers. They gave us the privilege of becoming one of only four or five video services with direct access to NASA’s source video feed, and we recently collaborated to premiere NASA’s August Women’s Equality Day  broadcast.The biggest highlight of this partnership so far was the opportunity to livestream the SpaceX launch. In a year where we all needed a bit more hope, being able to bring the live launch of a space shuttle to a wider audience was an amazing experience, inspiring so many to reach for the stars.Video of the stars starts with the cloudTo facilitate an event as large as a space launch with so many viewers takes a powerful infrastructure. To do all of this with our unique peer-to-peer blockchain system that rewards viewers and streamers for sharing bandwidth, we needed Google Cloud’s reliable, scalable, and stable infrastructure. With the strength of Google Cloud and their help on creating auto-scaling DevOps solutions, we were able to reach more viewers than ever without hitting the VM caps that previously caused issues with latency and customer experience. Previously, we’d faced challenges like infrastructure scaling limitations to meet application demands, high costs, and too much of our time wasted on managing and maintaining solutions.Google Cloud offers us better scalability, so we’re no longer capped by the number of active streamers we could have on our platform. Google Cloud gave us:Performance, and flexibility of implementationBreadth of capabilities and supportAbility to ingest streaming data for real-time insightsRelationship & communication with the Google account teamExpansive feature optionsPrice point compared to offered features/servicesOur partnership with Google Cloud has also let us reach viewers in regions that normally would have trouble accessing streaming video. Edge computing allows most of the computation work to be done near the source, improving response times and bandwidth usage—a perfect synergy leveraging Google and Theta Network’s core strengths. And with Google Cloud’s over 1600 nodes, we are able to get closer to our users than ever before.Running analytics on our skyrocketing dataBeyond the video streaming, Google Cloud’s enterprise data warehouse BigQuery gave us the capacity to do the typically difficult—if not impossible—task of sorting real-time data from the blockchain system. We have built a real-time pipeline for the viewership data using Dataflow, Pub/Sub and BigQuery. A Dataflow job continuously pulls the data from a Pub/Sub topic and ingests into BigQuery. We’ve seen Pub/Sub quickly ingest roughly 12,000 to 14,000 blocks of data containing 60,000 to 200,000 transactions daily into BigQuery for real-time analysis. We also used Pub/Sub and Dataflow to create the listener/subscriber for the topic our ETL pipeline publishes, then ingest that into BigQuery tables. By running fast queries in BigQuery, we were able to uncover findings such as: How many people watched and shared a certain video stream in the past hour;How many donations were made to a streamer in total;Which livestream has the highest donation to viewer ratio;What was the most impactful moment during a livestream.Prior to BigQuery, finding this information required writing customized scripts to analyze the blockchain raw data and the analysis used to take hours or even days of engineering time. Now we can gain such insights in a little as a few seconds, and effectively in real time. Now we can gather information to let streamers, advertisers, and partners know when more viewers were online and engaged. This means that NASA and other content creators could better find and reach their audiences. Results that let us grow and scale to the moon and beyondGoogle Cloud helps us better to forecast how many concurrent users we need to support during livestream events and predict multi-variable reputation scores for our network of thousands of edge and guardian nodes to identify and address bad actors and under-performing nodes. Today, our BigQuery environment has 45GB of data, which contains almost 7.5 million blocks and 57 million transactions and counting. We migrated to Google Cloud in less than six months, and saw the return on investment almost immediately. We’re able to bring top-notch connectivity, scalability and security capabilities for our branded content partners like NASA, enterprise validator partners including Google and community members that run Theta edge and guardian nodes, and we’re reducing costs over time.All of this is just the beginning of the ways we’re looking to spread more entertainment, science, and hope during these dark times. And thanks to Google Cloud’s strength and scalability, we’ll be able to keep growing, reaching even more audiences and partners.Learn more about Theta Labs here.
Quelle: Google Cloud Platform

Rate Limiting by the Numbers

As a critical part of Docker’s transition into sustainability, we’ve been gradually rolling out limits on docker pulls to the heaviest users of Docker Hub. As we near the end of the implementation of the rate limits, we thought we’d share some of the facts and figures behind our effort. Our goal is to ensure that Docker becomes sustainable for the long term, while continuing to offer developers 100% free tools to build, share, and run their applications.

We announced this plan in August with an effective date of November 1. We also shared that “roughly 30% of all downloads on Hub come from only 1% of our anonymous users,” illustrated in this chart:

This shows the dramatic impact that a very small percentage of anonymous, free users have on all of Docker Hub. That excessive usage by just 1%–2% of our users results not only in an unsustainable model for Docker but also slows performance for the other 98%–99% of the 11.3 million developers, CI services, and other platforms using Docker Hub every month. Those developers rely upon us to save and share their own container images, as well as to pull images from Docker Verified Publishers and our own trusted library of Docker Official Images, amounting to more than 13.6 billion pulls per month.

Based on our goal of ensuring the vast majority of developers can remain productive, we designed limits of 100 or 200 pulls in a 6-hour window for anonymous and authenticated free users, respectively. In the context of a developer’s daily workflow, 100 pulls in 6 hours amounts to a docker pull every 3.6 minutes on average, for 6 consecutive hours. We considered this more than adequate for an individual developer, while other use cases involving high pull rates such as CI/CD pipelines or production container platforms can decrease their usage or subscribe to a paid plan.

Over the course of a month, a single anonymous developer can (with the help of automation) make up to 12,000 docker pulls. By authenticating, that number increases to 24,000 docker pulls for free. As Docker container images vary in size from a few MB to well above 1 GB, focusing on pulls rather than size provides predictability to developers. They can pull images as they’re building applications, without worrying about their size but rather about their value.

Based on these limits, we expected only 1.5% of daily unique IP addresses to be included — roughly 40,000 IPs in total, out of more than 2 million IPs that pull from Docker Hub every day. The other 98.5% of IPs using Docker Hub can carry on unaffected — or more likely, receive improved performance as the heaviest users decreased.

As November 1st approached, we created a rollout plan that provided additional advance notice and decreased impact — even to developers we haven’t been able to reach through our emails or blog posts. We’ve put a few things in place to ease the transition for anyone affected:

Providing a grace period after November 1 prior to full enforcement for all usage, so only a small fraction of the heaviest users were limited early in our rollout;Progressive rollout of enforcement across the affected population, to provide for additional opportunities for communications to reach Docker developers, and to minimize any inadvertent impact; andTemporary time windows of full enforcement, to raise awareness of unknown reliance upon Docker Hub and to reach developers without Docker IDs who we could not otherwise.

On Wednesday, November 18, we expect to complete our progressive rollout to the new limits of 100 pulls and 200 pulls per 6-hour window for anonymous and authenticated free users, respectively. At that point, anyone who has not yet encountered the limits can reasonably conclude that their current usage of docker pulls is in that 98.5% of unaffected Docker Hub users.

As we’ve progressed down this path toward creating a sustainable Docker, we’ve heard multiple times from developers that the temporary full-enforcement windows were valuable. They surfaced unknown reliance upon Docker Hub, as well as areas where our paying customers had not yet authenticated their usage. We’ve also worked with customers to identify problems that were unknowingly causing some of the massive downloads, like runaway processes downloading once every 3 seconds. Alongside this, we’ve created additional paid offerings to support large enterprises, ISVs, and service providers with needs like IP whitelisting or namespace whitelisting.

We greatly appreciate the trust placed in Docker by the entire software community, and we look forward to helping you continue to build the great applications of the future!
The post Rate Limiting by the Numbers appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

Google Cloud, Harvard Global Public Health release improved COVID-19 Public Forecasts, share lessons learned

Harvard Global Health Institute and Google Cloud have been working the past three months to improve the COVID-19 Public Forecasts, to give first responders and healthcare organizations the best possible information to prepare for what lies ahead. These forecasts use AI to provide a projection of COVID-19 cases, deaths, and other metrics for U.S. counties and states. Since their original release, the COVID-19 Public Forecasts are now used by many organizations across the United States, and have been significantly improved in five major ways:1. Longer forecasts & confidence intervals. When initially launched, the COVID-19 Public Forecasts included predictions for 14 days in the future. They now include predictions for a 28-day horizon. Because predictions in general become more uncertain as they predict events further into the future, we have added confidence intervals to help users model that uncertainty.2. Improved model quality. A dedicated team has been continuously improving the model quality with cutting edge AI research. A white paper detailing some of these innovations was published in the machine learning conference, NeurIPS 2020. The model was one of the few to correctly forecast a surge in cases in October and November. The accuracy of the model has continuously improved over time and is retrained daily as more data becomes available. Since the forecasts were first published, we’ve seen the predictions improve by approximately 50%. “The COVID-19 Public Forecasts is an important public health tool for guiding the policy response to the COVID-19 pandemic. By providing an ‘early warning system’ of COVID-19 cases, hospitalizations, ICU admissions, ventilator utilization, and deaths, the COVID-19 Public Forecasts create the opportunity for public health officials and policymakers to move from a reactive to a proactive approach to suppress the pandemic,” said Dr. Thomas Tsai, MD, MPH, surgeon and health policy researcher at Harvard T.H. Chan School of Public Health. His research team is using the COVID-19 Public Forecasts to develop state and national testing targets to guide a testing strategy around screening of asymptomatic individuals to suppress the silent transmission of SARS-CoV-2.3. Ability to expand to other countries. We have added support for expanding the COVID-19 Public Forecasts to other countries, and today we are launching forecasts for Japan. As with the United States, these forecasts are free and based on public data such as the public COVID-19 Situation Report in Japan. The model predicts confirmed cases, deaths, recoveries, and hospitalizations, per day, and will look ahead 28 days into the future. Japan is made up of several prefectures and we offer these forecasts for each one. This information is available now on the Japan forecast dashboard. “We validated this COVID-19 forecast model for Japan from the academic perspective. The forecasts will be useful to professionals that understand the capabilities and constraints of the model, and will play a critical role for Japan’s public health and enhance our ability to understand and respond to the rapidly evolving COVID-19 pandemic. Coupled with other existing works such as Keio University’s COVID-19 response surveys partnering with Ministry of Health, Labour and Welfare and prefectures, this model will allow for more proactive and efficient public health interventions on a prefecture-by-prefecture basis,” said Prof. Miyata, Department of Health Policy and Management, School of Medicine, Keio University.4. Customized forecasts. Since the launch in August, we have worked with many organizations to better understand how these forecasts can help. In the process we have learned that many organizations have specific needs that go beyond just consuming our public forecast, such as wanting to use their own datasets as inputs. To that end, we have turned the initial forecasting model into a system that is customizable to new problems and datasets. We are working with public sector and healthcare leaders to help them create custom forecasts for their states and hospitals.5. What-if analysis for informing policy decisions. We have also seen significant interest in using the forecasting model to ask “what-if” questions to help make better-informed policy decisions. For example, you can see how the forecasts change in response to policy changes such as if non-pharmaceutical interventions (e.g. mask mandates) are introduced, reopening plans are changed, or vaccination policies are modified. To that end we have been developing a novel AI-driven what-if model to be used for COVID-19 and other infectious disease decision making. We hope that it will be helpful for organizations interested in doing vaccine rollout planning and other important decision making that may impact COVID-19 outcomes. If you or your organization are interested in exploring this tool, please contact us at COVID19-public-forecasts-feedback@google.com.Lessons learnedOver the course of the development of the COVID-19 forecasts, our team was faced with the risk of launching too quickly. On the one hand, the potential impact of more accurate and robust forecasts on the COVID-19 response was large, so launching quickly was important. However, we also needed to make sure that the quality was high enough to help inform decision makers as well as ensuring that it did not further any existing disparities through model bias. The following sections share some background about our journey leading up to the original launch of the COVID-19 Public Forecasts. Googlers across Alphabet came together in March and dove into literature to understand epidemiological forecasting, discover the best public datasets on which to train the model, build the infrastructure to train massive machine learning models, and design novel AI for time series forecasting. Over a hundred Googlers worked over many months to make sure the forecasts were robust, accurate, and fair.Combining cutting-edge machine learning with traditional epidemiological models. Most AI forecasting models learn from data, such as forecasting weather based on historical data. In contrast, most COVID-19 forecasting models do not learn from data but simulated spread according to human epidemiological assumptions. We designed a new kind of time series forecasting model that learns from both epidemiological human prior knowledge as well as data. Another significant challenge was to design systems that learn in a non-stationary environment: interventions such as mask and movement restrictions change frequently and sometimes in response to forecasts – progression of the disease influences public policy and individuals’ public behaviors and vice versa.Building on public Google Cloud products. From the beginning we knew that the tool we were creating would likely be shared broadly with the public and many organizations. That drove our decision to build this software on our public Google Cloud products, including Kubeflow pipeline, GCP hyper-parameter tuning, Kubernetes, BigQuery, Google Cloud Storage, and Cloud SQL. Using our own products helped us prepare our forecasts more quickly.Improving robustness of forecasts. To help make sure we were developing a useful forecast, we partnered with the Harvard Global Health Institute who guided us on how to maximize policy-making impact and ensure the forecasts would be useful to those who most needed it. Additionally, we partnered with a handful of early testers, including HCA Healthcare, to help us understand what should be forecasted, how it is formatted, and even test early versions of the forecasts. These efforts helped improve the forecasts before they were made available to the general public. We also brought in experts within Google with statistical and epidemiological expertise to ensure our work met the highest scientific standards. We designed a daily forecast launch process that first runs over 100 checks looking for any abnormalities, and we required a manual review for a qualitative analysis to check for issues. Every day our model training searches over hundreds of hyperparameter options, and the team works to ensure the best models reach our users.  Fair and equitable forecasts. It was important to us that the forecasts were reliable and robust. Given the disproportionate impact that COVID-19 has had on communities of color in the United States, we conducted a fairness analysis, looking at how both relative and absolute errors are different across various groups (particularly African American and Hispanic populations) and interpreting the results. We wrote up our findings in a public Fairness Analysis. We’re excited by what we were able to achieve in the eight months since this effort began, but more importantly, we’re proud of the people who have come together to make this small difference in the fight against the global pandemic and grateful for all of those on the front-lines saving lives, innovating, and bringing the world one step closer to returning to normal. If you have any questions about the COVID-19 Public Forecasts (g.co/covidforecast), customizations or what-if analysis, please contact us at COVID19-public-forecasts-feedback@google.com.Related ArticleGoogle Cloud AI and Harvard Global Health Institute Collaborate on new COVID-19 forecasting modelTo help first responders be better prepared for what lies ahead, Google Cloud is releasing the COVID-19 Public Forecasts.Read Article
Quelle: Google Cloud Platform

Wallaby vPTG Summaries

The OpenStack community had its second virtual Project Teams Gathering (PTG) following the Open Infrastructure Summit in October. Over 500 individuals and 46 teams (30+ OpenStack teams) across the globe, met and collaborated at the vPTG. Since the event concluded, several of those teams have posted summaries of the discussions they have had and the… Read more »
Quelle: openstack.org