4 best practices for ensuring privacy and security of your data in Cloud Storage

Cloud storage enables organizations to reduce costs and operational burden, scale faster, and unlock other cloud computing benefits. At the same time, they must also ensure they meet privacy and security requirements to restrict access and protect sensitive information. Security is a common concern we hear from companies as they move their data to the cloud, and it’s a top priority for all our products. Cloud Storage offers simple, reliable, and cost-effective storage and retrieval of any amount of data at any time, with built-in security capabilities such as encryption in transit and at rest and a range of encryption key management options, including Google-managed, customer-supplied, customer-managed and hardware security modules. Google has one of the largest private networks in the world, minimizing exposure of your data to the public internet when you use Cloud Storage. Best practices for securing your data with Cloud StorageSecuring enterprise storage data requires planning ahead to protect your data from future threats and new challenges. Beyond the fundamentals, Cloud Storage offers several security features, such as uniform bucket-level access, service account HMAC keys, IAM conditions, Delegation tokens, and V4 signatures. We wanted to share some security best practices for using these features to help secure and protect your data at scale: #1: Use org policies to centralize control and define compliance boundariesCloud Storage, just like Google Cloud, follows a resource hierarchy. Buckets hold objects, which are associated with projects, which are then tied to organizations. You can also use folders to further separate project resources. Org policies are settings that you can configure at the org, folder, or project level to enforce service-specific behaviors. Here are two org policies we recommend enabling: Domain-restricted sharing—This policy prevents content from being shared with people outside your organization. For example, if you tried to make the contents of a bucket available to the public internet, this policy would block that operation. Uniform bucket-level access—This policy simplifies permissions and helps manage access control at scale. With this policy, all newly created buckets have uniform access control configured at the bucket level governing access for all the underlying objects. #2: Consider using Cloud IAM to simplify access control  Cloud Storage offers two systems for granting permissions to your buckets and objects: Cloud IAM and Access Control Lists (ACLs). For someone to access a resource, only one of these systems needs to grant permissions. ACLs are object-level and grant access to individual objects. As the number of objects in a bucket increases, so does the overhead required to manage individual ACLs. It becomes difficult to assess how secure all the objects are within a single bucket. Imagine having to iterate across millions of objects to see if a single user has the correct access. We recommend using Cloud IAM to control access to your resources. Cloud IAM enables a Google Cloud wide, platform centric, uniform mechanism to manage access control for your Cloud Storage data. When you enable uniform bucket-level access control, object ACLs are disallowed, and Cloud IAM policies  at the bucket level are used to manage access—so permissions granted at a bucket-level automatically apply to all the objects in a bucket.#3: If you can’t use IAM Policies, consider other alternatives to ACLs We recognize that sometimes our customers continue to use ACLs for different reasons, such as multi-cloud architectures or sharing an object with an individual user. However, we don’t recommend putting end users on object ACLs. Consider these alternatives instead: Signed URLs—Signed URLs allow you to delegate time-limited access to your Cloud Storage resources. When you generate a signed URL, its query string contains authentication information tied to an account with access (e.g. a service account). For example, you could send a URL to someone allowing them to access a document, read it,  with access revoked after one week. Separate buckets—Audit your buckets and look for access patterns. If you notice that a group of objects all share the same object ACL set, consider moving them into a separate bucket so you can control access at the bucket-level. IAM conditions—If your app uses shared prefixes in object naming, you could also use IAM Conditions to shard access based on those prefixes.Delegation Tokens—You can use STS Tokens to grant time-limited access to Cloud Storage buckets and shared prefixes. #4 Use HMAC keys for service accounts, not user accounts A hash-based message authentication key (HMAC key) is a type of credential used to create signatures included in requests to Cloud Storage. In general, we suggest using HMAC keys for service accounts rather than user accounts. This helps eliminate the security and privacy implications of relying on accounts held by individual users. It also reduces the risk of service access outages as user accounts could be disabled when a user leaves a project or company.  To further improve security, we also recommend: Regularly changing your keys as part of a key rotation policy.Granting service accounts the minimum access to accomplish a task (i.e. the principle of least privilege). Setting reasonable expiration times if you’re still using V2 signatures (or migrating to V4 signatures, which automatically enforces a maximum one-week time limit). To learn more about Cloud Storage and more ways to keep your data safe and compliant, check out our access control documentation, and watch our breakout session from Cloud Next ‘20: OnAir.
Quelle: Google Cloud Platform

Going global: Workday uses Google Cloud AI to accelerate document processing

Scaling a business that sorts through millions of documents daily, across a global operation, is a tall order. Workday, with more than 3,400 core Workday Financial Management and Workday HCM customers, offers the Workday Expenses solution to provide a frictionless expense reporting experience for their customers. Here’s how they did it using Google Cloud’s Procurement DocAI.Solving for multi-language support for international expansionWorkday provides enterprise cloud applications for finance and human resources. Their in-house expense receipt parsing functionality needed to scale for international customers in multiple languages.Workday leverages Google Cloud’s Procurement DocAI machine learning solution to automate the processing of receipts with multi-language support as a key component of its expense receipt scanning capability. This partnership delivers data capture at scale, enabling Workday customers to gain deeper insights (e.g., fraud detection) and find new opportunities to keep their businesses moving forward. “At Workday, we believe enterprise software should simplify how organizations manage finance, HR, and planning. Google Cloud’s Procurement DocAI, combined with our document processing machine learning models, enables streamlined receipt intake for our customers at global scale.” – Shane Luke, VP Machine Learning & AI, WorkdayIncorporating Procurement DocAI’s multi-language support into Workday applications means faster innovation for customers to support their global deployments. Expanding languages to include French, German, and Spanish enables Workday to extend the reach of its solution to serve more customers around the world. It also offers higher data accuracy in data extraction, with an F1 score1 (measure of model’s accuracy) of > 85% for receipts, enabling a high level of satisfaction. Additional support in Workday for other languages and more documents like supplier invoices is in the near future.The partnership between Google Cloud and Workday is just one of the latest examples of how we’re providing AI-powered solutions to solve business problems.Procurement DocAI accelerates critical document processingProcurement DocAI is built on the recently announced Document AI platform, a unified console for document processing. Customers can easily create and customize all of the specialized parsers (e.g., invoices and receipt parsers) on the platform without the need to perform additional data mapping or training. All of Google Cloud’s specialized parsers are fine-tuned to achieve industry-leading accuracy, helping customers and partners confidently unlock insights from documents with machine learning.To learn more about how to get started with Procurement DocAI to automate your document workflows, check out this step-by-step guide.1. Google Internal DataRelated ArticleUsing Document AI to automate procurement workflowsShine a light on all your “dark” data with Google’s Document AI. Turn unstructured pdfs into fully automated workflows with machine learn…Read Article
Quelle: Google Cloud Platform

Google Cloud and SAP demonstrate massive scalability for financial services customers

The mission: Proving a scale-out cloud solution for SAP financial products subledger delivers performance, scale and reliabilityWhen financial firms run a critical, resource-intensive application like SAP’s financial products subledger (FPSL), they have traditionally adopted a monolithic, single-node, scale-up architecture. But this type of system isn’t designed to sustain long-term growth. As these systems grow, they get increasingly unwieldy and expensive.To demonstrate the cloud’s capabilities to run FPSL at scale, Google Cloud and SAP engineers conducted a six month proof of concept (POC), using SAP S/4HANA for financial products subledger on Google Cloud. The scale-out architecture was leveraged to run a payment scenario similar to those run by specialized payment providers such as PayPal.  While financial firms have been running this product for years on legacy single-node systems, SAP has optimized FPSL today to run on public clouds as an easily deployed scale-out solution. In a scale-out scenario as system demand grows, admins add new system instances that run in parallel as part of a distributed application architecture. But could a cloud-native, scale-out architecture handle the real-world demands of an application like FPSL, especially at extreme scale?Testing included a variety of Google Compute Engine VM instance types, plus network and storage resources. Both the SAP HANA database and application server environments spanned multi-node clusters that were scaled to a 96TB SAP HANA scale-out landscape. More details on the test environment can be found in the white paper. To give FPSL a real workout, the test data set generated for this purpose included a total of 10 million current accounts and daily postings of 40 million business payment transactions. The team used this data to simulate processing for all of the 2020 fiscal year plus half of FY2021—yielding a total of 12 billion payments and more than 200 billion subledger items intended for end-of-year accounting.Scale-out SAP S/4HANA FPSL on Google Cloud: 3 key findingsThe project yielded a wealth of hard data, based on internal measurements conducted by Google Cloud and SAP from April to September 2020. Performance, reliability and efficiency are key benefits for FPSL customers. 1. Performance and scale are no longer limitations for FPSL. System performance and scale was demonstrated on a number of levels with 200 billion records touched in 30 seconds for year to date balances. The system successfully maintained performance and throughput as it scaled with increasing parallel tasks from multiple application servers. A 40X acceleration in query runtime speed was achieved through parallelization. Engineers reduced runtime 2.7X—from two hours down to 44 minutes compared with performance levels for on premise scale-up systems. For peak data volume testing to simulate a Black Friday or Cyber Monday, loads were increased progressively from 40 million transactions to 160 million, quadrupling the number of line items per posting date from 500 million to 2 billion per day. Under these conditions, the figure below shows, the system responded, as expected, with slightly more than linear performance (volumes of 200% required slightly more than 200% runtime).OLTP Performance“The technology, products and services available through the SAP and Google Cloud partnership has given our Back Office significant agility and scalability. By moving to Google Cloud, we are positioning ourselves to keep pace with elevated transaction volumes, constant new financial product releases around the globe and M&A integration activities requiring swift operations.” —Dan Torunian, VP, Employee Technology and Experiences, PayPal2. Reliability was not compromised but improved due to Live Migration of Google Cloud VMs. The Google Cloud test environment ran actively for 24 weeks straight. The system ran continuous cycles of daily payment processing and year to date balances, with each day of actual system runtime handling the processing for multiple, simulated business days. When this process finished, the test environment started it again, without a second of intervening downtime.Not only were there no unplanned outages, but downtime as a result of system maintenance or applying security patches was also avoided. Google Cloud’s Live Migration capability keeps HANA VM database and application instances up and running by moving them seamlessly between servers when systems require maintenance or patching. The team also demonstrated plenty of headroom for additional workload demands with only 20-40% CPU utilization for the database and 15-30% CPU utilization for the application layer.Data protection for the environment was also put to the test. Multiple backups were performed and stored in Google Cloud’s multiregional Cloud Storage using the Cloud Storage Backint agent for SAP HANA. This resulted in 99.95% availability (or “11 nines” durability) with multi region replication, comprehensive encryption, and no egress cost. Engineers consistently measured speeds faster than 5 GB/second for backups to a multi-region-replicated cloud storage bucket (16.97 TB at 5.39 GB/second = 54 minutes).3. Enabling new optimizations like data tiering resulted in a 78% memory footprint reduction. The study also put the spotlight on data tiering as a strategy for managing storage costs. Powered by SAP HANA’s native storage extension data can be separated into “hot” and “warm” partitions for higher vs. lower priority access. The lower priority data is deemed “page loadable” and removed from memory, although it remains visible to the applications. Thanks to this capability, the test results showed a 78% reduction in memory footprint, in return for a very minor increase in data volume to the system’s disk storage—an important source of efficiency as transaction and data volumes continue to expand.”The SAP S/4HANA for financial products subledger solution on a 96 TB SAP HANA scale-out configuration brings the best of SAP and Google Cloud engineering together to deliver groundbreaking capabilities to PayPal. By combining Google Cloud’s expertise in building scalable, distributed systems with SAP’s relentless focus on delivering mission critical business process automation, PayPal now runs one of the world’s largest scale-out clusters of SAP HANA, with room to scale even further.” —Urs Hoelzle, SVP Technical Infrastructure, Google We’ve only scratched the surface of the findings here. If you’re looking to learn why more financial firms are trusting Google Cloud to host their most critical and sensitive core applications, download the white paper for even more insights into SAP and Google Cloud’s test results. Check out SAP’s perspective on this great achievement. Learn more about all Google Cloud solutions for SAP customers here.
Quelle: Google Cloud Platform

Stubhub’s path to retire the data center with Bare Metal Solution

Editor’s note: Today, we’re speaking with Austin Sheppard, CTO of StubHub, the world’s largest ticket marketplace. We’ll hear how Google Bare Metal Solution is helping StubHub on their journey into the cloud by reducing their dependency on Oracle, lowering overall costs, and driving performance. At StubHub, we’ve been at the very forefront of ticket marketplace innovation since 2000. We were the first to introduce a ticketing application, interactive seat mapping, 360-degree virtual views of seating, innovative price recommendation technology, and an algorithm for determining the best value of tickets—all features that have become standard.Ticket marketplaces go beyond standard ticket sales and distribution. They are in the business of helping buyers and sellers connect in a safe, convenient, and reliable environment. And when you serve millions of customers a year, data is not just an enhancement but one of the primary vehicles for unlocking opportunities and providing personalized experiences. At Stubhub, we have been on a journey to move all of our applications to the cloud and ultimately retire our data center. One of the most challenging migrations for us was going to be our Oracle databases, which power our ecommerce platform. We had been migrating the applications that have a large dependency on Oracle to Google Cloud, but we hadn’t been able to find an easy approach to moving the Oracle databases as well. Having Oracle on-premises and large chunks of the application layer in the cloud was causing latency issues and wasted team resources to synchronize data between the environments. To meet our performance requirements, we needed to have all of the data physically co-located (or close to it) with the application. Additionally, our Oracle databases are a legacy holdover and are still running on Solaris. This has caused performance challenges and it can be hard to get up-to-date service and maintenance. Our upgrade cycles require a colossal coordination effort and are difficult to do with minimized downtime and customer impact. We wanted to break free of this cycle and wanted a solution that would help to reduce our dependency on legacy technologies such as Oracle, while meeting our performance needs, and ultimately help to retire the data center.Moving to Google Cloud and the Bare Metal Solution was a great opportunity for us to take a leap into the future.Bare Metal Solution: The bridge off of OracleWe ultimately selected Google Cloud’s Bare Metal Solution (BMS) since it allows us to continue using Oracle databases without sacrificing performance. This gives us a fast way to move these workloads as-is to the cloud, which solves the latency issue and accelerates our ability to completely eliminate our on-premises footprint. BMS also seamlessly integrates with the rest of the Google Cloud services that we have already been using.For my team, it was critical that we were not just able to move to the cloud, but that we could also modernize what we used to run our business. Within the next year, we are looking to fully retire the data center. Once that is complete, we intend to modernize our Oracle databases onto the cloud-native Spanner for its unlimited scale, strong consistency, and high availability.Google Bare Metal Solution is our bridge to get off the Oracle database entirely. It gives us the flexibility to move the rest of our platform to the cloud so we can now modernize faster. We are planning to modernize these workloads on Cloud Spanner and this gives us the path to do so.Early results running Oracle on Google Cloud’s Bare Metal SolutionThe early results we’ve seen so far as we’ve started migrating have been promising. Our side-by-side performance tests between on-premises Solaris systems and the newly configured BMS servers are already showing a performance improvement. We have also projected that our overall operating costs would decrease due to consolidating environments and eliminating hardware support needs, and we’re already seeing lower licensing costs as well.We’re seeing some of our costs go down right away because we are able to do more with less. We can now run Oracle on a smaller hardware footprint because of the better, faster, cheaper Intel Cascade Lake processors, resulting in an overall lower licensing cost of Oracle database.The biggest benefit, though, is the independence BMS gives us as a business by reducing our dependency on legacy vendors like Oracle. We now have control over our own destiny as we move to a cloud-native organization.Explore Stubhub and learn more about migrating Oracle to Google Cloud.Related ArticleBare Metal Solution: Coming to a Google Cloud data center near youWith Bare Metal Solution, now you can run specialized databases in five new Google Cloud regions.Read Article
Quelle: Google Cloud Platform

Streaming augmented reality with Google Cloud

Every year at CES, people from around the world experience the latest and greatest that consumer tech has to offer. In 2021, CES will be in an all-digital format for the first time. So how can a virtual show like CES create immersive experiences for attendees tuning in remotely? That’s an interesting challenge for the cloud, and of course, every challenge presents an opportunity. Google Cloud and 5G help enterprises deliver new experiencesEarlier in 2020, we announced our enterprise telecommunications strategy to deliver workloads to the network edge on Google Cloud, and during our SearchOn event last October, we announced how cloud streaming technology can power augmented reality (AR) in consumer search results. Now, we’re merging the best of both worlds: Technology built for consumer search can take advantage of our enterprise edge capabilities. In light of the COVID-19 pandemic, this past year accelerated our support for enhanced consumer experiences across the board in novel ways. For example, we attempted to address questions such as how potential buyers can make a purchasing decision when they can’t see the product up close. This question becomes even more critical when considering a large purchase such as a new car. That’s exactly what Fiat Chrysler Automobiles (FCA) and Google Cloud are working together to solve. As part of FCA’s Virtual Showroom CES event, you can experience the new innovative 2021 Jeep Wrangler 4xe by scanning a QR code with your phone. You can then see an Augmented Reality (AR) model of the Wrangler right in front of you—conveniently in your own driveway or in any open space. Check out what the car looks like from any angle, in different colors, and even step inside to see the interior with incredible details.”As we continue our journey towards becoming a customer-centric mobility company, FCA is adopting emerging technologies that enable us to accelerate and deliver at the speed of our customers’ expectations,” said Mamatha Chamarthi, Chief Information Officer, FCA – North America and Asia Pacific. “Through our collaborative partnership with Google, we are able to expand our efforts to provide an immersive customer experience.” Harnessing the power of edge with 5GIn order to create a mixed-reality experience with a 3D car model, computer-aided design (CAD)-based data sources that represent a 3D vehicle with highly detailed geometry, depth, texture, and lighting were used. High-fidelity models, such as cars with full interiors, often mean large files (GBs in size). Traditionally, depending on your connection, this can result in long waiting times as assets are downloaded onto your phone. In addition, while mobile phones are more powerful than the Apollo Guidance Computer, they are no match for the power we have in the cloud. We still want to bring these high end experiences to everyone, regardless of their respective device or geographical location. We solve this problem by rendering the model in Google Cloud, then streaming it to the devices. Specifically, the Cloud AR tech uses a combination of edge computing and AR technology to offload the computing power needed to display large 3D files, rendered by Unreal Engine, and stream them down to AR-enabled devices using Google’s Scene Viewer. Using powerful rendering servers with gaming console grade GPUs, memory, and processors located geographically near the user, we’re able to deliver a powerful but low friction, low latency experience. This rendering hardware allows us to load models with tens of millions of triangles and textures up to 4k, allowing the content we serve to be orders of magnitude larger than what’s served on mobile devices (i.e., on-device rendered assets). Doing so leverages high-speed 5G connectivity and streams directly from Google Cloud’s distributed edge, delivering a rich, photorealist immersive experience. Customers like FCA benefit from Google’s years of investment and expertise in streaming technology (have you tried playing Cyberpunk2077 on Stadia yet?). With the expansion of 5G networks, not only will streaming enable the experience for anyone anywhere, but it will also cut the wait time of downloading large assets required for detailed AR/VR experiences, ultimately providing instant gratification.Applications and experiences are at the core of a winning edge propositionWe’re working to make these capabilities available to all enterprise customers to enable innovative use cases such as leveraging AR to help design teams collaborate, technicians perform machine diagnostics, creating next-generation, live video experiences for sports events, enabling new customer experiences across many industries, and supporting our customers in their digital transformation. Stay tuned!To see the Jeep Wrangler 4xe come to life whether you’re attending CES or not, scan the QR code below, or check out the FCA CES website. Depending on your OS, device, and network strength, you will see either a photorealistic, cloud-streamed AR model or an on-device 3D car model, both of which can then be placed in your physical environment.
Quelle: Google Cloud Platform

Building a self-service microservices architecture with Cloud SQL

Editor’s note: We’re hearing today from Entegral, an integrated software platform that enables communication and collaboration between tens of thousands of collision repair shops, insurance providers, and other industry professionals around the world. Owned by Enterprise Holdings, the world’s largest car rental provider and operator of the Enterprise Rent-A-Car brand, Entegral builds apps that make the claims process more efficient, using the best data technology and skills available. Here’s how they’re using Google Cloud to enable their teams to build faster.When we decided to make the move from on-premises to Google Cloud, this was our opportunity to not only revamp the technologies we used internally, but to also rethink how our teams could operate. At Entegral, that meant breaking our existing monolithic system and shifting to a microservices environment powered by Cloud SQL, which we use as a managed service for MySQL and PostgreSQL. Now, our internal teams have self-service access to the resources they need to develop new applications, and my team is free to focus our energies elsewhere.  Moving to a self-service access modelOur migration to Google Cloud was pretty straightforward. All of our on-premises databases were small enough to just do a simple export of the data and import into Google Cloud, so we were up and running quickly. To support our new microservices environment, we use Google Kubernetes Engine, and Cloud SQL for both MySQL and PostgreSQL as our main database. As part of moving to managed cloud services, we wanted to find ways to improve the operational efficiency of my infrastructure team. We wanted to give other teams the ability to provision their own Cloud SQL databases, all without manual intervention from my group.Prior to this, each request from other teams fell on my team. Depending on priorities, it could take days to get teams the credentials and resources they needed, and my team was responsible for managing those databases. Now our self-service model has turned all of this into a YAML configuration that takes minutes. All the security credentials are built in and teams can even select preferred engines and versions. This has dramatically decreased the manual intervention needed from our infrastructure team. This process has been completely disaggregated, with few requests coming in directly to my team, and we’ve maintained the ability to track instances across the company.Not only has Cloud SQL allowed us to move to a fully self-service model, but its benefits as a managed service have improved costs, reliability, and security. High availability is something we simply don’t need to think about any more, as it’s trivial to set up and build into the configuration. And Cloud SQL handles all of the on-demand scaling and upgrades to ensure teams always have what they need.Adding agile application developmentFor my colleague Patrick Tsai, team lead for application development, this new model was transformational. His team no longer needs to think a sprint ahead. They have access to the tools they need so they can start building quickly. They were building a spatial view of the company’s network of body shops by bringing together metadata. This allowed for an easy and fast way to visualize and manage their networks on Google Maps. Since this application heavily uses geospatial data, they opted for Cloud SQL for PostgreSQL and the popular PostGIS extension. In a matter of minutes instead of days, they can spin up a new database instance to support a variety of different APIs for this application. To date, Patrick’s team has five different environments with 15 different Cloud SQL instances each and doesn’t need to worry about scaling, upgrading, or maintaining any of them. They can just focus on building new functionality.Redefining how we operateWe’ve been thrilled with what we’ve been able to accomplish leveraging Cloud SQL and Google Cloud. It’s also changed how we evaluate new technologies. Now that we’re in the cloud, we believe all our services should be able to deliver the same level of self-service access with a single configuration.We’re continuing to evolve what we’ve designed to be even more dynamic and offer new secure connectivity defaults. And we’re excited to continue to empower Entegral’s teams to be more agile and transform the business in ever-evolving new ways. Learn more about Entegral and about Google Cloud’s database options.Related ArticleExtensions for connectivity and new data types now available in Cloud SQL for PostgreSQLCheck out new extensions and data types in open source database PostgreSQL, and why they matter when you’re using our Cloud SQL managed s…Read Article
Quelle: Google Cloud Platform

State of no-code app-development

It’s no secret that the world is changing faster than ever. Before 2020 became, well, 2020, businesses knew that digital transformation was critical, yet few were prepared to make the pivot. The events of last year caused a forcing function, accelerating the need for change in organizations around the globe. Problem solving required a democratized approach to the tools of innovation, one in which anyone in the workforce, whether skilled in coding or not, could leverage technology to create novel solutions. The solution stacks businesses were working with needed to be rethought. Though we typically associate app creation with traditional developers who write code, hundreds of thousands of apps were built this year by non-technical “citizen developer” app creators from around the globe. This democratization occurred because industries such as retail, construction, manufacturing, hospitality, telecom, education, real estate, IT services and more all sought digital transformation through no-code development. In this post, we’ll breakdown why no-code application development has become an important part of digital transformation by examining the following areas of impact: Speed and agility Productivity and collaborationGovernance and security Speed & agility were critical In a recent surveyof app creators using Google Cloud’s no-code application platform, AppSheet, some 32% of respondents cited speed of development as one of the greatest benefits to using no-code. Faster development times mean more responsive iterations, more problems being solved, and potentially greater impact for those that need tailored line-of-business solutions. The ability to pivot quickly has been incredibly important during a time in which the way we operated frequently needed to change quickly.  How exactly does no-code technology allow for faster development cycles? First and foremost, it does exactly what the name implies—it allows people to build applications and process automations without coding and therefore opens up innovation to a wider range of employees. AppSheet, for example, abstracts coding into an intuitive, natural-language based interface that anyone can use, even if they have never written a line of code—all while still giving IT the visibility and control it needs to keep data safe.  And while every no-code platform has a different approach to facilitating speedier and more democratic app creation, app creators that choose AppSheet are able to leverage the platform’s use of Google AI and ML in two different ways. Baked-in AI and ML help app creators create prototypes quickly, facilitating things like natural language-interfaces in which users need only type their intent to surface relevant functions, and also enable implementation of rich capabilities like Optical Character Recognition (OCR), for conducting activities such as inventory management or other use cases that involve barcodes, image recognition, and similar ML-driven capabilities. Collaboration transcends locationThe ways in which we work shifted an incredible amount in 2020. Those who generally worked in offices with their teams or in schools with their students were sent home. Those who continued to work on-location filled essential roles that often required agile, on-the-go access to critical data, whether it was stored on-device or in the cloud. As communities surrounding work, school, and family life became impacted and activated in ways we’ve arguably never seen, collaboration and connection became paramount to success in 2020.This need for collaboration was no different for those trying to find digital ways in which to support the communities in which they operate. For those who use no-code, there were two key ways in which this collaboration takes place. Once an app is shared with another person, whether it be a fellow creator or an end user, collaboration becomes something different. For a creator-to-creator relationship, it means ensuring the data source in use is designed for manageability, iteration, security, and friction-free access. For an end user-to-creator interaction, the relationship is more complex: app creators typically engage with an application in a development capacity on a desktop while end users typically engage with an application on their mobile devices. This is why our approach to no-code lets app creators build one app that’s compatible across them all— desktop, tablet, etc.— with no additional development required to support a range of  major operating systems or interaction models. This allows the creator to engage with their end users in a more direct and meaningful manner that not only increases collaboration, but can also increase productivity. BHI – a construction company based out of the US – is a great example of how to collaborate with a workforce through no-code. With 90% of their employees not only deskless, but also dispersed across worksites spanning 25 states, the company needs the ability to engage with data in real time. They take collaboration one step further by leveraging a combination of no-code development and Google Workspace. This combination has allowed BHI to free up their IT spend by 10% while embracing democratized digital innovation.  Unlocking data without compromising governance and security  Governance was one of 2020’s  most discussed IT topics, and for good reason: in 2019, research indicated that “anything from 30% to 50% of enterprise spend is linked to shadow IT.“ When an IT team can set policies and provide oversight for non-technical teams within the organization, employees on the ground can problem solve quickly without creating management and governance liabilities; enterprises want to empower employees to innovate and move fast—but they cannot afford to play fast-and-loose with data security. Neither no-code technology nor the citizen developers who leverage it are meant to replace traditional development practices within an organization. Rather, the intention is to work in concert to support digital transformation by both respecting governance needs and democratizing access to the tools of innovation and optimization.Belgium-based Solvay is a great example of governance at work. As we detailed in case study, ”Francis Boulu and the [Solvay] industrial team initially started using AppSheet to collect information from production operators to streamline checklists and inspections. Using AppSheet in conjunction with Google Sheets, they were able to rapidly demonstrate and roll out applications and work them into production.” Through word of mouth across the organization, the demand for no-code technology grew, which could easily have overwhelmed any IT team’s backlog. But because Solvay selected a platform that incorporated governance into its capabilities, IT was able to provide an infrastructure that empowered individual contributors to participate in the development process while still adhering to governance requirements. To date, Solvay has created approximately 1,000 apps with no-code that are used around the globe, including those they’ve built in response to support their COVID initiatives. Looking to the future As we move past 2020, the primary change in the state of work has been broader recognition of the need to empower non-technical employees to build custom solutions and optimizations. Work culture will continue to shift and the demands for democratizing technology will increase. So too will the demands for platforms such as Google Cloud’s AppSheet. The importance of AI will continue to grow, and as the technological capabilities for these platforms continue to evolve, governance will play a more important role than ever before to ensure those that need to build applications are able. 2021 will bring about new ways to engage with no-code, ways that we believe will help creators deepen their skills and create true impact for their organizations. Analysts agree: Gartner’s October 2019 report “The Future of Apps Must Include Citizen Development,” indicated that by 2023, “the number of active citizen developers at large enterprises will be at least four times the number of professional developers.”  If you’re not already evolving your work to include custom applications with no-code, join app creators around the globe and get started now.
Quelle: Google Cloud Platform

A year of API-driven digital transformation

2020 was a challenging year for many organizations as they faced sudden changes in consumer behavior and market dynamics. The shift to digital channels is nothing new, and even in 2019, digital was already the preferred option for commerce and collaboration across many industries—but in the wake of the global pandemic, these channels became the first and only option for many businesses. Preference gave way to necessity almost overnight. According to Adobe Analytics, Black Friday 2020 shopping witnessed a growth of 22% and Cyber Monday saw 15.1% increase compared to 2019.  More importantly, this is going to be a long term shift. According to McKinsey, over 60% of customers changed their shopping behavior in 2020. With in-person and brick-and-mortar operations disrupted across much of the world, the past year’s trends mean that more than ever, a business’s ability to operate relied on its ability to insert itself into digital channels and to engage with customers in new ways. APIs are the foundational mechanisms powering these digital interactions, and hence API management becomes a more urgent competency for many companies as they navigate the impacts of COVID-19.  The role of API management platformsWe have seen many organizations that invested in API management benefiting in different ways during these challenging times. Here are some examples:Deliver innovation faster: Companies challenged with legacy systems were able to unlock valuable data and services via APIs, allowing developers to build new experiences faster. To learn more, check out this video, from Google Cloud Next ‘20: OnAir, that explains how Australian Energy is leveraging APIs to build a hub for open data.Build business ecosystems: APIs allowed organizations to collaborate and quickly integrate with partner ecosystems to transform their business models. Read this article to learn how CHAMP Cargosystems executed ecosystem strategies to help airlines navigate COVID-19 disruptions. Automate operations with AI: During periods of peak online activity, such as Black Friday and Cyber Monday, AI-powered features like traffic prediction, anomaly detection, and API monitoring helped organizations to focus on business-critical concerns and keep services online without interruption. See this video to understand how TMobile has leveraged APIs, and the process automations they enable, across diverse technology ecosystems ranging from mobile services to machine learning to the Internet of Things. Develop new revenue streams: Many businesses suffered declines in top-line revenue during 2020, but monetization of API products helped some to stay competitive by unlocking new sources of revenue. To learn more, view this on-demand webinar that includes how MTN Group, a leading multinational telecommunications company, is harnessing APIs to unlock new revenue opportunities. Google Cloud’s continued innovation in API Management To help organizations navigate 2020’s challenges, as well as adapt to trends in enterprise IT that have been unfolding for years ahead, Google Cloud launched several new capabilities throughout the year to bolster how we help our customers with API management:API Gateway helps developers securely share their serverless workloads and monitor consumption on serverless APIs. Apigee datasource for AppSheet helps citizen developers accelerate digital transformation, empowering them to harness APIs, and the more powerful datasets and functionality to which they provide access, for no-code app development. Apigee Adapter for Envoy As enterprises adopt microservices-based architectures, Apigee was first to build an adapter for Istio to expose services outside the mesh or between services running entirely within the mesh. Apigee adapter for Envoy now integrates Envoy-based services into your Apigee environment, extending Envoy’s capabilities to include API management and letting developers expose services behind Envoy as APIs.Thought leadership, best practices, and engagement with the API community We continued our engagement with the communities of developers, technical, and business leaders looking to do more with APIs. For example we launched That Digital Show, a podcast series in which we share field-tested best practices and real-world use cases to help enterprises to build and scale their API programs. We also offered our Business Innovation Roadshow, which explored how to think like a futurist to evaluate emerging business opportunities, how to discern key technology signals, and when and how to act in order to build an API-first strategy. We  launched a new series of ebooks, The Path of Most Resilience, that explores three tangible ways in which APIs help build business resilience via improved operational efficiency, accelerated innovation, and the development of richer customer experiences. We also published a range of blog posts and articles— if you missed any, you can catch up on all the API goodness here. Our thought leadership efforts also encompassed several industry-specific discussions, including:APIs for Healthcare The Digital Transformation of Healthcare, which examines how the healthcare industry is navigating these difficult times and how patients will soon benefit from new digital services.Getting Digital with Healthcare, which involves a panel of experts discussing three key pillars the healthcare industry is leveraging to help organizations enable data interoperability and provide value-based care.APIs for Marketing: The Automotive Industry, which surveys how APIs are being used for marketing and digital transformation in the automotive industry.APIs for Media and Entertainment, which looks at the digital future of movies, television and gaming.API-powered Open banking, which discusses how APIs can accelerate compliance, create connected customer experiences, and launch new productsLast but not the least, in partnership with Oxford Economics, we surveyed over 1,000 CIOs of large organizations (i.e., those with over $2 billion in revenue), from around the world and across industries, to understand their digital business ecosystem strategies—and the benefits they derive from cultivating those relationships. Check out the key findings of the survey and the role of APIs in building and scaling digital ecosystems.What’s next?We believe most of the current shifts in market dynamics will have long-term implications on consumer behavior and the way businesses operate; even before the pandemic disrupted enterprises throughout the world and increased the urgency around digital transformation, results from our Oxford Economics survey indicated that business leaders were connecting APIs to  the modern digital experience and efficiencies required to be competitive. Digital experiences across a range of form factors—in offices, stores, homes, or the field—will increasingly define how work gets done, how customers interact with businesses, and how businesses interact with one another. Interactions and automations among data and functionality in difference systems will power these experiences—and APIs will continue to power those interactions. Going into 2021, in order to reinvent and globally scale their efforts, enterprises will need to think beyond digital transformation, as which has become an issue of table stakes rather than the mark of a differentiated leader. Instead, businesses must strive to pursue digital excellence—and that means API management becomes even more important. It’s not enough to simply use APIs, though that is a prerequisite for success. Businesses need to harness their APIs to pursue new business models, such as monetizing data for an outside audience, or new partnerships, such as connecting one’s own APIs to those of social networks in order to create richer, wider-reaching customer experiences. Businesses need to learn from APIs, applying machine learning to discern trends in developer and end user behavior and further fortify digital services against attackers. And they need to extend their APIs in new directions, opening them not only to traditional developers but also the new breed of no-code application builders. To boost our customers’ efforts towards digital excellence, we look forward to launching several new capabilities in the coming year that will let customers leverage Google Cloud’s industry-leading offerings to accelerate their API programs. We’re excited to see what you build with them.
Quelle: Google Cloud Platform

The Magic Of Distributed Joins in Cloud Spanner

Cloud Spanner is a relational database management system and as such it supports the relational join operation.  Joins in Spanner are complicated by the fact that all tables and indexes are sharded into splits.  Every split of a table or index is managed by a specific server and in general each server is responsible for managing many splits from different tables.  This sharding is managed by Spanner and it is an essential function that underpins Spanner’s industry leading scalability.  But how do you join two tables when both of them are divided into multiple splits managed by multiple different machines?  In this blog entry, we’ll describe distributed joins using the Distributed Cross Apply (DCA) operator.We’ll use the following schema and query to illustrate:If a table is not interleaved in another table then its primary key is also its range sharding key.  Therefore the sharding key of the Albums table is (SingerId, AlbumId).  The following figure shows the query execution plan for the given query.Here is a primer on how to interpret a query execution plan.  Each line in the plan is an iterator.  The iterators are actually structured in a tree such that the children of an iterator are displayed below it and at the next level of indentation.  So in our example, the second from the top line labelled Distributed cross apply has two children; Create Batch and, four lines below that, Serialize Result.  You can see that those children each have arrows pointing back to their parent, the Distributed cross apply.  Each iterator provides an interface to its parent with the API GetRow.  The call allows the parent to ask its child for a row of data.  An initial GetRow call made to the root of the tree starts execution.  This call percolates down the tree until it reaches leaf nodes.  That is where rows are retrieved from storage after which they travel up the tree to the root and ultimately to the application.  Dedicated nodes in the tree perform specific functions such as sorting rows or joining two input streams.In general, to perform a join, it is necessary to move rows from one machine to another.  For an index-based join, this moving of rows is performed by the Distributed Cross Apply operator.  In the plan you will see that the children of the DCA are labelled Input (the Create Batch) and Map (the Serialize Result).  The DCA will move rows from its Input child to its Map child.  The actual joining of rows is performed in the Map child and the results are streamed back to the DCA and forwarded up the tree.  The first thing to understand is that the Map child of a DCA marks a machine boundary.  That is, the Map Child is typically not on the same machine as the DCA.  In fact, in general, the Map side is not a single machine.  Rather, the tree shape on the Map side (Serialize Result and everything below it in our example) is instantiated for every split of the table on the Map side that might have a matching row.  In our example, that’s the Albums table, so if there are ten splits on the Albums table then there will be ten copies of the tree rooted at Serialize Result, each copy responsible for one split and executing on the server that manages that split.The rows are sent from the Input side to the Map side in batches.  The DCA uses the GetRow API to accumulate a batch of rows from its Input side into an in-memory buffer.  When that buffer is full, the rows are sent to the Map side.  Before being sent, the batch of rows is sorted on the join column.  In our example the sort is not necessary because the rows from the Input side are already sorted on SingerId but that will not be the case in general.  The batch is then divided into a set of sub-batches, potentially one for each split of the Map side table (Albums). Each row in the batch will be added to the sub-batch of the Map side split that could possibly contain rows that will join with it.  The sorting of the batch helps with dividing it into sub batches and also helps the performance of the Map side.The actual join is performed on the Map side, in parallel, with multiple machines concurrently joining the sub batch they received with the split that they manage.  They do that by scanning the sub-batch they received and using the values therein to seek into the indexing structure of the data that they manage.  This process is coordinated by the Cross Apply in the plan which initiates the Batch Scan and drives the seeks into the Albums table (see the lines labelled Filter Scan and Table Scan: Albums).Preserving input orderIt may have occurred to you that between sorting the batch and passing the rows between machines, any sort order the rows had in the Input side of the DCA might be lost – and you would be correct.  So what happens if you required that order to satisfy an ORDER BY clause – especially important if there is also a LIMIT clause attached to the ORDER BY?  There is an order preserving variant of the DCA and Spanner will automatically choose that variant if it will help the query performance.  In the order preserving DCA, each row that the DCA receives from its Input child is tagged with a number to record the order in which rows were received.  Then,  when the rows in a sub batch have generated some join result, they are re-sorted back to the original order.Left Outer JoinsWhat if you wanted an outer join?  In our example query, perhaps you want to list all singers, even those that don’t have any albums?   The query would look like this -There is a variant of DCA, called a Distributed Outer Apply (DOA) that takes the place of the vanilla DCA.  Aside from the name it looks the same as a DCA but provides the semantics of outer join.Learn moreTo get started with Spanner, create an instanceor try it out with a Spanner Qwiklab.
Quelle: Google Cloud Platform

Behind the scenes of Cloud Spanner’s ExecuteQuery request

This post is going to shed some light on the magic that happens when an application executes a query against Cloud Spanner. How does Spanner take an arbitrary SQL statement, locate the data, and return the response in milliseconds? We will take some of the concepts described inSIGMOD’17 paper and explain, step by step, how the execution occurs.Spanner data location primerThe requisite background for understanding the rest of this post is understanding data layout in Spanner. The data is organized into tables which can be root level or interleaved in another table. Interleaving instructs Spanner to store rows of “child” tables sharing a primary key prefix close to their “parent” rows (see Parent-child table relationships section in Schema and Data model doc). For example, using the Spanner sample schema with Singers and Albums tables:The data is stored into splits, where the primary key of the top level table determines the split point. The key ranges of the resulting splits are [Singers(-∞), Singers(1)) – not shown, [Singers(1), Singers(2)), [Singers(2), Singers(3))  and [Singers(3), Singer(+∞)) respectively. Each split is placed independently and may or may not be co-located with another split on a server. A split has multiple replicas to maintain distributed consensus and ensure quorum for transactions. Spanner coprocessor frameworkSpanner uses a coprocessor framework. This component allows Spanner to direct RPCs to a data location rather than a specific server, providing an abstraction between the data model and data placement.For example, a call would specify Singers(2) as its destination and not an IP address of the server. The coprocessor framework would then resolve the split that owns this data and route the request to the nearest replica by network latency from the client. This way, if the data moves, the client application doesn’t need to be aware of it and can rely on the coprocessor framework to route the request. PrepareQueryWhen the client calls executeQuery() in Java (or ReadOnlyTransaction.Query in Go), the first coprocessor call that is executed is PrepareQuery. Since the query optimizer logic is not hosted in the client, the call is routed to a random server which parses and analyzes the query, returning the Location Hint to the client. The location hint specifies the key of the top level table to send the query to. For example, if your query is “SELECT SingerId, AlbumId, AlbumTitle from Albums WHERE SingerId=2”, the Location Hint will be “Singers(2)”.The location hint computationThe location hint is computed by analyzing the compiled query representation and finding predicates on the top level table’s key columns. The location hint can contain parameters, so that it can be cached for parameterized queries. For example, for a query “SELECT * FROM Albums WHERE (SingerId = 1 AND AlbumId >= 10) OR (SingerId IN (2,3) AND AlbumId != 0)”, the location hint extraction logic will determine that the query addresses a table under the top level table Singers, thus it will attempt to extract the first predicate for the SingerID column from the WHERE clause. Upon discovering SingerId = 1 it will generate the location hint Singers(1). If the query contained SingerId = @id, the location hint would be Singers(@id). This parameterized form is then resolved for the actual value of @id query parameter to yield the data location (so id=1 would give Singers(1)).The location hint is cached using a hash of the SQL text as key. This means that if your application executes the same query for different end users, it’s better to create a parametrized query to increase the cache hit rate. If this cache is hit, the call to PrepareQuery is completely avoided, thus improving the overall performance and query latency.  This can be very significant for “simple” queries where the processing time is smaller than the round trip time; skipping a PrepareQuery call saves about half the latency. ExecuteQueryOnce the location hint is available, the next coprocessor call is ExecuteQuery. This call is routed based on the location hint, and the receiving server becomes the root server for the query. The server then compiles the query, creates the execution plan, and starts executing the required operations. The compiled query plan and execution plan are cached (so, again, parameterizing the queries is important to save this step on repeated queries).The execution plan will contain the Distributed Union operator to handle data access across one or more splits. In particular, there will be a Distributed Union on top of any execution tree. Additional Distributed Union operators might be inserted for other scans in the query (e.g. when joining two top level tables). The Distributed Union performs range extraction (described below), dispatches subqueries to the other splits, and streams results from the remote servers executing these subqueries to the caller. Distributed Union Range ExtractionSpanner compiles the predicates and builds a Filter Tree for efficient key predicate evaluation and range computation. For example, for the query from the above example  “SELECT * FROM Albums WHERE (SingerId = 1 AND AlbumId >= 10) OR (SingerId IN (2,3) AND AlbumId != 0)”, the FilterTree might look like this (the red text is described below) :The computation for the distribution range tries to extract keys for the top level table (SingerID). The FilterTree considers ranges as shown in red. For example, the node “SingerId = 1” produces the range of [1]. The node “AlbumID>=10” is not relevant to this key, so its range is (-∞, ∞). The AND node above intersects the two ranges, ending with [1]. It proceeds in a similar fashion, ending with the overall SingerId range of [1,3]. When the distribution keys cannot be extracted from the query (e.g., “SELECT * FROM T” without any key predicates), the query will be dispatched to every split containing a portion of the table. Sending ExecuteInternalQuerySince the distribution range [1,3] is enumerable (contains a small number of discrete keys), the Distributed Union can iterate over each value of SingerID and dispatch a subquery to that location. If several top level rows are on the same split, a single ExecuteInternalQuery RPC will be sent to that split. Note how the internal query also uses the coprocessor framework to locate the data. This allows flexibility to move portions of the data to ensure even distribution of the load on each server, and the coprocessor framework can route the new requests to the updated location without the need for the query processing component to worry about it.The Distributed Union operator is responsible for getting the results from the remote splits, stitching them together and returning the combined results to the caller. If the number of splits is large, the query can reduce its wall clock time by specifying the USE_ADDITIONAL_PARALLELISM query hint at the expense of higher resource consumption.ConclusionExecuting a query against Spanner requires taking into account data placement and movement, which is dynamically managed. Spanner’s coprocessor framework insulates the rest of Spanner’s components from the details of the data’s physical location and adaptively routes requests based on the logical data description.Distribution key extraction in the PrepareQuery call allows Spanner to start the execution on the first server that needs to process the query. Additional subqueries are sent by it, if needed, to other servers containing the rest of the data splits. This mechanism allows arbitrarily complex queries that can gather the required data from the distributed splits and presents a simple executeQuery API to the end user. This post has only scratched the surface of the complexity in executing the query. The paper also describes restarts mechanism, storage improvements,transactions and much more… brought to you by the Spanner team. Learn moreTo get started with Spanner, create an instanceor try it out with a Spanner Qwiklab.Related Article3 reasons to consider Cloud Spanner for your next projectLearn about the cloud database Spanner, which brings massive scale and strong consistency to your applications. Here’s why to consider al…Read Article
Quelle: Google Cloud Platform