Cloud Computing Köln - Seite 2333 von 7053 - Neues zu Cloud Computing, Internet of Things und Technologien

Editor’s note: Today we are hearing from Ben Hizak, co-founder of Cherre. He shares Cherre’s approach to consolidating multiple data sets into one data warehouse to power BI and analytics for their customers. He explains how the company uses Google Cloud and Cloud SQL to bring real estate datasets together and help customers make better data-driven decisions.Cherre’s quest to transform real estate investing and underwriting into a science requires us to connect thousands of datasets and make them accessible for investment, management and underwriting decisions. By delivering actionable insights based on fresh data, our clients can answer questions that they could never answer before. Each of our clients is interested in a different combination of public and paid datasets, which is then to be combined with that client’s private data. All of this has to be done in a secure, scalable and repeatable manner. We process this data in BigQuery and then store it in Cloud SQL for PostgreSQL, a single source of truth that our clients can use to train machine learning models, consume via an API which runs on Google Kubernetes Engine (GKE), or visualize using BI tools like Looker.A sample combination of public and paid datasets displayed in LookerBefore Cherre, this kind of consolidated data work was simply not available in the real estate industry. Before Cherre, datasets were never in one place, and each dataset contained different properties with different attributes. For example, data on transactions from one vendor and data on securitized debt from another lived in completely different worlds prior to Cherre. And private data vendors never before let raw data leave their systems. Building that trust so those vendors now allow their data onto our platform has been a unique honor and responsibility.Our clients are large organizations such as large real estate funds, banks, and insurance companies, which manage billions of dollars each. For them, the difference between a right decision and a wrong decision can make the difference between a successful business and a failed business. Their need for speed is insatiable. We enable our clients to make better decisions faster by putting clean, connected data at their fingertips. Cherre customers can evaluate opportunities and trends faster and more accurately, and efficiently manage their portfolios, while saving millions of dollars in manual data collection and analytics costs.Consolidating data to turn investing into a scienceConsolidating thousands of datasets and millions of data points requires both technical capabilities and human sensitivities. We serve as a center point for two sets of stakeholders — for large institutions that require the highest levels of security and for providers of proprietary data that stake their reputations on our ability to deliver their data on comparable or better infrastructure than they provide to their own clients. Our reputation relies on the strength of our technical infrastructure. Why PostgresWhen choosing a database to back our API, Postgres was the natural solution. It is feature-rich, fast, and has a strong community. Postgres also has strong geospatial abilities, which is an absolute requirement for clients in our industry.Postgres and GraphQL combine well to serve different use cases in a ‘wide data’ scenarioTo understand our stack, it helps to understand our data. Most data engineers are working in a “thin and fast” scenario; a small number of fields have to be processed very rapidly. Our case is the opposite. Real estate data is “wide and slow(er)”; our input consists of many thousands of tables. We reduce these to something more manageable, just over 100 tables and thousands of columns for certain clients, which they then can consume via our API and visualize in Looker. Some of our clients even run their calculations directly on our infrastructureREST doesn’t cut itEvery client asks different questions from their data. While some might care about specifics in a certain building — number of tenants, average occupancy or length of occupancy — others will be more interested in general information, such as fire safety codes. . Our clients expect to be able to filter on any column and traverse datasets without having to understand how they join together. In the classic world of REST APIs, clients would have to pull all the objects and know how to assemble them on the client’s side. However, that would be antithetical to our purpose — to make data ingestions as simple as possible. Data wrangling is our job. We had to look for a different approach, which required a very different architecture.GraphQL—a more modern type of API — comes in handy. Not only does it prevent over-fetching and under-fetching of data, but it also joins tables and traverses relationships (one-to-many, many-to-one, many-to-many) in a way that is seamless for the client. Real estate data can be difficult to wrangle in that it comes from thousands of authorities, in various formats and qualities. The data is built on various objects, often lacking a consistent key or identical data point, which makes for complex and very subtle joins. Cherre wanted to take on that burden, and not pass it to our clients. This is where GraphQL shines. Employing GraphQL lets Cherre focus on the minutiae so that our clients can focus on the big picture.GraphQL allows each of our clients to query only the objects and fields that they care about, and have them be joined, filtered and aggregated on the server side, yielding a response that is easy to consume. Take the following example in which we conduct a one-to-many join: buildings with their apartments (units), and filtering on both entities:In this example, we can see a many-to-one query, because each apartment belongs to only one building:And in this example we can see a one-to-many query that pulls apartments based on the building they belong to (in this case buildings that were built after 2010 and have a doorman).Disposable tables – Because you can’t always get it right the first timeSince Cherre is working on thousands of datasets and tens of thousands of input fields, the chances that we will misunderstand and miscalculate a field are so high as to be a certainty. To preempt this, we wanted to be able to rebuild our output from scratch. We were looking for something like “disposable infrastructure” but for datasets. We decided that Cherre would not operate on existing data and instead, would always recompute what we need from scratch. This approach is often called immutable tables. If there’s a mistake, you can fix the code and rerun — voila, mistake corrected.To achieve this, Cherre builds immutable tables that live for a short time. We then funnel some or all of the client traffic to the new table. We keep the old table in case we need to do a zero-downtime rollback, but ultimately delete the old table to conserve space. We delete data with full confidence that we can rebuild it from scratch, by running the appropriate version of the code.Immutable tables are more costly and take longer to compute, but they allow Cherre to iterate more quickly with the confidence that nothing is ever lost. It also allows us to compare two versions of our output.Why Cloud SQLAs you can see, our database needs are not trivial, which is why we chose Cloud SQL for PostgreSQL. We need to control our costs, and we do it using Cloud SQL committed-use discounts (CUDs and) and a cost monitoring tool from DoIT international. We also need high availability, read replicas, and backup management. Cloud SQL does all that for us so we don’t have to spend development time on it. It also saves us time we’d have otherwise had to spend patching and upgrading the software. We use Google Cloud projects to implement robust information security, so having a database that sits well within the Google projects system enables us to be consistent across the organization with the way we manage permissions and security. And we leverage Config Connector, which allows us to provision Google Cloud resources through Kubernetes. Since we want to use infrastructure-as-code for everything and we are using Google Kubernetes Engine (GKE), it was natural for us to use Config Connector for our operations.Looking aheadWe’re extremely proud of what we’ve built, but there’s always room for improvement. As technology continues to evolve and as we continue to develop, we’re seeking out new opportunities to become more efficient. We’re also looking to leverage deeper database optimization, which is much easier to do now, thanks to Query Insights. More information is becoming available every day. The buyers, sellers and other real estate professionals that can make smarter decisions based on that data will gain an edge in the marketplace. We’re committed to making the connections they need to get ahead. Read more about Cloud SQL, Cloud SQL insights and Looker in our documentation or take the Cloud SQL insights code lab.
Quelle: Google Cloud Platform

9. Oktober 2021

da Agency

Next ’21 is on: Top five things to do at Google Cloud Next

Google Cloud Next is less than a week away. Register now to join us October 12–14, 2021, for our no-cost, three-day flagship event to discover live and on-demand content that allows you to connect with experts, see the latest tech in action, and sharpen your skills around all things cloud.To make the most of Next ’21, be sure to check out as many of these top five things to do as possible:Kick off the first two days with a keynote. On October 12 at 9 AM PT, get your daily dose of cloud insights from leaders like Google and Alphabet CEO Sundar Pichai and Google Cloud CEO Thomas Kurian. On day two, be sure to catch the Developer keynote with Google Cloud SVP of Technical Infrastructure Urs Hölzle where he reveals our vision for the top three cloud technology trends for the next decade.Customize your content. Search and filter the catalog to find your favorite sessions, build your own playlists, and share them on Twitter, LinkedIn, and more – don’t forget to tag your posts with #GoogleCloudNext. Need some inspiration? Check out our personal playlists like Advanced app dev for developers! from Director of Product Management Aparna Sinha, Developer Advocate Stephanie Wong’s Looking for more technical data sessions? – and my own, Marketers who love data, with a side of collaboration. Or take a quick quiz and we’ll build a suggested playlist based on your availability and interests.Engage with experts. Get help solving your toughest business challenges. Participate in expert-led live Q&As with topics on data cloud, open cloud infrastructure, and more than 40 online interactive labs. Meet your peers in over 20 community conversations covering everything from building with Google Workspace to getting started with Vertex AI. Or chat with the Women Techmakers team, the authors of the 2021 State of DevOps report, or Google developers from APAC, EMEA and India.Dive in to demos. See how forward-thinking businesses use Google Cloud with our featured demos. Chess.com drives the future of the 1,500-year-old game by scaling to meet the demands of 11 million live matches per day with in-game, real-time player feedback. And international automaker Renault built its industry data management platform on Google Cloud to optimize production and operations worldwide. Chat with presenters at our live demos as you learn what’s new with Google Workspace, how to go beyond the basics of zero trust with BeyondCorp Enterprise, and more. Take action with Diversity, Equity & Inclusion (DEI) sessions. Learn how you can champion diversity, build community, and drive change with data-driven DEI discussions and sessions that explore how to turn core DEI values into real-world impact. Learn how healthcare providers enable accessible healthcare through technology to predict outcomes and craft personalized prevention plans. Or join Jim Hogan, principal innovation strategist at Google Cloud, in conversation with one of the founders of the internet, Vint Cerf, as they discuss how Cerf’s personal experiences with disability led to his belief that accessibility is the cornerstone of innovation.Be sure to tune in to Next ’21 on October 12–14 to get informed, be inspired, and expand your expertise.Related ArticleNext OnAir as it happens: All the announcements in one placeWe’ll be sharing lots of news, updates, and learning opportunities throughout Next OnAir. Check back here to see a running list of what’s…Read Article
Quelle: Google Cloud Platform

9. Oktober 2021

da Agency

BigQuery migrations made easy

Migrations are not easy: they take time, energy and effort to make them successful. BigQuery makes it easier with customizable tools and years of expertise to help out with your journey to the cloud. In line with our commitment to providing an open and flexible platform, we have built these migration tools following an open approach which enables flexibility and choice for you and your partners when migrating to BigQuery. Comprehensive solution for migration to BigQueryToday, we are announcing the preview of BigQuery Migration Service, a set of free-to-use tools to help you with your end-to-end migration needs. This service speeds up Teradata to BigQuery migrations with tooling covering migration planning, data transfer, automated SQL/script conversion and data verification. Support for additional data warehouses is coming soon.Assessment: Plan and manage migration risks and costsWhen we work with customers and partners on migrations, the most important step is to understand their ecosystem, requirements and business goals. We use this information to create a custom migration plan to help prepare and execute migrations. Over and over again, we have seen that identifying and addressing migration complexities ahead of time leads to reduced TCO and lower risk migrations.To that end, we are excited to announce private preview of our automated assessment tool as part of the BigQuery Migration Service. Assessment leverages our many years of experience helping some of the large organizations in the world modernize with BigQuery. It provides an easy and automated way to collect statistics from your legacy warehouse and generates a state analysis report consisting of: List of database objects, data I/O patterns and dependenciesAutomated query translation coverage and resultsQuery-to-Object mapping (e.g., which tables, views, functions it uses)User-to-Table mapping (e.g., which users access which tables)Table correlations (e.g., tables which are often joined or sub queried)List of BI/ETL tools in useThe summary report helps you efficiently prioritize and have a clear understanding of all the components and the amount of work required to execute a migration. In addition, all the underlying assessment data is made available within a BigQuery dataset for complete customization and ad-hoc analysis for you and your migration partners. SQL translation: Reduce manual effort, time and errors One of the hardest pieces of a data warehouse migration is modernizing legacy business logic, such as SQL queries, scripts and stored procedures. This process normally involves substantial manual query rewrites and verifications, which is time consuming and error prone. Today, we are excited to announce the public preview of batch and interactive SQL translation which helps automate much of this process, thus speeding up your path to a successful migration. Batch and interactive SQL translation provides fast, semantically correct and human readable translations of legacy objects with no ongoing dependencies post migration. It supports a broad range of Teradata artifacts including DML, DDL and BTEQ. Translations can be run in batch mode or ad-hoc directly from the BigQuery SQL workspace. Early users of SQL translation saw ~95% successful translations on 10M+ queries, leaving only ~5% of queries for manual review with their migration partners. Interactive SQL translation provides a split view within the BigQuery SQL editor. Users can type in SQL queries in non-BigQuery dialects and view the translated BigQuery SQL immediately. Interactive SQL translation provides users with a live, real time SQL translation tool which allows users to self-serve translation of their queries in parallel with a centralized large-scale SQL migration effort. This not only reduces the time and effort for analysts to migrate their queries, but also increases how quickly they learn to leverage the modern capabilities of BigQuery.Data Validation: Verify correctness of dataData validation is a crucial step in data warehouse migration projects. It compares structured and semi-structured data from the source and target to confirm data and logic has been moved correctly. The GCP Data Validation Tool (DVT) is an open sourced CLI tool that leverages open-source frameworks. It offers customized multi-level validation functions to compare source and target tables on the table level, column level, and row level. It is also flexible, meaning that new validation rules can easily be plugged in as you see fit. Furthermore, to facilitate automation, orchestration and scheduling, it can also be integrated with Cloud Functions, Cloud Run, and Composer for recurring validation.Customize BigQuery Migration Service to your needsWith BigQuery Migration Service we are fast tracking, simplifying and de-risking your migration so that you can modernize your data warehouse with BigQuery, a truly serverless and modern data warehouse, with confidence. We are starting with Teradata migration capabilities, but will add support for additional data warehouses soon. We have built end-to-end tools with openness top-of-mind which you and your migration partner of choice can customize to help ensure a successful migration. From assessment which streamlines metrics collection and exposes the raw dataset for full customization to our open source Data Validation Tool, where you and your partner can add custom validation logic, we are committed to giving you migrations tools you can customize to your unique needs. Our tools are freely available to help speed up your migration. If you would like to leverage our tools for an upcoming proof-of-concept or migration, reach out to your favorite GCP partner, your GCP sales rep or check out our documentation. We look forward to partnering with you on your journey to the cloud.
Quelle: Google Cloud Platform

9. Oktober 2021

da Agency

Analyzing Twitter sentiment with new Workflows processing capabilities

The Workflows team recently announced the general availability of iteration syntax and connectors! Iteration syntax supports easier creation and better readability of workflows that process many items. You can use a for loop to iterate through a collection of data in a list or map, and keep track of the current index. If you have a specific range of numeric values to iterate through, you can also use range-based iteration. Click to enlargeConnectors have been in preview since January. Think of connectors like client libraries for workflows to use other services. They handle authentication, request formats, retries, and waiting for long-running operations to complete. Check out our previous blog post for more details on connectors. Since January, the number of available connectors has increased from 5 to 20.The combination of iteration syntax and connectors enables you to implement robust batch processing use cases. Let’s take a look at a concrete sample. In this example, you will create a workflow to analyze sentiments of the latest tweets for a Twitter handle. You will be using the Cloud Natural Language API connector and iteration syntax.APIs for Twitter sentiment analysisThe workflow will use the Twitter API and Natural Language API. Let’s take a closer look at them.Twitter API To use the Twitter API, you’ll need a developer account. Once you have the account, you need to create an app and get a bearer token to use in your API calls. Twitter has an API to search for Tweets. Here’s an example to get 100 Tweets from the @GoogleCloudTech handle using the Twitter search API:Natural Language APINatural Language API uses machine learning to reveal the structure and meaning of text. It has methods such as sentiment analysis, entity analysis, syntactic analysis, and more. In this example, you will use sentiment analysis. Sentiment analysis inspects the given text and identifies the prevailing emotional attitude within the text, especially to characterize a writer’s attitude as positive, negative, or neutral.You can see a sample sentiment analysis response here. You will use the score of documentSentiment to identify the sentiment of each post. Scores range between -1.0 (negative) and 1.0 (positive) and correspond to the overall emotional leaning of the text. You will also calculate the average and minimum sentiment score of all processed tweets.Define the workflowLet’s start building the workflow in a workflow.yaml file.In the init step, read the bearer token, Twitter handle, and max results for the Twitter API as runtime arguments. Also initialize some sentiment analysis related variables:In the searchTweets step, fetch tweets using the Twitter API:In the processPosts step, analyze each tweet and keep track of the sentiment scores. Notice how each tweet is analyzed using the new for-in iteration syntax with its access to the current index.Under the processPosts step, there are multiple substeps. The analyzeSentiment step uses the Language API connector to analyze the text of a tweet and the next two steps calculate the total sentiment and keep track of the minimum sentiment score and index:Once outside the processPosts step, calculate the average sentiment score, and then log and return the resultsDeploy and execute the workflowTo try out the workflow, let’s deploy and execute it.Deploy the workflow:Execute the workflow (don’t forget to pass in your own bearer token):After a minute or so, you should see the see the result with sentiment scores:NextThanks to the iteration syntax and connectors, we were able to read and analyze Tweets in an intuitive and robust workflow with no code. Please reach out to @meteatamel and krisabraun@ for questions and feedback.Twitter sentiment analysis on GitHub.Share feedback, interesting use cases and customer requestsRelated ArticleIntroducing Workflows callbacksIntroducing Workflows callbacks. Thanks to callbacks, you can put a human being or autonomous system into the loop. If your processes req…Read Article
Quelle: Google Cloud Platform