Cloud Computing Köln - Seite 2312 von 7051 - Neues zu Cloud Computing, Internet of Things und Technologien

As you move data to the cloud, you face the important question of how to verifiably protect data from unauthorized access without limiting your options for storage and processing. Using public cloud services requires you to place inherent trust in your cloud provider, which can be uncomfortable for your most sensitive data and workloads. On Google Cloud Platform, you can use solutions such as Cloud External Key Manager (EKM)when encrypting data-at-rest to store and manage keys outside of Google’s infrastructure and Confidential Computing to encrypt data-in-use with keys that remain resident in the processor and unavailable to Google. However, while these solutions can reduce the level of implicit trust surrounding data at-rest or in-use, you still need to trust the cloud provider when data transitions from one state to another, or when the data is in-transit. So how do you deal with these challenges? At Cloud Next 2021, we announced a first of its kind solution that provides customers with ubiquitous data encryption which delivers unified control over data at-rest, in-use, and in-transit, all with keys that are under your control. With ubiquitous data encryption:You control the access to your data regardless of whether it’s on storage, in memory, or in flightYou can take full advantage of compute and storage power of GCPYou can reduce your level of implicit trust in GoogleTo build this solution, we leveraged Google Cloud’s confidential computing and Google Cloud EKM, working with partners, including Thales, to ensure that you can continue to use your existing EKM setup. In doing so, we made it possible to seamlessly encrypt your data as they are sent to the cloud, using your external key management solution, in a way that only a confidential VMcan decrypt and compute on it. In order to make sure the key can only be used in a confidential environment, we leverage Confidential VM’s attestation feature. How to setup and use ubiquitous data encryptionThe workflow to set up and use this capability is designed to be simple:Start by creating an encryption key outside GCP using your current external key management solution (for this solution, we currently support Thales Ciphertrust, with more EKM partner integrations to come)Grant access to your EKM encryption keys to the Confidential VMservice Use the gsutil tool to upload your data to Google Cloud Storage (GCS) using our lib. This will seamlessly encrypt your data using the key generated in Step 1.In your application running in a confidential VM, use gsutil to download the GCS data using our lib. This will seamlessly decrypt your data without revealing the key outside the confidential VM.If the application tries to access the GCS data on a non-confidential VM, it will fail when attempting to decrypt the data.How ubiquitous data encryption helps reduce implicit trustAdvanced configuration optionsYou can also add additional safeguards and optionally require more than one party to authorize access to your encryption key: for example, you can require a Cloud KMS key, in addition to your on-prem encryption key, to be present for every decryption operation. This provides even more control over the key access model, because it splits the ability to encrypt and decrypt across multiple parties.Customers handling highly regulated financial services data have started seeing immediate results from this integration between Confidential Computing and Cloud EKM: “Google’s new ubiquitous data encryption capabilities will allow us to bring more of our data and workloads to the cloud. Being able to encrypt data at-rest, in-use, and in-transit with a key that we control allows us to continue to meet our strict data security standards while being able to take advantage of the powerful storage and compute capabilities of Google Cloud.”– Jörn-Marc Schmidt, Vice President, Cryptography Engineering & Solutions, Deutsche BankTake the next stepIn summary, this new ubiquitous data encryption solution can help reduce your implicit trust in Google Cloud so you can bring even more of your sensitive data to GCP. Please see our documentation for more information and reach out via this form to get started or with comments or questions.Related ArticleIntroducing Google Cloud Confidential Computing with Confidential VMsGoogle Cloud will now offer the ability to encrypt data in use, while it’s being processed in a Google data center.Read Article
Quelle: Google Cloud Platform

20. Oktober 2021

da Agency

SenseData’s journey with Google Cloud’s managed database services

Editor’s note: Learn how SenseData helps Brazilian companies embrace customer success, rapidly evolving their products on Google Cloud and Google managed database services. SenseData has grown from a handful of customers to 140, manages multi-terabytes of data, and is now moving to Google BigQuery for data warehousing.SenseData is a customer success company, and we have one product. We gather information from our clients’ systems and aggregate it all in a single platform that they use to make smarter decisions about their business. Some clients want to increase sales, some want to reduce churn, and others want to see a more comprehensive picture of their customers.Our customer base has grown very quickly and so has the data our platform collects, manages, and makes consumable. Just 5 years ago, we had a minimum viable customer success product (MVP). Brazilian B2B customers don’t have a “default” software stack, so they use a mishmash of systems and software to manage the customer relationship. Our objective was to integrate the data from all these different systems. We were on the cloud for that reason, and our goal was to be cloud-agnostic, using open-source software like MySQL and other tools to manage data. Over time, our outlook has changed. It all started when we participated in the first Google Campus Residency for Startups. The residency introduced us to Google Cloud Platform and Google managed database services. After burning through our credits, we really haven’t looked back. We see the value of being a “Google Cloud shop.” As we have evolved, we’ve taken advantage of new managed database services from Google. We are really impressed with how Google also evolves with changes in data formats and storage. Best of all, with Cloud SQL for MySQL and PostgreSQL, and now BigQuery, we don’t have to worry about backups, restores, replicas, and everything else that database administrators must do. We can focus on using our talent to keep improving our platform.Oh, MySQL, how we’ve outgrown you: Building an ecosystem with Google servicesOur original architecture consisted of MySQL, an application server, and a cloud infrastructure from another vendor. During our campus residency, we moved to GCP and we started using Cloud SQL for MySQL because our clients’ data formats and sources were all over the map—Oracle, Microsoft SQL Server, Google Sheets, CSVs stored on other cloud infrastructure, and systems with only VPN access to data sources. As our clients grew, the fact that our platform was written so that all the customer data was completely separated sent us beyond the boundaries of MySQL, and we moved to PostreSQL through Cloud SQL. Some indexes and queries were not performing well for us in MySQL. In PostgreSQL, by doing nothing differently and with the same indexes and queries, the performance was consistently better. We have an ORM tool (SQLAlchemy) on top of the database layer of our app. So, it was very easy to migrate from MySQL to PostgreSQL.At the same time, we moved to Kubernetes with Google Kubernetes Engine. The result of that combination was an ecosystem that could accommodate various technical needs. For example, to build a firewall, we could start Kubernetes and create an egress rule that easily handled load balancing. Each customer has an external address using the same external IP, and inside Kubernetes, host rules can pinpoint and choose addresses.This ecosystem represented a big turning point for us. We had not planned on putting all our tech and data eggs in one basket, but our experience showed us the importance of having a first-rate database and managed database services—the lower latency, the support, and everything that comes with it. We quickly came to understand how Google can help companies during the growth process. We decided to take advantage of other GCP resources because Google makes it easy to access their first-rate services. Like two peas in a pod: Offering clients peace of mind with Google Cloud SQLAny client of a company like SenseData that has a cloud offering or uses cloud storage and data management is going to be concerned about its data security and whether other customers might get access to it. SenseData uses Google services to ensure that their data stays separate. On GCP with Cloud SQL for PostgreSQL, we have single tenancy per database, and multiple customer databases per instance. In other words, the instance is shared, but the logical database is not. We also have custom data that is JSON-B, which describes data binding relationships between JSON documents and hypermedia. If one of our clients is a SaaS company that sells consulting services, e-commerce storefronts, and a mobile app that calls a taxi service, we can easily perform joins of that custom data with all the various types of data collected by the client. Then we can deliver metrics and calculations that meet their needs.Separation of data is a piece of application layer cakeFor logical separation at the application layer, we use a cluster that has 13 nodes with 4 CPUs and 15 GB RAM. Inside the cluster, the logical separation uses namespaces. The onboarding and production are inside their own namespaces. Inside the config map of Kubernetes, there’s a patch that directs the customer to the database specific to that customer.In other words, the pod is the deployment method. The config map tells the pod to answer to the specific database. The service view has a named port or even a specific app selector that it serves. The egress rule of the host indicates the domains that go to each server and the port. This method allows us to have a different service for each customer. The BigQuery idea: Data warehousing in the cloud to help customers meet KPIsRecently we started to work with BigQuery. We made the decision to deploy because we were migrating the analytics from another vendor to Looker and wanted to improve performance and address the KPI needs of our biggest clients. These clients have a million customers, and the performance of some of their KPIs was not optimal. Many are ecommerce customers who want to track product sales against KPIs. For each product, they must look at historic sales data, hunt for specific SKUs, try to determine when the product was last purchased, and so on. Imagine doing that for multiple products concurrently. This gets very complicated in PostgreSQL BigQuery offers us a faster and easier way to address performance and increase scalability. All our calculations will migrate over to BigQuery. Once all the data is aggregated, it can go back to PostgreSQL. We use a Python client to get it from BigQuery to Cloud SQL. This evolution of Cloud SQL and data warehousing is impressive. It gives us the freedom to try new configurations and data management techniques. Two years down the road from now, we’re sure that if we have to change how we handle customer data, there will be an evolution of Cloud SQL or some other Google service that will help us make the switch. One isn’t the loneliest number: Cloud SQL makes database management easyCloud SQL and Google Cloud Platform help us by providing all the complicated database management services, plus observation, monitoring, and more. As a result, the SenseData infrastructure has been managed by just one person for about 6 years. Even though we have grown to 140 customers with terabytes of data, it’s still mostly a one-person job. How is this possible? The answer is simple. We don’t have to deal with backups, maintenance downtime, and resolving replication issues. Cloud SQL serves up everything for us instead. We don’t have to staff a team that includes a DBA, someone to manage networking, someone to administer VMs, and so on. That’s a big value for us. If we had stuck to our original plan to find cloud solutions no matter the vendor, we might not be able to stay so lean. The database services managed by Google, along with GCP and GKE, really make a difference. Learn more about SenseData and Cloud SQL. You can also take Cloud SQL for a spin with our free trial, And be sure to check out our recent blog on how Cloud SQL makes data protection easy.
Quelle: Google Cloud Platform

20. Oktober 2021

da Agency

Functions, events, triggers oh my! How to build event-driven app

Today, many developers want to design applications using an event-driven architecture (EDA). This software architecture paradigm promotes the production, detection, consumption of, and reaction to events. It’s an architectural pattern best applied to design and implement applications and systems that transmit events among loosely-coupled software components and services. You can build event-driven applications using Cloud Functions, our Functions-as-a-Service (FaaS) product, leveraging events and triggers. Events are all kinds of activities or things like changes to data in a database, files added to a storage system, or a new virtual machine instance being created etc., that happens within your cloud environment that you might want to take action on. You can evoke a function when any events occur via a trigger. Cloud Functions offers scalable pay-as-you-go functions as a service (FaaS) that can run your code with zero server management, and can work in events and triggers.This blog explains how you can use Cloud Functions, events, and triggers together to help build scalable event-driven architecture. Using an example of a recent news event, we’ll explain the architecture and the step-by-step process of building an entity-sentiment metadata repository for all known entities (such as person, location, organization, event, consumer goods, etc.) using Cloud Functions, events, triggers, and the Natural Language API. You can then use this repository to determine the sentiment (positive or negative) expressed about different entities within the news. For example, Google Cloud Next ‘21 took place October 12-14, 2021, where there were discussions about a broad range of Google Cloud products. Understanding how these products were received by various media outlets or developer blogs can be valuable information to understand the overall effectiveness of the event.Understanding Cloud Functions, events, and triggersBefore we go into our example, let’s review the various components of the architecture.Events are all kinds of activities or things like changes to data in a database, files added to a storage system, or a new virtual machine instance being created etc., that happen within a cloud environment that might require reactive action. Cloud Functions supports events from the following providers:HTTPCloud StorageCloud Pub/SubCloud FirestoreFirebase (Realtime Database, Cloud Storage, Analytics, Auth)Cloud LoggingFor example, a new message published to a Cloud Pub/Sub topic or a change to a Cloud Storage bucket can generate an event which can trigger an action. An event can result in a change to the application state. For example, when a customer purchases a product, the product state changes from “available” to “sold”. In addition to these primary event sources, many other sources may provide messages via Pub/Sub, such as Cloud Build notifications, or Cloud Scheduler Jobs.There are two distinct types of Cloud Functions: HTTP functions, and event-driven functions. Further, event-driven functions can be either background functions or CloudEvent functions, depending on which Cloud Functions runtime they are written for.HTTP functionsHTTP functions can be invoked from standard HTTP requests. Clients sending these HTTP requests wait for the response synchronously, and HTTP functions can support handling of common HTTP request methods like GET, PUT, POST, DELETE, and OPTIONS. Event-driven functionsCloud Functions can use event-driven functions to handle events from your Cloud infrastructure, such as messages on a Pub/Sub topic, or changes in a Cloud Storage bucket. Cloud Functions supports two sub-types of event-driven functions:Background functions: Event-driven functions written for the Node.js, Python, Go, and Java Cloud Functions runtimes are known as background functions. See Writing Background Functions for more details.CloudEvent functions: Event-driven functions written for the .NET, Ruby, and PHP runtimes are known as CloudEvent functions. See Writing CloudEvent Functions for more details.Event-driven functions such as Background Functions and CloudEvent Functions can be used when Cloud Functions are invoked indirectly in response to an event, such as a message on a Pub/Sub topic, a change in a Cloud Storage bucket, or a Firebase event. CloudEvent functions are conceptually similar to background functions. The principal difference between the two types of functions is that CloudEvent functions use an industry-standard event format known as CloudEvents. Another difference is that Cloud Functions itself can invoke CloudEvent functions using HTTP requests, which can be reproduced in other compute platforms. Taken together, these differences can enable CloudEvent functions to be moved seamlessly between compute platforms.We will use CloudEvent functions throughout this blog to build a demo that explains how to build event-driven architectures using Cloud Functions.Understanding event-driven architecture (EDA)Event-driven architecture (EDA) is a software architecture paradigm promoting the production, detection, consumption of, and reaction to events. This architectural pattern can be best applied to design and implement applications and systems that transmit events among loosely coupled software components and services. Putting everything together with an exampleThis example focuses on creating an event-driven architecture to build a service that analyzes RSS feeds content using managed services like Cloud Functions (event-driven), Cloud Scheduler, and the Google Cloud Machine Learning APIs.Goal With this example, we want to collect, analyze, and store RSS feeds and their Entity Sentiment Analysis at a given interval every day, using Machine Learning APIs. Entity Sentiment Analysis combines both entity analysis and sentiment analysis and attempts to determine the sentiment (positive or negative) expressed about entities within the text. This can be useful when:You want to identify all the entities like product, company, or person mentioned in any blog or news article.You want to discover sentiment expressed about the entity. With these kinds of use cases, data engineering teams can look at recent trends like entities mentioned the most in news, and their sentiment to understand how they are perceived in the market. Architecture:Click to enlargeThe above architecture illustrates the workflow of how an RSS feed can be analyzed as soon as it is collected. Here is the logic: Step 1: One RSS feed link will contain one or many URLs. Write a function to parse these URLs from RSS feed links and store them in a queue. Step 2: Write a function that reads URLs from the queue in Step 1, one by one, and download their web page contents. Store all web page contents in persistent storage.Step 3: Finally, write another function that reads web page contents from stored persistent storage in Step 2, analyze them, and write the results to another persistent storage target. Here is a step-by-step process to build the above steps using Google Cloud managed services: Step 1: Cloud Scheduler writes a sample message to Pub/Sub. This is just to invoke a cloud function that is configured to start as soon as this event occurs.A configuration like above writes a message to a Pub/Sub topic at 5 AM PDT every day. Step 2: An RSS feed contains one or many web page URLs. A message written to Pub/Sub in Step 1 triggers a cloud function that downloads the contents of the RSS feed and parses a list of web page URLs from its content. These URLs are then written to another Pub/Sub topic for further processing.With a configuration such as the one above, Cloud Functions starts executing at 5AM when the scheduler event arrives via the Pub/Sub trigger.Here is sample code written in C#, .NET core 3.1, to parse RSS feed(s) and collect web page URLs from it.Function PublishMessageWithRetrySettingsAsync can be found here.Step 3: A URL entry written to Pub/Sub in Step 2 triggers a new cloud function that downloads web page text and writes the text to a Cloud Storage bucket.With a configuration like the one above, a cloud function will execute for each web page URL collected from the RSS feed and written to Pub/Sub in Step 2. Here is sample code written in Python 3.9 to download web page text and store it in Cloud Storage. BeautifulSoup library is used to parse web pages and extract text.Step 3 will result in web page text stored in Cloud Storage as shown below:Step 4:A new file creation event in Step 3 triggers a new cloud function that does entity sentiment analysis on the web page text and writes results to another Cloud Storage bucket.With a configuration like this, a cloud function will execute for each web page text downloaded and stored in a Cloud Storage bucket.Here is sample code written in Nodejs 14 to read data from a Cloud Storage bucket, parse the download web page text, and store it in Cloud Storage.analyzeEntitySentiment function does Entity Sentiment Analysis. Natural Language API has several methods for performing analysis and annotation on text data. Here are examples of how to perform analysis on text data using Natural Language API.The analysis data collected can be used by data engineering teams to find out answers to questions like: ‘Which entities (person, product etc) were discussed the most in recent times?’ and ‘What is the sentiment associated with these entities?’ These questions can help in understanding the overall effectiveness of existing marketing campaigns or latest trends, to help create better marketing or public relations campaigns. Resources:All logs from above services will be available in Cloud Logging. Cloud Functions can be tested and debugged locally. All services are loosely coupled. Vertex AI can be used to analyze the meta data collected at the end..Summary: Cloud Functions can be powerful services that can be used to build event-driven applications with minimal effort. The above walkthrough demonstrates Cloud Functions events, trigger integrations with different Google Cloud services, and its multi-language support. To get started, check out the references below.References:Calling Cloud FunctionsTesting Event-Driven FunctionsCloud Functions Local DevelopmentDeploying Cloud FunctionsRelated ArticleTips for writing and deploying Node.js apps on Cloud FunctionsFollow these tips for writing performant, observable, Node.js applications that run on Cloud Functions.Read Article
Quelle: Google Cloud Platform