Cloud Computing Köln - Seite 2389 von 7059 - Neues zu Cloud Computing, Internet of Things und Technologien

OverviewCloud Spanner is a fully managed relational database built for scale with strong consistency and up to 99.999% availability. Key features include the following: ACID transactions, SQL queries (ANSI 2011 with extensions) and global scale.Automatic sharding – optimizes performance by automatically sharding the data based on request load and size of the data.Fully managed – Synchronous replication and maintenance are handled automatically.Flexible configurations- Depending on the workload, Cloud Spanner instances can be provisioned as either regional or multi-regional (spanning one continent or three continents).Online schema Changes – Cloud Spanner users can make a schema change, whether it’s adding a column or adding an index, while serving traffic with zero downtime.In this blog post, we will talk about how to deploy a sample application on Cloud Run with a Cloud Spanner backend using Terraform templates. We will also learn how to manage a production-grade Cloud Spanner instance using Terraform, by starting with a small instance and scaling up by adding more nodes or processing units.Terraform for Cloud Spanner What is Terraform?Terraform is a popular open-source infrastructure-as-code tool developed by HashiCorp, that provides a consistent CLI to manage hundreds of cloud services. It codifies cloud APIs into declarative configuration files. Terraform allows you to declare an entire GCP environment as code, which can then be version-controlled.Benefits of using Terraform with Cloud SpannerTerraform is very useful for provisioning scalable Cloud Spanner instances and databases in real-world, production environments. It allows for easier configuration management and version control and enables repeatability across regions and projects.Organizations using Terraform to manage their cloud infrastructure today can easily include Cloud Spanner into their existing infrastructure-as-code framework. The Cloud Spanner with Terraform codelab offers an excellent introduction to the provisioning of instances, creating and modifying databases, and scaling a Cloud Spanner instance with more nodes. The codelab is a great way to get started. In the next few paragraphs, we discuss a few details that go beyond what we cover in the codelab.Specifying a regional configurationCloud Spanner instances can be launched either regionally or in a multi-region. Take a look at Instances in the official documentation (under “regional configurations” and “multi-region configurations”) for a list of the configuration options available.For regional instances, add the “regional-” prefix to the instance names in the list below. Example: to provision an instance in Montreal, your Terraform template will have config = “regional-northamerica-northeast1″screenshot from: Google Cloud DocumentationFor multi-region instances, you can use the region name as-is from the documentation. Example – asia1 for a one-continent instance in Asia or nam-eur-asia1 for a three-continent instance. Modifying Cloud Spanner instance propertiesNote that properties such as the amount of compute capacity (via num_nodes or processing_units), display name, and labels can be modified when making changes to an existing Cloud Spanner instance via Terraform. However, changing the instance name (the instance identifier on the GCP console) will result in the cluster being destroyed and recreated.As mentioned above, compute capacity (a.k.a. the instance size) can also be defined in terms of processing_units. For example, the compute capacity defined below would be 1/10th of a node, while a value of 1000 would be one full node.Creating databases and executing DDL commandsFinally, Cloud Spanner Terraform resources also support creating databases and executing DDL as part of the template. The DDL support in the Terraform resource is a useful feature for initializing a database schema for smaller applications. Limitations of Terraform with Cloud SpannerManaging database schemasDatabase schemas tend to change periodically. Terraform has very limited support database schema updates – changes to the DDL, particularly those that are not append-only, will require dropping and re-creating the database. Therefore, we don’t recommend using Terraform to manage the schemas of Cloud Spanner databases. Instead, we recommend using a schema versioning tool like Liquibase. A Liquibase extension with Cloud Spanner support was released recently under the Cloud Spanner Ecosystem. Here is the official documentation. Deploying a sample appFor demonstration purposes, we’re going to use a sample stock price visualization app called OmegaTrade. This application stores the stock prices in Cloud Spanner and renders visualizations using Google Charts. To learn more about the app and its integration with Cloud Spanner, see this blog post. Now, to the fun part! We will deploy this application to Cloud Run using Terraform templates. We chose Cloud Run because it abstracts away infrastructure management and scales up or down automatically almost instantaneously depending on traffic. Let’s get started!As prerequisites, please ensure that you have:Access to a new or existing GCP project with one of the sets of roles listed below:OwnerEditor + Cloud Run Admin + Storage Admin Cloud Run Admin + Service Usage Admin + Cloud Spanner Admin + Storage AdminEnabled billing on the above GCP project.Installed and initialized the Google Cloud SDK.Installed and configured Docker on your machine.Installed and configured Git on your machine.NOTE – Please ensure that your permissions are not restricted by any organizational policies.We are deploying this application using Cloud Shell. If you are going through these steps on your local machine, and assuming you have already installed the Cloud SDK, you can execute the following command to authenticate.gcloud auth application-default loginChoose your Google account with access to the required GCP project and enter the Project ID when prompted. Next, we need to ensure the gcloud configuration is set up correctly. You may want to start with a new configuration by using the create command. Below, we are enabling authentication, unsetting any API endpoint URL set previously, and setting the GCP project we intend to use in the default gcloud configuration. Replace [Your-Project-ID] below with the ID of your GCP project.Before continuing further, let’s make sure our Terraform version is up to date (we need Terraform version 0.13.1 and above).If needed, you can go to the official Terraform download pageto download and install the latest version.Next, let’s enable Google Cloud APIs for Cloud Spanner, Container Registry, and Cloud Run. Note: you could also use Terraform to accomplish this, instead of running the commands manually.Now, let’s clone the repository that contains Terraform modules for Cloud Run, Cloud Spanner and GCE (for a Cloud Spanner emulator instance, which will be discussed in a future blog post). Take a look at the directory structure below. The examples folder has Cloud Spanner and Cloud Run Terraform examples, with their corresponding Terraform modules located in the modules folder. Note: By default we have defined the compute capacity in terms of number of nodes in these templates. In case you want to define your compute capacity using processing units instead of nodes, you will need to uncomment the lines of code for processing_units in each of the examples and modules, and comment out the corresponding lines for num_nodes. The two options are mutually exclusive. To find all the relevant files to modify, you can executeNote that as of the time of this writing, the smallest compute capacity available is 100 processing units (or 1/10th of a node).Launch Cloud Spanner We will now launch a single-node Cloud Spanner instance in the us-west1 region. The template also creates a database and the necessary tables for the OmegaTrade application using the DDL specified in the template.Take a look at the terraform.tfvars file to customize the compute capacity, region, and number of nodes or processing units. Enter your GCP project ID (without the square brackets), make any other changes you’d like, and save.In case you want to define your compute capacity using processing units instead of nodes, you can specify a value for spanner_processing_units instead of spanner_nodes and follow the instructions at the end of the previous section. Next, let’s initialize Terraform, and make sure that we have the correct versions of the providers installedAnalyze the execution plan,And apply the changes.Terraform will ask for your confirmation before applying.You should see the Cloud Spanner instance successfully provisioned .Click into the hamburger menu at the top left and select Spanner to verify the outcome on the GCP console. You can find the instance and database created, along with the necessary tables for the OmegaTrade application. The above example provisioned a single node instance in the us-west1 region. If you would like to launch in a different region (different instance configuration) with multiple nodes, simply edit the terraform.tfvars file and set a different instance configuration instead of spanner_config = “regional-us-west1″. Deploying the backend to Cloud RunWe are going to be deploying two different services to Cloud Run: omegatrade/frontendand omegatrade/backend.NOTE – Please ensure that your permissions are not restricted by any organizational policies, or you may run into an IAM-related issue at the apply stage later on.Now that the Cloud Spanner instance, database, and tables are in place, let’s build and deploy the backend service to Cloud Run. The frontend has a dependency on the backend URL, so we will start with the backend.In the backend folder, we will create a .env file and insert some seed data into the database we created in the previous section. We begin by setting the gcp-project-id, spanner-instance-id, and spanner-database-id to the appropriate values that we got from the GCP console (omitting the square brackets).Then, we run the following command to populate the seed data.Next, we build the image from the dockerfile and push it to GCR. We will change the commands below to reflect our GCP project ID and run them.We will now go back to the Terraform examples directory and provision the backend service of the OmegaTrade application.Like the Cloud Spanner example you have seen in the previous section, you can quickly edit the terraform.tfvars file to make changes according to your environment and deploy. Since the Terraform template adds a suffix to the instance name and DB name, you might want to get the exact instance name and database name from the GCP console. The backend_container_image_path is the same path that you used in the docker push command above.NOTE: In these templates, we follow the standard practice of using variables.tf or tfvars files to define variables and values. This is particularly useful when we have multiple resources with similar configuration, as while upgrading them the values need to be changed in only one place.Here is how my file looks with all the details except the project ID filled in.Next, let’s initialize Terraform and validate the plan.Now, we’re ready to deploy backend service. If you get an IAM-related error at this stage, it is likely because of an organizational policy of the organization that the project is hosted in. You may need to contact your organization admin or start over with a project in a different organization that does not have this restriction.Terraform will ask for your confirmation before applying. You should now see the backend service up and running.Check Cloud Run in the GCP console and locate the new service. Write down its URL. We will use it in the frontend configuration in the next section. Deploying the frontend to Cloud RunBefore we build the frontend service, we need to update the following file from the repo with the backend URL we got from the above step. Change the base URL to the backend URL noted down in the previous section.In the frontend folder, build the frontend service and push the image to GCR.Go to the Terraform example for CloudRun (frontend service).We will now provision the frontend service using the image we pushed to GCR above. Open the terraform.tfvars file and add the GCP project ID and frontend image path.Here is what the file should look like after filling in most of the details.Now, let’s deploy the frontend service.Once again, Terraform will ask for your confirmation before applying. You should now see the frontend service up and running.Check the services on the GCP console.You will now be able to go to the frontend URL and interact with the application! You can interact with existing visualizations or simulate write activity on Cloud Spanner by visiting the Manage Simulations view in the application. Choose an existing company or add a new company, and choose an interval and number of records. Scaling Cloud Spanner using TerraformTo scale the Cloud Spanner instance up or down, go back to the Cloud Spanner Terraform examples folder.We currently have a single node Cloud Spanner instance in the us-west1 region. For production environments, this configuration may not be sufficient. Scaling Cloud Spanner can be achieved using our Terraform template.Take a look at the terraform.tfvars file. We chose a regional instance with 1 node during the initial provisioning, running in the us-west1 region. We can now scale Cloud Spanner to the compute capacity necessary by running a terraform apply once again with an updated node count. An example .tfvars file to scale the instance is available as terraform.scale.tfvars as shown below. Specify the same instance ID and database name as initially used. Note that the Terraform template randomizes the names of the instance and database by adding a random suffix, but while scaling you just need to use the original names. Edit the terraform.scale.tfvars file with your project IDIn case you defined your compute capacity using processing units instead of nodes, you can follow the same steps to resize it, by specifying an updated value for spanner_processing_units instead of spanner_nodes and commenting/uncommenting the appropriate lines in the script as noted just above the Launch Cloud Spanner section earlier in this post.Apply changes to scale the instance from 1 node to 2 nodes.It may take a few seconds for the changes to take effect. Verify the changes on the GCP consoleConclusionWe have seen how easy it is to provision Cloud Spanner instances and create databases and tables using Terraform using a sample application deployment. We have also seen how to scale Cloud Spanner using Terraform after the initial deployment. Armed with this knowledge, you are ready to try out the code modules in this repository and set up your own Cloud Spanner instances with Terraform.To learn more about using Terraform with Cloud Spanner, visitThe Cloud Spanner documentationThe Terraform documentationThe Terraform with Cloud Spanner CodelabThe Google Cloud Platform Terraform repository on GitHubRelated ArticleMeasuring Cloud Spanner performance for your workloadIn this post, we will explore a middle ground to performance testing using JMeter. Performance test Cloud Spanner for a custom workload b…Read Article
Quelle: Google Cloud Platform

14. September 2021

da Agency

Chefkoch whips up handwritten recipes in the cloud with text detector

Editor’s note: When German cooking platform Chefkoch was looking to bring treasured hand-me-down recipes into the 21st century it found a scaleable, well-supported solution with Google’s data cloud. Here’s how it was cooked up. Whether it’s salad dressing or chicken soup, most households have a favorite dish passed down across the generations. These recipes are often scribbled on scraps of paper and this personal culinary heritage is heavily guarded. Recognizing the significance of handwritten or printed recipes, German cooking platform Chefkoch wanted to make it possible to quickly and easily parse, extract and digitalize these time-honored tasty morsels using Google Cloud augmented analytics and machine learning (ML) capabilities, to make it easy for users to share and access these recipes in a digital form.When considering the best way to develop the technology, Chefkoch undertook extensive market research of the global food market. It identified best practices within the industry, looked at working business models and upcoming food-tech trends, and applied this to an in-depth study of its own users’ motivations for using its platform. Finally, Chefkoch created a Kano Model that prioritized different features based on how likely they would be to satisfy its users.Chefkoch users each have their own Kochbuch (cookbook) on the platform where they can save, sort, and manage their Chefkoch recipes. On the back of its research, Chefkoch decided that this was the best place for its new proposition. It opted to develop the Kochbuch to make it possible to store any recipe within it, including offline ones. “To do this we needed to extract the text, be it handwritten or printed, and then separate the recipe title, the ingredients and the instructions,” explains Tim Adler, CTO of Chefkoch.APIs get reading recipesTo enable this, Chefkoch began assessing various text importing tools. In May 2021, it settled on Google Cloud’s ML services and Google Cloud Functions, because it offers scalable functions as a service (FaaS) with a number of APIs that enable code to be run without server management. “We screened the market for solutions in order to recognize text in scanned handwritten recipes,” says Adler. “Google’s solution convinced us, not only because we could work with the APIs and documentation easily, but also because Google’s team presented us with an impressive proof of concept with our own test-data.”Chefkoch chose to build this recipe-reading tool using the Vision and Natural Language services offered by Google Cloud because it can run across devices and be scaled cost-effectively. As seen in the diagram above, it uses the Cloud Vision API optical character recognition (OCR) tool, which is optimized for German and English text detection, to extract the text from a written or printed page. It then applies AutoML Natural Language Entity Extraction Models 1 and 2 and the Cloud Natural Language API to identify and segregate the different sections of the recipes to perfect the resulting on screen recipe, as shown below.Chefkoch worked closely with Google to perfect the solution. Our Google team initially made a demo for Chefkoch to help the team understand how everything works together, demonstrating, for example, how a dataset has to be structured to optimize the desired results after the model training. They presented a working end-to-end demo of a functioning API which takes in the image of a handwritten or printed recipe, and outputs the desired results: with the various components of the recipes cleanly extracted and separated. This offline-to-online recipe upload service is now being trialed on Chefkoch’s Kochbuch. “We are working on testing, improving, extending and producing the solution,” reveals Adler.Tweaking the recipe Results from early-stage testing are encouraging, with users rating the OCR feature with an A or B grade. In response to this feedback, small adjustments have already been made to the training of the model to get it to align with audience needs. The tool, which has been unofficially named the Handwritten Recipe Parser, can now pick up contextual spelling mistakes, for example, where a word is spelled incorrectly for the context, such as “meet” instead of meat.Cooking up users with analog to digital offeringThere are plans to expand the menu of features on Handwritten Recipe Parser too. Chefkoch is now developing a manual recipe extraction solution, where users can upload their own recipe images and add the title, ingredients and method and there are plans to enable users to amend existing Chefkoch recipes by adding their own text and written annotations. To learn more about Cloud AutoML and Vision API, visit our site.Related ArticleBusinesses realize the full value of visual data using Plainsight Vision AI on Google CloudAI startup Plainsight built its Vision AI offering on Google Cloud, enabling companies to extract accurate, actionable insights from vide…Read Article
Quelle: Google Cloud Platform

14. September 2021

da Agency

Run more workloads on Cloud Run with new CPU allocation controls

Cloud Run, Google Cloud’s serverless container platform, offers a very granular pay-per-use pricing, charging you only for CPU and memory when your app processes requests or events. By default, Cloud Run does not allocate CPU outside of request processing. For a class of workloads that expect to do background processing, this can be problematic. So today, we are excited to introduce the ability to allocate CPU for Cloud Run container instances even outside of request processing.This feature unlocks many use cases that weren’t previously compatible with Cloud Run:Executing background tasks and other asynchronous processing work after returning responsesLeveraging monitoring agents like OpenTelemetry that may assume access to CPU in background threadsUsing Go’s Goroutines or Node.js async, Java threads, and Kotlin coroutinesMoving Spring Boot apps that use built-in scheduling/background functionalityListening for Firestore changes to keep an in-memory cache up to dateEven if CPU is always allocated, Cloud Run autoscaling is still in effect, and may terminate container instances if they aren’t needed to handle incoming traffic. An instance will never stay idle for more than 15 minutes after processing a request (unless it is kept active using min instances).Combined with Cloud Run minimum instances, you can even keep a certain number of container instances up and running with full access to CPU resources. Together, these functionalities now enable new background processing use cases like using streaming pull with Cloud Pub/Sub or running a serverless Kafka consumer group.When you opt in to “CPU always allocated”, you are billed for the entire lifetimeof container instances—from when a container is started to when it is terminated. Cloud Run’s pricing is now different when CPU is always allocated: There are no per-request feesCPU is priced 25% lower and memory 20% lower Of course, the Cloud Run free tier still applies, and Committed Use Discounts can give you up to 17% discount for a one-year commitment.How to allocate always-on CPUYou can change your existing Cloud Run service to always have CPU allocated from the Google Cloud Console:or from the command line:gcloud beta run services update SERVICE-NAME –no-cpu-throttlingWe hope this change will allow you to run more workloads on Cloud Run successfully while still benefiting from its low-ops characteristics.To learn more about Cloud Run, check out our getting started guides.Related ArticleMaximize your Cloud Run investments with new committed use discountsCommitted use discounts in Cloud Run enable predictable costs—and a substantial discount!Read Article
Quelle: Google Cloud Platform

14. September 2021

da Agency

Push your code and see your builds happening in your terminal with "git deploy"

If you have used hosting services like Heroku before, you might be familiar with the user workflow where you run “git push heroku main”, and you see your code being pushed, built, and deployed. When your code is received by the remote git server, your build is started. With source-based build triggers in Cloud Build, the same effect happens: you “git push” your code, and this triggers a build. However, you don’t see this happen in the same place you ran your git push command. Could you have just one command that you run to give you that Heroku-like experience? Yes, you can. Introducing git deploy: a small Python script that lets you push your code and see it build in one command. You can get the code here: https://github.com/glasnt/git-deployThis code doesn’t actually do anything; it just shows you what’s already going on in Cloud Build. Explaining what this code doesn’t do requires some background knowledge about how git works, and how Cloud Build triggers work.git hooksHooks are custom scripts that are launched when various actions occur in git, and come in two categories: client-side, and server-side. You could set up client-side hooks to do, for example lint checks before you write your commit message, by creating a .git/hooks/pre-commit file that runs your linter of choice. For server-side hooks, however, those need to be stored on the server. You can see server-side hooks running when git returns logs with the “remote: ” prefix. Heroku uses server-side hooks to start deployments. GitHub also uses server-side hooks when you push a branch to a repo, returning the link you can use to create a pull request on your branch (for example: remote: Create a pull request for ‘mytopic’ on GitHub by visiting). However, since you as a developer do not have control over GitHub’s git server, you cannot create server-side hooks, so that solution isn’t possible in this instance. Instead, you can extend git on your machine.git extensionsWriting extensions for git is remarkably simple: you don’t actually change git at all, git just finds your script. When you run a command in git, it will first check if the command is one of it’s internal built-in functions. If the command is not built-in, it will check if there is an external command in its ‘namespace’ and run that. Specifically, if there is an executable on your system PATH that starts with “git-” (e.g. git-deploy), it will run that script when you call “git deploy”. This elegance means that you can extend git’s workflow to do anything you want while still ‘feeling’ like you’re in git (because you are. Kinda.)Inspecting Cloud Build in the command lineCloud Build has a rich user interface of its own in the Cloud console, and native integration into services like Cloud Run. But it also has a rich interface in gcloud, the Google Cloud command line. One of those functions is gcloud builds logs –stream, which allows you to view the progress of your build as it happens, much the same as if you were to view the build in the Google Cloud console. You can also use gcloud to list Cloud Build triggers, filtering by it’s GitHub owner, name, and branch. With that unique trigger ID, you can view what builds are currently running, and stream them. You can get the GitHub identifying information by inspecting git’s configured remote origin and branch.Putting it all togetherGiven all the background, we can now explain what the git deploy script does. Based on what folder you are currently in, it detects what branch and remote origin you have configured. It then runs the code push for you. It then checks to see what Cloud Build triggers are connected to that remote origin, and then waits until a build for that trigger has been started. Once it has, it just streams the logs to the terminal. Suffice to say that this script doesn’t actually do anything that’s not already being done, but it just shows you it all happening in the terminal. ✨(The choice to use Python for this script was mostly due to the fact I did not want to have to write regex parsers in bash. And even if I did, it wouldn’t work for users who use other shells. Git extensions can be written in any language, though!)Related ArticleIntegrating Google Cloud Build with JFrog Artifactory[Editor’s note: Today we hear from software artifact management provider JFrog about how and why to use Google Cloud Build in conjunction…Read Article
Quelle: Google Cloud Platform