Kategorie: Google Cloud Platform

USAA and Google Cloud work together to speed auto claims

When U.S. military personnel and their families want to obtain insurance or financial solutions, one of the first places they turn to is USAA. For nearly 100 years, this San Antonio, Texas-based financial services organization has helped families facilitate their financial security, with a focus on building deep relationships with its members. Another focus for USAA has been using technology to deliver easy and convenient digital experiences. In fact, USAA was the first financial services company to introduce mobile deposits, allowing its members to deposit their checks wherever they are. And, the company was a pioneer in the use of drones and aerial imagery to assess property damage in hard-to-access areas after major natural disasters, such as hurricanes. Now, in collaboration with Google Cloud, USAA is using machine learning (ML) to speed up the auto claims process, obtaining near-real-time damage estimates from digital images. We’re proud to work with USAA on their digital solutions, helping create faster and more cost-efficient estimates within the claims process.Traditionally, assessing auto damage claims has been a time-intensive, laborious process. A claims adjuster needs to physically inspect each vehicle, determine the parts that need replacement, and understand the labor required for repair. They then submit a report to the insurer, who must approve the estimate before sharing it with the repair shop and car owner. This process can sometimes take weeks from beginning to end.Digitizing and moving to the cloud greatly streamlines this process. The appraiser takes photos of the car and uploads them to Google Cloud where, by applying machine learning, they can identify the damage in real-time and send that data to USAA and then into the Mitchell Estimating platform. Mitchell’s software searches parts and labor databases, applies the information to their estimate, and returns that estimate directly to USAA. Appraisers will review the estimate and make adjustments as needed. The new, cloud-based process will allow for faster and more cost-efficient estimates.”This technology now gives claims teams the opportunity to focus more deeply on connecting and providing guidance to our members, during what can often be a trying time,” said Ramon Lopez, USAA VP Innovation. Early results are promising, as USAA has been able to predict high-level damage across a diverse vehicle set with great accuracy. We’ve also worked with USAA employees across the evolving digital environment, providing direct engineer-to-engineer collaboration with our own ML experts. USAA plans to add even more features to streamline the claims process for its customers, so that eventually, the majority of low-complexity assessments will be touchless—allowing appraisers to focus on more complex cases that best utilize their expertise.”Innovation at USAA is not about technology for the sake of technology,” said Lopez. “It’s about enabling our businesses to keep pace with member expectations and needs, all while maintaining the level of service our members deserve.”
Quelle: Google Cloud Platform

Building a document understanding pipeline with Google Cloud

Document understanding is the practice of using AI and machine learning to extract data and insights from text and paper sources such as emails, PDFs, scanned documents, and more.In the past, capturing this unstructured or “dark data” has been an expensive, time-consuming, and error-prone process requiring manual data entry. Today, AI and machine learning have made great advances towards automating this process, enabling businesses to derive insights from and take advantage of this data that had been previously untapped. In a nutshell, document understanding allows you to:Organize documents Extract knowledge from documents Increase processing speedAt Google Cloud we provide a solution, Document AI, which enterprises can leverage in collaboration with partners to implement document understanding. However, many developers have both the desire and the technical expertise to build their own document understanding pipelines on Google Cloud Platform (GCP)—without working with a partner—using the individual Document AI products.If that sounds like you, this post will take you step-by-step through a complete document understanding pipeline. The Overview section explains how the pipeline works, and the step-by-step directions below walk you through running the code.OverviewIn order to automate an entire document understanding process, multiple machine learning models need to be trained and then daisy-chained together alongside processing steps into an end-to-end pipeline. This can be a daunting process, so we have provided sample code for a complete document understanding system mirroring a data entry workflow capturing structured data from documents.Our example end-to-end document understanding pipeline consists of two components:A training pipeline which formats the training data and uses AutoML to buildImage Classification,Entity Extraction,Text Classification, andObject Detection models.A prediction pipeline which takes PDF documents from a specified Cloud Storage Bucket, uses the AutoML models to extract the relevant data from the documents, and stores the extracted data in BigQuery for further analysis.Training DataThe training data for this example pipeline is from a public dataset containing PDFs of U.S. and European patent title pages with a corresponding BigQuery table of manually entered data from the title pages. The dataset is hosted by the Google Public Datasets Project.Part 1: The Training Pipeline The training pipeline consists of the following steps:Training data is pulled from the BigQuery public dataset. The training BigQuery table includes links to PDF files in Google Cloud Storage of patents from the United States and European Union.The PDF files are converted to PNG files and uploaded to a new Cloud Storage bucket in your own project. The PNG files will be used to train the AutoML Vision models.The PNG files are run through the Cloud Vision API to create TXT files containing the raw text from the converted PDFs. These TXT files are used to train the AutoML Natural Language models.The links to the PNG or TXT files are combined with the labels and features from the BigQuery table into a CSV file in the training data format required by AutoML. This CSV is then uploaded to a Cloud Storage bucket. Note: This format is different for each type of AutoML model.This CSV is used to create an AutoML dataset and model. Both are named in the format patent_demo_data_%m%d%Y_%H%M%S. Note that some AutoML models can sometimes take hours to train.Part 2: The Prediction PipelineThis pipeline uses the AutoML models previously trained by the pipeline above. For predictions, the following steps occur:The patent PDFs are collected from the prescribed bucket and converted to PNG and TXT files with the Cloud Vision API (just as in the training pipeline).The AutoML Image Classification model is called on the PNG files to classify each patent as either a US or EU patent. The results are uploaded to a BigQuery table.The AutoML Object Detection model is called on the PNG files to determine the location of any figures on the patent document. The resulting relative x, y coordinates of the bounding box are then uploaded to a BigQuery table.The AutoML Text Classification model is called on the TXT files to classify the topic of the patent content as medical technology, computer vision, cryptocurrency or other. The results are then uploaded to a BigQuery table.The AutoML Entity Extraction model is called to extract predetermined entities from the patent. The extracted entities are applicant, application number, international classification, filing date, inventor, number, publication date and title. These entities are then uploaded to a BigQuery table.Finally, the BigQuery tables above are joined to produce a final results table with all the properties above.Step-by-Step DirectionsFor the developers out there, here’s how you can build the document understanding pipeline of your dreams. You can find all the code in our GitHub Repository.Before you begin:You’ll need a Google Cloud project to run this demo. We recommend creating a new project.We recommend running these instructions in Google Cloud Shell(Quickstart). Other environments will work but you may have to debug issues specific to your environment. A Compute Engine VM would also be a suitable environment.1. Git clone the repo and navigate to the patents example.2. Install the necessary system dependencies. Note that in Cloud Shell system-wide changes do not persist between sessions, so if you step away while working through these step-by-step instructions you will need to rerun these commands after restarting Cloud Shell.3. Create a virtual environment and activate it.When your virtual environment is active you’ll see patents-demo-env in your command prompt. Note: If your Cloud Shell session ends, you’ll need to reactivate the virtual environment by running the second command again.For the remaining steps, make sure you’re in the correct directory in the professional-services repo: /examples/cloudml-document-ai-patents/.4. Install the necessary libraries into the virtual environment.5. Activate the necessary APIs.Note: We use the gcloud SDK to interact with various GCP services. If you are not working in Cloud Shell you’ll have to set your GCP project and authenticate by running gcloud init.6. Edit the config.yaml file. Look for the configuration parameter project_id (under pipeline_project) and set the value to the project id where you want to build and run the pipeline. Make sure to enclose the value in single quotes, e.g. project_id: ‘my-cool-project’.Note: if you’re not used to working in a shell, run nano to open a simple text editor.7. Also in config.yaml, look for the configuration parameter creator_user_id (under service_acct) and set the value to the email account you use to log in to Google Cloud. Make sure to enclose the value in single quotes, e.g. creator_user_id: ‘cloud.user@fake.email.address.com’.8. Create and download a service account key with the necessary permissions to be used by the training and prediction pipelines.9. Run the training pipeline. This may take 3–4 hours, though some models will finish more quickly.Note: If Cloud Shell Closes while the script is still downloading, converting, or uploading the PDFs, you will need to reactivate the virtual environment, navigate to the directory, and rerun the pipeline script. The image processing should take about 15-20 minutes, so make sure Cloud Shell doesn’t close during that time.10. After training the models (wait about 4 hours), your Cloud Shell session has probably disconnected. Reconnect to Cloud Shell and run the following commands to reinstall dependencies, reactivate the environment, and navigate to the document understanding example code.11. Next, you need to use the AutoML UIs to deploy the object detection and entity extraction models, and to find the ids of your models (which you will enter into config.yaml). In the UIs, you can also view relevant evaluation metrics about the model and see explicit examples where your model got it right (and wrong).Note: Some AutoML products are currently in beta; the look and function of the UIs may change in the future.Go to the AutoML Image Classification models UI, and make sure you are in same project you ran the training pipeline in (top right dropdown). Note the ID of your trained image classification model. Also note the green check mark to the right of the model—if this is not a check mark, or if there is nothing listed, it means model training is still in progress and you need to wait.In Cloud Shell, edit config.yaml. Look for the line model_imgclassifier: and on the line below you’ll see model_id:. Put the image classification model id after model_id: in single quotes, so the line looks something like model_id: ‘ABC1234567890’Go to the AutoML Natural Language models UI, and similarly, make sure you are in the correct project, make sure your model is trained (green check mark) and note the model id of the trained text classification model.In Cloud Shell, edit config.yaml. Look for the line model_textclassifier: and on the line below you’ll see model_id:. Put the text classification model id after model_id: in single quotes, so the line looks something like ” model_id: ‘ABC1234567890′”Go to the AutoML Object Detection models UI, again making sure your project is correct and your model is trained. In this UI, the model id is below the model name. You also need to deploy the model, by opening the menu under the three dots at the far right of the model entry and selecting “Deploy model”. A UI popup will ask how many nodes to deploy to, 1 Node is fine for this demo. You need to wait for the model to deploy before running prediction (about 10-15 minutes), the deployment status is in the UI under “Deployed” and it will be “Yes” when the model is deployed.In Cloud Shell, edit config.yaml. Look for the line model_objdetect: and on the line below you’ll see model_id:. Put the object detection model id after model_id: in single quotes, so the line looks something like model_id: ‘ABC1234567890’Go to the AutoML Entity Extraction models UI, and make sure the project and region are correct. Verify the model training is complete by checking to make sure the Precision and Recall metrics are listed. Note the id as well.To deploy the model, click the model name and a new UI view will open. Click “DEPLOY MODEL” near the top right and confirm. You need to wait for the model to deploy before running prediction (about 10-15 minutes), the UI will update when deployment is complete.In Cloud Shell, edit config.yaml. Look for the line model_ner: and on the line below you’ll see model_id:. Put the entity extraction model id after model_id: in single quotes, so the line looks something like model_id: ‘ABC1234567890′.Before going further, make sure the model deployments are complete.12. To run the prediction pipeline, you’ll need some PDFs of patent first pages in a folder in Cloud Storage. You can provide your own PDFs in your own Cloud Storage location, or for demonstration purposes, you can run the following commands, which create a bucket with the same name as your project id (if it doesn’t already exist) and copy a small set of five patent first pages into a folder in the bucket called patent_sample.13. In the config.yaml file, fill in the configuration parameter labeled demo_sample_data (under pipeline_project) with the Cloud Storage location of the patents you want to process (in single quotes). If you’re following the example above, the parameter value is ‘gs://<YOUR-PROJECT-ID>/patent_sample’, substituting your project id.Also, fill in the parameter labeled demo_dataset_id, just under demo_sample_data. With the BigQuery dataset (in single quotes) in your project where the predicted entities will be collected. For example, you can use ‘patent_demo’. Note that this dataset must not exist already; the code will attempt to create the dataset and will stop running if the dataset already exists.Now, you are ready to make predictions!14. Run the prediction pipeline. This will process all the patent PDF first pages in the Cloud Storage folder specified in the demo_sample_data parameter, and upload predictions to (and create) BigQuery tables in the dataset specified by the  demo_dataset_id parameter.python3 run_predict.py15. Finally, go check out the dataset in the BigQuery UI and examine all the tables that have been created. There is a table called “final_view” which collects all the results in a single table. You should see something like this:Note, for this, or any of the other intermediate tables that were created during the prediction pipeline, you may need to query the table to see the results since it is so new. For example:If you are new to BigQuery, you can open a querying interface in the UI prepopulated with the table name by clicking the “QUERY TABLE” button visible when you select a table to view.ConclusionCongratulations! You now have a fully functional document understanding pipeline that can be modified for use with any PDF documents. To modify this example to work with your own documents, the data collection and training stages will need to be modified to pull documents from a local machine or another Cloud Storage bucket rather than a public BigQuery dataset. The training data will also need to be created manually for the specific type of documents you will be using. Now, if only we could create a robot that could scan paper documents to PDFs…For more information about the general process of document understanding, see this blog post from Nitin Aggarwal at Google Cloud. And for more information about the business use cases of Document Understanding on Google Cloud, check out this session from Google Cloud Next ’19.
Quelle: Google Cloud Platform

Moving a publishing workflow to BigQuery for new data insights

Google Cloud’s technology powers both our customers as well as our internal teams. Recently, the Solutions Architect team decided to move an internal process to use BigQuery to streamline and better focus efforts across the team. The Solutions Architect team publishes reference guides for customers to use as they build applications on Google Cloud. Our publishing process has many steps, including outline approval, draft, peer review, technical editing, legal review, PR approval and finally, publishing on our site. This process involves collaboration across the technical editing, legal, and PR teams.With so many steps and people involved, it’s important that we effectively collaborate. Our team uses a collaboration tool running on Google Cloud Platform (GCP) as a central repository and workflow for our reference guides. Increased data needs required more sophisticated toolsAs our team of solution architects grew and our reporting needs became more sophisticated, we realized that we couldn’t effectively provide the insights that we needed directly in our existing collaboration tool. For example, we needed to build and share status dashboards of our reference guides, build a roadmap for upcoming work, and analyze how long our solutions take to publish, from outline approval to publication. We also needed to share this information outside our team, but didn’t want to share unnecessary information by broadly granting access to our entire collaboration instance.Building a script with BigQuery on the back endSince our collaboration tool provides a robust and flexible REST API, we decided to write an export script which stored the results in BigQuery. We chose BigQuery because we knew that we could write advanced queries against the data and then use Data Studio to build our dashboards. Using BigQuery for analysis provided a scalable solution that is well-integrated into other GCP tools and has support for both batch and real-time inserts using the streaming API.We used a simple Python script to read the issues from the API and then insert the entries into BigQuery using the streaming API. We chose the streaming API, rather than Cloud Pub/Sub or Cloud Dataflow, because we wanted to repopulate the BigQuery content with the latest data several times a day. The Google API Python client library was an obvious choice, because it provides an idiomatic way to interact with the Google APIs, including the BigQuery streaming API. Since this data would only be used for reporting purposes, we opted to keep only the most recent version of the data as extracted. There were two reasons for this decision: Master data: There would never be any question about which data was the master version of the data. Historical data: We had no use cases that required capturing any historical data that wasn’t already captured in the data extract. Following common extract, transform, load (ETL) best practices, we used a staging table and a separate production table so that we could load data into the staging table without impacting users of the data. The design we created based on ETL best practices called for first deleting all the records from the staging table, loading the staging table, and then replacing the production table with the contents. When using the streaming API, the BigQuery streaming buffer remains active for about 30 to 60 minutes or more after use, which means that you can’t delete or change data during that time. Since we used the streaming API, we scheduled the load every three hours to balance getting data into BigQuery quickly and being able to subsequently delete the data from the staging table during the load process.Once our data was in BigQuery, we could write SQL queries directly against the data or use any of the wide range of integrated tools available to analyze the data. We chose Data Studio for visualization because it’s well-integrated with BigQuery, offers customizable dashboard capabilities, provides the ability to collaborate, and of course, is free. Because BigQuery datasets can be shared with users, this opened up the usability of the data for whomever was granted access and also had appropriate authorization. This also meant that we could combine this data in BigQuery with other datasets. For example, we track the online engagement metrics for our reference guides and load them into BigQuery. With both datasets in BigQuery, it made it easy to factor in the online engagement numbers to build dashboards.Creating a sample dashboardOne of the biggest reasons that we wanted to create reporting against our publishing process is to track the publishing process over time. Data Studio made it easy to build a dashboard with charts, similar to the two charts below. Building the dashboard in Data Studio allowed us to easily analyze our publication metrics over time and then share the specific dashboards with teams outside ours.Monitoring the load processMonitoring is an important part of any ETL pipeline. Stackdriver Monitoring provides monitoring, alerting and dashboards for GCP environments. We opted to use the Google Cloud Logging module in the Python load script, because this would generate logs for errors in Stackdriver Logging that we could use for error alerting in Stackdriver Monitoring. We set up a Stackdriver Monitoring Workspace specifically for the project with the load process. We then created a management dashboard to track any application errors. We set up alerts to send an SMS notification whenever errors appeared in the load process log files. Here’s a look at the dashboards in the Stackdriver Workspace:And this shows the details of the alerts we set up:BigQuery provides the flexibility for you to meet your business or analytical needs, whether they’re petabyte-sized or not. BigQuery’s streaming API means that you can stream data directly into BigQuery and provide end users with rapid access to data. Data Studio provides an easy-to-use integration with BigQuery that makes it simple to develop advanced dashboards. The cost-per-query approach means that you’ll pay for what you store and analyze, though BigQuery also offers flat-rate pricing if you have a high number of large queries. For our team, we’ve been able to gain considerable new insights into our publishing process using BigQuery, which have helped us both refine our publishing process and focus more effort on the most popular technical topics. If you haven’t already, check out what BigQuery can do using the BigQuery public datasets and see what else you can do with GCP in our reference guides.
Quelle: Google Cloud Platform

Container-native load balancing on GKE now generally available

Last year, we announced container-native load balancing, a feature that allows you to create services using network endpoint groups (NEGs) so that requests to your service get load balanced directly to the containers serving the requests. Since announcing the beta, we have worked hard to improve the performance, scalability and user experience of container-native load balancing and are excited to announce that it is now generally available.Container-native load balancing removes the second hop between virtual machines running containers in your Google Kubernetes Engine (GKE) cluster and the containers serving the requests, improving efficiency, traffic visibility and container support for advanced load balancer capabilities. The NEG abstraction layer that enables this container-native load balancing is integrated with the Kubernetes Ingress controller running on Google Cloud Platform (GCP). If you have a multi-tiered deployment where you want to expose one or more services to the internet using GKE, you can also create an Ingress object, which provisions an HTTP(S) load balancer and allows you to configure path-based or host-based routing to your backend services.Figure 1. Ingress support with instance groups vs. with network endpoint groups.Improvements in container-native load balancingThanks to your feedback during the beta period, we’ve made several improvements to container-native load balancing with NEGs. In addition to having several advantages over the previous approach (based on IPTables), container-native load balancing now also includes:Latency improvements: The latency of scaling down your load-balanced application to zero pod backends and then subsequently scaling back up is now faster by over 90%. This significantly improves response times for low-traffic services, which can now quickly scale back up from zero pods when there’s traffic. Improved Kubernetes integration: Using the Kubernetes pod readiness gate feature, a load-balancer backend pod is considered ‘Ready’ once the Load balancer health check for the pod is successful and the pod is healthy. This ensures that rolling updates will proceed only after the pods are ready and fully configured to serve traffic. Now, you can manage the load balancer and backend pods with native Kubernetes APIs without injecting any unnecessary latency. Standalone NEGs (beta): You can now manage your own load balancer (without having to create an HTTP/S based Ingress on GKE) using standalone NEGs, allowing you to configure and manage several flavors of Google Cloud Load Balancing. These include TCP proxy or SSL proxy based load balancing for external traffic, HTTP(S) based load balancing for internal traffic (beta) and global load balancing using Traffic Director for internal traffic. You can also create a load balancer with hybrid backends (GKE pods and Compute Engine VMs) or a load balancer with backends spread across multiple GKE clusters.Getting started with container-native load balancingYou can use container-native load balancing in several scenarios. For example, you can create an Ingress using NEGs with VPC-native GKE clusters created using Alias IPs. This provides native support for pod IP routing and enables advertising prefixes. Check out how to create an Ingress using container native load balancing. Then, drop us a line about how you use NEGs, and other networking features you’d like to see on GCP.
Quelle: Google Cloud Platform

3 steps to detect and remediate security anomalies with Cloud Anomaly Detection

Editor’s Note:This is the third blog in our six-part series on how to use Cloud Security Command Center. There are links to the first two blogs in the series at the end of this post. When a threat is detected, every second counts. But, sometimes it can be difficult to know if a threat is present or how to respond. Cloud Anomaly Detection is a built-in Cloud Security Command Center (Cloud SCC) feature that uses behavioral signals to detect security abnormalities, such as leaked credentials or unusual activity, in your GCP projects and virtual machines. In this blog, and the accompanying video, we’ll look at how to enable Cloud Anomaly Detection and quickly respond to threats. 1. Enable Cloud Anomaly Detection from Cloud Security Command CenterCloud Anomaly Detection is not turned on by default. You need to go to Security Sources from the Cloud SCC dashboard and activate it. Keep in mind, to enable a security source, you need to have the Organization Administrator Cloud IAM role. Once it’s turned on, findings will automatically be surfaced and displayed in the Cloud Anomaly Detection card on the Cloud Security Command Center dashboard.2. View findings in Cloud Security Command Center Cloud Anomaly Detection can surface a variety of anomalous findings, including:Leaked service account credentials: GCP service account credentials that are accidentally leaked online or compromised.Resource used for outbound intrusion: One of the resources or GCP services in your organization is being used for intrusion activities, like an attempt to break in to or compromise a target system. These include SSH brute force attacks, Port scans, and FTP brute force attacks.Potential compromised machine: A potential compromise of a resource in your organization.Resource used for crypto mining: Behavioral signals around a VM in your organization indicate that it might have been compromised and could be getting used for crypto mining.Unusual Activity/Connection: Unusual activity from a resource in your organization.Resource used for phishing: One of the resources or GCP services in your organization is being used for phishing.3. Remediate findings from Cloud Security Command Center After Cloud Anomaly Detection generates a finding, you can click on the finding for more information about what happened and use that information to fix the security issue.To learn more about Cloud Anomaly Detection, including how to turn it on and how it can help your organization, check out the video below.Previous blogs in this series:5 steps to improve your cloud security posture with Cloud Security Command CenterCatch web app vulnerabilities before they hit production with Cloud Web Security Scanner
Quelle: Google Cloud Platform

Google Cloud Firewall Rules Logging: How and why you should use it

Google Cloud Platform (GCP) firewall rules are a great tool for securing applications. Firewall rules are customizable software-defined networking constructs that let you allow or deny traffic to and from your virtual machine (VM) instances. To secure applications and respond to modern threats, firewall rules require monitoring and adjustment over time. GCP Firewall Rules Logging, which Google Cloud made generally available in February 2019, is a feature that allows the network administrators to monitor, verify and analyze the effects of firewall rules in Google Cloud. In this blog (the first of many on this topic), we’ll discuss the basics of Firewall Rule Logging, then look at an example of how to use it to identify mislabeled VMs and refine firewall rules with minimal traffic interruption.GCP Firewall Rules Logging: The BasicsFirewall Rules Logging provides visibility to help you better understand the effectiveness of rules in troubleshooting scenarios. It helps answer common questions, like: How can I ensure the firewall rules are doing (or not doing) what they were created for?How many connections match the firewall rules I just implemented? Are firewall rules the root cause of some application failures?Unlike VPC flow logs, firewall rules logs are not sampled. Every connection is logged (subject to some limits. Please refer to the Appendix for details). The Firewall Rule log format can be found here.Additionally, network administrators have the options to export firewall logs toGoogle Cloud Storage for long term log retention,to BigQuery for in-depth analysis using standard SQL, or to Pub/Sub to integrate with popular security information and event management software (SIEM), such assplunk for detecting/alerting traffic abnormalities and threats at near real time.For reference, GCP firewall rules are software-defined constructs with the following properties:GCP firewalls are VM-centric. Unlike traditional firewall devices, which are applied at the network edge, GCP firewall rules are implemented at VM level. This means the firewall rules can exist between your instances and other networks, and also between individual instances within the same VPC.GCP firewall rules always have targets. The targets are considered source VMs when defining egress firewall rules, and destination VMs when defining ingress firewall rules. Do not confuse “target” with the “destination” in the traditional firewall concept.GCP firewall rules are defined within the scope of aVPC network. There is no concept of subnets when defining firewall rules. However, you can specify source CIDR ranges, which give you better flexibility than subnets.Every VM has two immutable implied firewall rules: implied allow of egress, and implied deny of ingress at lowest priority. However, Firewall Rule Logging does not generate any entries for these implied firewall rules.While GCP firewall rules support many protocols—including TCP, UDP, ICMP, ESP, AH, SCTP, and IPIP—Firewall Rule Logging only logs entries for TCP and UDP connections.Firewall Best Practices Follow the least privilege principle — make firewall rules as tight as possible. Only allow well-documented and required traffic (ingress and egress), and deny all others. Use a good naming convention to indicate each firewall rule’s purpose.Use fewer and broader firewall rule sets when possible. Observe the standard quota of 200 firewall rules per project. The complexity of the firewall also matters. A good rule of thumb is to not throw too many atoms (tags/service accounts, protocols/ports, source/destination ranges) at the firewall rules. Please refer to the Appendix for more on firewall quota/limits.Progressively define and refine rules; start with the broader rules first, and then use rules to narrow down to a smaller set of VMs.Isolate VMs using service accounts when possible. If you can’t do that, use network tags instead. But do not use both. Service account access is tightly controlled by IAM. While network tags are more flexible, anyone with the instanceAdmin role can change them. More on filtering using service accounts versus network tags can be foundin our firewall rules overview.Conserve network space by planning proper CIDR blocks (segmentations) for your VPC network to group related applications in the same subnet.Use firewall rule logging to analyze traffic, detect misconfigurations, and report real-time abnormalities.In practice, there are many uses for Firewall Rule Logging. One common use case is to help identify mislabeled VM instances. Let’s walk through this scenario in more detail.Scenario: Mislabeled VM instancesACME, Inc. is migrating on-prem applications to Google Cloud. Their network admins implemented a shared VPC to centrally manage the entire company’s networking infrastructure. There are dozens of applications, run by multiple engineering teams, deployed in each GCP region with multi-tiered applications. User-facing proxies in the DMZ talk to the web servers, which communicate with the application servers, which, talk to database layers.This diagram represents one of many regions, each of which has multiple subnets. This region, US-EAST1, includes:acme-dmz-use1: 172.16.255.0/27acme-web-use1: 10.2.0.0/22acme-app-use1: 10.2.4.0/22acme-db-use1: 10.2.8.0/22Here are the traffic directions and firewall rules in place for this region:Proxy can access web servers in acme-web-use1:firewall rule: acme-web-use1-allow-ingress-acme-dmz-use1-proxyWeb servers can access app servers in acme-app-use1:firewall rule: acme-app-use1-allow-ingress-acme-web-use1-webserverApp servers can access database servers in acme-db-use1:firewall rule: acme-db-use1-allow-ingress-acme-app-use1-appsvrThis setup has granular partitions of network space that categorize compute resources by application function. This makes it possible for firewall rules to control the network space and lock the network infrastructure to comply with the least-privilege principle.For large organizations with thousands of VMs provisioned by dozens of application teams, we use service accounts or network tags to group the VMs, in conjunction with subnet CIDR ranges to define firewall rules. For simplicity, we use the network tags to demonstrate each use case.The problemIt’s not unusual for large enterprises to have hundreds of firewall rules due to the scale of the infrastructure and the complexity of network traffic patterns.With so much going on, it’s understandable, that application teams sometimes mislabel the VMs when they migrate them from on-prem to cloud and scale-up applications after migration. The consequences of mislabeling range from an application outage to a security breach. The same problem can also arise if we (mis)use service accounts.Going back to our example, an ACME application team mislabeled one of their new VMs as “appsrv” when it should actually be “appsvr”. As a result, the web server’s requests to access app servers are denied.Solution 1In order to identify the mislabeling and mitigate the impact quickly, we can enable the firewall rule logging for all the firewall rules using the following gcloud command.Next, we export the logs to BigQuery for further analysis with following steps: Create a BigQuery dataset to store the Firewall Rule Log entries:Create the BigQuery sink to export the firewall rules logs:You can also export individual firewall rules by adding filters on the jsonPayload.rule_details.reference. Here is a sample filter:When the BigQuery sink is created, a service account is generated automatically by the GCP. The service account follows the naming convention of “p<project_number>-[0–9]*@gcp-sa-logging.iam.gserviceaccount.com” and is used to write the log entries to the BigQuery table. We need to grant this service account the Data Editor role on the BigQuery dataset as following three steps:Once the logs are populated to the BigQuery sink, you can run queries to determine which rule denies what as following:The BigQuery table will be loaded with the log entries from the logFileName filter, which, in this case, contains all the firewall rules logs. The BigQuery table schema is directly mapped to the log entry’s json format. To keep it simple, for our example we’ll use the log viewer to inspect the log entries.Since we have implicit ingress and the denial rule is not being logged, we create a “deny all” rule with priority 65534 to capture anything that gets denied.The firewall rules for this scenario are:As we can see in the viewer,  “acme-deny-all-ingress-internal” is taking effect, and  “acme-allow-all-ingress-internal” is disabled, so we can ignore it.Below, we can see that the connection from websvr01 to the new appsvr02 (with the incorrect “appsrv” label) is denied.While this approach works for this example, it presents two potential problems:If we have a large amount of traffic, it will generate too much data for real-time analysis. In fact, one of our clients implemented this approach and ended up generating 5TB of firewall logs per day.Mislabeled VMs can cause traffic interruptions. The firewall is doing what it is designed to do, but nobody likes outages.So, we need a better approach to address both of the issues above.Solution 2To resolve the potential issues mentioned above, we can create another ingress rule to allow all traffic at priority 65533 and turn it on for a short period of time whenever there are new deployments.In this scenario, we don’t need to turn on all the Firewall Rules Logging. In fact, we could turn off most of it to save space.Any allowed evaluation logged by this firewall rule is a violator, and we do not expect many of them. The suspects are identified in real time.Now, we fix the label.As we can see, the connection from websvr01 to appsvr02 now works fine.After all the mislabels are fixed, we can turn off the allow and capture rule. Everyone is happy… until the next time new resources are added to the network.ConclusionWith Firewall Rules Logging, we can refine our firewall rules by following a few best-practices and identify undesired network traffic in near real-time. In addition to firewall rules logging, we’re always working on more tools and features to make managing firewall rules and network security in general easier. AppendixFirewall quotas and limitsThe default quota is 200 firewall rules per project, but can be bumped up to 500 through quota requests. If you need more than 200 firewall rules, we recommend  that you review the firewall design to see whether there is a way to consolidate the firewall rules. The upcoming hierarchical firewall rules can define firewall rules at folders/org level, and are not counted toward the per-project limit.The maximum number of source/target tags per firewall rule is 30/70, and the maximum number of source/target per service account is 10/10. A firewall rule can use network tags or service accounts, but not both.As mentioned, the complexity of the firewall rules also matters. Anything that is defined in firewall rules, such as source ranges, protocol/ports, network tags, and service accounts, counts towards an aggregated per-network hard limit. This number is at the scale of tens of thousands, so it doesn’t concern most customers, except in rare cases where a large enterprise may reach this limit.There are per-VM maximum number of logged connections in a 5-second interval depending on machine types: f1-macro (100), g1-small (250), and other machine types (500 per vCPU up to 4,000 in total).
Quelle: Google Cloud Platform