Cloud Computing Köln - Seite 2266 von 7048 - Neues zu Cloud Computing, Internet of Things und Technologien

Editor’s note: Over the next several weeks, you’ll see a series of blog posts focusing on best practices for writing Google Cloud Functions based on common questions or misconceptions as seen by the Support team. We refer to these as “anti-patterns” and offer you ways to avoid them. This article is the fourth post in the series.ScenarioYou notice that the data your Function saves to a database is either “undefined” or it is saving a cached value. For example, you have a Cloud Task that every hour invokes a Cloud Function that retrieves data from one database, transforms that data, and then saves the modified data to another database. Yet, you notice that your data is either undefined or a cached value. Most common root issueAn unhandled promise. One common anti-pattern we notice when using the then-able approach in a Cloud Function is that it is easy to overlook an unhandled Promise. For example, can you spot the issue in the following example?You will not know how long the call transformData(response.data) will take. And more likely, this call is probably an async method. The result is that the saveDataToDatabase method is executed before transformData() has completed. Thus, the variable dataNowTransformed is undefined upon saving to the database. How to investigateAre you using await? If not, we recommend using async and await keywords to help improve the readability of your code.If you cannot convert to await at this time, you’ll need to add a logging layer (see example below) to determine if you have an unhandled promise.How to add a logging layer using then-able functionsThere are two ways to log: sync logging by modifying your callbacksasync logging to avoid modifying your callbacks (i.e. adding a logging layer using then-able functions)Suppose you are okay with modifying your callbacks. You can add a synchronous logging layer just by using console.log(). We recommend creating a synchronous method called logData() function to keep your code clean and avoid numerous console.log() statements throughout your code.where logData() looks like: Now suppose you do not want to modify the code within your callbacks. We recommend adding an async logging method as follows: 1. Create an asynclogDataAsync() method in your Function.2. Call the logDataAsync method using the then-able() approachPlease see the helpful tips section for a more compact way to apply the async logging approach. How to handle the Promise using the then-able approachWe recommend that you perform one task at a time within a .then() callback. Going back to our original anti-pattern example, let’s update it to use the then-able approach. Here’s an end-to-end working example:But all these back to back .then() calls (called Promise chaining) make the code difficult to read and maintain. Please see the helpful tips section for a more compact way to write this code, in case you see it elsewhere.If possible, we suggest that you use awaits. Notice how the code in the Function event handler callToSlowRespondingAPI now becomes more succinct and readable. In addition, if anything goes wrong in these async method calls, an exception is thrown, in lieu of returning null or false in the return statement. Other helpful tipsAre you testing locally using the Functions Framework? You can follow this codelab to learn how to debug Node.js functions locally in Visual Studio Code.Whenever you are logging data, be aware of how much data you are logging (see log size limits) and whether your data has any sensitive information or personally identifiable information.Using the then-able() approach, you might often see code written as follows. This is functionally equivalent to the longer then-able() Promise-chaining version used above. However, we recommend using the async await approach for readability. Related ArticleAvoiding GCF anti-patterns part 3: How to establish outbound connections correctlyThird post in a series on how to avoid anti-patterns in Google Cloud Functions as seen by the Support team. This post explores how to mak…Read Article
Quelle: Google Cloud Platform

13. November 2021

da Agency

Modernizing compliance: Introducing Risk and Compliance as Code

Almost all publicly reported breaches in the cloud stem from misconfigurations, rather than from attacks that compromise underlying cloud infrastructure. Misconfigurations continue to be a source of security risk because most security and compliance practices play catchup – teams are involved later in the CI/CD process and misconfigurations are identified at runtime, instead of during the build process. Reliance on runtime security also creates friction between developers and security professionals because runtime tools, by their nature, are deployed at the end of the CI/CD process, and are therefore often seen as the final gate or blocker to production.To prevent and address the risk of misconfigurations and compliance violations earlier in the development process, security leaders have started to embrace security as code to achieve the speed and agility of DevOps, reduce risk, and more securely create value in the cloud. “Being able to precisely model and then continuously monitor the adoption and correct operation of controls in any environment is essential. In the software defined environment (i.e., cloud-native workloads) this is not only possible but more importantly it’s actually more easily achievable than other environments—and the more you do it the easier it becomes for continued monitoring.” —Phil Venables, Chief Information Security Officer, Google Cloud.Recognizing the need and opportunity to help customers prevent security misconfigurations and automate cloud compliance, the Google Cybersecurity Action Team is thrilled to announce the launch of our Risk and Compliance as Code (RCaC) Solution. The RCaC solution stack enables compliance and security control automation through a combination of Google Cloud Products, Blueprints, Partner Integrations, workshops and services to simplify and accelerate time to value:Existing products such as Assured Workloads, Security Command Center (SCC), and Risk Manager. Assured Workloads helps you define secure configurations and controls as code in your cloud architecture via APIs which are also expressed in some of our blueprints. SCC allows you to monitor for security misconfigurations and compliance violations on a continuous basis. Risk Manager gives you tools to leverage cyber insurance to deal with risks in the Google Cloud environment. A core set of blueprints such as Secure Foundations, Anthos Security blueprints, workload specific blueprints such as PCI DSS on GKE, and FedRAMP aligned 3-tier workload that codify infrastructure and policies. Blueprints can help you rapidly configure cloud environments in a secure and compliant manner. Partner integrations (such as Sysdig and others) with SCC to detect drift from blueprinted environments. These integrations expand the coverage beyond Google Cloud’s native controls to help deliver improved multi-cloud compliance and risk reduction.A policy library set mapped to common compliance frameworks such as NIST 800-53, PCI DSS, and ISO 27001 with preventative and detective controls that can be expressed as code. These policies communicate which controls can be codified from the above frameworks. Whitepapersand workshops for rapid security organization transformation and DevSecOps transformation. Professional services and partner-led accelerator programs that enable organizations to pilot the solution. Operationalizing Risk and Compliance as CodeThrough the RCaC solution, customers can introduce automation via IaC (Infrastructure as Code) and PaC (Policy as Code) in the form of blueprints. This lays the foundation of preventative controls. Additionally, customers can “shift-left” their security and compliance practices by evaluating IaC and PaC templates for security and compliance violations before they are used in a build.The next level of maturity is detection as code which involves monitoring for (security and compliance) drifts and applying remediations when an out-of-compliance infrastructure is identified. This forms a continuous monitoring loop that helps prevent misconfigurations. Cloud-native tooling helps to operate this model at scale.Three key benefits of Risk and Compliance as CodeWith RCaC, our hope is to provide our customers with the necessary components to express security and compliance requirements as code and shift left, leading to reduced risk and impact of misconfigurationsa continued compliance and security monitoring environment that is based on automation and code.Encourage a shift in the culture where there is reduced friction between developers and security and compliance teams Our goal with RCaC is to reduce the audit burden and fatigue that is experienced by GRC professionals as they modernize their infrastructure and at the same time continue to meet their compliance obligations.Implementing the Risk and Compliance as Code (RCaC) approachImplementing RCaC requires a substantial policy, architectural, and cultural change for almost all organizations. It requires a change in mindset from compliance being a reactive or a check-box exercise vs. addressing it proactively. Our solution helps organizations progress through this transition.For this reason, many have found it helpful to use the RCaC framework to classify workloads according to sensitivity and criticality to apply specific preventative controls based on workload risk and deployment type. Once this codification is realized, customers can leverage tools inside Security Command Center to continuously monitor for drift and non-compliance. Finally, customers can also build custom drift correction or leverage our Risk Protection Program with insurance providers to reduce security risk and gain access to an exclusive cyber insurance policy designed exclusively for Google Cloud customers. “Using blueprints to simplify meeting compliance standards is a real win for customers. Automation of the right configurations and controls as code helps reduce risk and accelerate cloud success. With tools like Sysdig Secure, Google Cloud customers can easily monitor for drift to make sure they remain secure and compliant over time, while also gaining runtime visibility,” said Omer Azaria, Sysdig VP of Research and Development.Furthermore, RCaC provides a model for future-state architecture and outlines key decisions necessary for various automation use cases. This approach also paves the path towards self-healing cloud native infrastructure and autonomic cloud security. To learn more about the solution, review the details at the website. For broader context, read the Google Cybersecurity Action Team paper “Assuring Compliance in the Cloud” and listen to our Google Cloud Security Podcast “Making Compliance Cloud-native” (episode 14). If you need the solution, request a briefing with Google Cloud Sales.Related ArticleModernizing SOC … Introducing Autonomic Security OperationsWe’ve launched the Autonomic Security Operations solution, a new approach to transforming Security Operations to protect against modern-d…Read Article
Quelle: Google Cloud Platform

13. November 2021

da Agency

How Google Cloud BigQuery enables big DevOps at JFrog

Editor’s note: Today we’re hearing from Mitali Bisht, Software Engineer at JFrog on how BigQuery and Data Studio powers operational analytics on the JFrog DevOps Platform. At JFrog, we know that keeping DevOps running smoothly requires knowing as much as you can about those operations. It’s a key principle of Artifactory, our artifact repository manager that powers the JFrog DevOps Platform. Information – in Artifactory’s case, artifact and build metadata – provides traceable paths through the complex systems we build every day. Data and the power to analyze it enables smart decisions by people and machines.So to better serve our JFrog Cloud customers running their SaaS subscriptions on Google Cloud Platform (GCP), we needed to be able to collect and analyze operational data on hundreds of their deployments.We wanted to gather statistics to serve metadata to users to make better decisions such as:Who is actively using their JFrog accounts, by IP address?Is there activity that suggests an attempted cyberattack?Which modules or packages people use the most?How efficiently are those resources being used?On a single-user scale, we’ve provided some facilities for our self-hosted customers through our JFrog Platform Log Analytics integrations, to configure for themselves and view their high-availability deployment’s activity through analytics programs like Splunk and DataDog.To monitor our SaaS operations on GCP, however, we needed to craft a solution that could extract and analyze this kind of data from multiple deployments on a much more massive scale.Among the many GCP services available, we were able to make particular use of Cloud Logging, BigQuery, and Data Studio to collect, analyze, and visualize such vast quantities of operations data in real time. Let’s dive into the architecture we used for this project at JFrog.Stage 1: Ingesting Data from LogsWe had two sources of logs to ingest data from:The nginx server serving our Artifactory SaaS instancesLogs streamed in from external cloud storageNGINX Access LogsFor the first, we already had the google-fluentd logging agent pod setup by default while setting up our Kubernetes cluster on GKE. The logging agent google-fluentd is a modified version of the fluentd log data collector. In its default configuration, the logging agent streams logs, as included in the list of default logs, to Cloud Logging. This default setup for nginx-access was sufficient; there was no need to customize the agent configuration to stream any additional logs.In Cloud Logging, all logs, including audit logs, platform logs, and user logs, are sent to the Cloud Logging API where they pass through the Logs Router. The Logs Router checks each log entry against existing rules to determine which log entries to discard, which log entries to ingest (store) in Cloud Logging, and which log entries to route to supported destinations using log sinks. Here we created the log sinks to export the logs into the BigQuery partitioned table. The object `Sink` holds the inclusion/exclusion filter and destination. You can create/view sinks under the Logging–>Logs router section of your GCP project. For example, our inclusion filter reads:External Cloud Storage LogsIn our external cloud storage, logs for multiple projects accumulate in the same bucket. To select only the logs related to our project, we created a custom Python script and scheduled it to run daily to perform these tasks:Authenticate, read and select the data related to our project.Process the data.Loadthe processed data into BigQuery.We used the BigQuery stream ingestion API to stream our log data directly into BigQuery. There is also BigQuery Data Transfer Service (DTS) which is a fully managed service to ingest data from Google SaaS apps such as Google Ads, external cloud storage providers such as Amazon S3 and transferring data from data warehouse technologies such as Teradata and Amazon Redshift. DTS automates data movement into BigQuery on a scheduled and managed basis. Stage 2: Storage in BigQueryBigQuery organizes data tables into units called datasets. These datasets are scoped to a GCP project. These multiple scopes — project, dataset, and table — help structure information logically. In order to refer to a table from the command line, in SQL queries, or in code, we refer to it by using the following construct: `project.dataset.table`.BigQuery leverages the columnar storage format and compression algorithm to store data in Colossus, optimized for reading large amounts of structured data. Colossus also handles replication, recovery (when disks crash) and distributed management (so there is no single point of failure). Colossus enables BigQuery users to scale to dozens of petabytes of data stored seamlessly, without paying the penalty of attaching much more expensive compute resources as in traditional data warehouses.Keeping data in BigQuery is a best practice if you’re looking to optimize both cost and performance. Another best practice is using BigQuery’s table partitioning and clustering features to structure the data to match common data access patterns.When a table is clustered in BigQuery, the table data is automatically organized based on the contents of one or more columns in the table’s schema. The columns you specify are used to collocate related data. When new data is added to a table or a specific partition, BigQuery performs automatic re-clustering in the background to restore the sort property of the table or partition. Automatic reclustering is completely free and autonomous for users.A partitioned table is a special table that is divided into segments, called partitions, that make it easier to manage and query your data. You can typically split large tables into many smaller partitions using data ingestion time or TIMESTAMP/DATE column or an INTEGER column. BigQuery supports the following ways of creating partitioned tables :Ingestion time partitioned tablesDATE/TIMESTAMP column partitioned tablesINTEGER range partitioned tablesWe used ingestion time partitioned BigQuery tables as our data storage. Ingestion time partitioned tables are:Partitioned on the data’s ingestion time or arrival time.BigQuery automatically loads data into daily, date based partitions reflecting the data’s ingestion or arrival time.Partition management is key to fully maximizing BigQuery performance and cost when querying over a specific range — it results in scanning less data per query, and pruning is determined before query start time. While partitioning reduces cost and improves performance, it also prevents cost explosion due to users accidentally querying really large tables in entirety.Ingested log data in BigQueryStage 3: Parse and Process DataBefore we can analyze the raw log data we’ve stored in BigQuery, we need to process it so that it can be more easily queried.Parsing the DataWe used a Python script to massage the raw log data. Our script reads the raw logs we stored in BigQuery partitioned tables, parses them to break down the data, and then stores those refined results in a new BigQuery partitioned table store with more defined columns. We also integrated with MaxMind IP geolocation services to perform IP reverse lookup, and better visualize usage by organization. There are client libraries available for most of the popular languages in order to make API calls to BigQuery.Our Python script runs daily to process the ingested data and return it to BigQuery:Parsed Data in BigQueryAnalyzing the DataBigQuery is highly efficient running multiple concurrent complex queries in very large datasets. The BigQuery compute engine is Dremel, a large multi-tenant cluster that executes SQL queries. Dremel dynamically apportions slots to queries on an as-needed basis, maintaining fairness for concurrent queries from multiple users. A single user can get thousands of slots to run their queries. In between storage and compute is ‘shuffle’, which takes advantage of Google’s Jupiter network to move data extremely rapidly from one place to another.When we run queries in BigQuery, the result sets can be materialized to create new tables rather than storing in temp tables. In this way, we can join data from multiple tables and store in new ones with just one click and hand it over to anybody who does not have access to all those datasets tables by exporting it to GCS or exploring with Google Sheets or Data Studio.Stage 4: Visualize To visualize this processed data, we used GCP Data Studio, a free service that has petabyte-scale processing power and end-to-end integration with the rest of Google Cloud Platform.Data Studio supports 14 Google ecosystem connectors, including BigQuery. One of the unique and beneficial features of Google Data Studio is that it promotes collaboration with other Google Workspace apps. This made it a perfect choice for our BI tool. We created a datasource by selecting the project, dataset, and table we want to visualize. Clicking Explore with Data Studio creates a new report page with options to add charts, filters, and metrics.Big Data for Big DevOpsWith the ability to collect, process, and visualize this huge amount of log data, our JFrog operations teams are better able to understand the evolving needs of our SaaS customers, and to keep their accounts running smoothly on Google Cloud Platform.Through this, we’re even more keenly aware of how JFrog’s enterprise customers rely on the 100s of terabytes of artifact data that collectively pass through their artifact repositories each day to practice DevOps in the cloud. BigQuery’s ability to process this big data helps keep us — and our customers — running at peak efficiency.To accelerate your releases and delivery at enterprise scale through cloud DevOps, check out an Enterprise+ Cloud subscription to the JFrog Platform on Google Cloud Marketplace.
Quelle: Google Cloud Platform

13. November 2021

da Agency

Docker’s Developer Community: Wind In Our Sails

Two years ago, in November 2019, we refocused our company on the needs of developers. At the time, we recognized the growing adoption of microservices, the explosion in the number of tools, and the many opportunities to simplify these complexities. Little did we know that within months the world would face a global pandemic and economic recession, and that our company would quickly shift to 100% virtual, work-from-home operations. We also didn’t anticipate how the pandemic would increase the demand for application development, further accelerating the importance of developer productivity.

And throughout all these ups and downs the Docker team stayed the course. I thank them for their unwavering focus on serving our developer community. And I thank our Docker Captains, our community leaders, our ecosystem partners, our customers and the wider Docker community for their feedback, loyalty, and trust. Your support these last two years enabled us to not just survive, but to thrive.

Delivering Collaboration, Speed, and Security for Developers

The global tragedy of the pandemic accelerated digital initiatives in every industry while the “new normal” of work-from-home raised the importance of collaboration within “virtual-first” development teams. In response, to simplify the complexities of modern app development facing these teams we shipped Docker Development Environments. With it, team members can quickly and easily share with each other reproducible development environments, enabling them to spend more time writing code instead of installing tools, setting environment variables, and untangling dependencies.

This “need for speed” triggered by the pandemic is somewhat addressed by the decade-long rise in custom silicon. Yet how can developers take advantage of custom silicon performance without re-writing their applications or learning unfamiliar tools? Since our re-focusing in November 2019 we’ve shipped new capabilities that address this challenge, including support for Arm-based Apple M1 silicon in Docker Desktop, NVIDIA GPU support in Docker Desktop and Docker Compose-to-AWS, and RISC-V support in BuildKit. In doing so, we’re helping developers to get the speed benefits of custom silicon “for free” without having to make changes to their app or learn a new toolchain.

Sadly, the rise in online activity catalyzed by the pandemic is attracting the attention of criminals, resulting in an increase in the frequency and sophistication of attacks on organizations’ software supply chains. For this challenge, given the heterogeneity of customers’ environments Docker and our partners have taken a multi-vendor, open standards-based approach. This includes delivering CNCF-based Notary v2 for digital signatures, Docker Hub-interoperable container registry partnerships with AWS, Mirantis, and JFrog, and trusted content partnerships with Canonical, Red Hat, VMware, and other leading commercial ISVs.

The results? More development teams than ever are using Docker as the fastest, most secure way to build, share, and run modern applications. In fact, since our refocusing on developers two years ago our community has grown to 15.4 million monthly active developers sharing 13.7 million apps at a rate of 14.7 billion pulls per month. Moreover, for the second year in a row Stack Overflow’s Developer Survey ranked Docker as the #1 most wanted development tool, and JetBrains’ annual survey rated Docker Compose as the most popular container development tool, used by 58% of respondents.

We’re Just Getting Started

While we’re humbled by the results of these first two years, we also believe we’re just getting started – there’s still so much to do. In particular, the pandemic-triggered demand for apps is accelerating the demand for developer talent. Specifically, the demand for developers is growing 8X faster than the average demand for other occupations, with the market projected to be 45 million developers by the end of this decade. What are the implications for our industry and for Docker?

First, to state the obvious, more developers means more apps. And whether it’s the modernization of traditional apps or the creation of new, Kubernetes-destined ones both mean increased complexity. The result is a larger attack surface of software supply chains for criminals to attack.

With our presence at both ends of the supply chain, Docker is uniquely positioned to help. As the source of trusted content at the beginning of the supply chain – Docker Official Images, Docker Verified Publisher images of commercial ISV partners, and Docker-sponsored open source projects – developers can trust the foundational building blocks of their apps from the start. On the other end of the supply chain, Docker Desktop, we’re providing tools for developers to discover trusted content, verify its integrity, and ensure its ongoing freshness. And for their managers, we give them SaaS-delivered visibility and controls for images pulled from registries as well as created within Docker Desktop. This allows development teams to ship quickly and safely – no need to trade one off for the other.

Second, increasingly developers want the freedom to choose best-of-breed tools in order to take advantage of the latest innovations in app development. In working with ecosystem partners on open standards like OCI, compose-spec, CNCF Distribution, and others, we’ve delivered innovation for developers such as Compose-to-cloud tooling for AWS and Azure, image build automations with GitHub and Bitbucket repos and GitHub Actions, image vulnerability scanning with Snyk, and more.

Going forward, so as to provide developers even more choice in ecosystem partner tools we are making it even easier for partners to integrate with Docker tools, services, and content by expanding the breadth and depth of our product interfaces. Around this we are providing discovery and assurance services so that developers know these ecosystem partner integrations are safe, maintained, and supported. Furthermore, our SaaS-delivered management plane gives managers visibility into integration usage and a means to set and enforce integrations policy. This combination gives developers the freedom to choose their tools, safely.

Third, daily more and more developers are joining hundreds of thousands of Docker community members freely sharing with each other their time, expertise, and joy in using Docker for app development. And whether it’s our public roadmap, awesome-compose contributions, community Slack, community meetups, or 80,000-participant DockerCons you’ll find a friendly, enthusiastic crowd. Moreover, it’s a community that welcomes newcomers just getting started, and thus plays an important role in sustainably scaling Docker adoption as tens of millions more new developers join the community in the years to come.

But besides stars and reviews on image repos on Docker Hub, the wellspring of community recommendations, best practices, and cool hacks isn’t accessible directly in the product to other community members. Wouldn’t it be cool if, for example, the Dockerfile optimization someone in your company a continent away just figured out was automatically made visible and available to everyone in your company working with a similar image? We think so, too! We see so much potential in enabling community members to help and learn from each other, and we can’t wait to share more with you.

Fair Skies Ahead

In these last two years since our refocusing on developers, we’re humbled by the non-stop growth in the Docker community, the enthusiastic feedback and adoption of the new features we’ve shipped, and the positive support for the business changes we’ve made to enable us to sustainably scale Docker to tens of millions more developers. And with the accelerating demand for new apps and more developers to build, share, and run them, we’re incredibly excited about the next chapters in our journey together with you!

Thank you, and let’s keep shipping!

PS – We can’t wait to share details of the above and more with you at next year’s DockerCon on Tuesday, May 10, 2022. Save the date, it’s gonna be a blast! Register today at https://www.docker.com/dockercon/

The post Docker’s Developer Community: Wind In Our Sails appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/