Cloud Computing Köln - Seite 1778 von 7007 - Neues zu Cloud Computing, Internet of Things und Technologien

Dataflow has provided a number of capabilities to improve utilization and efficiency by automatically provisioning and scaling resources for your job. The following are some of the examples:Horizontal Autoscaling that automatically scales the number of workers.Streaming Engine, which decouples storage from the workers. This also gives workers access to unbounded storage and more responsive Horizontal Autoscaling.Dynamic work rebalancing that splits work across available workers based on work progress. Building on this solid and differentiated foundation, we recently launched Dataflow Prime, a new next generation serverless, no-ops, autotuning platform for your data processing needs on Google Cloud. Dataflow Prime introduces a new industry-first resource optimization technology, Vertical Autoscaling, which automatically scales worker memory in order to remove the need to do manual tuning of worker configuration. With Vertical Autoscaling, Dataflow Prime automatically determines the right worker configuration for your job. Current user challenges With Dataflow, you write your data processing logic using the Apache Beam SDK or Dataflow Templates and let Dataflow handle the optimization, execution and scalability of your pipeline. While in many cases your pipeline executes well, for some cases you have to manually select the right resources like memory for best performance and cost. For many users, this is a time consuming trial and error process and a single worker configuration is unlikely to be optimal for the pipeline. In addition, there was the risk of static configurations becoming outdated when data processing requirements changed. We have designed Vertical Autoscaling to solve these challenges and allow you to focus on your application and business logic. How does Vertical Autoscaling work?Vertical Autoscaling observes out of memory (OOM) events and memory usage of your streaming pipeline over time and triggers memory scaling based on this. This makes your pipeline resilient to out of memory errors without any manual intervention.With Vertical Autoscaling, if there is high memory utilization, all workers in your job are replaced with workers with larger memory capacity. In the following illustration we see that workers 1, 2, and 3 have high memory utilization and a capacity of 4GB. After Vertical Autoscaling, all workers have a memory capacity of 5 GB, which gives them sufficient memory headroom.This process is iterative, and it can take up to a few minutes to replace the workers.Similarly, if there is low memory usage, Vertical Autoscaling downscales the workers to lower memory capacity, thus improving utilization and saving cost. It relies on historical usage data per pipeline to know when it is safe to scale down, prioritizing pipeline stability. You may observe a long period of time (12 hours or more) where no downscaling occurs after a spike in memory utilization. Vertical Autoscaling takes a conservative approach to downscaling in order to keep pipelines processing with minimal disruption.Things to know about Vertical Autoscaling How does Vertical Autoscaling impact my job?As the workers are replaced, you may observe a temporary drop in throughput, but impact to a running pipeline (i.e. backlog, watermark, throughput metrics) will not be significantly different from a Horizontal Autoscaling event.Horizontal Autoscaling is disabled during and up to 10 minutes after Vertical Autoscaling.As with horizontal scaling, some backlog may accumulate during the scaling process, if this backlog cannot be cleared in a timely fashion, horizontal scaling may occur to clear that backlog.Does Vertical Autoscaling remove all OOMs?It is important to note that Vertical Autoscaling is designed to react to OOMs and high memory usage, but cannot necessarily prevent OOMs, especially if there is a fast spike in memory usage on a worker which results in an OOM.When OOMs occur, Vertical Autoscaling automatically detects them and resize worker memory to address issues. As a consequence, you will see a few OOM errors in the worker logs but these can be ignored if those are followed by upscale events.It is also important to note that some OOMs may be happen as a result of downscale events where Dataflow reduced the amount of memory because of underutilization. In such cases, Dataflow will automatically upsize if it detects OOMs. Again, it is safe to ignore these OOM messages if they are followed by upscale events.If the OOM messages are not followed by an upscale event, you may have hit the memory scaling limit. In this case you may need to optimize your pipeline’s memory usage or use resource hints.If you see OOM messages continuously and have not observed a job message indicating you have hit the memory scaling limit, please contact the support team. Note that if OOMs occur very rarely (e.g. once every few hours per pipeline), Vertical Autoscaling may choose to not scale up the workers to avoid introducing additional disruption.How to enable Vertical Autoscaling ?Vertical Autoscaling is only available for Dataflow Prime jobs. See the instructions to launch Dataflow Prime jobs and how to enable Vertical Autoscaling. You don’t have to make any code changes to run your existing Apache Beam pipeline on Dataflow Prime. Additionally, you don’t have to specify the worker type when launching a Dataflow Prime job. However if you want to control the initial worker’s resource configuration you can use resource hints.You can confirm if Vertical Autoscaling is running on your pipeline by looking for the following job log:Vertical Autoscaling is enabled. This pipeline is receiving recommendations for resources allocated per worker. How to monitor Vertical Autoscaling ?Whenever Vertical Autoscaling updates workers with more or less memory, the following job logs are generated in Cloud Logging: Vertical Autoscaling is enabled. This pipeline is receiving recommendations for resources allocated per worker. Vertical Autoscaling update triggered to change per worker memory limit for pool from X GiB to Y GiB. You can read more about these logs in this section.Additionally you can visually monitor Vertical Autoscaling by looking at worker capacity under the ‘Max worker memory utilization’ chart in Dataflow metrics UI.Following is a chart for a Dataflow worker that Vertically Autoscaled. We see three Vertical Autoscaling events for this job. Whenever the memory used was close to memory capacity, Vertical Autoscaling triggered and scaled up the worker memory capacity.SummaryTryVertical Autoscaling for your streaming jobs on Dataflow Prime for improved resource optimization and cost savings.There is no code change required to run your existing Apache Beam pipeline on Dataflow Prime.There is no additional cost associated with using Vertical Autoscaling. Dataflow Prime jobs continue to be billed based on the resources it consumes.Related ArticleThe next generation of Dataflow: Dataflow Prime, Dataflow Go, and Dataflow MLDataflow is GCP’s Cloud Native way for all data processing workloads, powered by the universal batch and streaming model of Apache Beam.Read Article
Quelle: Google Cloud Platform

30. August 2022

da Agency

Introducing on-demand backup, schema extension support for Google Cloud’s Managed Microsoft AD

Managed Service for Microsoft Active Directory (Managed Microsoft AD) is a Google Cloud service that offers highly available, hardened Microsoft Active Directory running on Windows virtual machines. We recently added on-demand backup and schema extension capabilities that can help Google Cloud users more easily and effectively manage AD tasks. Managed Microsoft AD is a fully managed service with automated AD server updates, maintenance, and security configuration, and needs no hardware management or patching. The service is constantly evolving, adding new capabilities to effectively manage your cloud-based, AD-dependent workloads. Here’s a closer look at the benefits for Google Cloud users of the new on-demand backup and schema extension capabilities.Flexibility to manage your AD domain with on-demand backup and restoreManaged Microsoft AD already offers scheduled backups which are taken automatically every 12 hours. Now with on-demand backup and restore, customers will have the ability to create checkpoints (snapshots) at any point in time and restore back to that state when needed. The new on-demand backup and restore functionality is now generally available in addition to the scheduled backups. This functionality can provide flexibility for customers to initiate backup and recovery based on their unique needs. Here are two scenarios where on-demand backup and recovery can be used:Critical domain changes now can be done anytime without aligning to the next backup schedule. Users can restore to a point back in time from backups without having to raise a support request.With this release, users can create up to five on-demand backups. Managed Microsoft AD APIs also offer management functionalities for backups that includes listing of all backups (both on-demand and scheduled,) restoring to a selected backup, updating labels, and deleting a backup. All these capabilities help users to effectively manage their backup administrative tasks. Power application integrations with Schema Extension supportNote: Schema Extension feature is in public preview and covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Active Directory (AD) relies on schema to organize and store the directory data. The AD schema contains a formal definition of every attribute and class that can exist in an Active Directory object. When you create a Managed Microsoft AD instance, it creates a default schema on the domain controller as well. However, there can be a situation where you want to customize the classes or attributes. Such a need arises when you have applications that require new types of information to be stored in Active Directory (e.g., to support single sign-on capabilities). Managed Microsoft AD now supports schema extension and enables modification of the existing schema to customize attributes via API using an LDAP Data Interchange Format (LDIF) file. The following LDIF change types are supported: add, modify, modrdn and moddn. It is generally recommended to do a domain backup before schema changes are applied. To simplify this, Managed Microsoft AD initiates a backup every time schema changes are triggered. This schema extension support enables additional context for users and for integrating with applications that are dependent on specific classes or attributes.Use case: Schema extension for LAPSYou can store and rotate the local account passwords of domain-joined computers in AD using Local Administrator Password Solution (LAPS), a Microsoft tool for password management. Any device that LAPS is deployed to can randomize the local administrator password, store that password in Active Directory, and then change that password on a set schedule. For LAPS to work with Active Directory, it needs the schema to be extended for storing the required attributes. For this use case, we assume that you have already installed LAPS and have your Managed Microsoft AD up and running. LAPS requires the following two additional attributes:ms-Mcs-AdmPwd – This attribute stores the local administrator passwordms-Mcs-AdmPwdExpirationTime – This attribute stores the expiration time of administrator passwordLet’s now look at how to add the required attributes using the Managed Microsoft AD schema extension feature.Step 1: Prepare an LDIF file to add ms-Mcs-AdmPwd and ms-Mcs-AdmPwdExpirationTime attributes.code_block[StructValue([(u’code’, u’dn: CN=ms-Mcs-AdmPwd,CN=Schema,CN=Configuration,dc=example,dc=comrnchangetype: addrnobjectClass: attributeSchemarnldapDisplayName: ms-Mcs-AdmPwdrnadminDisplayName: ms-Mcs-AdmPwdrnadminDescription: LAPS PasswordrnattributeId: 1.2.840.113556.8000.9999.2.2rnattributeSyntax: 2.5.5.5rnoMSyntax: 19rnisSingleValued: TRUErnsystemOnly: FALSErnsearchFlags: 904rnschemaIdGuid:: 64e85e0a-f479-4206-880d-ecbf73e2babbrnrnrndn: CN=ms-Mcs-AdmPwdExpirationTime,CN=Schema,CN=Configuration,dc=example,dc=comrnchangetype: AddrnobjectClass: attributeSchemarnldapDisplayName: ms-Mcs-AdmPwdExpirationTimernadminDisplayName: ms-Mcs-AdmPwdExpirationTimernadminDescription: LAPS Password Expiration TimernattributeId: 1.2.840.113556.8000.9999.2.3rnattributeSyntax: 2.5.5.6rnoMSyntax: 65rnisSingleValued: TRUErnsystemOnly: FALSErnsearchFlags: 0rnschemaIdGuid:: b3fea135-c39a-4169-aec7-c618cc8cb6ffrnrndn:rnchangetype: modifyrnadd: schemaUpdateNowrnschemaUpdateNow: 1′), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3eaa5b788850>)])]Step 2: Login as a delegated administrator to your VM hosted in Google Cloud that was domain-joined with Managed Microsoft AD.Step 3: Extend the schema by running the following gCloud CLI command:code_block[StructValue([(u’code’, u’gcloud beta active-directory domains extend-schema DOMAIN_NAME –ldif-file=LDIF_FILE_PATH –description=u201dSample descriptionu201d –project=PROJECT_ID’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3eaa5b4f0d50>)])]Managed Microsoft AD creates a backup automatically when you initiate schema extension. You can use this backup to perform an authoritative restore, which returns the domain to a previous point before addition of these attributes. Step 4: To verify the schema changes, run the following command in Windows PowerShell:code_block[StructValue([(u’code’, u”get-adobject -Identity ‘cn=ATTRIBUTE,cn=Schema,cn=Configuration,dc=example,dc=com -Properties *”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3eaa5b4f0410>)])]The Managed Microsoft AD schema is now extended with the required attributes for configuring LAPS. You can now proceed with the rest of the LAPS setup as usual, including password settings, access permissions, and GPO configuration.These new features make it now easier to integrate applications with your Managed Microsoft AD and provide flexibility for operations like backup and restore. Here are additional resources where you can learn more about Managed AD and these new features.Managed Service for Microsoft AD documentationBackup and restore a domain in Managed Microsoft ADIntroduction to schema extension in Managed Microsoft ADExtend the schema in a Managed Microsoft ADRelated ArticleAnnouncing support for on-premises Windows workloads with Certificate Authority ServiceTo mitigate the challenges in managing and migrating digital certificates, Google Cloud is debuting a new technology partnership with CA …Read Article
Quelle: Google Cloud Platform