Cloud Computing Köln - Seite 1952 von 7019 - Neues zu Cloud Computing, Internet of Things und Technologien

Google Cloud is a great home for your high performance computing (HPC) workloads. As with all things Google Cloud, we work hard to make complex tasks seem easy. For HPC, a big part of user friendliness is support for popular tools such as schedulers.If you run high performance computing (HPC) workloads, you’re likely familiar with the Slurm workload manager. Today, with SchedMD, we’re announcing the newest set of features for Slurm running on Google Cloud, including one-click hybrid configuration, Google Cloud Storage data migration support, real-time configuration updates, Bulk API support, improved error handling, and more. You can find these new features today in the Slurm on Google Cloud GitHub repository and on the Google Cloud Marketplace.Slurm is one of the leading open-source HPC workload managers used in TOP 500 supercomputers around the world. Over the past five years, we’ve worked with SchedMD, the company behind Slurm, to release ever-improving versions of Slurm on Google Cloud. Here’s more information about our newest features:Turnkey hybrid configurationYou can now use a simple hybrid Slurm configuration setup script for enabling Google Cloud partitions in an existing Slurm controller, allowing Slurm users to connect an on-premise cluster to Google Cloud quickly and easily.Google Cloud Storage data migration supportSlurm now has a workflow script that supports Google Cloud Storage, allowing users to define data movement actions to and from storage buckets as part of their job. Note that Slurm can handle jobs with input and output data pointing to different Google Cloud Storage locations.Real-time Configuration UpdatesSlurm now supports post-deployment reconfiguration of partitions, with responsive actions taken as needed, allowing users to make changes to their HPC environment on-the-fly.Bulk API supportBuilding on the Bulk API integration completed in the Slurm scripts released last year, the newest scripts now support Bulk API’s Regional Endpoint calls, Spot VMs, and more.Clearer error handlingThis latest version of Slurm on Google Cloud will indicate the specific place (e.g. job node, node info, filtered log file, etc.) where an API error has occurred, and expose any underlying Google API errors directly to users. The scripts also add an “installing” animation and guidance on how to check for errors during the installation process if it takes a longer time than expected.Billing tracking in BigQuery and StackdriverYou can now access usage data in BigQuery, which you can merge with Google Cloud billing data to compute the costs of individual jobs, and track and display custom metrics for Stackdriver jobs.Adherence to Terraform and Image Creation best practicesThe Slurm image creation process has now been converted to a Packer-based solution. The necessary scripts are incorporated into an image and then parameters are provided via metadata to define the Ansible configuration, all of which follows Terraform and Image Creation best practices. All new Terraform resources now use Cloud Foundation Toolkit modules where available, and you can use bootstrap scripts to configure and deploy Terraform modules.Authentication ConfigurationYou can now enable or disable oslogin and install LDAP libraries (e.g. OSLogin, LDAP, Disabled, etc) across your Slurm cluster. Note that the admin must manually configure non-oslogin auth across the cluster.Support for Instance TemplatesFollowing on the Instance Template support launched in last year’s Slurm on Google Cloud version, you can now use additional Instance Template features launched in the intervening year (e.g. hyperthreading, Spot VM).Enhanced customization of partitionsThe latest version of Slurm on Google Cloud adds multiple ways to customize your deployed partitions including: Injection of custom prolog and epilog scripts, pre-partition startup scripts, and the ability to configure more Slurm capabilities on compute nodes.Getting startedThe Slurm experts at SchedMD built this new release. You can download this release in SchedMD’s GitHub repository. For more information, check out the included README. If you need help getting started with Slurm check out the quick start guide, and for help with the Slurm features for Google Cloud check out the Slurm Auto-Scaling Cluster codelab and the Deploying a Slurm cluster on Google Compute Engine and Installing apps in a Slurm cluster on Compute Engine solution guides. If you have further questions, you can post on the Slurm on Google Cloud Google discussion group, or contact SchedMD directly.Related ArticleIntroducing the latest Slurm on GCP scriptsThe latest version of Slurm for Google Cloud includes support for Terraform, the HPC VM Image, placement policies, Bulk API and instance …Read Article
Quelle: Google Cloud Platform

18. Mai 2022

da Agency

Twitter takes data activation to new heights with Google Cloud

Twitter is an open, social platform that’s home to a world of diverse people, perspectives, ideas, and information. We aim to foster free and global conversations that allow people to consume, create, distribute, and discover information about the topics they care about the most.Founded in 2006, Twitter keeps a watchful eye on emerging technologies to maintain a modern platform that can meet the needs of the changing times. These early investments helped accelerate Twitter’s product but predated modern open source equivalents. As a result of its desire to leverage more open source technologies to keep up with the changing times, Twitter wanted to use the data it collected to maximize the user experience. However, its past generation of operational tools highlighted a need to create less time-consuming and more reliable data processing techniques that allowed Twitter developers to automate complex, manual tasks to relieve developer burden. This presented an opportunity for Twitter to modernize its tools and glean valuable insights that would be transformative for the evolution of its products and partnerships with advertisers. With the plan to standardize and simplify its approach to data processing across its operations, Twitter progressively migrated its operations to BigQuery on Google Cloud.In the complex, competitive world of programmatic advertising, the relevance, quality, and interpretation of data insights are critical in a company’s ability to stay ahead of ever-changing needs. The ability to streamline its approach to large-scale data processing quickly became an anchor in Twitter’s plan to better align its goals with those of its advertisers and customers. With the recent migration of its advertising data from on-premises to Google Cloud, Twitter has leveraged several Google Cloud solutions, notably BigQuery and Dataflow, to facilitate this greater alignment.Leveraging BigQuery for improved advertising partnerships and data extractionAligning the goals of advertisers and customers with those of a company is a considerable challenge, but for a company with hundreds of millions of avid users like Twitter, developing and executing an approach that balanced the needs of all parties was proving to be a complex task. Pradip Thachile, a senior data scientist responsible for Twitter’s revenue team’s adoption of Google Cloud, likened the process to a kind of flywheel that allows the Twitter team to work in collaboration with advertising partners to develop and test hypothetical approaches that center its goals and those of advertising partners. He explained the essential role of the BigQuery solution in the synthesis of these goals with an eye on the optimization of business growth for all involved. “Mating all this is a nontrivial problem at scale. The only way we can accomplish it is by being able to build this kind of scientific learning flywheel. BigQuery is a critical component, because the velocity with which we can go from hypothesizing to actual action through BigQuery is huge.”As the anchoring service for the ingestion, movement, and the extraction of valuable insights from all data at Twitter, BigQuery is the engine of Twitter’s recent optimization of internal productivity and revenue growth.Data modeling for optimized productivity and value extraction with DataflowAs a fully managed streaming analytics service, Dataflow has proven to be a time-saving solution that contributes significantly to the enhancement of productivity at Twitter. Through the reduction of the time invested in manual tasks for scaling, Dataflow facilitates the seamless and effortless organization and templatization of the movement of the archetypal data sets at Twitter. With less time devoted to the calibration of operational tools, Twitter’s team can focus on the higher-value tasks related to the discovery and development of innovative ways to further leverage its data insights. Reliable support with data expertise from GoogleNotable for its expertise in data, Google Cloud contributed substantial technical support to Twitter. The Twitter team routinely accessed the Google Cloud product team for guidance on ingestion velocity as they leveraged the sizable ingestion capabilities of BigQuery for its data. At a higher level, the Google Cloud support team supplied valuable resources including white papers and use cases that could enhance Twitter’s performance. Thachile describes the value of Google Cloud’s support, “Google Cloud provides a very effective stratified layer of support. They can be as close to the problem as you’d like them to be.”For more of the story about how Twitter is using BigQuery, read this blog from Twitter.Related ArticleNow generally available: BigQuery BI Engine supports many BI tools or custom applicationLearn about BigQuery BI Engine and how to analyze large and complex datasets interactively with sub-second query response time and high c…Read Article
Quelle: Google Cloud Platform

18. Mai 2022

da Agency

Introducing Autonomic Security Operations for the U.S. public sector

As sophisticated cyberattack campaigns increasingly target the U.S. public and private sectors during the COVID era, the White House and federal agencies have taken steps to protect critical infrastructure and remote-work infrastructure. These include Executive Order 14028 and the Office of Management and Budget’s Memorandum M-21-31, which recommend adopting Zero Trust policies, and span software supply chain security, cybersecurity threat management, and strengthening cyberattack detection and response.However, implementation can be a challenge for many agencies due to cost, scalability, engineering, and a lack of resources. Meeting the requirements of the EO and OMB guidance may require technology modernization and transformational changes around workforce and business processes. Today we are announcing Autonomic Security Operations (ASO) for the U.S. public sector, a solution framework to modernize cybersecurity analytics and threat management that’s aligned with the objectives of EO 14028 and OMB M-21-31. Powered by Google’s Chronicle and Siemplify, ASO helps agencies to comprehensively manage cybersecurity telemetry across an organization, meet the Event Logging Tier requirements of the White House guidance, and transform the scale and speed of threat detection and response. ASO can support government agencies in achieving continuous detection and continuous response so that security teams can increase their productivity, reduce detection and response time, and keep pace with – or ideally, stay ahead of – attackers. While the focus of OMB M-21-31 is on the implementation of technical capabilities, transforming security operations will require more than just technology. Transforming processes and people in the security organization is also important for long-term success. ASO provides a more comprehensive lens through which to view the OMB event logging capability tiers, which can help drive a parallel transformation of security-operations processes and personnel.Modern Cybersecurity Threat Detection and ResponseGoogle provides powerful technical capabilities to help your organization achieve the requirements of M-21-31 and EO 14028:Security Information & Event Management (SIEM) – Chronicle provides high-speed petabyte-scale analysis, and is capable of consuming log types outlined in the Event Logging (EL) tiers in a highly cost-effective manner.Security Orchestration, Analytics, and Response (SOAR) – Siemplify offers dozens of out-of-box playbooks to deliver agile cybersecurity response and drive mission impact, including instances of automating 98% of Tier-1 alerts and driving an 80% reduction in caseload.User and Entity Behavior Analytics (UEBA) – For agencies that want to develop their own behavioral analytics, agencies can use BigQuery, Google’s petabyte scale data lake, to store, manage, and analyze diverse data types from many sources. Telemetry can be exported out of Chronicle, and custom data pipelines can be built to import other relevant data from disparate tools and systems, such as IT Ops, HR and personnel data, and physical security data. From there, users can leverage BQML to readily generate machine learning models without needing to move the data out of BigQuery. For Google Cloud workloads, our Security Command Center Premium product offers native, turnkey UEBA across GCP workloads.Endpoint Detection and Response (EDR)– For most agencies, EDR is a heavily adopted technology that has broad applicability in Security Operations. We offer integrations to many EDR vendors. Take a look at our broad list of Chronicle integrations here.Threat intelligence – Our solution offers a native integration with VirusTotal, has the ability to operationalize threat intelligence feeds natively in Chronicle, and integrates with various TI and TIP solutions.Community Security AnalyticsTo increase collaboration across public-sector and private-sector organizations, we recently launched our Community Security Analytics (CSA) repository, where we’ve partnered with the MITRE Engenuity Center for Threat-Informed Defense, CYDERES, and others to develop open-source queries and rules that support self-service security analytics for detecting common cloud-based security threats. CSA queries are mapped to the MITRE ATT&CK® framework of tactics, techniques and procedures (TTPs) to help you evaluate their applicability in your environment and include them in your threat model coverage.“Deloitte is excited to collaborate with Google Cloud on their transformational public sector Autonomic Security Operations (ASO) solution offering. Deloitte has been recognized as Google Cloud’s Global Services Partner of the Year for four consecutive years, and also as their inaugural Public Sector Partner of the Year in 2020,” said Chris Weggeman, managing director of GPS Cyber and Strategic Risk, Google Cloud Cyber Alliance Leader, Deloitte & Touche LLP. “Our deep bench of more than 1,000 Google Cloud certifications, capabilities spanning the Google Cloud security portfolio, and decades of delivery experience in the government and public sector makes us well-positioned to help our clients undertake critical Security Operations Center transformation efforts with Google Cloud ASO.”Cost-effective for government agenciesTo help Federal Agencies meet the requirements of M-21-31 and the broader EO, Google’s ASO solutions can drive efficiencies and help manage the overall costs of the transformation. ASO can make petabyte-scale data ingestion and management more viable and cost-effective. This is critical at a time when M-21-31 is requiring many agencies to ingest and manage dramatically higher volumes of data that had not been previously budgeted for. PartnersWe’re investing in key partners who can help support U.S. government agencies on this journey. Deloitte and CYDERES both have deep expertise to help transform agencies’ Security Operations capabilities, and we continue to expand our partners to support the needs of our clients. A prototypical journey can be seen below.“Cyderes shares Google Cloud’s mission to transform security operations, and we are honored to deliver the Autonomic Security Operations solution to the U.S. public sector. As the number one MSSP in the world (according to Cyber Defense Magazine’s 2021 Top MSSPs List) with decades of advisory and technology experience detecting and responding to the world’s biggest cybersecurity threats, Cyderes is uniquely positioned to equip federal agencies and departments to go far beyond the requirements of the executive order to transform their security programs entirely via Google’s unique ASO approach,” said Robert Herjavec, CEO of CYDERES. “As an original launch partner of Google Cloud’s Chronicle, our deep expertise will propel our joint offering to modernize security operations in the public sector, all with significant cost efficiency compared to competing solutions.” said Eric Foster, President of CYDERES.Embracing ASOAutonomic Security Operations can help U.S. government agencies advance their event logging capabilities in alignment with OMB maturity tiers. More broadly, ASO can help the U.S. government undertake a larger transformation of technology, process, and people, toward a model of continuous threat detection and response. As such, we believe that ASO can help address a number of challenges presently facing cybersecurity teams, from the global shortage of skilled workers, to the overproliferation of security tools, to poor cybersecurity situational awareness and analyst burnout caused by an increase of data without sufficient context or tools to automate and scale detection and response.We believe that by embracing ASO, agencies can help agencies achieve:10x technology, through the use of cloud-native tools that help agencies meet event logging requirements in the near term, while powering a longer-term transformation in threat management; 10x process, by redesigning workflows and using automation to achieve Continuous Detection and Continuous Response in security operations; 10x people, by transforming the productivity and effectiveness of security teams and expanding their diversity; and10x influence across the enterprise through a more collaborative and data-driven approach to solving security problems between security teams and non-security stakeholders.To learn more about Google’s Autonomic Security Operations solution for the U.S. public sector, please read our whitepaper. More broadly, Google Cloud continues to provide leadership and support for a wide range of critical public-sector initiatives, including our work with the MITRE Engenuity Center for Threat-Informed Defense, the membership of Google executives on the President’s Council of Advisors on Science and Technology and the newly established Cyber Safety Review Board; Google’s White House commitment to invest $10 billion in Zero Trust and software supply chain security, and Google Cloud’s introduction of a framework for software supply chain integrity. We look forward to working with the U.S. government to make the nation more secure.Visit our Google Cloud for U.S. federal cybersecurity webpage.Related posts:Autonomic Security Operations for the U.S. Public Sector Whitepaper“Achieving Autonomic Security Operations: Reducing toil”“Achieving Autonomic Security Operations: Automation as a Force Multiplier”“Advancing Autonomic Security Operations: New resources for your modernization journey”Related ArticleRead Article
Quelle: Google Cloud Platform

18. Mai 2022

da Agency

Microsoft and AT&T demonstrate 5G-powered video analytics

In November 2021, Microsoft and AT&T announced the launch of Azure public MEC (multi-access edge compute) with a site in Atlanta, Georgia. The Azure public MEC solution enables low-latency applications at the edge of the mobile operator’s network, providing Azure compute services integrated with 5G connectivity. Azure public MEC is designed to run AI and machine learning workloads that require intensive compute and low latency network. The access to these resources is over high-quality 5G connections from phones, smart cameras, IoT devices, and other equipment. Enterprises and developers can build and run these low-latency applications and manage their workloads using the same tools they are using to run applications in the Azure public cloud.

To light up new compelling applications with Azure public MEC that benefit from low latency 5G connectivity, we are making available a video analytics library under the umbrella of Edge Video Services.

Edge Video Services

Edge Video Services (EVS) is a Microsoft platform for developing video analytics solutions that can be deployed on Azure public MEC. For example, consider some smart city applications like our Vision Zero work with the City of Bellevue, which enabled a new generation of real-time traffic flows leading to substantial improvements in the day-to-day lives of commuters. Similarly, real-time video analytics can make cities safer by controlling traffic lights for situations such as allowing a person in a wheelchair to safely cross the street. A related application, which we demonstrated at Hannover Messe 2016, integrated an early version of EVS into traffic light cameras and those in self-driving cars to analyze videos to help reduce accidents and fatalities. Other new applications that are coming soon include improving transportation systems, monitoring air quality, street lighting, smart parking, crowd management, and emergency management. Beyond smart cities, EVS can provide modern smart enterprises with end-to-end experiences with video analytics for mixed reality as a natural component of 5G network solutions. Additional examples include managing machines and robots in connected factories, handling customer demands and services in retail stores and restaurants or tracking pedestrian traffic in sports arenas.

Figure 1: EVS architecture stack.

As shown in Figure 2 below, 5G compute infrastructure has a hierarchy of intelligent components including Azure Percept devices, Azure private MEC, and Azure public MEC. EVS integrates with all of these solutions and provides these features:

Inter-edge orchestrator to manage network traffic involving multiple public MECs. It deploys application containers across the edge hierarchy for high availability and fault tolerance.
Network monitoring and adaptation to continuously monitor the dynamic wireless and wired network connections, adapting application demands accordingly.
Dynamic resource allocation for video machine learning containers. This adapts based on the load generated from the mobile network and the workloads deployed in the on-premises edge location.

Smart cities deployment at Azure public MEC with AT&T in Atlanta

Working with AT&T, Microsoft demonstrated the value of EVS on the Azure public MEC connected to the AT&T’s 5G network in Atlanta. The setup consisted of an on-premise edge device, managed by Azure IoT Hub and an Azure Kubernetes cluster as shown in the diagram below.

Figure 2: Azure public MEC and AT&T deployment.

The EVS orchestrator places the various containers across the on-premises edge and Azure public MEC. This split execution requires only lightweight compute power on-premises, and also removes the need to provision high bandwidth connectivity out of the on-premises edge.

In our Atlanta deployment, we demonstrated EVS’s split architecture, with lightweight execution at the on-premises edge. It transferred 230MB of data over the 5G link out of the on-premise edge over 24 hours, by contrast, 9.5GB of data would have been sent if all the encoded video were transferred out. In other words, EVS reduced the network utilization by 42x. This network saving was obtained with a CPU-only edge on-premises with no loss in accuracy. Our measurements also showed that network latencies to the Azure public MEC were about 6x lower at the median compared to the nearest Azure region, which translated to faster responses for the application.

EVS is integrated with AT&T’s network APIs to obtain real-time information about the 5G network. As a result, EVS adapts the amount of traffic transferred between the edges, depending on any fluctuations to the latency and bandwidth of the 5G link. EVS uses Azure Traffic Manager to support automatic failover for the Azure public MEC to the nearest Azure region, thus ensuring no disruption to the video application. When failing over to the Azure region, EVS adapts to the changed and increased latency by adjusting the amount of traffic sent out of the on-premises edge via changes to the encoder and machine learning model parameters with minimal impact on application accuracy. EVS is also cognizant of other containers executing at the edges and can elastically scale up or down its compute requirements.

EVS on Azure public MEC: Try it out today

For your video scenarios, we encourage you to try out EVS on Azure public MEC with your own on-premises edge devices. The reference architecture and instructions are available in our GitHub repository. The repository also includes a sample video of cars entering a parking lot that you can use to test EVS for counting cars. To submit feedback about EVS, please email to evs-support@microsoft. Please note this is only for submitting feedback, you will not be contacted.
Quelle: Azure

18. Mai 2022

da Agency

Announcing Azure DNS Private Resolver: Now in preview

A quick overview of Azure DNS

We offer two types of Azure DNS Zones—private and public—for hosting your private DNS and public DNS records.

Azure Private DNS: Azure Private DNS provides a reliable and secure DNS service for your virtual network. Azure Private DNS manages and resolves domain names in the virtual network without the need to configure a custom DNS solution. By using private DNS zones, you can use your own custom domain name instead of the Azure-provided names during deployment.
Azure Public DNS: DNS domains in Azure DNS are hosted on Azure's global network of DNS name servers. Azure DNS uses anycast networking. Each DNS query is answered by the closest available DNS server to provide fast performance and high availability for your domain.

More information on additional services part of the Azure DNS offering can be found in the Azure DNS product page.

What is being announced today?

Azure Private DNS Resolver enables you to query Azure Private DNS Zones from an on-premises environment and vice versa without provisioning IaaS-based DNS solutions.

Azure DNS Private Resolver preview is being announced to all customers and will have regional availability in the following regions:

Australia East
UK South
North Europe
South Central US
West US 3
East US
North Central US
Central US EUAP
East US 2 EUAP
West Central US
East US 2
West Europe

All of these regions support Availability Zones and will help with the regional and global resiliency story of customer workloads!

What will customers be able to do today that they couldn’t do before?

Customers will no longer need to provision IaaS-based solutions on their virtual networks to resolve names registered on Azure Private DNS Zones and will be able to do conditional forwarding of domains back to on-premises, across multicloud providers, and public DNS servers.

This solution will also work with your existing Azure ExpressRoute, Azure VPN or Azure Bastion setup in a seamless way.

Customers will also be able to manage their DNS settings at a Virtual Network level in a very simplified way by linking rules to each of their Virtual Networks and enabling conditional forwarding at scale.

Private access to your Private DNS Zones

Conditionally forward from your virtual networks to any reachable DNS server and from on-premises to Azure Private DNS Zones.

Plus, the following benefits

Zero Maintenance: Fully managed service which does not require you to patch or plan for any downtime of your service.
Cost Reduction: Run at a fraction of traditional IaaS solutions which typically would require planning for high availability, resiliency, and backup of configurations.
Highly Available: Built-in high availability, zone redundancy. You will no longer need to plan for availability zones awareness nor how many instances to provision per region Azure availability zones are physically separate locations within each Azure region that are tolerant to local failures and are connected by a high-performance network with a round-trip latency of less than 2ms.
DevOps Friendly: Build your pipelines with Terraform, Azure Resource Manager, REST API support, Go, Typescript/Javascript. This will allow you to keep a consistent configuration and experience across regions and different instances of your service.

Get started and share your feedback

You can try Azure DNS Private Resolver today. For more information about the capabilities available, please visit the Azure DNS Private Resolver technical documentation. Post your ideas and suggestions on the networking community page. More information on additional services part of the Azure DNS offering can be found in the Azure DNS product page.
Quelle: Azure