Introducing the latest Slurm on Google Cloud scripts

Google Cloud is a great home for your high performance computing (HPC) workloads. As with all things Google Cloud, we work hard to make complex tasks seem easy. For HPC, a big part of user friendliness is support for popular tools such as schedulers.If you run high performance computing (HPC) workloads, you’re likely familiar with the Slurm workload manager. Today, with SchedMD, we’re announcing the newest set of features for Slurm running on Google Cloud, including one-click hybrid configuration, Google Cloud Storage data migration support, real-time configuration updates, Bulk API support, improved error handling, and more. You can find these new features today in the Slurm on Google Cloud GitHub repository and on the Google Cloud Marketplace.Slurm is one of the leading open-source HPC workload managers used in TOP 500 supercomputers around the world. Over the past five years, we’ve worked with SchedMD, the company behind Slurm, to release ever-improving versions of Slurm on Google Cloud. Here’s more information about our newest features:Turnkey hybrid configurationYou can now use a simple hybrid Slurm configuration setup script for enabling Google Cloud partitions in an existing Slurm controller, allowing Slurm users to connect an on-premise cluster to Google Cloud quickly and easily.Google Cloud Storage data migration supportSlurm now has a workflow script that supports Google Cloud Storage, allowing users to define data movement actions to and from storage buckets as part of their job. Note that Slurm can handle jobs with input and output data pointing to different Google Cloud Storage locations.Real-time Configuration UpdatesSlurm now supports post-deployment reconfiguration of partitions, with responsive actions taken as needed, allowing users to make changes to their HPC environment on-the-fly.Bulk API supportBuilding on the Bulk API integration completed in the Slurm scripts released last year, the newest scripts now support Bulk API’s Regional Endpoint calls, Spot VMs, and more.Clearer error handlingThis latest version of Slurm on Google Cloud will indicate the specific place (e.g. job node, node info, filtered log file, etc.) where an API error has occurred, and expose any underlying Google API errors directly to users. The scripts also add an “installing” animation and guidance on how to check for errors during the installation process if it takes a longer time than expected.Billing tracking in BigQuery and StackdriverYou can now access usage data in BigQuery, which you can merge with Google Cloud billing data to compute the costs of individual jobs, and track and display custom metrics for Stackdriver jobs.Adherence to Terraform and Image Creation best practicesThe Slurm image creation process has now been converted to a Packer-based solution. The necessary scripts are incorporated into an image and then parameters are provided via metadata to define the Ansible configuration, all of which follows Terraform and Image Creation best practices. All new Terraform resources now use Cloud Foundation Toolkit modules where available, and you can use bootstrap scripts to configure and deploy Terraform modules.Authentication ConfigurationYou can now enable or disable oslogin and install LDAP libraries (e.g. OSLogin, LDAP, Disabled, etc) across your Slurm cluster. Note that the admin must manually configure non-oslogin auth across the cluster.Support for Instance TemplatesFollowing on the Instance Template support launched in last year’s Slurm on Google Cloud version, you can now use additional Instance Template features launched in the intervening year (e.g. hyperthreading, Spot VM).Enhanced customization of partitionsThe latest version of Slurm on Google Cloud adds multiple ways to customize your deployed partitions including: Injection of custom prolog and epilog scripts, pre-partition startup scripts, and the ability to configure more Slurm capabilities on compute nodes.Getting startedThe Slurm experts at SchedMD built this new release. You can download this release in SchedMD’s GitHub repository. For more information, check out the included README. If you need help getting started with Slurm check out the quick start guide, and for help with the Slurm features for Google Cloud check out the Slurm Auto-Scaling Cluster codelab and the Deploying a Slurm cluster on Google Compute Engine and Installing apps in a Slurm cluster on Compute Engine solution guides. If you have further questions, you can post on the Slurm on Google Cloud Google discussion group, or contact SchedMD directly.Related ArticleIntroducing the latest Slurm on GCP scriptsThe latest version of Slurm for Google Cloud includes support for Terraform, the HPC VM Image, placement policies, Bulk API and instance …Read Article
Quelle: Google Cloud Platform

Cutting-edge disaster recovery for critical enterprise applications

Enterprise data backup and recovery has always been one of the most compelling and widely adopted public cloud use cases. That’s still true today, as businesses leverage the cloud to protect increasingly critical applications with stricter RTO/RPO requirements.Veeam and Google Cloud have long been leaders at providing reliable, verifiable, cloud-based recovery solutions across any environment or application. And now, we’re taking another step in that direction with the introduction of Continuous Data Protection (CDP) disaster recovery for business-critical Tier One applications. Veeam Backup & Replication (VBR) and Veeam Backup for Google Cloud (VBG), available on Google Cloud Marketplace, offer enterprises a faster, simpler, and more cost-effective way to level up your company’s backup and recovery capabilities. Enterprise customers can take control and craft a backup and storage strategy based on their SLA requirements and RTO/RPO goals, rather than cost, capacity, or scalability constraints. And with Google Cloud, enterprises get the secure, global cloud infrastructure and applications they need to achieve value with digital transformation.3 ways Veeam and Google Cloud elevate your company’s backup and recovery gameMore than ever, businesses are adopting cloud migration and modernization strategies to cut costs, simplify and streamline IT overhead, and enable innovation. And with four out of five organizations planning to use either cloud storage or a managed backup service within the next two years¹, many will be looking to understand just how and why the cloud can help them protect their businesses and serve their big-picture cloud objectives.There are a lot of different ways to tackle these questions when it comes to leveraging VBR and VBG on Google Cloud infrastructure. We’ll focus here on a few that appear to be top of mind with many of our customers.Cloud-based CDP for business-critical applications. Disaster recovery (DR) for critical Tier One applications doesn’t leave much room for error: Many of these applications will measure RTOs and RPOs in minutes or even seconds to avoid a major business disruption.In some cases, these applications use dedicated, high-availability infrastructure to maintain independent disaster recovery capabilities. In many others, however, it falls upon IT to maintain an on-prem CDR solution, running on dedicated DR infrastructure, to ensure near real-time RTOs/RPOs for enterprise Tier One applications.VBR on Google Cloud gives these enterprises a complete and fully managed CDR solution delivering RPOs measured in seconds. And by running VBR on Google Cloud’s highly secure, global cloud infrastructure, even the most advanced enterprise IT organizations can deploy a DR environment that will match or exceed their on-prem capabilities — with none of the CapEx, overhead costs, or management headaches.Right-sizing your enterprise backup strategy. Of course, many enterprise applications don’t require this level of protection, especially in terms of RPOs. In many cases, snapshot-based replication, typically with 6-12-hour RPOs, is enough for a business to recover less critical systems without suffering a major business setback.Veeam customers get the flexibility they need to choose the right type of protection for their applications and business data. They can easily store both VM replicas and an unlimited number of Veeam backups in Google Cloud, and restore from either source. Google’s Archive tier of object storage gives VBG customers one of the industry’s most cost-effective long-term storage solutions—while still achieving relatively fast RTOs.Running Veeam on Google Cloud also solves the scalability challenges that so many enterprises face when they manage on-prem systems. With Veeam and Google Cloud, an organization’s DR and backup capabilities will always align seamlessly with business needs.For example, resizing a Google Cloud VMware Engine (GCVE) cluster or spinning up additional clusters is something that can happen on the fly to accommodate restores and migrations. There’s no need to worry about overprovisioning and, with Veeam’s Universal Licensing, no additional licenses are required to migrate to the cloud. Customers can make DR and backup decisions based entirely on risk and business considerations, rather than on budget constraints or arbitrary resource limitations.Getting out of the data center game. Finally, running VBR on Google Cloud can be a major step towards retiring costly, resource-intensive, on-prem IT assets. Most enterprises today are moving aggressively to retire data centers and migrate applications to the public cloud; virtually all of them are now managing hybrid cloud environments that make it easier to move workloads between on-prem and public cloud infrastructure.By leveraging the cloud as a DR target, Veeam on Google Cloud reduces some of the costs and IT resources associated with maintaining on-prem data centers, servers, storage, and network infrastructure. Setting the stage for digital transformationDisaster recovery has always been a frustrating initiative for enterprise IT. It’s a demanding, expensive, resource-intensive task, yet it’s also one where dropping the ball can be a catastrophic mistake. We can’t take DR — or backup and recovery in general — off an IT organization’s list of priorities. But Veeam and Google Cloud can make it much simpler, easier, and less expensive for our customers to maintain world-class backup and recovery capabilities while putting themselves in a great position to achieve their broader digital transformation goals.Google Cloud Marketplace makes procurement easier, too: Buying VBR and VBG on Google Cloud Marketplace helps fast-track corporate technology purchases by allowing you to purchase from an approved vendor, Google. All Marketplace purchases are included in your single Google Cloud bill, while drawing down any monthly spend commitment you may already have with Google Cloud. To learn more about how Veeam and Google Cloud work together to help you keep your critical applications protected, visit veeam.com/google-cloud-backup.Related ArticleCIS hardening support in Container-Optimized OS from GoogleOur latest Container-Optimized OS release supports CIS benchmark compliance and can provide continuous CIS scanning capabilities.Read Article
Quelle: Google Cloud Platform

Introducing Autonomic Security Operations for the U.S. public sector

As sophisticated cyberattack campaigns increasingly target the U.S. public and private sectors during the COVID era, the White House and federal agencies have taken steps to protect critical infrastructure and remote-work infrastructure. These include Executive Order 14028 and the Office of Management and Budget’s Memorandum M-21-31, which recommend adopting Zero Trust policies, and span software supply chain security, cybersecurity threat management, and strengthening cyberattack detection and response.However, implementation can be a challenge for many agencies due to cost, scalability, engineering, and a lack of resources. Meeting the requirements of the EO and OMB guidance may require technology modernization and transformational changes around workforce and business processes. Today we are announcing Autonomic Security Operations (ASO) for the U.S. public sector, a solution framework to modernize cybersecurity analytics and threat management that’s aligned with the objectives of EO 14028 and OMB M-21-31. Powered by Google’s Chronicle and Siemplify, ASO helps agencies to comprehensively manage cybersecurity telemetry across an organization, meet the Event Logging Tier requirements of the White House guidance, and transform the scale and speed of threat detection and response. ASO can support government agencies in achieving continuous detection and continuous response so that security teams can increase their productivity, reduce detection and response time, and keep pace with – or ideally, stay ahead of – attackers. While the focus of OMB M-21-31 is on the implementation of technical capabilities, transforming security operations will require more than just technology. Transforming processes and people in the security organization is also important for long-term success. ASO provides a more comprehensive lens through which to view the OMB event logging capability tiers, which can help drive a parallel transformation of security-operations processes and personnel.Modern Cybersecurity Threat Detection and ResponseGoogle provides powerful technical capabilities to help your organization achieve the requirements of M-21-31 and EO 14028:Security Information & Event Management (SIEM) – Chronicle provides high-speed petabyte-scale analysis, and is capable of consuming log types outlined in the Event Logging (EL) tiers in a highly cost-effective manner.Security Orchestration, Analytics, and Response (SOAR) – Siemplify offers dozens of out-of-box playbooks to deliver agile cybersecurity response and drive mission impact, including instances of automating 98% of Tier-1 alerts and driving an 80% reduction in caseload.User and Entity Behavior Analytics (UEBA) – For agencies that want to develop their own behavioral analytics, agencies can use BigQuery, Google’s petabyte scale data lake, to store, manage, and analyze diverse data types from many sources. Telemetry can be exported out of Chronicle, and custom data pipelines can be built to import other relevant data from disparate tools and systems, such as IT Ops, HR and personnel data, and physical security data. From there, users can leverage BQML to readily generate machine learning models without needing to move the data out of BigQuery. For Google Cloud workloads, our Security Command Center Premium product offers native, turnkey UEBA across GCP workloads.Endpoint Detection and Response (EDR)– For most agencies, EDR is a heavily adopted technology that has broad applicability in Security Operations. We offer integrations to many EDR vendors. Take a look at our broad list of Chronicle integrations here.Threat intelligence – Our solution offers a native integration with VirusTotal, has the ability to operationalize threat intelligence feeds natively in Chronicle, and integrates with various TI and TIP solutions.Community Security AnalyticsTo increase collaboration across public-sector and private-sector organizations, we recently launched our Community Security Analytics (CSA) repository, where we’ve partnered with the MITRE Engenuity Center for Threat-Informed Defense, CYDERES, and others to develop open-source queries and rules that support self-service security analytics for detecting common cloud-based security threats. CSA queries are mapped to the MITRE ATT&CK® framework of tactics, techniques and procedures (TTPs) to help you evaluate their applicability in your environment and include them in your threat model coverage.“Deloitte is excited to collaborate with Google Cloud on their transformational public sector Autonomic Security Operations (ASO) solution offering. Deloitte has been recognized as Google Cloud’s Global Services Partner of the Year for four consecutive years, and also as their inaugural Public Sector Partner of the Year in 2020,” said Chris Weggeman, managing director of GPS Cyber and Strategic Risk, Google Cloud Cyber Alliance Leader, Deloitte & Touche LLP. “Our deep bench of more than 1,000 Google Cloud certifications, capabilities spanning the Google Cloud security portfolio, and decades of delivery experience in the government and public sector makes us well-positioned to help our clients undertake critical Security Operations Center transformation efforts with Google Cloud ASO.”Cost-effective for government agenciesTo help Federal Agencies meet the requirements of M-21-31 and the broader EO, Google’s ASO solutions can drive efficiencies and help manage the overall costs of the transformation. ASO can make petabyte-scale data ingestion and management more viable and cost-effective. This is critical at a time when M-21-31 is requiring many agencies to ingest and manage dramatically higher volumes of data that had not been previously budgeted for. PartnersWe’re investing in key partners who can help support U.S. government agencies on this journey. Deloitte and CYDERES both have deep expertise to help transform agencies’ Security Operations capabilities, and we continue to expand our partners to support the needs of our clients. A prototypical journey can be seen below.“Cyderes shares Google Cloud’s mission to transform security operations, and we are honored to deliver the Autonomic Security Operations solution to the U.S. public sector. As the number one MSSP in the world (according to Cyber Defense Magazine’s 2021 Top MSSPs List) with decades of advisory and technology experience detecting and responding to the world’s biggest cybersecurity threats, Cyderes is uniquely positioned to equip federal agencies and departments to go far beyond the requirements of the executive order to transform their security programs entirely via Google’s unique ASO approach,” said Robert Herjavec, CEO of CYDERES. “As an original launch partner of Google Cloud’s Chronicle, our deep expertise will propel our joint offering to modernize security operations in the public sector, all with significant cost efficiency compared to competing solutions.” said Eric Foster, President of CYDERES.Embracing ASOAutonomic Security Operations can help U.S. government agencies advance their event logging capabilities in alignment with OMB maturity tiers. More broadly, ASO can help the U.S. government undertake a larger transformation of technology, process, and people, toward a model of continuous threat detection and response. As such, we believe that ASO can help address a number of challenges presently facing cybersecurity teams, from the global shortage of skilled workers, to the overproliferation of security tools, to poor cybersecurity situational awareness and analyst burnout caused by an increase of data without sufficient context or tools to automate and scale detection and response.We believe that by embracing ASO, agencies can help agencies achieve:10x technology, through the use of cloud-native tools that help agencies meet event logging requirements in the near term, while powering a longer-term transformation in threat management; 10x process, by redesigning workflows and using automation to achieve Continuous Detection and Continuous Response in security operations; 10x people, by transforming the productivity and effectiveness of security teams and expanding their diversity; and10x influence across the enterprise through a more collaborative and data-driven approach to solving security problems between security teams and non-security stakeholders.To learn more about Google’s Autonomic Security Operations solution for the U.S. public sector, please read our whitepaper. More broadly, Google Cloud continues to provide leadership and support for a wide range of critical public-sector initiatives, including our work with the MITRE Engenuity Center for Threat-Informed Defense, the membership of Google executives on the President’s Council of Advisors on Science and Technology and the newly established Cyber Safety Review Board; Google’s White House commitment to invest $10 billion in Zero Trust and software supply chain security, and Google Cloud’s introduction of a framework for software supply chain integrity. We look forward to working with the U.S. government to make the nation more secure.Visit our Google Cloud for U.S. federal cybersecurity webpage.Related posts:Autonomic Security Operations for the U.S. Public Sector Whitepaper“Achieving Autonomic Security Operations: Reducing toil”“Achieving Autonomic Security Operations: Automation as a Force Multiplier”“Advancing Autonomic Security Operations: New resources for your modernization journey”Related ArticleRead Article
Quelle: Google Cloud Platform