cloud computing News, Entwicklungen, Updates, HowTos - Seite 1959 von 7107

Storing state with containers Kubernetes has become the preferred choice for running not only stateless workloads (e.g., web services) but also for stateful applications (e.g., e-commerce applications). According to the Data on Kubernetes report, over 70% of Kubernetes users run stateful applications in containers. Additionally, there is a rising trend of managed data services like MariaDB and Databricks using Google Kubernetes Engine to power their SaaS businesses to benefit from the portability of Kubernetes, built-in auto-upgrade features such as blue-green deployments, backup for GKE and out-of-the-box cost efficiency for better unit economics. All of this means that container-native storage on GKE is increasingly important. Specifically, storage that can be seamlessly attached and detached to containers as they churn (because the average container lifetime is much shorter than VMs) and remain portable across zones to stay resilient. That’s where Filestore Enterprise fits in. Customers get a fully managed regional file system with four 9s of availability. Storage is instantaneously attached to containers as they churn and zonal failovers are handled seamlessly. The rest of this blog explores multiple storage options with containers and how Filestore Enterprise fits in to help guide customers to make decisions of the best storage option that meets their needs.External persistent state for “stateless” containers (left) vs. persistent containers with CSI managed state within persistent volumes (right)Storage optionsThree storage models (from left to right): local file system, SAN and NAS.To understand the lay of the land, let’s explore three options for common patterns for attached storage with containers (note: Cloud Storage is accessed via the application code in a container and not covered here). Local file system over a local SSD device: A local file system (over local ssd block device) is the simplest to set up and can be very cost-effective and provide good performance (over local SSD), but in most cases it lacks enterprise storage capabilities such as snapshots, backups, and asynchronous DR. Also it provides limited reliability and redundancy as the state is host local. This model is well suited for scratch space/ephemeral storage use cases, but much less so for production-level, mission-critical use cases.Local file system over a remote/shared block device (SAN): The SAN (Storage Area Network) model is powerful and well known. A SAN-backed remote volume can provide good performance, advanced storage services, and good reliability. As the volume is external to the containers’ host, the persistent volume can be reattached (mounted) to a different host in case of container migration or if the original one failed, but is predominantly limited to only one host and Pod at a time. In the cloud world, SAN devices are replaced by networked block services, such as Google Cloud Persistent Disk (PD).Remote/networked file system (NAS): The NAS (Network Attached Storage) model is semantically a powerful storage model as it also allows read-write sharing of the volume across several containers. In such a model the file system logic is implemented in a remote filer and accessed via a dedicated file system protocol, most commonly Network File System (NFS). In the cloud world, NAS devices are commonly replaced by file system services such as Filestore.GCP block and file storage backendsIn Google Cloud non-local storage can be implemented using either PD or Filestore. PD provides flexible SSD- or HDD-backed block storage, while Filestore provides NFSv3 file volumes. Both models are CSI (Container Storage Interface) managed and fully integrated into the GKE management system. The main advantages and disadvantages of both models (depicted below) are as follows:PD provides capacity-optimized storage (HDD) and good price-performance variants (SSD, Balanced). PD provides flexible sizes and zonal volumes. On the other hand, PD based volumes do not support read-write sharing. This means multiple containers can’t read and write to the same volume. Customers can choose Regional support (RePD) but this is limited to active-passive models. PD-backed volumes support container migration and failover (after host failures), but such migration or failover may require time and expertise to implement.Filestore provides similar HDD and SSD variants and active-active regional (enterprise) variants. All Filestore variants support the read-write sharing model and almost instantaneous container migration and failover. Because of this increased functionality, Filestore-backed volumes have higher cost compared to the PD-backed volumes and have a minimum size limit of 1TB.Main Google Cloud storage models PD & FilestoreFilestore as fully managed container storageBoth PD and Filestore support container native operations such as migrating containers across hosts for use cases such as upgrades or failover. Customers on PD get best-in-class price/performance with extensive selection of multiple PD types. That’s why PD is popular with many GKE customers, as they benefit from price-performance and capabilities. However, with PD, customers need to have expertise in storage systems. In PD, the file system logic is built into the host. This coupling means during migration the host must cleanly shut down the container, unmount the file system, reattach the PD to the target host, mount the file system and only then boot the container. While GKE manages a lot of these operations automatically, in the case of failover there are potential file system and disk corruption issues. Users will need to run some cleanup processes (“fsck”) on the mounted volume before it can be used. With Filestore, customers get a fully managed regional file system that is decoupled from the host. Customers don’t need any expertise to operate storage and failovers are handled seamlessly as there are no infrastructure operations to attach/detach volumes. In addition, customers also benefit from storage that can be simultaneously read and written to by multiple containers.In addition to the general value of the Filestore as a GKE backend, Filestore Enterprise supports mission-critical and medium-to-large stateful deployments as it adds regional (four 9s) availability, active-active zone access, instantaneous snapshots, and smaller SSD entry point for each volume. Summary and conclusionsGoogle Cloud offers several fully managed options for GKE persistent volumes. In addition to the PD-based volumes, Filestore Enterprise is a first-class citizen storage backend for GKE and can also serve mission-critical use cases where (active/active) regional redundancy and fast failover/migration are important. Furthermore, Filestore Enterprise is just getting started on delivering better price-performance efficiency for customers. For example, customers can access a private preview to drive higher utilization of Filestore Enterprise instances by bin packing volumes as shares. Summary tableLinksAccessing file shares from Google Kubernetes Engine clusters | FilestoreHow persistent container storage works — and why it mattersDisk and image pricing | Compute Engine: Virtual Machines (VMs) | Google CloudPersistent disksService tiers Using the Filestore CSI driver1. The full list of PD models and pricing can be found here: https://cloud.google.com/compute/disks-image-pricing#disk
Quelle: Google Cloud Platform

9. August 2022

da Agency

Accelerating migrations to Google Cloud with migVisor by EPAM

Application modernization is quickly becoming one of the pillars of successful digital transformation and cloud migration initiatives. Many organizations are becoming aware of the dramatic benefits that can be achieved by moving legacy, on-premises apps and databases into cloud native infrastructure and services, such as reduced Total Cost of Ownership (TCO), elimination of expensive commercial software licenses, and improved performance, scalability, security and availability.The complexity of applications and databases to a cloud-centric architecture requires a rapid, accurate, and customized assessment of modernization potential and identification of challenges. Addressing business and functional drivers, TCO calculations, uncovering technological challenges and cross-platform incompatibilities, preparation of migration, and rollback plans can be essential to the success and outcome of the migration. These cloud migration initiatives are often divided into three high-level phases: Discovery: identifying and cataloging the source inventory. Output is usually an inventory of source apps, databases, servers, networking, storage, etc. The discovery of existing assets within a data center is usually straightforward and can often be highly automated. Pre-migration readiness: the planning phase. This includes the analysis of the current portfolio of the databases and applications for migration readiness, determining the target architecture, identifying technological challenges or incompatibilities, calculating TCO, and preparing detailed migration plans. Migration execution: where the rubber hits the road. During this phase of the migration process, database schemas are actively converted, the application data access layer is refactored, data is replicated from source to target, often in real-time, and the application is deployed in its determined compute platform(s). Successful evaluation and planning phase as part of the pre-migration readiness phase can bolster confidence in investment towards modernization. Skipping or inaccurately completing the pre-migration phase can lead to a costly and sub-optimal result. Relying on manual pre-migration assessments can lead to long migration timelines, reduced success rates and poor confidence in the post-migration state, increased risk and total migration cost. Some of the commonly asked question during pre-migration include:How compatible are my source databases, which are often commercial and proprietary in nature, with their open-source cloud-native alternatives? For example, how compatible are my Oracle workloads and usage patterns with Cloud SQL for PostgreSQL? What’s my degree of vendor lock-in with my current technology stack? Are proprietary features and capabilities being used that are incompatible with open-source database technologies?How tightly-coupled are my applications with my current database engine technology? Can my applications be deployed as-is, refactored for cloud readiness with ease, or will it be a big undertaking? How much effort will my migration require? How expensive will it be? What will be my run-rate in Google Cloud post-migration and my ROI?Can we identify quick-win applications and databases to start with?There is a direct association between the accuracy and speed of the pre-migration phase and the outcome of the migration itself. The faster and more accurately organizations complete the required pre-migration analysis, the more cost efficient and successful the migration itself will usually be. EPAM Systems, Inc., a leader in digital transformation, worked with Google Cloud as a preferred partner to accelerate cloud migrations beginning with pre-migration assessments. Leveraging EPAM’s migVisor for Google Cloud—a unique pre-migration accelerator that automates the pre-migration process—and EPAM’s consulting and support services, organizations can quickly generate a cloud migration roadmap for rapid and systematic pre-migration analysis. This approach has resulted in the completion of thousands of database assessments for hundreds of customers.migVisor is agentless, non-intrusive, and hosted in the EPAM cloud. migVisor seamlessly connects to your source databases and runs SQL queries to ascertain the database configuration, code, schema objects and infrastructure setup. Scanning of source databases is done rapidly and without interruption to production workloads.migVisor prepares customers to land applications in Google Cloud and its managed suite of databases services and platforms such as Cloud SQL, bare metal hosting, Spanner and Cloud Bigtable. migVisor supports re-hosting (lift-and-shift), re-platforming, and re-factoring. “EPAM’s recent application assessment update to its migration tooling system, migVisor, will bring a new level of transparency to the entire application and database modernization process”, said Dan Sandlin, Google Cloud Data GTM Director at Google Cloud. “This enables organizations to make the most of digital technologies and provides a clear IT ecosystem transformation that allows our customers to build a flexible foundation for future innovation.”Previously, migVisor focused on assessments of the source databases and the compatibility of customers’ existing database portfolio with cloud-centric database technologies. Coming this quarter, migVisor adds support for application assessments, augmenting its existing and class-leading capabilities in the database space. The addition of application modernization assessment functionality in migVisor, combined with EPAM’s certification and specialization in Google Cloud Data Management and hands-on engineering experience, strengthens EPAM’s position as a leader for large-scale digital transformation projects and migVisor as a trusted product for cloud migration assessments to Google Cloud customers. EPAM provides customers an end-to-end solution for faster and more cost-effective migrations. Assessments that used to take weeks can now be completed in mere days. Within minutes of registering for an account, anyone can start usingmigVisor by EPAM to automatically assess applications and application code. Visit themigVisor page to learn more and sign up for your account.Related ArticleAccelerate Google Cloud database migration assessments with EPAM’s migVisorThe Database Migration Assessment is a Google Cloud-led project to help customers accelerate their deployment to Google Cloud databases w…Read Article
Quelle: Google Cloud Platform

9. August 2022

da Agency

Bluware and Microsoft Azure develop OSDU-enabled interactive AI seismic interpretation solution for energy super major

This blog post has been co-authored by Kapil Raval, Principal Program Manager, Microsoft.

Bluware, which develops cloud-native solutions to help oil and gas operators to increase exploration and production workflow productivity through deep learning by enabling geoscientists to deliver faster and smarter decisions about the subsurface and today announced its collaboration with Microsoft for its next-generation automated interpretation solution, InteractivAI™, which is built on the Azure implementation of the OSDU™ Data Platform.

The two companies are working together to provide comprehensive solutions combining Microsoft Cloud implementation of OSDU™ Data Platform with Bluware’s subsurface knowledge. As the world’s energy companies retool for the future, they are juggling priorities between new forms of energy, carbon emissions, and maintaining the growing demand for fossil fuels. Innovative solutions such as cloud computing and machine learning are playing an important role in this transition.

To address an energy super major’s seismic interpretation challenges, Bluware is providing an interactive deep learning solution that runs natively on Azure, called InteractivAI™.

InteractivAI™ is utilized by the organization’s exploration and reservoir development teams to accelerate seismic interpretations and improve results by assisting geoscientists in identifying geological and geophysical features that may have been previously missed, incorrectly interpreted, or simply too time-consuming to interpret.

Using a data-centric approach, the application is unique in its ability, allowing users to train and infer simultaneously. Imagine running deep learning in real-time where the interpreter is providing feedback that the operator can actually see as the network suggests on-the-fly interpretations. This even includes results on data that is either not readily visible to the human eye or very difficult to see. This interactive workflow delivers more precise and comprehensive results in hours compared to months resulting in higher quality exploration and reservoir development.

The interactive deep learning approach

Bluware is pioneering the concept of ‘interactive deep learning’, wherein the scientist remains in the figurative ‘driver’s seat’ and steers the network as it learns and adapts based on the interpreter’s teachings. The adjustment and optimization of training the data set provides immediate feedback to the network, which in turn adjusts weights and biases accordingly in real-time.

Bluware differs from other deep learning approaches which use a neural network that has been pre-trained on multiple data sets. Users must rely on a network that was trained on data they have not seen, created with a set of unknown biases, and therefore something they have no control over.

The basic parameterization exposed to scientists in these traditional approaches gives the illusion of network control without really ceding any significant control to the user. Processing times can be days or weeks, and scientists can only supply feedback to the network once the training is complete, at which point training will need to run again from scratch.

The interactive deep learning approach is a data-specific approach that focuses on creating the best learning and training model for the geology the user is working with. Unlike traditional deep learning approaches, the idea is to start with a blank, untrained network and train it while labeling to identify any feature of interest. This approach is not limited to salt or faults, but can also be used to capture shallow hazards, injectites, channels, bright spots, and more. This flexibility allows the expert to explore the myriad of possibilities and alternative interpretations within the area of interest.

The energy company initially conducted a two-month evaluation with multiple experts across their global asset teams. The results were remarkable, and the organization is continually adding users. Additionally, Bluware has provided a blueprint for the company’s IT team for an Azure Kubernetes Service (AKS) implementation which will accelerate and expand this Azure-based solution.

A seismic data format designed for the cloud

As companies continue to wrestle with enormous, complex data streams such as petabytes of seismic data, the pressure to invest in digital technology intensifies. Bluware has adapted to this imperative, delivering a cloud-based format for storing seismic data called Volume Data Store™ (VDS). Microsoft and Bluware have worked together to natively enable VDS as part of the Microsoft Cloud implementation of OSDU™ Data Platform, where developers and customers can connect to the seismic data stored and provide interactive AI-driven seismic interpretation workflows by using the InteractivAI™ SaaS from the Azure Appsource.

Bluware and Microsoft are collaborating in parallel to support major energy customers through their seismic shift initiatives including moving petabytes of data to Azure Blob storage in a cloud-native VDS environment.

Revolutionizing the way energy companies store and use seismic data

Bluware designed InteractivAI™ not only with seismic workflows in mind but also with an eye on the trends shaping the future of the energy sector. Creating a cloud-native data format makes it scalable for energy companies to do more with their data while lowering costs and speeding up workflows, allowing them to arrive at more accurate decisions faster leveraging the power of Azure.

About Bluware

In 2018, a group of energy-focused software companies, namely Bluware, Headwave, Hue, and Kalkulo AS merged to become Bluware Corp. to empower change, growth, and a sustainable future for the energy sector.

As companies pivot from fossil fuels to cleaner energy sources, the combination of new industry standards, cloud computing, and AI will be critical for companies to adapt quickly, work smarter, and continue to be profitable. Companies that adapt faster, will have a significant advantage over their competition. For more information, visit Bluware’s website.
Quelle: Azure

9. August 2022

da Agency

Azure Lab Services August 2022 update—Improved classroom and training experience

The new updated Azure Lab Services allows you to set up and configure Cloud labs for your classroom and training scenarios. You don’t have to worry about setting up, expanding, or managing on-premises labs anymore. We provide a managed service and take the hassle out of managing and maintaining these labs. The updated service comes with improved performance and enhanced backend reliability. With the introduction of virtual network (VNet) injection and more control of the virtual network, you can now unlock key training and classroom scenarios such as lab-to-lab communication and utilize the service to teach a wide range of courses requiring complex configurations. With this new update you also now have an option to integrate the service with the Canvas learning management system.

The introduction of additional roles, Azure policies, and enhanced cost tracking and management capability provides the features you need to fully understand, manage and maintain your service. The availability of a .NET SDK, Python SDK, Azure PowerShell module, and Azure Resource Manager (ARM) templates makes it easy for IT and administrators to automate and manage all aspects of the service. Learn more about the Azure Lab Services update and how to use it.

With major reliability and performance enhancements to the original service, this major update is bringing a whole slew of additional features for IT organizations, administrators, educators, and students.

The update is bringing features and functionality for all personas of the service including administrators, educators, and students.

New features help IT departments and administrators automate and manage

For the IT staff and the service administrators, now there is a concept of creating a lab plan instead of a lab account in the Azure portal to start the process of creating labs. A lab plan is used to create, configure settings, and manage the labs. For ease of administration of the lab, new roles have been created to provide granular control to different people in the organization who will manage and maintain the labs. We are also introducing default and custom Azure policies with this update to help administrators with more control over the management of the labs.

Similar to the older service, you will have to request additional virtual processors (vCPUs), depending on your Azure subscription, and how many labs and virtual machines you want to create in the labs. With this updated release, there is an improved vCPU capacity management for your subscription, and you don't share the vCPU capacity with other customers when using the service.

With the new release, it is also easier to track costs for your labs or the virtual machines utilizing Azure Cost Management. On the networking front, we are introducing Virtual Network Injection compared to virtual network peering, which was offered in the older service. Virtual Network Injection provides you with control of Azure NSG (Network Security Group) and load balancer for your virtual network. Virtual Network Injection supports some of the common scenarios such as lab-to-lab communication, access to Azure or on-premises license server, and utilizing Azure File services.

In order to make it easy for administrators to manage and maintain the service, we are offering a range of tools including a .NET SDK, Python SDK, Azure PowerShell module, and ARM templates. These tools will not only help you with automating and managing your service but can also be utilized to build value-add services on top of our service for your customers.

In alignment with all the global compliance and regulatory laws around data residency, the customers now have a choice to deploy the labs and related virtual machines in their region of choice, so their data stays local to where they want.

More options and flexibility for educators

Educators and instructors are also getting features and new functionality to improve their experience in the service. The updated service can also be integrated with Canvas, a popular learning management system. This makes it easy for educators to stay in Canvas to create, manage, and maintain their labs, and students can also access the labs and virtual machines from within Canvas. Educators now have the option to create labs with virtual machines and assign students to them with non-admin access.

The auto-shutdown feature of the virtual machines has now been improved to work across both Windows and Linux virtual machines. In addition, there are improvements around virtual machine idle detection based on resource usage and user presence. The update also provides additional flexibility to the educator to skip the virtual machine template creation process if they already have an image to use and don’t want to customize it. Using an already existing image or the default image from the Azure marketplace allows for fast creation of the lab compared to when the educator wants to create a lab with an image but will further customize it after the lab is created.

Faster, easier access for students

The updated service has also introduced improvements to the student experience. Students can now troubleshoot any virtual machine access issues by redeploying their virtual machine without losing data. If the lab is set up to use Azure Active Directory (AAD) group sync, there is no longer a need to send an invitation email to the students to register for the lab and get access to the virtual machine. Now, a virtual machine is automatically assigned to the student and they can access it immediately.

Learn more

Enable your educational, learning, and training scenarios today no matter what industry, by using the service. Get started today to use the enhanced experience and new features by utilizing the Azure Lab Services August 2022 update!
Quelle: Azure

9. August 2022

da Agency

Dear Moby: CronJobs and Kubernetes and Ocean Puns, Oh My!

Whalecome, dear reader, to the first issue of Dear Moby — my new advice column where I, Moby Dock, will be answering real developer questions from you, the Docker community. Ever hear of the Dear Abby column? Well, this one is better, because it’s just for developers.
Since we announced this column (and its video counterpart with my friends, Kat and Shy), we’ve received a tidal wave of questions. (You can submit your own questions here!)
Despite my whaleth of knowledge and passion for all things app development, I’m only one whale — and I don’t have all the answers! Many, but not all. So, I’ve commissioned a crew of fellow Docker experts to voyage alongside me in pursuit of the answers you seek.
So without further ado, let’s dive into the fray…

Our first question comes from shaileshb who asks:
“Hey! I’m creating a CronJob for my kubernetes cluster. Currently, I am confused as to whether I should put database connection strings and the main logic inside the CronJob itself, or whether those should exist in an API that the CronJob calls.”
Today’s commissioned experts: Director of Engineering Shawn Axsom and Principal Software Engineer Josh Newman
Dear shaileshb,
The best approach depends on your specific circumstances, but there are important considerations around performance and security you have to take into account with every deployment to a cluster you create.
Whether you use an additional API or not, you should secure connection strings and other secrets.
You want to keep it secret and keep it safe, so try these best practices:

Don’t put connection strings in environmental variables that someone could access while breaching the container or inspecting container or pod metadata.
Set identity access management policies based on the Principle of Least Privilege. (More about PoLP here.)
Consolidate database access to a single service or limited subset.
Consider a secrets manager, regardless of what deployment approach you take. (Take a deep dive into Kubernetes secret storage with this post from Conjur!)

Next, when it comes to performance, it really depends on your circumstance. Either approach can give you good performance, but your choice needs to take into account things like how often the CronJob runs, whether it runs in parallel, whether caching is involved, etc. And if designing for scale, you’ll want to consider connection limits and connection pooling.
When going the CronJob route, we suggest considering an external connection pool — but be sure it’s set up properly to avoid exhausting it. This might be an advantage of the API route. Depending on your tech stack and usage, you can get better connection pooling within the API service. If you want to brush up on your connection pooling considerations, check out this Stack Overflow article.
In the long run, simplicity is a virtue and will always reign supreme.
An understandable, secure, and scalable system is a one with a cohesive design that isn’t over-engineered. It’s often best to start small (and also consider existing services that may benefit from the functionality) and keep the code contained in one location where secrets are stored securely — especially if there aren’t performance or scalability concerns (or a need to reuse it). And if these concerns do rear their ugly heads, that simplicity makes it easier to refactor. Truth be told, after observing the code in production (or getting feedback), you might even opt for a different approach, so the simpler you start, the easier you make things for yourself later.
Above all else, it’s always good to design for resiliency and extensibility. If best practices aren’t put in place from the start, make sure to design the CronJob or API in a modular and composable way so these practices can be put into practice later without a rewrite.

Well, that does it for our first issue of Dear Moby! Many thanks to Josh and Shawn.
Have another question for me and the Docker dev team to tackle? Submit it here!
Until next time,
Moby Dock
Quelle: https://blog.docker.com/feed/