medium.com – In general, design patterns are implemented to solve and reuse common well thought out architectures. Design patterns also introduce efficiency into your application and for your developers, reducing…
Quelle: news.kubernauts.io
Apache Hadoop and the big data ecosystem around it has served businesses well for years, offering a way to tackle big data problems and build actionable analytics. As these on-prem deployments of Hadoop and Apache Spark, Presto, and more moved out of experiments and into thousand-node clusters, cost, performance, and governance challenges emerged. While these challenges grew on-prem, Google Cloud emerged as a solution for many Hadoop admins looking to decouple compute from storage to increase performance while only paying for the resources they use. Managing costs and meeting data and analytics SLAs, while still providing secure and governed access to open source innovation, became a balancing act that the public cloud could solve without large upfront machine costs. How to think about on-prem Hadoop migration costsThere is no one right way to estimate Hadoop migration costs. Some folks will look at their footprint on-prem today, then try to compare directly with the cloud, byte for byte and CPU cycle for CPU cycle. There is nothing wrong with this approach, and when you consider opex and capex, and discounts such as those for sustained compute usage, the cost case will start to look pretty compelling. Cloud economics work! But what about taking a workload-centric approach? When you run your cloud-based Hadoop and Spark proof of concepts, consider the specific workload by measuring the units billed to run just that workload. Spoiler: it is quite easy when you spin up a cluster, run the data pipeline, and then tear down the cluster after you are finished. Now, consider making a change to that workload. For example, use a later version of Spark and then redeploy. This is a seemingly easy task—but how would you accomplish it today on your on-prem cluster, and what would have been the cost to plan and implement such a change? These are all things to consider when you are building a TCO analysis of migrating your entire, or just a piece of, your on-prem Hadoop cluster. Where to begin your on-prem Hadoop migrationIt’s important to note that you are not migrating a cluster, but rather the user and workloads, from a place where you shoulder the burden of maintaining and operating a cluster to a place where you share that responsibility with Google. Starting with these users and workloads allows you to build a better, more agile experience.Consider the data engineer who wants to update their pipeline to use the latest Spark APIs. When you migrate their code, you can choose to run it on its own ephemeral cluster—you are not forced to update the code for all your other workloads. They can run on their own cluster(s) and continue to leverage the previous version of the Spark APIs.Or for the data analyst who may need additional resources to run their Hive query in time to meet a reporting deadline, you can choose to enable autoscaling. Or for a data scientist who has been wanting to decrease their ML training job duration, you can provide them with a familiar notebook interface and spin up a cluster as needed with GPUs attached.These benefits all sound great, but there is hard work involved in migrating workloads and users. Where should you start? In the Migrating Data Processing Hadoop Workloads to Google Cloud blog post, we start the journey by helping data admins, architects, and engineers consider, plan, and run a data processing job. Spoilers: You can precisely select which APIs and versions are available for any specific workload. You can size and scale your cluster as needed to meet the workload’s requirements.Once you’re storing and processing your data in Google Cloud, you’ll want to think about enabling your analysis and exploration tools, wherever they are running, to work with your data. The work required here is all about proxies, networking, and security—but don’t worry, this is well-trodden ground. In Migrating Hadoop clusters to GCP – Visualization Security: Part I – Architecture, we’ll help your architects and admins to enable your analysts. For data science workloads and users, we have recently released Dataproc Hub, which enables your data scientists and IT admins to access on-demand clusters tailored to their specific data science needs as securely as possible. The Apache Hadoop ecosystem offers some of the best data processing and analytical capabilities out there. A successful migration is one in which we have unleashed them for your users and workloads; one in which the workload defines a cluster, and not the other way around. Get in touch with your Google Cloud contact and we’ll make your Hadoop migration a success together.
Quelle: Google Cloud Platform
Kaggle is a data science community of nearly 5 million users. In September of 2019, we found ourselves under a sudden siege of spam traffic that threatened to overwhelm visitors to our site. We had to come up with an effective solution, fast. Using AutoML Natural Language on Google Cloud, Kaggle was able to train, test, and deploy a spam detection model to production in just eight days. In this post, we’ll detail our success story about using machine learning to rapidly solve an urgent business dilemma.A spam dilemmaMalicious users were suddenly creating large numbers of Kaggle accounts in order to leave spammy search engine optimization (SEO) content in the user bio section. Search engines were indexing these bios, and our existing spam detection heuristics were failing to flag them. In short, we faced a growing and embarrassing predicament. Our problem was context. Kaggle is a community focused on data science and machine learning. As a result of our topical data-science focus, a user bio that seems harmless in isolation may be the work of a spammer. Here is a real example of one such bio:I am a personal injury lawyer in Chicago. I help individuals and families in cases involving serious injuries and wrongful death. Many of my cases involve car accidents, nursing home abuse, and medical malpractice.Such a bio may fit in on a forum of legal professionals, but on the Kaggle site it’s a mark of an SEO spammer. This content also lacks the typical keywords and unsavory topics that one might expect to find in spam. This context meant that stopping the spam required more than a generic model; we needed a solution that could take our Kaggle-specific context into account.We had the intuition that machine learning could handle this problem, but building natural language models to deal with spam was not anyone at Kaggle’s day job. We feared weeks of late nights slogging towards a good-enough solution—spam models require very high accuracy because of the high cost of miscategorizing a legitimate user. Even with a usable prototype running in R or Python, there was the looming frustration of deploying it in Kaggle’s C# codebase. As we planned out our options, we had an unconventional idea: what about trying AutoML?Enter AutoMLTrue to its name, AutoML performs automated machine learning: evaluating huge numbers of neural network architectures to determine the most effective model for a problem. We first witnessed the potential of the AutoML suite of products when a Google team used it to take second place at the 2019 KaggleDays hackathon. On a whim, we decided to pass our bio problem through the AutoML Natural Language Classification API. We could readily generate a labeled training dataset because we had existing examples of bios belonging to known-legitimate users:After uploading these bios, clicking the “Start Training” button, and waiting a few hours, we received an email that training was complete. Building models is normally a process that involves many failures, but the results were astoundingly impressive for a first attempt, with precision (how “accurate” the model is) and recall (how “thorough” the model is) above 99%.We manually inspected the performance, ran test examples through the model, and determined it would be immediately suitable to deploy in production. It successfully picked up on a wide variety of spammy content types (some identifying information and language is blurred out):Returning to our previous example on the importance of context, the model gives the personal injury lawyer a 98% confidence of being spam:Meanwhile, it has full confidence that the data scientist equivalent is allowable:On top of being accurate, AutoML afforded a major advantage when the time came to deploy the model. When training was finished, the model was already hosted and exposed via an API. Kaggle simply had to write a quick shim to call this API from our application.It took only eight days from when we started working on this problem to when we deployed a model serving live traffic. It required no advanced skills in deep learning or natural language processing. The model has since made thousands of correct decisions and greatly reduced our spam-related traffic.While this story was about spam detection, the takeaway isn’t just that you can use AutoML for spam. AutoML has the potential to replicate this success story across the thousands of bespoke image, text, or tabular problems that businesses face. AutoML can step in when off-the-shelf models are insufficient, when you want to test a hunch but don’t have months to dedicate to it, or if you’re simply not a deep learning expert. The combination of high accuracy, rapid iteration, and smooth deployment can make AutoML an attractive approach to developing machine learning solutions for a wide range of business problems and needs.
Quelle: Google Cloud Platform
Enterprises and teams are adopting DevOps technologies combined with people and processes to deliver high-quality code, with faster release cycles and continuous delivery of value, to achieve higher levels of satisfaction for their own customers.
However, it can often get difficult to craft CI/CD pipelines by editing multiple YAMLs to stitch your code to cloud automation workflows. Teams end up spending considerable time and effort setting up and switching between different discrete tools during their day-to-day development cycles.
In November, GitHub Actions for Azure became generally available to automate deploying your app code in GitHub to Azure directly from their repositories. Building on this, at Microsoft Build 2020 we announced that GitHub Actions for Azure are now integrated into Visual Studio Code, Azure CLI, and the Azure Portal simplifying the experience of deploying to Azure from your preferred entry points. Download the new Visual Studio Code extension or install the Azure Command-Line Interface (CLI) extension for GitHub Actions.
GitHub Actions for Azure can now deploy any enterprise application
GitHub Actions gives you the flexibility to build an automated software development lifecycle workflow. To help development teams easily create workflows to build, test, package, release, and deploy to Azure, more than 30 GitHub Actions for Azure are published on GitHub Marketplace, with more planned to roll out in the coming months.
These actions enable deployments to multiple Azure services, from web applications to serverless functions and Kubernetes, as well as Azure SQL and MySQL databases.
We also support Azure login actions that can serve as a generic step that lets customers use scripting for a breadth of Azure resources using Azure CLI or Azure PowerShell. Various utility actions like Azure Key Vault, App Service Settings, and more are also published that help developers target Azure to deploy even their complex enterprise applications while following all the DevOps best practices. Check out the sample application Rock, Paper, Scissors, Lizard, Spock, a multilanguage application built with Visual Studio and Visual Studio Code, deployed with GitHub Actions and running on Azure Kubernetes Service (AKS).
Easily get started with Actions for Azure
Various starter templates are made available to deploy your apps created with popular languages and frameworks such as .NET, Node.js, Java, PHP, Ruby, or Python in containers or running on any operating system. To simplify the onboarding experience with deploying web applications, we’ve also included sample repositories which can help you get started in four easy steps:
Fork the sample repository (example, Python sample).
Click on Deploy to Azure in the readme file to create an Azure Web App.
Configure the required GitHub Repo Secrets.
Update the workflow YAML with the Web App configuration and commit the changes.
These steps will trigger your CI/CD workflow to build and deploy an app to Azure using GitHub Actions.
Create Action workflows from Visual Studio Code, Azure Portal, or Azure CLI
Today there are millions of developers using Visual Studio Code targeting Azure. Similarly, there are millions of developers on Azure Portal as well. We want to meet Azure developers where they are and provide the best end-to-end developer experiences using all our developer tools. With the new integrations that we’re announcing for Actions into various tool extensions, you can now deploy to Azure effortlessly using GitHub Actions from your favorite tools. This will significantly reduce ramp-up time on GitHub Actions, avoid frequent context switching, and help your teams be more productive with built-in extensions in your favorite tools. We’re excited to announce three new tooling integrations:
In Azure Portal, GitHub Actions has now been added as a build provider in the App Service Deployment Center and Azure Kubernetes Service, making it easier for you to set up CI/CD workflows with GitHub Actions.
The Visual Studio Code Deploy to Azure extension helps you set up continuous build and deployment for Azure App Service or Azure Kubernetes Service without leaving the editor.
Azure CLI extension can be installed by running the command az extension add –name deploy-to-azure, and it supports deployments to Azure Kubernetes Service and Azure Container Instance via the az aks app up and az container app up commands.
You can use any of these tool integrations to set up an automatically generated and fully customizable CI/CD workflow that’s triggered for every code push. The YAML file is pre-populated with build and release steps, which you can edit as needed. As part of creating the workflow, all the relevant Azure and GitHub repository-related configurations are set up, without you needing to worry about plumbing the two systems.
Get started
Check out the starter templates and the documentation for Deploy to Azure CLI extension, Visual Studio Code extension, and GitHub Actions for Azure to get started. If you have any changes you’d like to see or suggestions for these features, then we’d love your feedback as well as contributions in the respective GitHub repositories–we’re taking pull requests! If you encounter a problem with any specific action, you can also open an issue on the action repository.
Learn more
View more GitHub integrations with Azure to automate your code-to-cloud workflow.
Check out the comprehensive list of GitHub Actions.
Quelle: Azure
Customizing virtual machine (VM) images to meet security and compliance requirements and achieve faster deployment is a strong need for many enterprises, but most don't enjoy the process and energy needed for determining the right tooling, building the right pipeline, and maintaining it continuously.
We built Azure Image Builder service to make building customized images easy in Azure.
Azure Image Builder service offers unification and simplification for your image building process across Azure and Azure Stack with an automated image building pipeline. Whether you want to build Windows or Linux virtual machine images, you can use existing image security configurations to build compliant images for your organization and patch existing custom images using Linux commands or Windows Update. Azure Image Builder supports images from multiple Linux distributions, Azure Marketplace, and Windows Virtual Desktop environments and you can build images for specialized VM sizes, including creating images for GPU VMs.
After you build the image, you can manage it with Shared Image Gallery and integrate your CI/CD pipeline with Azure Image Builder service. When you use Azure DevOps or other DevOps solutions, this gives you easy image patching, versioning, and regional replication capabilities.
Finally, Azure Image Builder service offers unmatched governance and compliance where role-based access control is integrated so you can determine who has access in which images and connect your existing VNET to access routable resources, servers, and services, including configuration servers (DSC, Chef, Puppet, and more). Deploying Azure Image Builder does not require a public IP address, which ensures the safety and gives you full control of the asset you’re building.
We’ve designed this service to take on the heavy-lifting when you’re building your next customized image, to meet the corporate and regulatory compliance rules, and preconfiguring VMs with applications for faster deployment without the hassle they used to require. You don't need to spend time learning how to build or maintain image pipelines, learn new tools, or have different tools. Simply describe your image configuration in a template, using your new or existing commands, scripts, build artifacts, and Azure Image Builder will create it for you.
Azure Image Build is expected to be generally available in Q3 2020.
Next steps with Azure Image Builder
Try Azure Image Builder in preview today with these resources:
Learn more about Azure Image Builder service.
Learn how to use Azure Image Builder with an image gallery for Linux virtual machines and Windows virtual machines.
Watch the Azure Image Builder Ignite webcast for a deep-dive into how Azure Image Builder and Shared Image Gallery can help you.
Quelle: Azure
How businesses respond in times of uncertainty is as varied as the businesses themselves. Many slow down operations to operate more cost-effectively, while others lean into new opportunities that didn’t exist before. Regardless of how you respond, ensuring your organization can cost-effectively adapt and scale to rapidly changing conditions is key.
When it comes to migrating your data, you have a variety of options to consider, and it’s important to have the flexibility to choose a path that helps you respond to uncertainty in a way that makes the most sense for your business. Azure SQL is here to help.
Introducing Azure SQL
Azure SQL is a family of fully managed, secure, and intelligent SQL database services that support a wide range of application patterns, from re-hosting and modernizing existing SQL Server workloads to modern cloud application development.
Because the entire Azure SQL family is built upon the same SQL Server database engine, you can migrate applications with ease and continue to use the tools, languages, and resources you’re familiar with. You’ll discover that your skills and experience transfer easily to the cloud, as the innovative features in Azure SQL help you operate more efficiently and save money along the way.
Azure SQL helps you do more
Azure SQL has options for any budget. You can choose managed services that are automatically patched, updated, and backed-up for you, so you can refocus resources onto higher priorities. A recent Forrester study found operational and financial benefits of modernizing applications on Azure SQL, citing a three-year 238 percent return on investment and up to a 40 percent increase in database administrator (DBA) productivity, among other benefits.1
Azure SQL also helps you stay agile in an ever-changing world by reducing your costs and simplifying performance management. Serverless compute, for example, continuously right-sizes resources to meet workload demand. Hyperscale storage is built on a flexible, cloud native architecture that allows it to grow as needed, rapidly scaling up to 100 TB. Built-in AI powers intelligent features like automatic tuning and Advanced Threat Protection, which maintain peak performance and data protection on your behalf.
SQL Server on Azure Virtual Machines offers similar efficiency and cost-effectiveness with Azure BlobCache, which is automatically provisioned for Azure Marketplace images and gives fast, free reads for customers. Given that typical SQL Server workloads are read-heavy, this provides tremendous savings.
Azure SQL comes with industry leading offers
In addition to the operational and financial benefits that come with managed services, Azure SQL can help reduce your upfront costs with industry-leading pricing and special offers, such as:
Azure Hybrid Benefit—If you have SQL Server and Windows licenses with active Software Assurance, you can reuse those licenses in the cloud and save up to 82 percent2 off pay-as-you-go rates on SQL Server on Azure Virtual Machines, Azure SQL Managed Instance, and Azure SQL Database.
Reservation pricing—Get more cloud for less cost when you commit upfront to a one- or three-year term. With reservation pricing, you can save up to 80 percent3 versus pay-as-you-go pricing, and you can exchange or cancel unused reservations at any time.
Azure Dev/Test Pricing—Use your Visual Studio subscriptions to get discounted pricing and Azure credits for non-production scenarios, with savings up to 55 percent off pay-as-you-go pricing on Azure SQL Database and Azure SQL Managed Instance and Ubuntu Linux rates on SQL Server on Azure Virtual Machines.
Get started
Need help with next steps? We can guide you to the right Azure SQL service that best meets the needs of your database workload, and you can get started today.
1 “The Total Economic ImpactTM of Migration to Azure SQL Managed Databases,” a commissioned study conducted by Forrester Consulting in March 2020 on behalf of Microsoft.
2 Calculations based on scenarios running 744 hours/month for 12 months at 3-year Reserved Instances or Reserved Capacity. Prices as of 10/24/2018, subject to change. Azure Windows VM calculations based on one D2V3 Azure VM in US West 2 region at the SUSE Linux Enterprise Basic rate. AWS calculations based on one m5.Large VM in US West (Oregon) using Windows Server pay-as-you-go rate for Reserved Instances under Standard 3-year term, all upfront payment. SQL Server calculations based on 8 vCore Azure SQL Database Managed Instance Business Critical in US West 2 running at Azure Hybrid Benefit rate. AWS calculations based on RDS for SQL EE for db.r4.2xlarge on US West (Oregon) in a multi AZ deployment for Reserved Instances under Standard 3-year term, all upfront payment. Extended security updates cost used for AWS is based on Windows Server Standard open NL ERP pricing in USD. Actual savings may vary based on region, instance size, and performance tier. Savings exclude Software Assurance costs, which may vary based on Volume Licensing agreement. Contact your sales representative for details.
3 The 80% saving for SQL Server on Azure Virtual Machines is based on the combined cost of Azure Hybrid Benefit for Windows Server and 3-year Azure Reserved Instance. The estimate does not include Software Assurance costs. Sample annual cost comparison of two D2V3 Windows Server VMs. Savings based on two D2V3 VMs in US West 2 Region running 744 hours/month for 12 months; Base compute rate at SUSE Linux Enterprise rate for US West 2. Azure pricing as of 04/24/2018. Prices subject to change. Actual savings may vary based on location, instance type, or usage. The 80% savings for Azure SQL Database and Azure SQL Managed Instance is based on eight vCore SQL Database Business Critical in West 2 US Region, running 730 hours per month. Savings are calculated from on demand full price (license included) against base rate with Azure Hybrid Benefit plus 3-year reserved capacity commitment. Savings excludes Software Assurance cost for SQL Server Enterprise edition, which may vary based on EA agreement. Actual savings may vary based on region, instance size and performance tier. Prices as of November 2018, subject to change.
Quelle: Azure
Do you remember the first time you used Docker? I do. It was about six years ago and like many folks at the time it looked like this:
docker run -it redis
I was not using Redis at the time but it seemed like a complicated enough piece of software to put this new technology through its paces. A quick Docker image pull and it was up and running. It seemed like magic. Shortly after that first Docker command I found my way to Docker Compose. At this point I knew how to run Redis and the docs had an example Python Flask application. How hard could it be to put the two together?
version: ‘3’
services:
web:
build: .
ports:
– “5000:5000″
redis:
image: “redis”
I understood immediately how Docker could help me shorten my developer “commute.” All the time I spent doing something else just to get to the work I wanted to be doing. It was awesome!
As time passed, unfortunately my commute started to get longer again. Maybe I needed to collaborate with a colleague or get more resources then I had locally. Ok, I can run Docker in the cloud, let me see how I can get a Docker Engine. Am I going to use another tool, set up one manually, automate the whole thing? What about updates? Maybe I should use one of the managed container services? Well then I’d have to use a different CLI and perhaps a different file format. That would be totally different from what I’m using locally. Seeing my commute bloat substantially, my team and I began to work on a solution and went on a path to find collaborators with the cloud service providers.
I am excited to finally be able to talk about the result of a collaborative set of ideas that we’ve been working on for a year to once again shorten your developer commute. Docker is expanding our strategic partnership with Microsoft and integrating the Docker experience you already know and love with Azure Container Instances (ACI).
What does that mean for you? The same workflow in Docker Desktop and with the Docker CLI and the tools you already have with all the container compute you could want. No infrastructure to manage. No clusters to provision. When it is time to go home, docker rm will stop all the meters. We will be giving an early preview of this work on stage at DockerCon tomorrow; so please register here and watch the keynote.
Let me give you a sense of how simple the process will be. You will even be able to log into Azure directly from the Docker CLI so you can connect to your Azure account. The login experience will feel very familiar, and odds are you have used it before for other services:
docker login azure
Once you are logged in, you just need to tell Docker that you want to use something besides the local engine. This is my favorite part– it is where in my opinion the magic lives. About a year ago we introduced Docker Context. Originally, it let you switch between engines (local or remote), Swarm, and Kubernetes. When it launched, I thought we needed to make this happen for any service that can run a container. If you want to shorten a developer’s commute, this is the way to do it.
docker context create aci-westus aci –aci-subscription-id xxx –aci-resource-group yyy –aci-location westus
All you need is a set of Azure credentials. If you have an Azure resource group you want to use you will be able to select it or we can create one for you. Once you have your Docker Context, you can tell Docker to switch to using it by default.
docker context use aci-westus
Once you have the context selected, it is just Docker. You can run individual containers. And you can run multiple containers with Docker Compose; look at Awesome Compose to find a compose file to try out. Or fire up Visual Studio Code and get back to doing what you wanted to do– writing code. As part of this strategic partnership with Microsoft, we are working closing with the Visual Studio Code teams to make sure the Docker experience is awesome.
Docker and Microsoft’s partnership has a long history. I am proud to be able to talk about what I have been working on for the last year. Together we are working on getting a beta ready for release in the second half of 2020. You can register for the beta here.
If there are other providers you would like to see come on board to bring the simplicity of the Docker experience to the cloud, then please let us know on our public roadmap. I am looking forward to telling you more about what else we have been working on soon!
The post Shortening the developer commute with Docker and Microsoft Azure appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/
Der Winzling von HP hat trotz seiner Kompaktheit viele Ports und ist aufrüstbar. (HP, Desktop-Management)
Quelle: Golem
Wer wieder einmal Halo 2 wie früher spielen möchte, kann das möglicherweise demnächst in einer nachgebauten Version des ersten Xbox Live tun. (Xbox Live, Microsoft)
Quelle: Golem
5G – der ganz große Sprung steht hier noch aus. Doch die Datenrate wird besser, auch bei LTE. (5G, Long Term Evolution)
Quelle: Golem
