Cloud Computing Köln

Scaling Microsoft Kaizala on Azure

This post was co-authored by Anubhav Mehendru, Group Engineering Manager, Kaizala.

Mobile-only workers depend on Microsoft Kaizala—a simple and secure work management and mobile messaging app—to get the work done. Since COVID-19 has forced many of us to work from home across the world, Kaizala usage has surged close to 3x from pre-COVID-19. While this is a good opportunity for the product to grow, it has increased pressure on the engineering team to ensure that the service scales along with the increased usage while maintaining the customer promised SLA of 99.99 percent.

Today, we’re sharing share some of the learnings about managing and scaling an enterprise grade secure productivity app and the backend service behind it.

Foundation of Kaizala

Kaizala is a productivity tool primarily targeted for mobile-only users and is based on Microservice architecture with Microsoft Azure as the core cloud platform. Our workload runs on Azure Cloud Services, with Azure SQL DB and Azure Blob Storage used for primary storage. We use Azure Cache for Redis to handle caching, and Azure Service Bus and Azure Notification Hub manages async processing of events. Azure Active Directory (Azure AD) is used for our user authentication. We use Azure Data Explorer and Azure Monitoring for data analytics. Azure Pipelines is used for automated safe deployments where we can deploy updates rapidly multiple times in a week with high confidence.

We follow a safe deployment process, ensuring minimal customer impact, and stage wise release of new feature and optimizations with full control on exposure and rollback ability.

In addition, we use a centralized configuration management system where all our config can be controlled, such as exposure of a new feature to a set of users/groups or tenants. We fine grained control on msg processing rate, receipt processing, user classification, priority processing, slow down non-core functionalities etc. This allows us to rapidly prototype new feature and optimization over a user set.

Key resiliency strategies

We employ the following key resilience strategies:

API rate limit

To protect our existing service from misuse, we need to control the incoming calls coming from multiple clients within a safe limit. We incorporated a rate limiter entirely based on in-memory caching that does the work with negligible latency impact on customer operations.

Optimized caching

To provide optimal user experience, we created a generic in-memory caching infra where multiple compute nodes are able the quickly sync back the state changes using Azure Redis PubSub. Using this a significant number of external API calls were avoided which effectively reduced our SQL load.

Prioritize critical operations

In case of overload of service due to heavy customer traffic, we prioritize the critical customer operations such as messaging over non-core operations such as receipts.

Isolation of core components

Core system components that support messaging are now totally isolated from other non-core parts so that any overload does not impact the core messaging operations. The isolation is done at every resource level such as separate compute nodes, separate service bus for event processing and totally separate storage for non-core operations.

Reduction in intra node communication

We made multiple enhancements in our message processing system where we significantly reduced scenarios of intra node communication that caused a heavy intra node dependency and slows down the entire message processing.

Controlled service rollout

We made several changes in our rollout process to ensure controlled rollout of new features and optimizations to minimize and negative customer impact. The deployments moved to early morning slots where the customer load is minimal to prevent any downtime.

Monitoring and telemetry

We setup specific monitoring dashboards to give a quick overview of service health that monitor important parameters, such as CPU consumption, thread count, garbage collection (GC) rate, rate of incoming messages, unprocessed messages, lock contention rate, and connected clients.

GC rate

We have finetuned the options to control the rate of Gen2 GC happening in a cloud service as per the needs of the web and worker instances to ensure minimal latency impact of GC during customer operations.

Node partitioning

Users need to be partitioned across multiple nodes to distribute the ownership responsibility using a consistent hashing mechanism. This master ownership helps in ensuring that only required user's information is stored in the in-memory cache on a particular node.

Active passive user

In large group messaging operations, there are always users who are actively using the app while a lot of users are not active. Our idea is to prioritize message delivery for active users so that the last bucket active user received the message fast.

Serialization optimization

Default JSON serialization is costly when the input output operations are very frequent and burn precious CPU cycles. ProtoBuf offers a fast binary serialization protocol that was leveraged to optimize the operations for large data structures.

Scaling compute

We re-evaluated our compute usage in our internal multiple test and scale environments and judiciously reduced the compute node SKU to optimize as per the needs and optimize cost of goods sold (COGS). While most of our traffic in an Azure region is during the day time, there is minimal load at the night where we leverage to do heavy tasks, such as redundant data cleanup, cache cleanups, GC, database re-indexing, and compliance jobs.

Scaling storage

With increasing scale, the load of receipts became huge on the backend service and was consuming a lot of storage. While critical operations required highly consistent data, the requirement is less for non-critical operations. We moved the receipt to highly available No-SQL storage, which costs a tenth of the SQL storage.

Queries for background operations were spread out lazily to reduce the overall peak load on SQL storage. Certain non-critical Operations were moved from being strongly consistent to eventual consistency model to flatten the peak storage load, thus creating more capacity for additional users.

Our future plans

As the COVID-19 situation continues to be grave, we are expecting an accelerated pace of Kaizala adoption from multiple customers. To keep up with the increase in messaging load and high customer usage, we are working on new enhancements and optimizations to ensure that we remain ahead of the curve including:

Developing alternative messaging flows where users actively using the app can directly pull group messages even if the backend system is overloaded. Message delivery is prioritized for active users over passive users.
Aggressively working on distributed in-memory caching of data entities to enable fast user response and alternative designs to keep cache in sync while minimizing stale data.
Moving to container-based deployment model from the current virtual machine (VM)-based model to bring more agility and reduce operational cost.
Exploring alternative storage mechanism which scale well with massive write operations for large consumer groups supporting batched data flush in a single connection.
Actively exploring ideas around active-active service configuration to minimize downtime due to data center outages and minimize Recovery Time Objective (RTO) and Recovery Point Objective (RPO).
Exploring ideas around moving some of the non-core functionalities to passive scale units to utilize the standby compute/storage resources there.
Evaluating the dynamic scaling abilities of Azure Cloud services where we can automatically reduce the number of compute nodes during nighttime hours where our user load is less than fifth of the peak load.

Quelle: Azure

Azure Cost Management + Billing updates – September 2020

Whether you're a new student, thriving startup, or the largest enterprise, you have financial constraints, and you need to know what you're spending, where, and how to plan for the future. Nobody wants a surprise when it comes to the bill, and this is where Azure Cost Management + Billing comes in.

We're always looking for ways to learn more about your challenges and how Azure Cost Management + Billing can help you better understand where you're accruing costs in the cloud, identify and prevent bad spending patterns, and optimize costs to empower you to do more with less. Here are a few of the latest improvements and updates based on your feedback:

Simplify financial reporting with cost allocation, now in preview.
Connector for AWS is now generally available.
Get pay-as-you-go rates for all Azure products and services.
What's new in Cost Management Labs.
Expanded availability of resource tags in cost reporting.
15 ways to optimize your Azure costs.
New ways to save money with Azure.
Upcoming changes to Azure usage data.
Documentation updates.

Let's dig into the details.

 

Simplify financial reporting with cost allocation, now in preview

Managing cloud costs can be challenging, especially if your organization needs to break down costs for an internal chargeback. You might have separate business units, or you might need to facilitate external billing for distinct customer solutions. This becomes even more difficult when you employ shared services to reduce costs, since there may not be a clear way to break those shared services down by business unit or customer. This is where the Azure Cost Management + Billing cost allocation preview for Enterprise Agreement (EA) and Microsoft Customer Agreement (MCA) accounts comes in.

Cost allocation is the process of breaking down and distributing shared costs. A couple examples of this are networking infrastructure, shared by multiple virtual machines, or shared databases or storage accounts, used by different teams or customers. If you're sharing services across business units or customers, take a look at how cost allocation can help you drive more accountability and streamline cost reporting within your organization.

Learn more about the cost allocation preview and let us know what you'd like to see next.

 

Connector for AWS is now generally available

In 2019, we announced the preview of AWS connectors for Azure Cost Management + Billing, which allows you to view and manage your Azure and AWS costs from a single pane of glass in the Azure portal. Support for AWS is now generally available. This new connector simplifies handling different cost models and numerous billing cycles so you can visualize and always stay up-to-date with your costs across clouds.

For more details and a walkthrough for how to get started, see the Connector for AWS general availability announcement.

 

Get pay-as-you-go rates for all Azure products and services

Have you ever needed access to pay-as-you-go (PAYG) prices for different Azure services, but didn't know where to start? Maybe you're looking to optimize costs with the cheapest SKU or location combination or generating a savings report to show how much you've saved with discounted rates. Perhaps you want to get notified about price changes or build your own pricing calculator. Whatever the need, the Retail Prices API gives you a simple, unauthenticated way to get PAYG rates for all Azure products and services. Give it a spin and let us know what you'd like to see next.

 

What's new in Cost Management Labs

With Cost Management Labs, you get a sneak peek at what's coming in Azure Cost Management and can engage directly with us to share feedback and help us better understand how you use the service, so we can deliver more tuned and optimized experiences. Here are a few features you can see in Cost Management Labs:

Dark theme support in dashboard tiles – Now available in the public portal
Pinned dashboard tiles now support the dark theme.
Pin the Cost Management overview for quick access to scopes – Now available in the public portal
Similar to pinning cost analysis, you can also pin the Cost Management overview to get quick access to the scopes you need. Each pinned tile will remember and pre-select the desired scope, helping you get to where you need to be quicker than ever.
Improved getting started
Expanded getting started experience covering more options across Cost Management + Billing.

Of course, that's not all. Every change in Azure Cost Management is available in Cost Management Labs a week before it's in the full Azure portal. We're eager to hear your thoughts and understand what you'd like to see next. What are you waiting for? Try Cost Management Labs today.

 

Expanded availability of resource tags in cost reporting

Tagging is the best way to organize and categorize your resources outside of the built-in management group, subscription, and resource group hierarchy. Add your own metadata and build custom reports using cost analysis. While most Azure resources support tags, some resource types do not. Here are the latest resource types which now support tags:

Data Factories
Databricks workspaces

Remember tags are a part of every usage record and are only available in Cost Management reporting after the tag is applied. Historical costs are not tagged. Learn more about resource tagging limitations and update your resources today for the best cost reporting.

 

15 ways to optimize your Azure costs

With a global health pandemic continuing to challenge us to find a new normal, cost optimization is at the forefront of many minds. We've talked a lot about the many different ways to optimize costs, but I'd like to offer a quick checklist for anyone getting started with cost optimization in Azure, as well as offer up a couple service-specific checklists for those using Azure SQL or Azure Backup.

First, let's start with basics:

Before you start architecting your solutions, review and factor proven practices from the Azure Well-Architected Framework into your solutions.
If you already have solutions deployed, your next step is to review and take action on cost-saving recommendations in Azure Advisor. 
Look for additional savings, up to 72 percent, for over 18 Azure and third-party services with Azure reservations. 
Take advantage of unused on-premises Windows and SQL Server licenses with Azure Hybrid Benefit.
Make sure you understand how the services you use are charged with the Azure Pricing Calculator and architect your solutions in ways that maximize investments while minimizing costs.

If you're using SQL in Azure, on-premises, or from another cloud provider, here are a few extra tips that might help you save even more on top of reservations, Azure Hybrid Benefit, and other Advisor recommendations:

Maintain business continuity in the cloud with free SQL Server licenses.
Shift capex to opex with SQL Server on Azure Virtual Machines.
Protect your data with free security updates.
Boost productivity with fully managed Azure SQL database services.
Pay only for the resources you use with per-second billing on the SQL Database serverless.

And lastly, if you're using a backup solution today, here are a few ways Azure Backup can help you reduce costs:

Clean up backups for deleted resources using the inactive resources report.
Consider using daily differential backups with weekly full backups to save compared to daily full backups.
Consider shorter retention durations to reduce costs while still meeting compliance requirements.
Backup only what you need with selective disk backup to include or exclude certain virtual machine data disks.
Opt for locally-redundant storage (LRS) for dev/test or other workloads that don't need geo-redundant storage (GRS) replication to cut storage costs in half.

If you're interested in more details on any of the above, here are a few resources you may want to check out:

How to optimize your Azure workload costs.
8 ways to optimize costs on Azure SQL.
5 ways to optimize your backup costs with Azure Backup.

 

New ways to save money with Azure

Lots of cost optimization improvements over the past month! Here are a few you might be interested in:

Look for new and updated cost-saving recommendations in Azure Advisor:

Right-sizing underutilized MariaDB, MySQL, and PostgreSQL database server resources.
Reservations for Cosmos DB, SQL PaaS, Blob storage, App Service Stamp Fee, MariaDB, MySQL, PostgreSQL, and Synapse analytics.
CPU, memory, and network utilization are now available in via API and Azure Resource Graph.

Save on Dsv3, Dsv4, Ddsv4, Esv3, Esv4, and Edsv4 virtual machines with Azure Dedicated Host.
Save up to 94 percent compared to other providers with Azure Synapse Analytics.

 

Upcoming changes to Azure usage data

Many organizations use the full Azure usage and charges to understand what's being used, identify what charges should be internally billed to which teams, and/or to look for opportunities to optimize costs with Azure reservations and Azure Hybrid Benefit, just to name a few. If you're doing any analysis or have setup integration based on product details in the usage data, please update your logic for the following services.

The following Azure Bastion meter IDs for Azure Gov are changing effective October 1:

Azure Bastion Basic:
Old: 0e3e1208-1bcd-41a4-a98b-a82d5928e448
New: 077ecfba-a126-5c32-8c15-506554457f05
Azure Bastion data transfer out:
Old: 43f56dc9-e2c7-47bc-935e-56e24f531111
New: 88d1ca2d-4cd2-50a9-b104-d83f41e8a2dc

Also, remember the key-based Enterprise Agreement (EA) billing APIs have been replaced by new Azure Resource Manager APIs. The key-based APIs will still work through the end of your enrollment, but will no longer be available when you renew and transition into Microsoft Customer Agreement. Please plan your migration to the latest version of the UsageDetails API to ease your transition to Microsoft Customer Agreement at your next renewal.

 

Documentation updates

Lots of documentation updates this month. Here are a few you might be interested in:

Subscription transfer for CSP partners.
Selecting a language for budget alerts.
Documented invoice status values shown in the Azure portal.
Added note about subscription quota changes after subscription transfers.
Billing admins can see reservation recommendations.
Updated the list of cost recommendations.
Set the default management group for new subscriptions.
New management group tutorials using the portal, Azure CLI, PowerShell, or API.

Want to keep an eye on all of the documentation updates? Check out the Cost Management + Billing doc change history in the azure-docs repository on GitHub. If you see something missing, select Edit at the top of the document and submit a quick pull request.

 

What's next?

These are just a few of the big updates from last month. Don't forget to check out the previous Azure Cost Management + Billing updates. We're always listening and making constant improvements based on your feedback, so please keep the feedback coming.

Follow @AzureCostMgmt on Twitter and subscribe to the YouTube channel for updates, tips, and tricks. And, as always, share your ideas and vote up others in the Cost Management feedback forum.

We know these are trying times for everyone. Best wishes from the Azure Cost Management + Billing team. Stay safe and stay healthy!
Quelle: Azure