da Agency

BigQuery authorized views permissions via Terraform, avoiding the chicken & egg problem

Enterprises that use Terraform for spinning up their Infrastructure, including the instantiation of Google BigQuery, can run into a chicken & egg problem if using the IAM access permissions resource blocks for both their Datasets and Authorized Views. This problem can cause BigQuery operational issues across an organization, creating an unpleasant experience for the end-user due to the momentary loss of access to the data. End users without access to “private data” are likely to rely on the Authorized views to a great extent. This blog post shows how to avoid running into the problem and provides a step-by-step guide to correctly managing Authorized View permissions via Terraform. This publication has three components; Use case, problem statement, and solution.1. Use caseThe use case at hand involves 2 products, Google Cloud BigQuery and Hashicorp Terraform. Let’s look at both in light of the use case, one by one.BigQuery is Google Cloud’s fully managed enterprise data warehouse that helps you manage and analyze your data with built-in features like machine learning, geospatial analysis, and business intelligence. To consume and take advantage of BigQuery, you need datasets. Datasets are logical containers (contained within a specific project) that are used to organize and control access to your BigQuery resources. Datasets are similar to schemas in other database systems. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery. Cloud IAM can restrict members’ access to table levels but not to “parts of a table.” Suppose you have a use case where you want a member with a data viewer role to query / access specific information in a table, like an employee’s name and job title by department, without having access to the address of every employee. In that case, you can create a BigQuery authorized view. An authorized view lets you share query results with particular users and groups without giving them access to the underlying source data.The industry standard for infrastructure provisioning on Google Cloud is via Terraform tool by HashiCorp.Terraform is used to instantiate all infrastructure components and supports BigQuery resources. To manage IAM policies for BigQuery datasets, Terraform has three different resources: google_bigquery_dataset_iam_policy, google_bigquery_dataset_iam_binding, and google_bigquery_dataset_iam_member. 2. Problem statementThese BigQuery resources are intended to convert the permissions system for BigQuery datasets to the standard IAM interface. Still, there is a warning note as part of the Terraform documentation: “Using any of these resources will remove any authorized view permissions from the dataset. To assign and preserve authorized view permissions, use the google_bigquery_dataset_access instead.”As the note said, these resources work well in some scenarios but not for “Authorized Views” permissions. The Google Terraform resources to manage IAM policy for a BigQuery dataset each have respective unique use cases:google_bigquery_dataset_iam_policy: Authoritative. Sets the IAM policy for the dataset and replaces any existing policy already attached.google_bigquery_dataset_iam_binding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the dataset are preserved.google_bigquery_dataset_iam_member: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the dataset are preserved.Using any of these resources together with an authorized view will remove the permissions from the dataset. If any of these resources are used in conjunction with the “google_bigquery_dataset_access” resource or the “access” field on the “google_bigquery_dataset” resource, we will end up in a race condition where these resources will fight over which permissions take precedence. So, this essentially means that if we try to create and assign permissions to authorized views simultaneously as dataset creation from within the Terraform code, we will end up with a chicken & egg problem where there will be a dispute between the dataset and authorized views policy, causing the authorized views permissions to be wiped out as a result.Lets see the issue re-creation in action below.Terraform BigQuery  – dataset, table and authorized view resourcesTerraform BigQuery  – table IAM policy resourceWe can confirm the creation works with following query and Console screenshot:From the Google Cloud console we can see the created dataset, the authorized view and the dummy SAGoogle Cloud console  – Authorized view BQ datasetGoogle Cloud console  – Authorized view permissionsNow we add a new user to the source dataset with the following code.This revokes the authorized view and the “dummy terraform” SA loses its previously functional access.Google Cloud console  – Authorized view BQ datasetAs we discussed previously, this will be the behavior due to how IAM is implemented on BQ datasets; we need to consider all constraints around the IAM policy for BigQuery dataset and design our Terraform with the google_bigquery resource that best fits our needs. For our scenario, the resource that helped us resolve this issue is google_bigquery_dataset_access; this resource gives dataset access for a single entity and is intended to be used in cases where it is not possible to compile a complete list of access blocks to include in a google_bigquery_dataset resource and is the recommended resource when creating authorized views.Referring to the HCL code below, we have created a module for the dataset access resource; due to the nature of google_bigquery_dataset_access of giving access to a single entity. We are looping through a list of datasets and passing the dataset details to the module; this helped us avoid removing any authorized views from that dataset.Terraform – module/dataset_access/main.tfTerraform – module/dataset_access/output.tfTerraform – module/dataset_access/variables.tfTerraform – example/main.tfTerraform – example/terraform.tfvarsIn conclusion, how BigQuery implements IAM via Terraform is unique and different from how we do IAM for other Google Cloud services. It is essential first to understand the architecture of a specific BigQuery implementation and then feed that into deciding which BQ TF IAM resource(s) to use. We encourage you to read more about creating Authorized views and take a look at all the available Terraform blueprints for Google Cloud at the following links. Create an authorized viewTerraform blueprints and modules for Google Cloud
Quelle: Google Cloud Platform

da Agency

Microsoft Cost Management updates—January 2023

Whether you're a new student, a thriving startup, or the largest enterprise, you have financial constraints and you need to know what you're spending, where it’s being spent, and how to plan for the future. Nobody wants a surprise when it comes to the bill, and this is where Microsoft Cost Management comes in.

We're always looking for ways to learn more about your challenges and how Microsoft Cost Management can help you better understand where you're accruing costs in the cloud, identify and prevent bad spending patterns, and optimize costs to empower you to do more with less. Here are a few of the latest improvements and updates based on your feedback:

Manage your Enterprise Agreement billing account in the Azure portal.
Recent and pinned views in the Cost analysis preview.
Consistent global pricing for the Microsoft Cloud.
Help shape the future of invoice experiences.
Help shape the future of cost management for cloud services.
What's new in Cost Management Labs.
New ways to save money with Microsoft Cloud.
New videos and learning opportunities.
Documentation updates.

Let's dig into the details.

Manage your Enterprise Agreement billing account in the Azure portal

In March, we announced the general availability of the Enterprise Agreement (EA) billing experience in the Azure portal for direct customers working with Microsoft. Now that same experience is generally available for our indirect customers working with partners. All the same EA tools are available from Cost Management and Billing in the Azure portal:

Seamlessly create and manage departments, accounts, and subscriptions.
Manage access to departments, accounts, and subscriptions.
View properties and manage policies, like the ability to view charges and purchase reservations.
View notification contacts for enrollment emails.
View your monthly Azure usage and charges.
Generate and manage API access keys.

Looking beyond account management, you’ll also see new tools to help you monitor and manage costs:

View and download consolidated usage and charges, including options for amortized reservation charges.
Analyze and drill into your costs in the portal or schedule automated exports.
Enable tag inheritance to streamline tag-based cost analysis within your account.
Split shared costs to drive more visibility and accountability throughout the organization with cost allocation.
Configure budgets to get alerted before costs exceed predefined thresholds.

With these updates, EA billing account administrators should start to use the Azure portal for all account management needs. Account management from the EA portal will no longer be available for indirect customers starting on February 20, 2023.

Stay tuned for more updates, including support for indirect partners. To learn more, see EA billing administration on the Azure portal or check out the EA billing administration video series.

Recent and pinned views in the Cost analysis preview

Cost analysis is your tool for interactive analytics and insights. It should be your first stop when you need to explore or get quick answers about your costs. Over the past year, you've seen the addition of new smart views and capabilities, like anomaly detection, that offer more insights and help you understand costs more easily in the Cost analysis preview, but many of you have asked where you should start–in Cost analysis or the Cost analysis preview? Now, you don’t have to choose. The Cost analysis preview lets you decide where to start and remembers which views you use most, helping you jump back in and get the answers you need quicker than ever.

Cost analysis comes with various built-in views that summarize:

Cost of your resources at various levels.
Overarching services spanning all your resources.
Amortized reservation usage.
Cost trends over time.

Cost analysis has two types of views: smart views that offer intelligent insights and more details by default and customizable views you can edit, save, and share to meet your needs. The first time you open the Cost analysis preview, you start with a list of all available cost views.

Smart views open in tabs within the Cost analysis preview, allowing you to switch between views as you investigate issues. To open a second view, select the + to the right of the list of tabs. Customizable views open outside of the tabs in Cost analysis, a customizable view editor.

As you explore the different views, you’ll notice that the Cost analysis preview remembers which views you’ve used in the Recent section. Switch to the All views section to explore all built-in and saved views. If there’s a specific view you’d like quick access to, select Pin to recent from the All views list. You also have quick access rename, subscribe, copy a link to, or delete views from this list.

We encourage you to check out these updates and let us know what you’d like to see next. We’re eager to get your feedback as we continue to evolve the experience for you.

Consistent global pricing for the Microsoft Cloud

Earlier this month, we announced that we are taking several steps to align the pricing of our Microsoft Cloud products (such as, Azure, Microsoft 365) globally, meaning organizations will have consistent pricing reflecting the exchange rate of the local currency to the US dollar (USD). Starting April 1, 2023, pricing for Microsoft Cloud products will be adjusted in the following currencies:

GBP: +9%
DKK, EUR and NOK: +11%
SEK: +15%

In the future, we will assess pricing in local currency as part of a regular twice-a-year cadence, taking into consideration currency fluctuations relative to USD. This will provide increased transparency and predictability globally and move to a pricing model that is most common in our industry.

The Microsoft Cloud continues to be priced competitively, and Microsoft remains deeply committed to the success of its customers and partners. We will continue to invest to enable customers to innovate, consolidate and eliminate operating costs, optimize business performance and efficiency, and provide the foundation for a strong security strategy that customers around the world have come to rely on.

Help shape the future of invoice experiences

Do you view, manage, or pay invoices within the Azure portal or Microsoft 365 admin center? We're exploring new capabilities to improve your invoice experience and would love to get your feedback.

If you are interested in chatting about your experience, please sign up here.

Help shape the future of cost management for cloud services

Are you responsible for purchasing, managing, and optimizing cloud solutions and software for your organization? Does your daily job role involve understanding and monitoring cloud spending, discovering services, acquiring or updating licenses and subscriptions, analyzing resource utilization, or paying invoices?

If so, we’d love to talk to you and learn more about your job role in a 60-minute discussion. Please send an email to CE_UXR@microsoft.com and we will get back to you.

What's new in Cost Management Labs

With Cost Management Labs, you get a sneak peek at what's coming in Microsoft Cost Management and can engage directly with us to share feedback and help us better understand how you use the service, so we can deliver more tuned and optimized experiences. Here are a few features you can see in Cost Management Labs:

New: Remember preview features across sessions.
Select the preview features you're interested in from the Try preview menu and you'll see them enabled by default the next time you visit the portal. No need to enable this option—preview features will be remembered automatically in the preview portal.
New: Customers view for Cloud Solution Provider partners.
View a breakdown of costs by customer and subscription in the Cost analysis preview. Note this view is only available for CSP billing accounts and billing profiles. You can enable this option from the Try preview menu.
New: Total KPI tooltip.
View additional details about what costs are included in the Cost analysis preview. You can enable this option from the Try Preview menu.
Update: Recent and pinned views in the cost analysis preview—Now available in the public portal.
Show all classic and preview views in the cost analysis preview and streamline navigation by prioritizing recently used and pinned views. You can see this in the Cost Management Labs or by opting in using Try Preview.
Recommendations view.
View a summary of cost recommendations that help you optimize your Azure resources in the cost analysis preview. You can opt in using Try Preview.
Forecast in the cost analysis preview.
Show your forecast cost for the period at the top of the cost analysis preview. You can opt in using Try preview.
Group related resources in the cost analysis preview.
Group related resources, like disks under VMs or web apps under App Service plans, by adding a “cm-resource-parent” tag to the child resources with a value of the parent resource ID.
Charts in the cost analysis preview.
View your daily or monthly cost over time in the cost analysis preview. You can opt in using Try Preview.
View cost for your resources.
The cost for your resources is one click away from the resource overview in the preview portal. Just click View cost to quickly jump to the cost of that resource.
Change scope from the menu.
Change scope from the menu for quicker navigation. You can opt-in using Try Preview.

Of course, that's not all. Every change in Microsoft Cost Management is available in Cost Management Labs a week before it's in the full Azure portal or Microsoft 365 admin center. We're eager to hear your thoughts and understand what you'd like to see next. What are you waiting for? Try Cost Management Labs today.

New ways to save money in the Microsoft Cloud

This month I’ll share a few updates spread across the Microsoft Cloud:

General availability: DR secondary free with SQL Server on Azure Virtual Machines.
General availability: Arm-based VMs now available in four additional Azure regions.
Preview: License Geo-redundant Disaster Recovery for SQL Managed Instance for free.
Forrester study finds 228 percent ROI when modernizing applications on Azure PaaS.
Microsoft 365 Basic and more.
Microsoft 365 expands data residency offerings.
Dynamics 365 and Power Platform help you do more with less.

New videos and learning opportunities

Sharing a recent Azure Friday video that does a good job of providing an overview of cost management and optimization for Azure:

Managing, reporting, and reducing your costs in Azure (26 minutes).

Follow the Microsoft Cost Management YouTube channel to stay in the loop with new videos as they’re released and let us know what you'd like to see next.

Want a more guided experience? Start with Control Azure spending and manage bills with Microsoft Cost Management.

Documentation updates

As usual, there were plenty of documentation updates since our last update. Here are a few documents that were updated that you might be interested in:

Group and allocate costs using tag inheritance.
Buy an Azure savings plan.
Microsoft Customer Agreement Azure usage and charges file terms.
Assign roles to Azure Enterprise Agreement service principal names.
Troubleshoot a declined card.
Troubleshoot common Cost Management errors.
30 updates based on your feedback.

Want to keep an eye on all documentation updates? Check out the Cost Management and Billing documentation change history in the azure-docs repository on GitHub. If you see something missing, select Edit at the top of the document and submit a quick pull request. You can also submit a GitHub issue. We welcome and appreciate all contributions!

What's next?

These are just a few of the big updates from the last couple of months. Don't forget to check out the previous Microsoft Cost Management updates. We're always listening and making constant improvements based on your feedback, so please keep the feedback coming.

Follow @MSCostMgmt on Twitter and subscribe to the YouTube channel for updates, tips, and tricks. You can also share ideas and vote up others in the Cost Management feedback forum or join the research panel to participate in a future study and help shape the future of Microsoft Cost Management.
Quelle: Azure