Microsoft Azure News, Entwicklungen, Updates, HowTos - Seite 210 von 259

Since its release at NAB two years ago, Azure media Player has grown significantly in robustness and in richness of features. We have been working hard addressing feedback from our fantastic customers (that’s you!) to enhance and improve a player that everyone can benefit from. For AMP’s 2nd birthday I am incredibly excited to announce our first major release its initial; welcome AMP 2.0!

What’s new in AMP 2.0?

Advertisement support

Azure Media Player version 2.1.0 and higher supports the insertion of pre- mid- and post- roll ads in all your on demand assets. The player inserts ads in accordance to the IAB’s VAST standard and allows you to configure options like ad position and skipabilty. To learn more about video ads with Azure Media Player check out my blog post: Monetizing Your Content With Azure Media Player

A new skin

We released a new skin along with as a counterpart to “AMP-Default” called “AMP-Flush”. You can enable AMP flush by simply changing two point in your code:

1) update the CSS your application loads

from

2) updating the class in your videotag

from

making these changes should result in the following new skin:

A more accessible player

We are always working towards creating a more accessible and user friendly player. The team has been working hard to improve your experience with the player in use cases like:

interfacing with assistive technologies (like JAWS or Narrator)
playback in High Contrast mode
navigating without a mouse (or Tab To Navigate)

New plugins

This release comes with some new plugins you can load from our plugin gallery as well as new functionality baked into the player like Playback speed (icon in the screencap above). The details for these new features and additional APIs can all be found in our documentation. You can utilize them to customize the player to support the playback scenario you want to achieve. Plugin development is a very community driven operation and if you have any questions about creating plugins, modifying the ones in the gallery, or contributing them to the player, please email me at saraje@microsoft.com

Making the Switch to AMP 2.0

Transitioning to AMP 2.0 is an incredibly simple process. Make sure to update your CDN endpoints to point to 2.1.0 like so:

Providing Feedback

Azure Media Player will continue to grow and evolve, adding additional features and enabling new scenarios. You can request new features, provide ideas or feedback via UserVoice. If you have and specific issues, questions or find any bugs, drop us a line at ampinfo@microsoft.com.

Additional resources

Learn more
License
Documentation
Samples
Demo page
Plugin Gallery
UserVoice
Sign up

Quelle: Azure

25. April 201725. April 2017

da Agency

“How to make a movie the secure way” at NAB Show 2017

At the NAB Show 2017 Conference this week, we’ll be reprising our session “Securing the Making of the Next Hollywood Blockbuster” (Las Vegas | April 25, 2017 | 1:30 PM – 2:00 PM in the Cybersecurity & Content Protection Pavilion – Booth C3830CS – Central Hall). Azure’s very own Joel Sloss will regale the audience on the transition to secure movie production in the cloud, made possible by Microsoft Azure and a cadre of ISV partners who’ve ported their solutions to the platform.

Content comes in many forms and must be stored in many places, including on servers, workstations, mobile storage, archives, etc., which presents a massive security challenge. Securing this ever-moving data on top of the added challenge of properly handling personal information such as health records, contract details, and paystubs, strains an already complicated data governance situation.

In this session, we’ll look at an end-to-end workflow leveraging Azure and the combined wizardry of Avid, 5th Kind, Contractlogix, Docusign, MarkLogic, and SyncOnSet. Lulu Zezza, Physical Production Executive and the driving force behind the end-to-end workflow, will be co-presenting with Mr. Sloss. She noted that, “Moving to the cloud is the best way to implement security controls across so many different physical and logical environments, locations, and data types. We have people working all over the world, for different companies, using different systems, all contributing to the same production. In the past, it’s been like a free-for-all, with contractors getting access to things they shouldn’t, information being duplicated and stored in the wrong places, and sensitive content left out in the open.”

The new digital workflow enables a secure “script-to-screen” experience for the management of both production data and the crew’s personal HR information, to which new global privacy standards apply. Metadata captured from contracts, scripts, and camera files is associated with filming days, scenes and takes recorded, and later to the final edit of the film, reducing the need for document sharing and film screenings. Plus communications are kept protected and confidential. It’s a whole new way to make movies.

Join Joel at his session where you’ll hear about:

Architectural considerations for multi-domain cloud environments
Secure access and device management for BYOD users
Content protection and privacy in connected and disconnected networks

Learn more information about Microsoft’s activities at NAB Show 2017.
Quelle: Azure

25. April 201725. April 2017

da Agency

How Azure Security Center detects a Bitcoin mining attack

Azure Security Center helps customers deal with myriads of threats using advanced analytics backed by global threat intelligence. In addition, a team of security researchers often work directly with customers to gain insight into security incidents affecting Microsoft Azure customers, with the goal of constantly improving Security Center detection and alerting capabilities.

In the previous blog post "How Azure Security Center helps reveal a Cyberattack", security researchers detailed the stages of one real-world attack campaign that began with a brute force attack detected by Security Center and the steps taken to investigate and remediate the attack. In this post, we’ll focus on an Azure Security Center detection that led researchers to discover a ring of mining activity, which made use of a well-known bitcoin mining algorithm named Cryptonight.

Before we get into the details, let’s quickly explain some terms that you’ll see throughout this blog. “Bitcoin Miners” are a special class of software that use mining algorithms to generate or “mine” bitcoins, which are a form of digital currency. Mining software is often flagged as malicious because it hijacks system hardware resources like the Central Processing Unit (CPU) or Graphics Processing Unit (GPU) as well as network bandwidth of an affected host. Cryptonight is one such mining algorithm which relies specifically on the host’s CPU. In our investigations, we’ve seen bitcoin miners installed through a variety of techniques including malicious downloads, emails with malicious links, attachments downloaded by already-installed malware, peer to peer file sharing networks, and through cracked installers/bundlers.

Initial Azure Security Center alert details

Our initial investigation started when Azure Security Center detected suspicious process execution and created an alert like the one below. The alert provided details such as date and time of the detected activity, affected resources, subscription information, and included a link to a detailed report about hacker tools like the one detected in this case.

We began a deeper investigation, which revealed the initial compromise was through a suspicious download that got detected as “HackTool: Win32/Keygen". We suspect one of the administrators on the box was trying to download tools that are usually used to patch or "crack" some software keys. Malware is frequently installed along with these tools allowing attackers a backdoor and access to the box.

Based on our log analysis, the attack began with the creation of a user account named “*server$”.
The “*server$” account then created a scheduled task called "ngm”. This task launched a batch script named "kit.bat” located in the "C:WindowsTempngmtx" folder.
We then observed process named "servies.exe“ being launched with cryptonight related parameters.
Note: The ‘bond007.01’ represents the bitcoin user’s account behind this activity and ‘x’ represents the password.

Two days later we observed the same activity with different file names. In the screenshot below, sst.bat has now replaced kit.bat and mstdc.exe has replaced servies.exe . This same cycle of batch file and process execution was observed periodically.

These .bat scripts appear to be used for making connections to the crypto net pool (XCN or Shark coin) and launched by a scheduled task that restarts these connections approximately every hour.

Additional Observation: The downloaded executables used for connecting to the bitcoin service and generating the bitcoins are renamed from the original, 32.exe or 64.exe, to “mstdc.exe” and “servies.exe” respectively. These executable’s naming schemes are based on an old technique used by attackers trying to hide malicious binaries in plain sight. The technique attempts to make files look like legitimate benign-sounding Windows filenames.

Mstdc.exe: “mstdc.exe” looks like “msdtc.exe” which is a legitimate executable on Windows systems, namely Microsoft Distributed Transaction Coordinator required by various applications such as Microsoft Exchange or SQL Server installed in clusters.
Servies.exe: Similarly, “services.exe” is a legitimate Service Control Manager (SCM) is a special system process under the Windows NT family of operating systems, which starts, stops and interacts with Windows service processes. Here again attackers are trying to hide by using similar looking binaries. “Servies.exe” and “services.exe”, they look very similar, don’t they? Great tactic used by attackers.

As we did our timeline log analysis, we noted other activity including wscript.exe using the “VBScript.Encode” to execute ‘test.zip’.

On extraction, it revealed ‘iissstt.dat’ file that was communicating with an IP address in Korea. The ‘mofcomp.exe’ command appears to be registering the file iisstt.dat with WMI. The mofcomp.exe compiler parses a file containing MOF statements and adds the classes and class instances defined in the file to the WMI repository.

Recommended remediation and mitigation steps

The initial compromise was the result of malware installation through cracked installers/bundlers which resulted in complete compromise of the machine. With that, our recommendation was first to rebuild the machine if possible. However, with the understanding that this sometimes cannot be done immediately, we recommend implementing the following remediation steps:

1. Password Policies: Reset passwords for all users of the affected host and ensure password policies meet best practices.

Best Practices for Enforcing Password Policies
Selecting Secure Passwords
Using Strong Passwords

2. Defender Scan: Run a full antimalware scan using Microsoft Antimalware or another solution, which can flag potential malware.

3. Software Update Consideration: Ensure the OS and applications are being kept up to date. Azure Security Center can help you identify virtual machines that are missing critical and security OS updates.

4. OS Vulnerabilities & Version: Align your OS configurations with the recommended rules for the most hardened version of the OS. For example, do not allow passwords to be saved. Update the operating system (OS) version for your Cloud Service to the most recent version available for your OS family. Azure Security Center can help you identify OS configurations that do not align with these recommendations as well as Cloud Services running outdates OS version.

5. Backup: Regular backups are important not only for the software update management platform itself, but also for the servers that will be updated. To ensure that you have a rollback configuration in place in case an update fails, make sure to back up the system regularly.

6. Avoid Usage of Cracked Software: Using cracked software introduces unwanted risk into your home or business by way of malware and other threats that are associated with pirated software. Microsoft highly recommends evading usage of cracked software and following legal software policy as recommended by their respective organization.

More information can be found at:

Educate yourself on software piracy risk.
Learn more by reading “SIRv13: Be careful where you go looking for software and media files”.

7. Email Notification: Finally, configure Azure Security Center to send email notifications when threats like these are detected.

Click on Policy tile in Prevention Section.
On the Security Policy blade, you pick which Subscription you want to configure Email Alerts for.
This brings us to the Security Policy blade. Click on the Email Notifications option to configure email alerting.

An email alert from Azure Security Center will look like the one below.

To learn more about Azure Security Center, see the following:

Azure Security Center detection capabilities — Learn about Azure Security Center’s advanced detection capabilities.
Managing and responding to security alerts in Azure Security Center — Learn how to manage and respond to security alerts.
Managing security recommendations in Azure Security Center — Learn how recommendations help you protect your Azure resources.
Security health monitoring in Azure Security Center — Learn how to monitor the health of your Azure resources.
Monitoring partner solutions with Azure Security Center — Learn how to monitor the health status of your partner solutions.
Azure Security Center FAQ — Find frequently asked questions about using the service.
Azure Security blog — Get the latest Azure security news and information.

Quelle: Azure

24. April 201724. April 2017

da Agency

Announcing new Azure Services in the UK

We’re pleased to announce the following services which are now available in the UK!

HDInsight – HDInsight is a 100% compatible Hadoop service that allows you to easily provision and manage Hadoop clusters for big data processing in Azure.

HDInsight is the only fully-managed cloud Hadoop offering that provides optimized open source analytic clusters for Spark, Hive, MapReduce, HBase, Storm, Kafka, and R Server backed by a 99.9% SLA. Each of these big data technologies and ISV applications are easily deployable as managed clusters with enterprise-level security and monitoring. Learn more about HDInsight.

Azure Import/Export service now also available in UK is the perfect companion service to use with HDInsight – the combination allows you to easily ingest, process and optionally export a limitless amount of data.

Azure Import/Export – Import/Export Service is now live in UK South! The Azure Import/Export Service enables you to move large amounts of on-premises data into and out of your Azure Storage accounts. It does this by enabling you to securely ship hard disk drives directly to our Azure data centers. Once we receive the drives we’ll automatically transfer the data to or from your Azure Storage account. This enables you to import or export massive amounts of data more quickly and cost effectively (and not be constrained by available network bandwidth).

Customers can now use Azure Import/Export Service to copy data to and from Azure Storage by shipping hard disk drives to Azure UK South data center.

For more information about creating import/export jobs see, Use the Microsoft Azure Import/Export Service to transfer data to Blob Storage.

Azure Container Registry – Azure Container Registry is a private registry for hosting container images. Using the Azure Container Registry, customers can store Docker-formatted images for all types of container deployments. Azure Container Registry integrates well with orchestrators hosted in Azure Container Service, including Docker Swarm, DC/OS and Kubernetes. Users can benefit from using familiar tooling capable of working with the open source Docker Registry v2.

Customers can now create one or more container registries in their Azure subscription. Each registry is backed by a standard Azure storage account in the same location. Take advantage of local, network-close storage of your container images by creating a registry in the same Azure location as your deployments. Learn more about Azure Container Registry.

We are excited about these additions, and invite customers using the UK Azure region to try them today!
Quelle: Azure

21. April 201721. April 2017

da Agency

How Microsoft builds massively scalable services using Azure DocumentDB

This week at Microsoft Data Amp we covered how you can harness the incredible power of data using Microsoft’s latest innovations in its Data Platform. One of the key pieces in the Data Platform is Azure DocumentDB, Microsoft’s globally distributed NoSQL database service. Released in 2015, DocumentDB is being used virtually ubiquitously as a backend for first-party Microsoft services for many years.

DocumentDB is Microsoft's multi-tenant, globally distributed database system designed to enable developers to build planet scale applications. DocumentDB allows you to elastically scale both, throughput and storage across any number of geographical regions. The service offers guaranteed low latency at P99, 99.99% high availability, predictable throughput, and multiple well-defined consistency models, all backed by comprehensive SLAs. By virtue of its schema-agnostic and write optimized database engine, by default DocumentDB is capable of automatically indexing all the data it ingests and serve SQL, MongoDB, and JavaScript language-integrated queries in a scale-independent manner. As a cloud service, DocumentDB is carefully engineered with multi-tenancy and global distribution from the ground up.

In this blog, we cover case studies of first-party applications of DocumentDB by the Windows, Universal Store, and Azure IoT Hub teams, and how these teams could harness the scalability, low latency, and flexibility benefits of DocumentDB to innovate and bring business value to their services.

Microsoft DnA: How Microsoft uses error reporting and diagnostics to improve Windows

The Windows Data and Analytics (DnA) team in Microsoft implements the crash reporting technology for Windows. One of their components runs as a Windows Service in every Windows device. Whenever an application stops responding on a user&039;s desktop, Windows collects post-error debug information and prompts the user to ask if they’re interested in finding a solution to the error. If the user accepts, the dump is sent over the Internet to the DnA service. When a dump reaches the service, it is analyzed and a solution is sent back to the user when one is available.

Windows error reporting diagnostic information

Windows&039; need for fast key-value lookups

In DnA’s terminology, crash reports are organized into “buckets”. Each bucket is used to classify an issue by key attributes such as Application Name, Application Version, Module Name, Module Version, and OS Exception code. Each bucket contains crash reports that are caused by the same bug. With the large ecosystem of hardware and software vendors, and 15 years of collected data about error reports, the DnA service has over 10 billion unique buckets in its database cluster.

One of the DnA team’s requirements was rather simple at face value. Given the hash of a bucket, return the ID corresponding to its bucket/issue if one was available. However, the scale posed interesting technical challenges. There was a lot of data (10 billion buckets, growing at 6 million a day), high volume of requests and global reach (requests from any device running Windows), and low latency requirements (to ensure a good user experience).

To store “Bucket Dimensions”, the DnA team provisioned a single DocumentDB collection with 400,000 request units per second of provisioned throughput. Since all access was by the primary key, they configured the partition key to be the same as the “id”, with a digest of the various attributes as the value. As DocumentDB provided <10 ms read latency and <15ms write latency at p99, DnA could perform fast lookups against buckets and lookup issues even as their data and request volumes continued to grow over time.

Windows cab catalog metadata and query

Aside from fast real-time lookups, the DnA team also wanted to use the data to drive engineering decisions to help improve Microsoft and other vendors’ products by fixing the most impactful issues. For example, the team has observed that addressing the top 1 percent of reliability issues could address 50 percent of customers’ issues. This analysis required storing the crash dump binary files, “cabs”, extracting useful metadata, then running analysis and reports against this data. This presented a number of interesting challenges on its own.

The team deals with approximately 600 different types of reliability-incident data. Managing the schema and indexes required a significant engineering and operational overhead on the team.
The cab metadata was also a big volume of data. There were about 5 billion cabs, and 30 million new cabs were added every day.

The DnA team could migrate their Bucket Dimension and Cab Catalog stores to DocumentDB from their earlier solution based on an on-premises cluster of SQL Servers. Since shifting the database’s heavy lifting to DocumentDB, DnA benefited from the speed, scale, and flexibility offered by DocumentDB. More importantly, they could focus less on maintenance of their database and more on improving user experience on Windows.

You can read the case study at Microsoft’s DnA team achieves planet-scale big-data collection with Azure DocumentDB.

Microsoft Global Homing Service: How Xbox Live and Universal Store build highly available location services

Microsoft’s Universal Store team implements the e-commerce platform that is used to power Microsoft’s storefronts across Windows Store, Xbox, and a large set of Microsoft services. One of the key internal components in the Universal Store backend is the Global Homing Service (GHS), a highly reliable service that provides its downstream consumers with the ability to quickly retrieve location metadata associated with one to many, arbitrary large number of, IDs.

Global Homing Service (GHS) using Azure DocumentDB across 4 regions

GHS is on a hot path for the majority of its consumer services and receives hundreds of thousands of requests per second. Therefore, the latency and throughput requirements for the service are strict. The service had to maintain 99.99% availability and predictable latencies under 300ms end-to-end at the 99.9th percentile to satisfy requirements of its partner teams. To reduce latencies, the service is geo-distributed so that it is as close as possible to calling partner services.

The initial design of GHS was implemented using a combination of Azure Table Storage and various levels of caches. This solution worked well for the initial set of loads, but given the critical nature of GHS and increased adoption of the service from key partners, it became apparent that the existing SLA was not going to meet their partners’ P99.9 requirements of <300ms with a 99.99% reliability over 1 minute. Partners with a critical dependency on the GHS call path found that even if the overall reliability was high, there were periods of time where the number of timeouts would exceed their tolerances and result in a noticeable degradation of the partner’s own SLA. These periods of increased timeouts were given the name “micro-outages” and key partners started tracking these daily.

After investigating many possible solutions, such as LevelDB, Kafka, MongoDB, and Cassandra, the Universal Store team chose to replace GHS’s Azure Table backend and the original cache in front of it with an Azure DocumentDB backend. GHS deployed a single DocumentDB collection with 600,000 request units per second deployed across four geographic regions where their partner teams had the biggest footprint. As a result of the switch of DocumentDB, GHS customers have seen p50 latencies under 30ms and a huge reduction in the number and scale of micro-outages. GHS’s availability has remained at or above 99.99% since the migration. In addition to the increase in service availability, overall latencies significantly improved as well for most of GHS call patterns.

Number of GHS micro-outages before and after DocumentDB migration

Microsoft Azure IoT Hub: How to handle the firehose from billions of IoT devices

Azure IoT Hub is a fully managed service that allows organizations to connect, monitor, and manage up to billions of IoT devices. IoT Hub provides reliable communication between devices, the a queryable store for device metadata and synchronized state information, and provides extensive monitoring for device connectivity and device identity management events. Since IoT Hub is at the ingestion point for the massive volume of writes coming from IoT devices across all of Azure, they needed a robust and scalable database in their backend.

IoT Hub provides device-related information, “device twins”, as part of its APIs that device and back ends can use to synchronize device conditions and configuration. A device twin is a JSON document that includes tags assigned to the device in the backend, a property bag of “reported properties” which include device configuration or conditions, and a property bag of “desired properties” that can be used to notify the device to perform a configuration change. The IoT Hub team choose Azure DocumentDB over Hbase, Cassandra, and MongoDB because DocumentDB provided functionality that the team needed like guaranteed low latency, elastic scaling of storage and throughput, provide high availability via global distribution, and rich query capabilities via automatic indexing.

IoT Hub stores the device twin data as JSON documents and performs updates based on the latest state reported by devices in near real-time. The architecture uses a partitioned collection that uses a compound key constructed by concatenating the Azure account (tenant) ID and the device ID to elastically scale to handle massive volumes of writes. IoT Hub also uses Service Fabric to scale out devices across multiple servers, each server communicating with a 1-N DocumentDB partitions. This topology is replicated across each Azure region that IoT Hub is available.

Next steps

In this blog, we looked at a couple of first-party use cases of DocumentDB and how these Microsoft teams were able to utilize Azure DocumentDB to improve user experience, improve latency, and reliability of their services.

Learn more about global distribution with DocumentDB.
Create a new DocumentDB account from the Azure Portal or download the DocumentDB Emulator.
Stay up-to-date on the latest DocumentDB news and features by following us on Twitter @DocumentDB or reach out to us on the developer forums on Stack Overflow.

Quelle: Azure

21. April 201721. April 2017

da Agency

How Microsoft builds massively scalable services using Azure DocumentDB

Microsoft DnA: How Microsoft uses error reporting and diagnostics to improve Windows

Windows error reporting diagnostic information

Windows&039; need for fast key-value lookups

Windows cab catalog metadata and query

You can read the case study at Microsoft’s DnA team achieves planet-scale big-data collection with Azure DocumentDB.

Microsoft Global Homing Service: How Xbox Live and Universal Store build highly available location services

Global Homing Service (GHS) using Azure DocumentDB across 4 regions

Number of GHS micro-outages before and after DocumentDB migration

Microsoft Azure IoT Hub: How to handle the firehose from billions of IoT devices

Next steps

Quelle: Azure

21. April 201721. April 2017

da Agency

Announcing Cloud Partner Portal: Public preview for single-virtual machine offers

Today, we are excited to announce the public preview of the Cloud Partner Portal for publishing single virtual machine offers. The Cloud Partner Portal enables our publisher partners to create, define, publish and get insights for their single virtual machine offerings on the Azure Marketplace.

With this announcement, new and existing publisher partners who wish to publish virtual machines onto Azure will be able to use the new Cloud Partner Portal to perform any of the above actions. This new portal will soon support all other offer types and will replace the current publishing portal in time.

The new improved Cloud Partner Portal

Today’s release has several new features that make publishing onto the Azure Marketplace a lot faster, simpler and easier.

Features of Today’s Release:

1. Org Id login support – This has been an ask from our publisher partners for a long time and we are adding support for Org Id to the Cloud Partner Portal. Additionally, the new publishing portal would support RBAC so that offers remain secure and publishers don’t have to make all contributors co-admins giving them only the level of access needed.

2. Get it right the first time – Everyone hates do-overs. There is nothing worse than spending time defining an offer and thinking you are done only an issue with the offer downstream. To prevent this, your offer is validated as you type. This reduces unwanted surprises after publishing the offer.

Additionally, we anticipate an overall reduction in time from starting defining an offer to actually publishing an offer.

We have spent a considerable amount of time in writing validations for every field within the offer to ensure when publishers click publish, their offer will publish successfully. Even as we ship, we are adding new validations with every release which make the process a lot more predictable.

3. Simplified publishing workflow – The new publishing portal has a simplified publishing workflow providing one path to offer publishing. There are no separate production and staging slots exposed to publishers. Publishers just need to ‘Publish’ their offers, and we take care of the rest.

Before an offer goes live, publishers are given a chance to review it and ensure that everything is working as expected.

4. Be more informed – The new Cloud Partner Portal lets publishers know even before they publish their offer about the steps their offer would go through along with estimated execution times. Along with the guidance around the workflows, we have notifications built into the portal which keep the publishers informed on your offer’s progress to getting listed on Azure.

5. Insights in the portal – The Cloud Partner Portal provides a direct link into the insights of an offer. These insights provide a quick glance and drilldowns into an offer’s health and performance on the Azure Marketplace. The insights portal also has an onboarding video and rich documentation that helps publishers familiarize themselves with its features.

6. Feedback is just a click away – The send a smile/frown button will be ubiquitous in the new portal. In a matter of clicks publishers can send feedback directly to the engineering team.

I can keep writing about the host of new features and capabilities of the new publishing portal, however the best way to discover these features is to take the portal for a spin.

If you are an existing Azure Publisher with a Virtual Machine offer, your account for the new publishing portal is already created. Please visit the Cloud Partner Portal and login using your current credentials. Please refer to our documentation if you would need any help in getting started.

Existing publishers can also let us know if they would like to get their offers migrated following the steps available to registered publishers. We also have a brand new seller guide that can help you navigate the Azure Marketplace better and get most value out of it.

If you are a new Publisher looking to publish onto the Azure platform, please fill up the nomination form here and we will be in touch with you.

As you try out the new cloud partner portal, please keep the steady stream of feedback coming in. We hope you enjoy using the portal as much as we enjoyed creating it for you.
Quelle: Azure

21. April 201721. April 2017

da Agency

Cloudera now supports Azure Data Lake Store

With the release of Cloudera Enterprise Data Hub 5.11, you can now run Spark, Hive, and MapReduce workloads in a Cloudera cluster on Azure Data Lake Store (ADLS). Running on ADLS has the following benefits:

Grow or shrink a cluster independent of the size of the data.
Data persists independently as you spin up or tear down a cluster. Other clusters and compute engines, such as Azure Data Lake Analytics or Azure SQL Data Warehouse, can execute workload on the same data.
Enable role-based access controls integrated with Azure Active Directory and authorize users and groups with fine-grained POSIX-based ACLs.
Cloud HDFS with performance optimized for analytics workload, supporting reading and writing hundreds of terabytes of data concurrently.
No limits on account size or individual file size.
Data is encrypted at rest by default using service-managed or customer-managed keys in Azure Key Vault, and is encrypted with SSL while in transit.
High data durability at lower cost as data replication is managed by Data Lake Store and exposed from HDFS compatible interface rather than having to replicate data both in HDFS and at the cloud storage infrastructure level.

To get started, you can use the Cloudera Enterprise Data Hub template or the Cloudera Director template on Azure Marketplace to create a Cloudera cluster. Once the cluster is up, use one or both of the following approaches to enable ADLS.

Add a Data Lake Store for cluster wide access

Step 1: ADLS uses Azure Active Directory for identity management and authentication. To access ADLS from a Cloudera cluster, first create a service principal in Azure AD. You will need the Application ID, Authentication Key, and Tenant ID of the service principal.

Step 2: To access ADLS, assign the permissions for the service principal created in the previous step. To do this, go to the Azure portal, navigate to the Data Lake Store, and select Data Explorer. Then navigate to the target path, select Access and add the service principal with appropriate access rights. Refer to this document for details on access control in ADLS.

Step 3: Go to Cloudera Manager -> HDFS -> Configuration. Add the following configurations to core-site.xml:

Use the service principal property values obtained from Step 1 to set these parameters:

<property>
<name>dfs.adls.oauth2.client.id</name>
<value>Application ID</value>
</property>
<property>
<name>dfs.adls.oauth2.credential</name>
<value>Authentication Key</value>
</property>
<property>
<name>dfs.adls.oauth2.refresh.url</name>
<value>https://login.microsoftonline.com/<Tenant ID>/oauth2/token</value>
</property>
<property>
<name>dfs.adls.oauth2.access.token.provider.type</name>
<value>ClientCredential</value>
</property>

Step 4: Verify you can access ADLS by running a Hadoop command, for example:

hdfs dfs -ls adl://<your adls account>.azuredatalakestore.net/<path to file>/

Specify a Data Lake Store in the Hadoop command line

Instead of, or in addition to, configuring a Data Lake Store for cluster wide access, you could also provide ADLS access information in the command line of a MapReduce or Spark job. With this method, if you use an Azure AD refresh token instead of a service principal, and encrypt the credentials in a .JCEKS file under a user’s home directory, you gain the following benefits:

Each user can use their own credentials instead of having a cluster wide credential
Nobody can see another user’s credential because it’s encrypted in .JCEKS in the user’s home directory
No need to store credentials in clear text in a configuration file
No need to wait for someone who has rights to create service principals in Azure AD

The following steps illustrate an example of how you can set this up by using the refresh token obtained by signing in to the Azure cross platform client tool.

Step 1: Sign in to Azure cli by running the command “azure login”, then get the refreshToken and _clientId from .azure/accessTokens.json under the user’s home directory.

Step 2: Run the following commands to set up credentials to access ADLS:

export HADOOP_CREDSTORE_PASSWORD=<your encryption password>
hadoop credential create dfs.adls.oauth2.client.id -value <_clientId from Step 1> -provider jceks://hdfs/user/<username>/cred.jceks
hadoop credential create dfs.adls.oauth2.refresh.token -value ‘<refreshToken from Step 1>’ -provider jceks://hdfs/user/<username>/cred.jceks

Step 3: Verify you can access ADLS by running a Hadoop command, for example:

hdfs dfs -Ddfs.adls.oauth2.access.token.provider.type=RefreshToken -Dhadoop.security.credential.provider.path=jceks://hdfs/user/<username>/cred.jceks -ls adl://<your adls account>.azuredatalakestore.net/<path to file>
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar teragen -Dmapred.child.env="HADOOP_CREDSTORE_PASSWORD=$HADOOP_CREDSTORE_PASSWORD" -Dyarn.app.mapreduce.am.env="HADOOP_CREDSTORE_PASSWORD=$HADOOP_CREDSTORE_PASSWORD" -Ddfs.adls.oauth2.access.token.provider.type=RefreshToken -Dhadoop.security.credential.provider.path=jceks://hdfs/user/<username>/cred.jceks 1000 adl://<your adls account>.azuredatalakestore.net/<path to file>

Limitations of ADLS support in EDH 5.11

Only Spark, Hive, and MapReduce workloads are supported on ADLS. Support for ADLS in Impala, HBase, and other services will come in future releases.
ADLS is supported as a secondary storage. To access ADLS, use fully qualified URLs in the form of adl://<your adls account>.azuredatalakestore.net/<path to file> .

Additional resources

Cloudera documentation on ADLS support

Quelle: Azure