At The Start Of The Trump Era Facebook And Apple Spent More On Lobbying Than They Ever Have

Justin Sullivan / Getty Images

As Facebook has been tinkering with brain-computer interfaces and an augmented reality camera platform, it&;s also been cultivating some good old fashioned Washington influence. According to federal lobbying disclosures filed Thursday, the social network set it&039;s own all-time record high for spending in a single quarter.

Facebook spent $3.2 million lobbying the federal government in the first months of the Trump era. During the same period last year, Facebook spent $2.8 million (about 15% less). The company lobbied both chambers of Congress, the White House, and six federal agencies on issues including high-tech worker visas, network neutrality, internet privacy, encryption, and international taxation. Facebook was the twelfth highest spender out of any company and second highest in tech.

“Our work in Washington demonstrates our commitment to protecting the people who use our service and promoting the value of innovation to the US economy,” a Facebook spokesperson told BuzzFeed News. “As with the first quarter last year, our expenditure shows an increase that is mainly related to the timing of stock-based compensation for a number of employees.”

While Facebook&039;s first quarter figures for 2016 were the largest for that year, that was not the case in 2015. And outside firms that lobbied on behalf of Facebook spent more money in the first quarter of 2017 than in any other period in the past two years.

Apple spent $1.4 million, just $50,000 more than during the final months of the Obama presidency, when it set its previous record, but the most it has ever spent in a single quarter. Apple lobbied on issues including government requests for data, the regulation of mobile health apps, and self-driving cars.

Google, once again, outspent every other technology company. It was tenth overall, tallying $3.5 million.

“We think it is important to have a strong voice in the debate and help policymakers understand our business and the work we do to keep the Internet open and fuel economic growth,” a Google spokesperson told BuzzFeed News.

While the search giant decreased its lobbying spending compared to this time last year, Amazon, Microsoft, and Uber all boosted their beltway budgets for the first three months of 2017.

Amazon spent nearly $3 million on lobbying, behind only Facebook and Google, and was 17th out of all companies in or out of tech. Amazon met with government officials to discuss net neutrality, drone air cargo, drone privacy, and the flow of data across borders, among other issues. Microsoft claimed $2.3 million as the fourth biggest spender in tech and 27th overall.

Uber continues to dwarf its ride-hail competitor Lyft in lobbying heft. Lyft spent $110,000 in the first quarter of this year, a massive increase over its amount last year, at $20,000. But Uber towered over its rival, spending $370,000, continuing an upward trend.

SpaceX has also amplified its spending compared to the first quarter of last year. Founded by Elon Musk, who was tapped by President Trump as an economic adviser, the company spent $480,000. SpaceX lobbied NASA, the Department of Defense, the National Security Council, and Congress on issues including commercial space launches, NASA funding, and defense space transportation.

The second biggest spender out of any company was AT&T, which shelled out $4.6 million. The company&039;s proposed $85 billion merger with Time Warner, now under review by the Justice Department, is being closely watched by industry analysts and political observers. The outcome of the deal may serve as a test for President Trump, who vowed to squash the proposal if he were elected. But since winning the White House, Trump has softened his remarks.

Next week, during a confirmation hearing on Capitol Hill, Senators will scrutinize Trump&039;s pick to lead the Justice Department&039;s antitrust division, Makan Delrahim. As the country&039;s top competition enforcer, Delrahim would play a key role in AT&T&039;s merger review and in the bubbling debate over market concentration in tech, media, and the broader American economy. Before serving as deputy counsel in Trump&039;s White House, Delrahim worked as a lawyer and lobbyist at the law firm Brownstein Hyatt Farber Schrecka. Qualcomm and Comcast were among his lobbying clients.

Quelle: <a href="At The Start Of The Trump Era Facebook And Apple Spent More On Lobbying Than They Ever Have“>BuzzFeed

Inside The Hunt For Russia’s Hackers

SAN FRANCISCO — Just past 8 a.m. on March 14, police trod quietly through the snow to the double-fronted doors of Karim Baratov’s lavish home in Ancaster, Ontario. The officers passed by the garage where Baratov’s jet-black Mercedes Benz and Aston Martin DBS were parked, two of the only outward indications that the 22-year-old had money to spend. Minutes later, they took the Canadian-Kazakh hacker away into custody — a subdued end to an international cyber drama that involved the highest levels of the US government, Russian spies, a global cybercrime syndicate, and hundreds of millions of unsuspecting Americans.

The baby-faced Baratov is currently awaiting trial in the US on charges that he helped hack into half a billion Yahoo accounts — the largest known hack in history. His co-conspirators are Alexsey Belan, 29, a notorious Russian hacker still at large, and two Russian intelligence officers, Dmitry Aleksandrovich Dokuchaev, 33, and Igor Anatolyevich Sushchin, 43. The case against them is the starkest public example of the ways in which the Russian government works with cybercriminals to achieve its global agenda through cyberwarfare, and why those attacks have proven so difficult for governments around the world to track, let alone prosecute.

Left to right: Baratov, Dokuchaev, and Sushchin.

Courtesy FBI

Baratov, according to accounts given by US law enforcement, was a hacker for hire. It appears he simply took the wrong job.

“The Yahoo hack is a great example of the US government coming forward and saying we know what you are doing and we can prove it,” said Milan Patel, the former chief technology officer of the FBI’s cyber division and now managing director at the K2 Intelligence cybersecurity firm. “In the past the US and Russia engaged in a lot of tit-for-tat covert operations. But with Russia now, a lot is coming to the forefront and being made public about how they run their cyberactivities.”

“We would tip them off about a person we were looking for, and they would mysteriously disappear, only to appear later on working for the Russian government.”

That’s not always how it was. In the mid-2000s, FBI agents tried to work with their counterparts in the FSB, Russia&;s Federal Security Service, to investigate hackers, with regular bilateral meetings featuring US and Russian agents working together in the hope that the two countries could stem the growing tide of online crime. At least that’s how the Americans saw it.

“We would tip them off about a person we were looking for, and they would mysteriously disappear, only to appear later on working for the Russian government,” Patel said. “We basically helped the FSB identify talent and recruit by telling them who we were after.”

The arrest of Baratov and his co-conspirators signals a broader US government crackdown on Russian cybercriminals. For years, cybersecurity researchers and US authorities have traced the ties between cybercriminals and the Russian state, including how malware first developed for criminal enterprises has made its way into state-sponsored cyberattacks on Russia’s neighbors, and how botnet armies created by hackers have been repurposed to launch attacks on Russian targets. Now, they appear ready to strike. Earlier this month, Spanish authorities acting on behalf of the US arrested Pyotr Levashov, long known to authorities as one of the world’s most prolific spam kingpins. Five months ago, the US named a number of well-known Russian hackers as being behind the hacks on the Democratic National Committee, which they say were aimed at influencing the US elections. For those following the murky dealings of the world’s top hackers, the names did not come as a surprise. What was new was the willingness of US officials to publicly name the hackers, and to aggressively pursue Russian cybercriminals who aid Russia’s increasingly aggressive strides into cyberwarfare.

“Russia is playing with different rules — or maybe just without rules.”

Three Russian hackers told BuzzFeed News over the last month that there was “panic” about how far the arrests would go, and for how long hackers would be pursued by US authorities. US security officials told BuzzFeed News that they would do well to be scared, as “the gloves were coming off” with Russian hackers.

“We’ve reached a boiling point with Russia. They are the closest competitor to the US when it comes to cyberespionage and cyberattacks,” Patel said. “But Russia is playing with different rules — or maybe just without rules.”

Erik Carter for BuzzFeed News

Ask Americans to describe a typical Russian hacker who targets the US and they will likely describe a scruffy Russian teenager in a dimly lit basement, or a chiseled military figure in a warehouse-like room filled with hundreds of hackers, pounding away at their keyboards as they plot to take down the US. The truth is that Russian cyber operations are far more complex than either of those scenarios, with the Russian state relying on a network of hackers it hires within its military and intelligence divisions, as well as cybercriminal networks and hackers for hire it can recruit or co-opt as it needs.

“It’s a multilayered system, and it is very flexible. That’s what makes it so hard to track,” said one FBI agent who currently works within the bureau’s cyber division. He asked to speak off the record so that he could discuss open cases with BuzzFeed News. “Let’s say, for instance that Russian intelligence decide they want to hack into eBay to try and find information about a certain person. They might do that through an existing team they have in place, or they might go to a hacker, who has already infected a computer they want compromised and tell him to give them access or else … or they might just pay a guy who has previously hacked eBay to do it for them again.”

That flexibility makes it very difficult for the FBI, or any other law enforcement agency, to track what is being hacked, and why, the FBI agent said.

“They will use whatever method they need to use to get in, and they have no lines between criminals who are hacking for profit and those who are hacking for the government,” he said. “They might be going into eBay to steal credit cards, or they might be doing it as part of a covert op to target a US member of Congress. They might be doing both, really. It makes it hard to know when a hack is a matter of national security and when it is not.”

The hack on Yahoo that compromised the information of more than 500 million people lays out the complex relationship between the hackers and their targets. The accounts were hacked in 2014, with Yahoo only discovering the compromised accounts in September 2016. Just a few months later, Yahoo announced it had discovered a second, earlier breach, which had affected an additional 500 million people in 2013. Together, the hacks cost the company roughly $350 million, as users fled from the platform amid security concerns. It was, cybersecurity experts said, a death blow for Yahoo.

A spokesman for Yahoo did not answer a request for comment from BuzzFeed News. In a public statement published soon after the indictment was issued, Yahoo wrote: “The indictment unequivocally shows the attacks on Yahoo were state-sponsored. We are deeply grateful to the FBI for investigating these crimes and the DOJ for bringing charges against those responsible.”

For weeks, cybersecurity researchers investigating the hacks believed they were looking at a case of corporate espionage. But as the scope of the breach was discovered, researchers began to fear that an enemy of the US was compiling a massive database of all US nationals, complete with personal details and email accounts they could mine for vulnerable information. The indictments issued last month against Baratov, Belan, and the FSB officers revealed that the group had breached Yahoo looking for both political targets and financial targets. The hundreds of millions of other people who had been caught up in the breach were just collateral damage.

The hundreds of millions of other people who had been caught up in the breach were just collateral damage.

“The guys who did this to Yahoo, they were criminals. They could have turned around and sold the entire database to the highest bidder,” the FBI agent said. “We are lucky they didn’t.”

Enough is known about the four men to sketch a rough timeline of how they came together to carry out the hack. Dokuchaev was once known in hacker circles as “Forb,” and he spoke openly about hiring out his services until he was recruited into government work, as the Russian newspaper RBC has reported. At the FSB, Dokuchaev was partnered with Sushchin, and the two recruited Belan, a Latvian-born hacker who had been on a list of the FBI’s most wanted since 2012.

“This is the way it goes: They trap one hacker and then they get him to trap his friends,” said one Russian hacker, who agreed to speak to BuzzFeed News via an encrypted app on condition of anonymity. The hacker, who recently served time in a Russian prison and had fled the country once he was released, said the “pressure was intense” to do work on behalf of Russian intelligence officers. “They press on you. It’s not, like, a nice request. It’s a knock on your door and maybe a knock on your ass. If they can’t threaten you they threaten your family.”

Amedeo DiCarlo, lawyer for Karim Baratov, arrives at the courthouse in a chauffeured Rolls-Royce in Hamilton, Ontario, Canada, on Wednesday, April 5.

Robert Gillies / AP

It’s unclear how the men were connected to Baratov, who immigrated to Canada from Kazakhstan with his family in 2007. Investigators say Baratov was a hacker for hire. In a July 14, 2016, post on his Facebook page, Baratov wrote that he first discovered how profitable hacking could be when he was expelled from his high school for “threatening to kill my ex-friend as a joke.” The time off school “allowed me to work on my online projects 24/7, and really move my businesses to the next level.” The post, which included photos of a BMW, Audi, and Lamborghini, claims he made “triple and even quadruple the normal amount” of income. He ended the post with “Taking shortcuts doesn&039;t mean shortcutting the end result.”

Once the group had gained access to Yahoo, its targets included an economic development minister of a country bordering Russia, an investigative reporter who worked for Russian newspaper Kommersant, and a managing director of a US private equity firm, court documents show. FBI investigators believe that in addition to searching for the political targets requested by the FSB, Belan also used the Yahoo database to line his own pockets by searching for credit card information and devising various schemes to target Yahoo users. In November 2014, he began tampering with the Yahoo database so that anyone interested in erectile dysfunction treatments was redirected to his own online pharmacy store, from which he got a commission for driving traffic to the site.

“It’s a knock on your door and maybe a knock on your ass. If they can’t threaten you they threaten your family.”

“When you look at this case, you realize it has national security and criminal elements. It doesn’t fit neatly into one box or the other,” the FBI agent involved in the case said.

Patel said that the FBI often had difficulty distinguishing between cyber cases that were criminal in nature, versus those which were politically motivated, or had ties to the Russian state. “The government is making an effort to bridge the gap between investigations that involve classified national security issues, and those which are criminal because those worlds aren’t separate anymore,” he said, explaining that departments were trying to form more joint task forces and share classified information when possible.

It’s unclear who within the FSB was responsible for the group, or if their orders ultimately came from another arm of Russia’s government. In December 2016, Dokuchaev was arrested in Russia and accused of treason. His arrest appeared to be part of a roundup of Russian military and cybersecurity figures, though little information has emerged since their arrests.

Andrei Soldatov, a Russian investigative journalist and co-author of The Red Web, a book about the Kremlin’s online activities, said that while the Russian government’s tactic of outsourcing cyber operations to various groups is helpful in distancing themselves (and ultimately providing deniability), it also left them vulnerable to hackers running amuck.

“Hackers are not people who are traditionally easy to control,” said Soldatov. “They might disobey you sometimes.”

Erik Carter for BuzzFeed News

When asked why they first started hacking, many Russian hackers say you’ve asked them to solve the question of which came first, the chicken or the egg.

“I hacked because I wanted to get online, and then I was online because I was hacking,” said one Russian hacker, who considers himself a veteran of the Russian hacking scene due to his early involvement in credit cards schemes in the 1990s. He agreed to speak with BuzzFeed News on condition of anonymity, as he was concerned for the safety of himself and his family. “In the &039;90s you could only afford the internet in Russia if you were rich, or a hacker.”

Russians visit a cybercafé on July 25, 1997, in Moscow.

Andres Hernandez / Getty Images

The internet came to Russia after the fall of the Soviet Union. A devastated economy and uncertain political times meant that few had access to the internet, which could cost hundreds of dollars to surf for just a few hours. The Russian hacker said he and his friends got involved in early credit card schemes as a way of paying for internet use, which they then used to discover more about burgeoning online crime.

“We were baby hackers. Nobody knew what was possible,” he said. “But when the internet came to Russia, so did the hackers.”

Police initially ignored cybercriminals, and a de facto rule came into effect that as long as the hackers were targeting people and institutions outside of Russia, they would be left alone by the state.

“We were baby hackers. Nobody knew what was possible,” he said. “But when the internet came to Russia, so did the hackers.”

Quelle: <a href="Inside The Hunt For Russia’s Hackers“>BuzzFeed

Teslas Is Recalling 53,000 Cars Over Parking Brake Issues

Spencer Platt / Getty Images

Tesla has issued a recall for 53,000 Model S and Model X electric cars it manufactured in between February and October 2016. The recall, which concerns issues with the cars&; parking brakes, accounts for two-thirds of the nearly 84,000 cars the company produced in 2016.

The problem affects a gear in the electric parking brake that could, if it malfunctioned, cause the brake to stick in place, according to Tesla&039;s statement. An Italian company, Brembo SpA, manufactured the part. Tesla will replace the brakes in affected cars and said that while customers wait, it&039;s still safe to use their vehicles.

The company said in a statement that less than 5% of cars were likely affected and that no accidents or injuries have been reported. “In order to be overly cautious, we are going to be proactively replacing these parts,” Tesla said.

The company also recalled 90,000 Model S cars in 2015 over concerns about the car&039;s seat belts.

Tesla&039;s stock has surged 700% in recent years, even climbing past Ford&039;s this month. However, Tesla is still not profitable. Ford turned a profit of $4.6 billion in 2016. Tesla&039;s stock fell 1% at the news of the recall and Ford&039;s is up 2.5%, but Tesla&039;s valuation is still higher.

The company is expected to ship its Model 3 in the summer of 2018, though analysts fear delays, which have plagued the company before. CEO Elon Musk has said he plans for Tesla to make one million vehicles in 2018.

Quelle: <a href="Teslas Is Recalling 53,000 Cars Over Parking Brake Issues“>BuzzFeed

How Microsoft builds massively scalable services using Azure DocumentDB

This week at Microsoft Data Amp we covered how you can harness the incredible power of data using Microsoft’s latest innovations in its Data Platform. One of the key pieces in the Data Platform is Azure DocumentDB, Microsoft’s globally distributed NoSQL database service. Released in 2015, DocumentDB is being used virtually ubiquitously as a backend for first-party Microsoft services for many years.

DocumentDB is Microsoft&;s multi-tenant, globally distributed database system designed to enable developers to build planet scale applications. DocumentDB allows you to elastically scale both, throughput and storage across any number of geographical regions. The service offers guaranteed low latency at P99, 99.99% high availability, predictable throughput, and multiple well-defined consistency models, all backed by comprehensive SLAs. By virtue of its schema-agnostic and write optimized database engine, by default DocumentDB is capable of automatically indexing all the data it ingests and serve SQL, MongoDB, and JavaScript language-integrated queries in a scale-independent manner. As a cloud service, DocumentDB is carefully engineered with multi-tenancy and global distribution from the ground up.

In this blog, we cover case studies of first-party applications of DocumentDB by the Windows, Universal Store, and Azure IoT Hub teams, and how these teams could harness the scalability, low latency, and flexibility benefits of DocumentDB to innovate and bring business value to their services.

Microsoft DnA: How Microsoft uses error reporting and diagnostics to improve Windows

The Windows Data and Analytics (DnA) team in Microsoft implements the crash reporting technology for Windows. One of their components runs as a Windows Service in every Windows device. Whenever an application stops responding on a user&039;s desktop, Windows collects post-error debug information and prompts the user to ask if they’re interested in finding a solution to the error. If the user accepts, the dump is sent over the Internet to the DnA service. When a dump reaches the service, it is analyzed and a solution is sent back to the user when one is available.

Windows error reporting diagnostic information

 

Windows&039; need for fast key-value lookups

In DnA’s terminology, crash reports are organized into “buckets”. Each bucket is used to classify an issue by key attributes such as Application Name, Application Version, Module Name, Module Version, and OS Exception code. Each bucket contains crash reports that are caused by the same bug. With the large ecosystem of hardware and software vendors, and 15 years of collected data about error reports, the DnA service has over 10 billion unique buckets in its database cluster.

One of the DnA team’s requirements was rather simple at face value. Given the hash of a bucket, return the ID corresponding to its bucket/issue if one was available. However, the scale posed interesting technical challenges. There was a lot of data (10 billion buckets, growing at 6 million a day), high volume of requests and global reach (requests from any device running Windows), and low latency requirements (to ensure a good user experience).

To store “Bucket Dimensions”, the DnA team provisioned a single DocumentDB collection with 400,000 request units per second of provisioned throughput. Since all access was by the primary key, they configured the partition key to be the same as the “id”, with a digest of the various attributes as the value. As DocumentDB provided <10 ms read latency and <15ms write latency at p99, DnA could perform fast lookups against buckets and lookup issues even as their data and request volumes continued to grow over time.

Windows cab catalog metadata and query

Aside from fast real-time lookups, the DnA team also wanted to use the data to drive engineering decisions to help improve Microsoft and other vendors’ products by fixing the most impactful issues. For example, the team has observed that addressing the top 1 percent of reliability issues could address 50 percent of customers’ issues. This analysis required storing the crash dump binary files, “cabs”, extracting useful metadata, then running analysis and reports against this data. This presented a number of interesting challenges on its own.

The team deals with approximately 600 different types of reliability-incident data. Managing the schema and indexes required a significant engineering and operational overhead on the team.
The cab metadata was also a big volume of data. There were about 5 billion cabs, and 30 million new cabs were added every day.

The DnA team could migrate their Bucket Dimension and Cab Catalog stores to DocumentDB from their earlier solution based on an on-premises cluster of SQL Servers. Since shifting the database’s heavy lifting to DocumentDB, DnA benefited from the speed, scale, and flexibility offered by DocumentDB. More importantly, they could focus less on maintenance of their database and more on improving user experience on Windows.

You can read the case study at Microsoft’s DnA team achieves planet-scale big-data collection with Azure DocumentDB.

Microsoft Global Homing Service: How Xbox Live and Universal Store build highly available location services

Microsoft’s Universal Store team implements the e-commerce platform that is used to power Microsoft’s storefronts across Windows Store, Xbox, and a large set of Microsoft services. One of the key internal components in the Universal Store backend is the Global Homing Service (GHS), a highly reliable service that provides its downstream consumers with the ability to quickly retrieve location metadata associated with one to many, arbitrary large number of, IDs.

Global Homing Service (GHS) using Azure DocumentDB across 4 regions

GHS is on a hot path for the majority of its consumer services and receives hundreds of thousands of requests per second. Therefore, the latency and throughput requirements for the service are strict. The service had to maintain 99.99% availability and predictable latencies under 300ms end-to-end at the 99.9th percentile to satisfy requirements of its partner teams. To reduce latencies, the service is geo-distributed so that it is as close as possible to calling partner services.

The initial design of GHS was implemented using a combination of Azure Table Storage and various levels of caches. This solution worked well for the initial set of loads, but given the critical nature of GHS and increased adoption of the service from key partners, it became apparent that the existing SLA was not going to meet their partners’ P99.9 requirements of <300ms with a 99.99% reliability over 1 minute. Partners with a critical dependency on the GHS call path found that even if the overall reliability was high, there were periods of time where the number of timeouts would exceed their tolerances and result in a noticeable degradation of the partner’s own SLA. These periods of increased timeouts were given the name “micro-outages” and key partners started tracking these daily.

After investigating many possible solutions, such as LevelDB, Kafka, MongoDB, and Cassandra, the Universal Store team chose to replace GHS’s Azure Table backend and the original cache in front of it with an Azure DocumentDB backend. GHS deployed a single DocumentDB collection with 600,000 request units per second deployed across four geographic regions where their partner teams had the biggest footprint. As a result of the switch of DocumentDB, GHS customers have seen p50 latencies under 30ms and a huge reduction in the number and scale of micro-outages. GHS’s availability has remained at or above 99.99% since the migration. In addition to the increase in service availability, overall latencies significantly improved as well for most of GHS call patterns.

Number of GHS micro-outages before and after DocumentDB migration

Microsoft Azure IoT Hub: How to handle the firehose from billions of IoT devices

Azure IoT Hub is a fully managed service that allows organizations to connect, monitor, and manage up to billions of IoT devices. IoT Hub provides reliable communication between devices, the a queryable store for device metadata and synchronized state information, and provides extensive monitoring for device connectivity and device identity management events. Since IoT Hub is at the ingestion point for the massive volume of writes coming from IoT devices across all of Azure, they needed a robust and scalable database in their backend.

IoT Hub provides device-related information, “device twins”, as part of its APIs that device and back ends can use to synchronize device conditions and configuration. A device twin is a JSON document that includes tags assigned to the device in the backend, a property bag of “reported properties” which include device configuration or conditions, and a property bag of “desired properties” that can be used to notify the device to perform a configuration change. The IoT Hub team choose Azure DocumentDB over Hbase, Cassandra, and MongoDB because DocumentDB provided functionality that the team needed like guaranteed low latency, elastic scaling of storage and throughput, provide high availability via global distribution, and rich query capabilities via automatic indexing.

IoT Hub stores the device twin data as JSON documents and performs updates based on the latest state reported by devices in near real-time. The architecture uses a partitioned collection that uses a compound key constructed by concatenating the Azure account (tenant) ID and the device ID to elastically scale to handle massive volumes of writes. IoT Hub also uses Service Fabric to scale out devices across multiple servers, each server communicating with a 1-N DocumentDB partitions. This topology is replicated across each Azure region that IoT Hub is available.

Next steps

In this blog, we looked at a couple of first-party use cases of DocumentDB and how these Microsoft teams were able to utilize Azure DocumentDB to improve user experience, improve latency, and reliability of their services.

Learn more about global distribution with DocumentDB.
Create a new DocumentDB account from the Azure Portal or download the DocumentDB Emulator.
Stay up-to-date on the latest DocumentDB news and features by following us on Twitter @DocumentDB or reach out to us on the developer forums on Stack Overflow.

Quelle: Azure

How Microsoft builds massively scalable services using Azure DocumentDB

This week at Microsoft Data Amp we covered how you can harness the incredible power of data using Microsoft’s latest innovations in its Data Platform. One of the key pieces in the Data Platform is Azure DocumentDB, Microsoft’s globally distributed NoSQL database service. Released in 2015, DocumentDB is being used virtually ubiquitously as a backend for first-party Microsoft services for many years.

DocumentDB is Microsoft&;s multi-tenant, globally distributed database system designed to enable developers to build planet scale applications. DocumentDB allows you to elastically scale both, throughput and storage across any number of geographical regions. The service offers guaranteed low latency at P99, 99.99% high availability, predictable throughput, and multiple well-defined consistency models, all backed by comprehensive SLAs. By virtue of its schema-agnostic and write optimized database engine, by default DocumentDB is capable of automatically indexing all the data it ingests and serve SQL, MongoDB, and JavaScript language-integrated queries in a scale-independent manner. As a cloud service, DocumentDB is carefully engineered with multi-tenancy and global distribution from the ground up.

In this blog, we cover case studies of first-party applications of DocumentDB by the Windows, Universal Store, and Azure IoT Hub teams, and how these teams could harness the scalability, low latency, and flexibility benefits of DocumentDB to innovate and bring business value to their services.

Microsoft DnA: How Microsoft uses error reporting and diagnostics to improve Windows

The Windows Data and Analytics (DnA) team in Microsoft implements the crash reporting technology for Windows. One of their components runs as a Windows Service in every Windows device. Whenever an application stops responding on a user&039;s desktop, Windows collects post-error debug information and prompts the user to ask if they’re interested in finding a solution to the error. If the user accepts, the dump is sent over the Internet to the DnA service. When a dump reaches the service, it is analyzed and a solution is sent back to the user when one is available.

Windows error reporting diagnostic information

 

Windows&039; need for fast key-value lookups

In DnA’s terminology, crash reports are organized into “buckets”. Each bucket is used to classify an issue by key attributes such as Application Name, Application Version, Module Name, Module Version, and OS Exception code. Each bucket contains crash reports that are caused by the same bug. With the large ecosystem of hardware and software vendors, and 15 years of collected data about error reports, the DnA service has over 10 billion unique buckets in its database cluster.

One of the DnA team’s requirements was rather simple at face value. Given the hash of a bucket, return the ID corresponding to its bucket/issue if one was available. However, the scale posed interesting technical challenges. There was a lot of data (10 billion buckets, growing at 6 million a day), high volume of requests and global reach (requests from any device running Windows), and low latency requirements (to ensure a good user experience).

To store “Bucket Dimensions”, the DnA team provisioned a single DocumentDB collection with 400,000 request units per second of provisioned throughput. Since all access was by the primary key, they configured the partition key to be the same as the “id”, with a digest of the various attributes as the value. As DocumentDB provided <10 ms read latency and <15ms write latency at p99, DnA could perform fast lookups against buckets and lookup issues even as their data and request volumes continued to grow over time.

Windows cab catalog metadata and query

Aside from fast real-time lookups, the DnA team also wanted to use the data to drive engineering decisions to help improve Microsoft and other vendors’ products by fixing the most impactful issues. For example, the team has observed that addressing the top 1 percent of reliability issues could address 50 percent of customers’ issues. This analysis required storing the crash dump binary files, “cabs”, extracting useful metadata, then running analysis and reports against this data. This presented a number of interesting challenges on its own.

The team deals with approximately 600 different types of reliability-incident data. Managing the schema and indexes required a significant engineering and operational overhead on the team.
The cab metadata was also a big volume of data. There were about 5 billion cabs, and 30 million new cabs were added every day.

The DnA team could migrate their Bucket Dimension and Cab Catalog stores to DocumentDB from their earlier solution based on an on-premises cluster of SQL Servers. Since shifting the database’s heavy lifting to DocumentDB, DnA benefited from the speed, scale, and flexibility offered by DocumentDB. More importantly, they could focus less on maintenance of their database and more on improving user experience on Windows.

You can read the case study at Microsoft’s DnA team achieves planet-scale big-data collection with Azure DocumentDB.

Microsoft Global Homing Service: How Xbox Live and Universal Store build highly available location services

Microsoft’s Universal Store team implements the e-commerce platform that is used to power Microsoft’s storefronts across Windows Store, Xbox, and a large set of Microsoft services. One of the key internal components in the Universal Store backend is the Global Homing Service (GHS), a highly reliable service that provides its downstream consumers with the ability to quickly retrieve location metadata associated with one to many, arbitrary large number of, IDs.

Global Homing Service (GHS) using Azure DocumentDB across 4 regions

GHS is on a hot path for the majority of its consumer services and receives hundreds of thousands of requests per second. Therefore, the latency and throughput requirements for the service are strict. The service had to maintain 99.99% availability and predictable latencies under 300ms end-to-end at the 99.9th percentile to satisfy requirements of its partner teams. To reduce latencies, the service is geo-distributed so that it is as close as possible to calling partner services.

The initial design of GHS was implemented using a combination of Azure Table Storage and various levels of caches. This solution worked well for the initial set of loads, but given the critical nature of GHS and increased adoption of the service from key partners, it became apparent that the existing SLA was not going to meet their partners’ P99.9 requirements of <300ms with a 99.99% reliability over 1 minute. Partners with a critical dependency on the GHS call path found that even if the overall reliability was high, there were periods of time where the number of timeouts would exceed their tolerances and result in a noticeable degradation of the partner’s own SLA. These periods of increased timeouts were given the name “micro-outages” and key partners started tracking these daily.

After investigating many possible solutions, such as LevelDB, Kafka, MongoDB, and Cassandra, the Universal Store team chose to replace GHS’s Azure Table backend and the original cache in front of it with an Azure DocumentDB backend. GHS deployed a single DocumentDB collection with 600,000 request units per second deployed across four geographic regions where their partner teams had the biggest footprint. As a result of the switch of DocumentDB, GHS customers have seen p50 latencies under 30ms and a huge reduction in the number and scale of micro-outages. GHS’s availability has remained at or above 99.99% since the migration. In addition to the increase in service availability, overall latencies significantly improved as well for most of GHS call patterns.

Number of GHS micro-outages before and after DocumentDB migration

Microsoft Azure IoT Hub: How to handle the firehose from billions of IoT devices

Azure IoT Hub is a fully managed service that allows organizations to connect, monitor, and manage up to billions of IoT devices. IoT Hub provides reliable communication between devices, the a queryable store for device metadata and synchronized state information, and provides extensive monitoring for device connectivity and device identity management events. Since IoT Hub is at the ingestion point for the massive volume of writes coming from IoT devices across all of Azure, they needed a robust and scalable database in their backend.

IoT Hub provides device-related information, “device twins”, as part of its APIs that device and back ends can use to synchronize device conditions and configuration. A device twin is a JSON document that includes tags assigned to the device in the backend, a property bag of “reported properties” which include device configuration or conditions, and a property bag of “desired properties” that can be used to notify the device to perform a configuration change. The IoT Hub team choose Azure DocumentDB over Hbase, Cassandra, and MongoDB because DocumentDB provided functionality that the team needed like guaranteed low latency, elastic scaling of storage and throughput, provide high availability via global distribution, and rich query capabilities via automatic indexing.

IoT Hub stores the device twin data as JSON documents and performs updates based on the latest state reported by devices in near real-time. The architecture uses a partitioned collection that uses a compound key constructed by concatenating the Azure account (tenant) ID and the device ID to elastically scale to handle massive volumes of writes. IoT Hub also uses Service Fabric to scale out devices across multiple servers, each server communicating with a 1-N DocumentDB partitions. This topology is replicated across each Azure region that IoT Hub is available.

Next steps

In this blog, we looked at a couple of first-party use cases of DocumentDB and how these Microsoft teams were able to utilize Azure DocumentDB to improve user experience, improve latency, and reliability of their services.

Learn more about global distribution with DocumentDB.
Create a new DocumentDB account from the Azure Portal or download the DocumentDB Emulator.
Stay up-to-date on the latest DocumentDB news and features by following us on Twitter @DocumentDB or reach out to us on the developer forums on Stack Overflow.

Quelle: Azure

Ubuntu 12.04 (Precise Pangolin) nearing end-of-life

Ubuntu 12.04 "Precise Pangolin" has been with us from the beginning, since we first embarked on the journey to support Linux virtual machines in Microsoft Azure. However, as its five-year support cycle is nearing an end in April 2017 we must now move on and say "goodbye" to Precise. Ubuntu posted the official EOL notice back in March. The following is an excerpt from one of the announcements:

This is a reminder that the Ubuntu 12.04 (Precise Pangolin) release is nearing its end of life. Ubuntu announced its 12.04 (Precise Pangolin) release almost 5 years ago, on April 26, 2012. As with the earlier LTS releases, Ubuntu committed to ongoing security and critical fixes for a period of 5 years. The support period is now nearing its completion and Ubuntu 12.04 will reach its end of life near the end of April 2017. At that time, Ubuntu Security Notices will no longer include information or updated packages, including kernel updates, for Ubuntu 12.04.

The supported upgrade path from Ubuntu 12.04 is via Ubuntu 14.04. Users are encouraged to evaluate and upgrade to our latest 16.04 LTS release via 14.04. Ubuntu 14.04 and 16.04 continue to be actively supported with security updates and select high-impact bug fixes.

For users who can&;t upgrade immediately, Canonical is offering Ubuntu 12.04 ESM (Extended Security Maintenance), which provides important security fixes for the kernel and the most essential user space packages in Ubuntu 12.04. These updates are delivered in a secure, private archive exclusively available to Ubuntu Advantage customers.

Users interested in Ubuntu 12.04 ESM updates can purchase Ubuntu Advantage.

Existing UA customers can acquire their ESM credentials by filing a support request.
Quelle: Azure

Twitter Locks Trump Associate Roger Stone's Account, Again

Roger Stone is back in Twitter&;s penalty box.

Stone, a confidant to President Donald Trump and former adviser to his campaign, had his Twitter account temporarily locked again this week after tweeting “I&039;m watching you and know what you&039;re up to. Better watch your ass” to Media Matters communications director Laura Allison Keiter on Wednesday afternoon. Twitter locked Stone&039;s account in March following another rules violation.

Reached via Twitter, Keiter forwarded an email she received from Twitter Thursday morning confirming it locked Stone&039;s account.

Laura Allison Keiter

Twitter&039;s in the midst of a scotched earth campaign against harassment on its platform. In recent months, the company has rolled out a number of anti-harassment tools including keyword filters and a new disciplinary measure that temporarily throttles the reach of users it believes are targeting others for abuse. In December, Twitter CEO Jack Dorsey said fighting harassment is the company&039;s top priority.

Roger Stone did not immediately reply to a request for comment.

Quelle: <a href="Twitter Locks Trump Associate Roger Stone&039;s Account, Again“>BuzzFeed

Why It's So Hard For Riders To Sue Uber

Why It's So Hard For Riders To Sue Uber

AP/Julio Cortez

Uber users who sign up for the app and agree to its terms of service have been given sufficient notice that they have given up their right to sue the company, Uber said in a Massachusetts appeals court Monday.

A group of Massachusetts riders who sued the company for charging them an $8.75 airport ride fee, were “expressly and conspicuously informed” of Uber&;s terms and conditions once they clicked the “done” button to enter their payment information, Uber said. Those terms include giving up the right to bring a class action lawsuit against the company and an agreement to settle disputes out of court, it said.

“Reasonably communicated notice of terms, coupled with an opportunity to review those terms via hyperlink, satisfies the Massachusetts inquiry notice standard,” the company argued. Whether the rider “bothers to access and read those terms is irrelevant.”

A screenshot of Uber’s notification to consumers of its terms of service and privacy policy.

A screenshot of Uber's notification to consumers of its terms of service and privacy policy.

Uber / Via documentcloud.org

Last year a district court upheld Uber&039;s arbitration clause in a decision, but the Boston users are appealing that ruling, claiming Uber attempts to “obscure” its terms, which “abrogate basic legal rights,” including your constitutional right to a jury trial and any obligation to provide a safe
vehicle, or a safe driver.

“When Uber wants to notify consumers about surge pricing, it makes sure they know about the price hike and requires that they specifically agree to it,” Matthew Wessler, a principal at Gupta Wessler representing the Boston riders, told BuzzFeed News. “But when it comes to requiring the waiver of important constitutional rights, companies are much less likely to provide that kind of clear notice.”

Uber declined to comment to BuzzFeed News.

In a number of cases brought by consumers challenging contractual language that prohibits customers from suing, courts have upheld the contract. But a California court on Monday ruled that Uber obscured its terms of service and privacy policy on its sign up screen, meaning the rider suing the company was not reasonably notified that he was giving up his right to a class action lawsuit.

Uber is far from alone in using so-called arbitration clauses, which prohibit consumers from taking the company to court. They&039;ve become widespread across corporate America.

Companies like Starbucks, In-N-Out and Netflix all have agreements that prohibit class action lawsuits and push people into private arbitration. A number of credit cards, loan products and telecommunications companies like AT&T and Verizon also require consumers to agree to an arbitration provision.

In arbitration, instead of going to court, consumers pay fees that can be as much as $1,450 to resolve their complaints in front of an arbitrator in private proceedings, according to the Consumer Financial Protection Bureau, which released a study of arbitration in 2015. The process is similar to court proceedings, except it is less formal and any award is ultimately decided by the arbitrator, who is not a judge.

Consumers resolved 341 cases through arbitration between filed in 2010 and 2011, but only 32 of those cases ended with an award. The total amount awarded to consumers was $172,433, according to the study.

Following a series of state court decisions, arbitration has grown so common that nearly all consumer contracts contain some type of clause that prohibits class actions and forces people into a private dispute resolution process.

AP/Eric Risberg

Colin Marks, a professor at St. Mary&039;s University School of Law, told BuzzFeed News that courts have typically sided with companies when these clauses are challenged.

“It’s always on the consumer,” he said. “I don&039;t know if the average Uber user knows what arbitration means, but it assumes you know what it means.”

Marks said in this case, Uber met the minimum standard to tell users that they have terms and conditions at some point during the sign up process. Federal and state law does not require companies to disclose key points in the terms, including whether you&039;re giving up your right to sue.

“It&039;s not whether or not it&039;s a legal requirement as an ethical requirement,” Jennifer Bennett, an attorney with the consumer advocacy group Public Justice, told BuzzFeed News. “What companies are doing is sneaking in terms that they know consumers aren’t going to be able to find or know or understand which forces consumers to give up their right to go to court.”

Daniel Simons / Via youtube.com

Bennett said psychological research shows that people tend to overlook details while they are occupied with completing a particular task.

In one experiment, half of people who were asked to count how many times a basketball was being passed around were so focused on that task at hand that they didn&039;t notice a person walking by in a gorilla costume.

In Uber&039;s case, this could mean if a user is presented with the terms of service as a hyperlink while they are typing in payment information, it is unlikely they will click through to read the terms, she said.

“They keep trying to put the burden on the consumer to go around hunting for terms,” Bennett said. “It seems wrong for businesses to do.”

Quelle: <a href="Why It&039;s So Hard For Riders To Sue Uber“>BuzzFeed