Tuning for Zero Packet Loss in Red Hat OpenStack Platform – Part 3

In Part 1 of this series Federico Iezzi, EMEA Cloud Architect with Red Hat covered the architecture and planning requirements to begin the journey into achieving zero packet loss in Red Hat OpenStack Platform 10 for NFV deployments. In Part 2 he went into the details around the specific tuning and parameters required. Now, in Part 3, Federico concludes the series with an example of how all this planning and tuning comes together!

Putting it all together …
So, what happens when you use the cpu tuning features?
Well, it depends on the hardware choice of course. But to see some examples we can use Linux perf events to see what is going on. Let’s look at two examples.
Virtual Machine
On a KVM VM, you will have the ideal results because you don’t have all of the interrupts from the real hardware:
$ perf record -g -C 1 — sleep 2h
$ perf report –stdio -n
# To display the perf.data header info, please use –header/–header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 100  of event ‘cpu-clock’
# Event count (approx.): 25000000
#
# Children      Self  Command  Shared Object      Symbol               
# ……..  ……..  …….  ……………..  …………………
#
  100.00%     0.00%  swapper  [kernel.kallsyms]  [k] default_idle
           |
           —default_idle
              native_safe_halt
  100.00%     0.00%  swapper  [kernel.kallsyms]  [k] arch_cpu_idle
           |
           —arch_cpu_idle
              default_idle
              native_safe_halt
  100.00%     0.00%  swapper  [kernel.kallsyms]  [k] cpu_startup_entry
           |
           —cpu_startup_entry
              arch_cpu_idle
              default_idle
              native_safe_halt
  100.00%   100.00%  swapper  [kernel.kallsyms]  [k] native_safe_halt
           |
           —start_secondary
              cpu_startup_entry
              arch_cpu_idle
              default_idle
              native_safe_halt
  100.00%     0.00%  swapper  [kernel.kallsyms]  [k] start_secondary
           |
           —start_secondary
              cpu_startup_entry
              arch_cpu_idle
              default_idle
              native_safe_halt
Physical Machine
On physical hardware, it’s quite different. The best results involved backlighting a bunch of ipmi and watchdog kernel modules:
$ modprobe -r iTCO_wdt iTCO_vendor_support
$ modprobe -r i2c_i801
$ modprobe -r ipmi_si ipmi_ssif ipmi_msghandler
Note: If you have a different watchdog than the example above, (iTCO is for Supermicro motherboards), check out the kernel modules folder where you can find the whole list: /lib/modules/*/kernel/drivers/watchdog/
Here’s the perf command and output for physical:
$ perf record -F 99 -g -C 2 — sleep 2h
$ perf report –stdio -n
# To display the perf.data header info, please use –header/–header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 4  of event ‘cycles:ppp’
# Event count (approx.): 255373
#
# Children      Self       Samples  Command  Shared Object      Symbol                                        
# ……..  ……..  …………  …….  ……………..  ……………………………………….
#
   99.83%     0.00%             0  swapper  [kernel.kallsyms]  [k] generic_smp_call_function_single_interrupt
           |
           —generic_smp_call_function_single_interrupt
              |          
               –99.83%–nmi_restore
   99.83%     0.00%             0  swapper  [kernel.kallsyms]  [k] smp_call_function_single_interrupt
           |
           —smp_call_function_single_interrupt
              generic_smp_call_function_single_interrupt
              |          
               –99.83%–nmi_restore
   99.83%     0.00%             0  swapper  [kernel.kallsyms]  [k] call_function_single_interrupt
           |
           —call_function_single_interrupt
              smp_call_function_single_interrupt
              generic_smp_call_function_single_interrupt
              |          
               –99.83%–nmi_restore
   99.83%     0.00%             0  swapper  [kernel.kallsyms]  [k] cpuidle_idle_call
           |
           —cpuidle_idle_call
              call_function_single_interrupt
              smp_call_function_single_interrupt
              generic_smp_call_function_single_interrupt
              |          
               –99.83%–nmi_restore
   99.83%     0.00%             0  swapper  [kernel.kallsyms]  [k] arch_cpu_idle
           |
           —arch_cpu_idle
              cpuidle_idle_call
              call_function_single_interrupt
              smp_call_function_single_interrupt
              generic_smp_call_function_single_interrupt
              |          
               –99.83%–nmi_restore
Using mpstat, and excluding the hardware interrupts, the results are as follows:
Please note: one CPU core per socket has been excluded – in this case using two Xeon E5-2640 V4.
$ mpstat -P 1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,31,32,32,34,35,36,37,38,39 3600
Linux 3.10.0-514.16.1.el7.x86_64 (ws1.localdomain)      04/20/2017      _x86_64_     (40 CPU)

03:05:10 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       9    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      13    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      15    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      17    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      19    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      21    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      23    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      24    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      25    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      26    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      27    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      28    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      29    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      31    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      32    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      34    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      35    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      36    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      37    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      38    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:      39    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Cool, right? Want to know more about Linux perf events? Check out the following links:

Linux perf Examples – Read both the web page and the presentation, also listen to the youtube video
SCALE13x: Linux Profiling at Netflix
Tracing and Profiling – Yocto Project
Tutorial – Perf Wiki

Zero Packet Loss, achieved …
As you can see, using tuned and the cpu-partitioning profile is exceptional in that it exposes a lot of deep Linux tuning which usually only a few people know about.

And with a combination of tuning, service settings, and plenty of  interrupt isolations (over 50% of the total settings are about interrupt isolations!) things really start to fly.
Finally, once you make sure PMD threads and VNF vCPUs do not get interrupted by other threads allowing for proper CPU core allocation, the zero packet loss game is achieved.
Of course, there are other considerations such as the hardware chosen, the VNF quality, and the number of PMD threads, but, generally speaking, those are the main requirements.
Further Reading …
Red Hat Enterprise Linux Performance Tuning Guide
Network Functions Virtualization Configuration Guide (Red Hat OpenStack Platform 10)

Check out the Red Hat Services Webinar Don’t fail at scale: How to plan, build, and operate a successful OpenStack cloud today! 

The “Operationalizing OpenStack” series features real-world tips, advice and experiences from experts running and deploying OpenStack.
Quelle: RedHat Stack

Twitter Is Still Dismissing Harassment Reports And Frustrating Victims

Ariel Davis / BuzzFeed News

On July 3, Maggie H. opened up her Twitter mentions and found her face photoshopped into the crosshairs of a gunsight. The image was a screengrab of her Twitter profile page, taken by a user she had blocked. It showed her face directly in the center of a target above a caption that read, “@[username redacted] BTFO by cantbeatkevin kill #390 #noscope” (BTFO is shorthand for “blown the fuck out”).<br /></p><p>Harassment on Twitter was nothing new for Maggie, but this latest threat unnerved her. One day before receiving the photoshopped target image, Maggie had argued with a Twitter troll account by the name of @LowIQCrazyMika. After a contentious back-and -forth, @LowIQCrazyMika tweeted that they’d found Maggie’s Facebook account. A subsequent tweet named the small, rural town in which Maggie lives: “Youre [sic] speaking like the child of an insestual relationship from the remote woods of [town name redacted].”</p><p>Maggie accused the account of stalking her and filed an abuse report to Twitter. Shortly after that, she received the photoshop of her inside the target from a different user. The tweet was retweeted by <b>LowIQCrazyMika. Maggie filed an abuse report for this tweet as well.

Four days after filing the first report, Maggie received a form email from Twitter. It said @LowIQCrazyMika had not violated Twitter’s rules by alluding to her location.

“I'm scared and there's no accountability.”

“I'm scared and there's no accountability,” Maggie told BuzzFeed News. “I even sent Twitter support a copy of my license with an explanation that tweeting that I live in a small town is akin to giving away my exact location, and they're not doing anything.” On July 7, BuzzFeed News contacted Twitter about Maggie’s harassment reports. Twitter declined comment, citing its policy of not commenting on individual accounts. But soon after the tweet with Maggie’s face inside a gun target disappeared, the account that broadcast it was suspended, and Maggie received an email from Twitter noting the company had taken action.

Though the suspension ultimately granted Maggie some peace of mind, her process of getting justice is one of many examples that show a frustrating pattern for victims — one in which Twitter is slow or unresponsive to harassment reports until they’re picked up by the media.

After a decade-long failure to effectively address harassment on its platform, Twitter has finally begun making efforts to curb its abuse problem. Last November it rolled out a keyword filter and a mute tool for conversation threads, as well as a “hateful conduct” report option. In February, the company made changes to its timeline and search designed to hide “potentially abusive or low-quality” tweets, and added a policy update intended to crack down on abusive accounts from repeat offenders. Just last week, Twitter rolled out a few more muting tools for users, including the ability to mute new (formerly known as egg) accounts, as well as accounts that don't follow you.

And yet targeted harassment of the sort Maggie experienced continues. That may be because Twitter’s recent abuse prevention controls are a largely cosmetic solution to a systemic problem. And Twitter’s inconsistent enforcement of harassment reports suggest that perhaps the company’s algorithmic moderation systems simply aren’t as effective as the company would like to think. It’s in these situations that Maggie — and others — have looked for a third party to intervene.

“Twitter isn't taking this problem seriously at all,” Maggie told BuzzFeed News after her troll’s account was suspended. “Can you please help me hold them accountable?” she wrote.

There are no shortage of examples of this pattern. In late June, a BuzzFeed engineer stumbled across a tweet that read “if BuzzFeed headquarters was destroyed in an explosion and every one that worked there was all of a sudden dead, that’d be a good thing.” Five days after reporting the tweet, the engineer received a form email from Twitter stating that the tweet didn’t violate the company’s terms of service. When a BuzzFeed News reporter asked Twitter why this was the case, the tweet was flagged as a violation and removed. Twitter did not explain why the tweet was initially dismissed as not in violation of Twitter’s terms of service.

Twitter has long been criticized for being slow to respond to incidents of abuse on its platform unless they go viral or are flagged by reporters or celebrities. In August of 2016, Twitter told software engineer Kelly Ellis that a string of 70 tweets calling her a “psychotic man hating ‘feminist’” and wishing that she’d be raped did not violate company rules forbidding “targeted abuse or harassment of others.” Shortly after BuzzFeed News published a report on the tweets, however, they were taken down. There were also multiple other instances of tweets being removed only after reports from BuzzFeed News and other outlets.

Given Twitter’s size and the volume of tweets it broadcasts — estimates are old but one report from an app developer in early 2016 clocked over 303 million tweets per day — the social network could never review each and every suspicious tweet with a human eye. But Twitter’s history of opaque protocols for addressing abuse casts its inaction in a different light. In September, a BuzzFeed News survey of over 2,700 users found that 90% of respondents said that Twitter didn’t do anything when they reported abuse. As one victim of serial harassment told BuzzFeed News following the survey, “It only adds to the humiliation when you pour your heart out and you get an automated message saying, ‘We don’t consider this offensive enough.’”

But even with a sharper focus on abuse in 2017, a concerning number of reports of clear-cut harassment still seem to slip through the cracks and return a form email telling the victims that their case did not rise to a level Twitter considers to be a violation of its terms of service. In a cursory search over the last seven months, BuzzFeed News turned up 27 examples of clear rules violations — including the unauthorized publishing of personal information (such as addresses and screenshots of apartment buildings) of journalists, threats of physical violence, and extensive, targeted harassment — that were met with a “did not violate” response from the social network. Similarly, during an open call on Twitter for examples of clear harassment that were dismissed by the company, this writer received 89 direct messages from users alleging that they received at least one improper dismissal of their harassment claim. In more than half of these cases, the users provided BuzzFeed News with a screenshot of the form email they received from the company.

In a number of examples provided to BuzzFeed News, the harassment that was dismissed by Twitter was hardly subtle. One female sportswriter provided seven dismissed harassment reports from Twitter. One account she reported has only posted 78 tweets — all directed at the sportswriter in order to troll her. Another of the accounts she reported uses her likeness in its profile picture. Each and every tweet from the account that BuzzFeed News viewed mocks her or those close to her, including crass tweets about family members transitioning, and harassing tweets about children.

Twitter’s hateful conduct policy explicitly states that “we also do not allow accounts whose primary purpose is inciting harm towards others.” Despite her report to Twitter and a request from BuzzFeed News for Twitter to explain why the account is not in violation of Twitter’s rules, however, the account is still up. All of the accounts the sportswriter reported and shared with BuzzFeed News are still active.

Numerous other accounts reported for serial harassment are still active. @Trap4Von, for example, persists despite a number of abusive tweets including statements like, “i'll follow you home an [sic] leave my children inside of you.” Twitter’s hateful conduct policy explicitly forbids “violent threats,” noting that “you may not promote violence against or directly attack or threaten other people.”

Twitter did not respond to the 27 explicit examples of harassment on which BuzzFeed News requested comment. The company did, however, provide a statement via a spokesperson:

Twitter has undertaken a number of updates, through both our technology and human review, to reduce abusive content and give people tools to have more control over their experience on Twitter. We've also been working hard to communicate with our users more transparently about safety. We are firmly committed to continuing to improve our tools and processes, and regularly share updates at @TwitterSafety. We urge anyone who is experiencing or witnessing abuse on Twitter to report potential violations through our tools so we can evaluate as quickly as possible and remove any content that violates Twitter user rules.

But despite Twitter’s renewed commitment to anti-abuse tools, repeat victims of harassment are frustrated that Twitter’s harassment workflow often requires a cheat code of media involvement.

“There's no way to appeal to them and tell them why they got the decision not to remove tweets wrong, so people who are threatened basically have no choice but to go to someone with a bigger platform,” Maggie told BuzzFeed News.

Kelly Ellis — the software engineer — echoed Maggie’s frustration. “I often feel like, how are other users supposed to escalate stuff,” she said in a recent email. As a victim of continued harassment, Ellis has looked not only to the media, but to Twitter employees.

“I will even sometimes DM people I know there,” Ellis said. “One case that happens pretty frequently is if someone is harassing me with multiple accounts and all the reports will come back as Twitter saying it's not abusive. But then I talk to a friend at Twitter who says it definitely is and helps get it taken care of for me. There's some disconnect going on internally there with their training, I think.”

Just last month a Twitter engineer apologized in a string of tweets for harassment reports falling through the cracks of the company’s reporting system. The apology came after writer Sady Doyle posted a tweet about a troll with the handle @misogyny who’d once tweeted a threatening picture of a gun at Doyle. In her June tweet, Doyle posted screenshots showing that the troll had created a new account and bragged he’d “beat[en] the case” against him. Doyle then posted a screenshot of her plea to Twitter to ban the user alongside Twitter’s response — a form letter saying the account did not violate Twitter rules.

Doyle’s tweet was retweeted over 6,200 times. Shortly thereafter, Twitter reversed its ruling and suspended the user. On Twitter, Doyle’s followers lamented the company’s fickle enforcement. “Basically nothing is worth a ban, unless you get enough people tweeting about it,” one user replied.

The Twitter engineer followed with a sincere apology. “That's often because an employee spots it and internally escalates it. I hate seeing things like this; we have to do better. Sorry.”

Doyle tweeted back that she earnestly appreciated the escalation, but that — like many others — she was troubled by the need to have assistance from a third party.

“When this stuff happens to me, I have a platform, so it's easy to publicize — but other people aren't that lucky.”

Quelle: <a href="Twitter Is Still Dismissing Harassment Reports And Frustrating Victims“>BuzzFeed

Release Announcement: Open Service Broker API 2.12

Learn about the new release of Open Service Broker API 2.12 (OSB 2.12). Released on 30-June-2017, this is the first release of the API developed by the new Open Service Broker API working group as part of an effort by the Kubernetes community to create an integration between Kubenetes and the OSB API.
Quelle: OpenShift

Amazon Kinesis Streams Server-Side Encryption Now Available Broadly

Earlier this month, we announced support for Server-Side Encryption in Kinesis Streams in select AWS regions. Today, we are expanding the availability of this feature to all public AWS regions except the China (Beijing) region, and to the AWS GovCloud region. Server-side encryption makes it easy to meet strict data management requirements by encrypting your data at rest within Kinesis Streams. For more information, see the Kinesis Streams FAQ page and Server-Side Encryption in the Kinesis Streams Developer Guide.
Quelle: aws.amazon.com

IBM Cloud as-a-service revenue up by 32 percent

In its second-quarter earnings released Tuesday, IBM posted yet another strong quarter for cloud, with 15 percent growth year over year and a total of $3.9 billion in revenue in the quarter ($15.1 billion over the past 12 months).
Particularly strong was IBM as-a-service offerings, which saw a 32 percent year-over-year boost and an annual exit run rate of $8.8 billion.
“In the second quarter, we strengthened our position as the enterprise cloud leader and added more of the world’s leading companies to the IBM Cloud,” said Chairman, President and CEO Ginni Rometty. Clients including Lloyds Banking Group, American Airlines, Bombardier, BMW, Comcast Business and NVIDIA started or expanded work with IBM Cloud.
For more about IBM Cloud revenue, including how strategic imperatives generated big growth, check out the infographic below.

 
The post IBM Cloud as-a-service revenue up by 32 percent appeared first on Cloud computing news.
Quelle: Thoughts on Cloud