BLOG.info(self) - Performance

Upstream OpenStack Performance and Release-Shaming

bnemec — Mon, 03 Dec 2018 19:01:38 +0000

These topics may seem like strange bedfellows, but trust me: there's a method to my madness. Originally this was going to be part of my Berlin summit post, but as I was writing it got rather long and I started to feel it was important enough to deserve a standalone post. Since there are two separate but related topics here, I've split the post. If you're interested in my technical thoughts on upstream performance testing, read on. If you're only interested in the click-baity release-shaming part, feel free to skip to that section. It mostly stands on its own.

Performance

This came out of a session that was essentially about

This is of some interest to me because my first job out of college was on a performance team. That also means I have a pretty good understanding of what it takes to extensively performance test a major piece of software. Spoiler alert: It's a lot!

We had a team of 4 or 5 people dedicated primarily to performance testing and improvement. And that was just for my department's specific aspect of the product. There was a whole separate "core" performance team that was responsible for the overall product.

Besides people, there were also racks and racks of hardware needed. Though somewhat less once I got the load testing client software running on Linux instead of Windows, which reduced the hardware requirements to drive the tests by 5 or 10x. Still, we had a sizable chunk of a data center set aside exclusively for our use.

You also need software that can run tests and collect the results in a consumable format. Fortunately this seems to be a solved problem for OpenStack as

That's three major pieces - people, hardware, and software - that you need to do performance testing well. OpenStack currently has one of those things that is available upstream. People and hardware? Not so much. I suspect part of the problem is that OpenStack vendors see their performance testing and tuning as a value-add and thus don't tend to publish their results. Certainly Red Hat is doing performance testing downstream, but to my knowledge the results aren't publicly available.

Is this the ideal situation? That depends on how you look at it. From a technical standpoint it would be far better if we had a dedicated upstream team doing regular performance testing against bleeding edge versions of OpenStack. That would allow us to catch performance regressions much faster than we generally do now. Downstream testing is likely happening against the last stable release, and thus is ~6 months behind at all times. On the business side, though, it makes more sense. To a large extent, what Red Hat (for example) is selling is expertise. That includes the expertise of our performance team. If you want their help tuning your cloud, then you pay us for a subscription and you get access to that knowledge. The software is free, but anything beyond that is not. So, ideal? No, but the reality of corporate-sponsored open source probably necessitates it.

The hardware side is also tricky. Upstream ~~OpenStack~~OpenDev infra is exclusively populated by public cloud resources. These are inherently unsuited to performance testing because they are all on shared hardware whose performance can vary significantly based on the amount of load on the cloud. To do proper performance testing you need dedicated hardware with as few variables from run to run as possible. Even when upstream infra has had bare hardware donated to them, in many cases it didn't last. Apparently it's more common for companies to take back hardware donations than cloud resources. Seems odd, I know, but that was the experience related in this session.

So, what will it take to improve the state of upstream performance testing? Probably someone with moderately deep pockets to pay for the time and hardware needed, and who has a vested interest in improving performance upstream. Not a terribly promising answer, I realize, but that's the nature of the beast. This isn't a problem someone can solve by going heads down on it for a week. It takes an ongoing investment, and unless someone's revenue stream is dependent on it I'm not sure I see how it will happen.

Release-Shaming

However, as you can see from the title there was one other thing in this session that I wanted to touch on. Specifically, a rather extended digression where participants in the discussion browbeat

Further, this touches on some feedback we've gotten from operators in the past. Specifically that they don't always feel comfortable discussing things with developers if they're not on the latest release. That's why they like the ops meetup - it's a safe space, if you will, for them to discuss their experience with OpenStack and not have to worry about someone jumping on them for not CD'ing master. I exaggerate, but you see my point. This discussion, and apparently previous discussions, with developers was unnecessarily hostile toward the operators. You know, the people we're writing our software for.

The funny thing is that the discussion kind of shined a light on the exact problem the session was trying to solve. At one point there was basically a list of changes that have been made to improve performance since Mitaka. Which is fine, but did that actually improve performance or do you just think it did? In my experience OpenStack performance does not necessarily improve every cycle. Granted, I no longer operate a cloud at any sort of scale, but as a user of OpenStack I actually find that things seem more sluggish on recent releases than they did 3 years ago. As a result I feel the burden of proof is on the developers to show that their changes actually improved performance. How do you prove that? Performance testing! What the session was about in the first place!

In the interest of fairness, I do agree that it would be better on a number of levels if everyone stayed up to date on the most recent OpenStack releases. Heck, OVB on public cloud was blocked for years by the fact that public clouds were running versions of OpenStack that were too old to support some features I needed. I do get that argument. But I also understand that not every company is in a position to jump on the perpetual upgrade treadmill. The fast forward upgrades work that has been going on for a few cycles is a recognition of this, and it's something we need to keep in mind in other areas as well.

Overall I would say that the OpenStack developer community is a very civil, even downright friendly place most of the time. I hope we can extend the same to our colleagues in the operator community.

Host Filesystem Impact on Tempest Performance in OpenStack

bnemec — Mon, 06 Oct 2014 15:20:31 +0000

As I mentioned in a previous post, about a year ago I picked up a 1U server from EBay to use as a local single-node OpenStack environment. In general I was quite happy with it, but at some point I got tired of paying for electricity to run a fairly power-hungry server that sits idle, or close to it, about 95% of the time. The fans also picked up an annoying whine somewhere along the line, so once I discovered how much more efficient a modern desktop processor would be, while actually performing better than the old dual server ones, I decided it was time for a new box. This post will be the story of my journey to get on a Fedora 20 host

* -ish in the interest of not opening that can of worms ;-)

In summary, there were two big things that were impacting the performance of my Tempest runs: nested virt and the host filesystem/disk performance. Spoilers, I guess. ;-)

Nested Virt

The nested virt topic is the simple one. I just turned it off and immediately saw a huge increase in performance. One particular scenario test that I was using as a benchmark went from taking around 3 minutes with nested virt to just 1 using unaccelerated qemu. This seems a bit counterintuitive since you would expect the accelerated nested virt to be faster, if anything, but I think it demonstrates the immaturity of nested virt at this time. I also had issues with the entire system freezing when I was running nested instances that went away after I stopped doing that. This was a little unfortunate since part of the reason I had gone with an AMD processor was that I understood their nested virt to be more mature than Intel's and I was hoping to take advantage of that. Not the end of the world though.

Note that I'm told there have been significant improvements made to nested virt recently, but given that my two experiences with it so far have been failures (nested virt on the old 1U Intel server did not work right either) I can't say I'll be in a huge hurry to try it again right away. Maybe in another year or two.

In any case, that was the easy problem to find and fix. While disabling nested virt improved performance significantly, I still found that my full Tempest runs were unacceptably slow. I was seeing my hard drive activity pretty much pegged from the start of a run to the end, and because of that the CPU was almost sitting idle much of the time. While my mid-range processor might not be able to run with the big boys, it wasn't to blame here.

Host Filesystems

As it turns out, the biggest culprit was XFS. I've used XFS for quite a number of things over the years, and given that VM storage involves a lot of big files (usually touted as a strength of XFS) I thought it was a natural fit for a virtualization host. In some cases this might be true, but for running Tempest it was very much not. All levels of RAID, including 0, actually made this worse. I don't know if Fedora was improperly configuring the stripe size settings or something, but that didn't appear to be the case and I had the same issue in RAID 1, which doesn't have stripes at all. In fact, both RAID 0 and RAID 1 were much slower than a simple single-drive setup with XFS. I did try some filesystem/NCQ/scheduler tweaking, but none of it seemed to make a significant difference.

I'll include all of my test run results below, but you can see the extremely poor performance of XFS in RAID 0 in comparison to ext4 in RAID 0. It also appears my Seagate drives are slower than my Western Digital ones. In the interest of fairness, I should mention that for some things XFS was in fact noticeably faster, in particular doing package installs. It just kind of fell over when doing Tempest runs.

In addition, you will see some SSD results toward the end. I had initially hoped that I could get away with simply RAIDing more spinny disks to get acceptable performance, but ultimately I decided to bite the bullet and pick up a 256 GB SSD for my performance critical VM's and the OS. The others I am now booting from volume on an LVM RAID 5 array made up of the leftover rotational hard drives. This way I can have instances with fast disk, or instances with practically unlimited space (it's unlikely I would run through multiple TB of space doing any of the things I use this system for).

As you can see, the SSD results are quite good, although once again ext4 is much faster than XFS. Either Tempest runs are a pathologically bad use case for XFS, or the fact that the instances are all ext4 themselves has some impact (these are qcow2 images though, so I'm skeptical of that).

Just to add one more data point, I ran Tempest on a boot from volume instance too. After what I had seen from my RAID 0 runs, the results were surprising, to say the least. Even with the volumes on a software RAID 5 array, not a configuration renowned for excellent performance, it wasn't that far behind the SSD results and was comparable to the previous RAID 0 runs. My theory is that getting the OS (both operating system and OpenStack) disk activity off the rotational disks allowed them to perform better for the Tempest run. It probably doesn't hurt that the LVM volume is using the raw mdraid device either, so there's no underlying filesystem involved at all.

Conclusion

Anyway, I don't know how directly applicable this will be to most people running OpenStack since it's typically going to be deployed on server-class hardware with real RAID controllers, higher quality drives, and dedicated compute and database nodes to reduce disk contention. In addition, my results are far from scientific. All of the numbers below are single runs, so it's possible one or more of them could have been influenced by a transient load on the system, but given how long it takes to do a full Tempest run (especially in the configurations where performance was bad), I wasn't willing to go for a test methodology that would hold up to peer review. I just wanted to find what worked. :-)

However, I did spend quite a bit of time (reinstalling Fedora, RDO, and running Tempest for all of these different configurations was a multi-hour process) coming up with this information, so I thought I'd write it up in case it might be useful to someone else.

After months of off and on experimention, I'm quite happy with my new system's performance now. It ended up costing a bit more than I planned due to the extra drives and SSD, but the new system uses well under half the power of the old one, runs much, much quieter, and does perform better. I can be happy with that.

Raw results

Timing of tox -e full runs

RAID 0 WD/Seagate ext4
======
Totals
======
Run: 2359 in 6377.707648 sec.
 - Passed: 2156
 - Skipped: 200
 - Failed: 3

==============
Worker Balance
==============
 - Worker 0 (700 tests) => 0:37:34.151293s
 - Worker 1 (534 tests) => 0:45:04.037967s
 - Worker 2 (602 tests) => 0:38:58.548955s
 - Worker 3 (523 tests) => 0:44:16.987106s

 RAID 0 WD/WD ext4
 ======
Totals
======
Run: 2359 in 5190.060175 sec.
 - Passed: 2157
 - Skipped: 200
 - Failed: 2

==============
Worker Balance
==============
 - Worker 0 (700 tests) => 0:30:20.502750s
 - Worker 1 (534 tests) => 0:36:06.530701s
 - Worker 2 (602 tests) => 0:30:47.064148s
 - Worker 3 (523 tests) => 0:35:38.674856s


 RAID 0 WD/WD xfs
 ======
Totals
======
Run: 2361 in 11881.111973 sec.
 - Passed: 2144
 - Skipped: 200
 - Failed: 17

==============
Worker Balance
==============
 - Worker 0 (700 tests) => 1:13:16.662514s
 - Worker 1 (536 tests) => 1:28:40.118105s
 - Worker 2 (602 tests) => 1:06:12.777629s
 - Worker 3 (523 tests) => 1:21:01.110717s

 RAID 0 Seagate/Seagate ext4
 ======
Totals
======
Run: 2359 in 6316.422507 sec.
 - Passed: 2158
 - Skipped: 200
 - Failed: 1

==============
Worker Balance
==============
 - Worker 0 (700 tests) => 0:39:26.093815s
 - Worker 1 (534 tests) => 0:45:20.647021s
 - Worker 2 (602 tests) => 0:39:05.222058s
 - Worker 3 (523 tests) => 0:42:10.433892s

 SSD xfs
 ======
Totals
======
Run: 2361 in 5973.290727 sec.
 - Passed: 2156
 - Skipped: 200
 - Failed: 5

==============
Worker Balance
==============
 - Worker 0 (700 tests) => 0:33:53.633396s
 - Worker 1 (536 tests) => 0:42:07.423013s
 - Worker 2 (602 tests) => 0:33:22.217886s
 - Worker 3 (523 tests) => 0:39:16.427730s

 SSD ext4
 ======
Totals
======
Run: 2359 in 3600.612752 sec.
 - Passed: 2157
 - Skipped: 200
 - Failed: 2

==============
Worker Balance
==============
 - Worker 0 (700 tests) => 0:20:22.467067s
 - Worker 1 (534 tests) => 0:24:00.071754s
 - Worker 2 (602 tests) => 0:19:43.647173s
 - Worker 3 (523 tests) => 0:25:15.156410s

 RAID 5 volume
 ======
Totals
======
Run: 2236 in 5290.762352 sec.
 - Passed: 2041
 - Skipped: 193
 - Failed: 2

==============
Worker Balance
==============
 - Worker 0 (624 tests) => 0:39:18.886647s
 - Worker 1 (529 tests) => 0:30:19.590476s
 - Worker 2 (571 tests) => 0:24:39.270117s
 - Worker 3 (512 tests) => 0:40:11.858793s

Using pypi-mirror with devtest

bnemec — Thu, 27 Mar 2014 16:45:43 +0000

A full run of TripleO's export DIB_COMMON_ELEMENTS="stackuser pip-cache pypi"

Then I ran devtest.sh --trash-my-machine and compared the results to a previous run. Here are the contents of devtest_perf.log (which you can find in your TRIPLEO_ROOT directory after a successful run) before and after setting up the mirror:

[2014-03-24 20:27:31] Run comment : No Comment [2014-03-24 20:27:31] Total runtime: 3206 s [2014-03-24 20:27:31] ramdisk : 165 s [2014-03-24 20:27:31] seed : 700 s [2014-03-24 20:27:31] undercloud : 790 s [2014-03-24 20:27:31] overcloud : 1539 s [2014-03-24 20:27:31] DIB_COMMON_ELEMENTS=stackuser pip-cache pypi-openstack use-ephemeral [2014-03-27 15:37:02] Run comment : No Comment [2014-03-27 15:37:02] Total runtime: 2675 s [2014-03-27 15:37:02] ramdisk : 124 s [2014-03-27 15:37:02] seed : 539 s [2014-03-27 15:37:02] undercloud : 639 s [2014-03-27 15:37:02] overcloud : 1320 s [2014-03-27 15:37:02] DIB_COMMON_ELEMENTS=stackuser pip-cache pypi use-ephemeral

Around 9 minutes less for the full run, and that's comparing to a hot cache. If you were running with a cold/no cache I expect the difference would be much larger. I left the pip-cache element since the drawbacks to that element don't matter to me, and I figured it couldn't hurt.

Now that I've got my mirror set up and added the appropriate env vars to my tripleo rc file, I shouldn't have to do much to take advantage of this added speed, which should be a big win for my future development. If I can find a way to avoid hitting an external mirror for package installs (I don't think running a full distro mirror locally is an option, but we'll see), I might be able to keep my image installs entirely on my local gigabit network. Stay tuned. :-)

BLOG.info(self) - Performance

Upstream OpenStack Performance and Release-Shaming

Performance

Release-Shaming

Tags:

Host Filesystem Impact on Tempest Performance in OpenStack

Nested Virt

Host Filesystems

Conclusion

Raw results

Tags:

Using pypi-mirror with devtest

Tags: