After posting about the inadequacy of a recent test report I thought it would be interesting to conduct my own informal tests of Vista vs XP performance. I do not run a computer laboratory, but I guess my tests have the benefit of being real-world.
I tested several conditions on three computers. On two of them I was able to test XP 32-bit vs Vista 32-bit. I tried various combinations of Aero on or off, visual effects on or off, and UAC (User Account Control) on or off. I also tried setting Vista to run only basic services, using Msconfig.
The test suite I used was PassMark Performance Test 6.1. You can use this free for 30 days, so anyone can repeat the tests.
Caveat: perfect tests are hard to achieve. A big problem with Vista is that it has all sorts of background services that are meant to maintain your system, perform backups, check for updates, and so on. Further, if you have third-party software installed (and who doesn’t?) then it may also be running background tasks. The only compensation I applied was to wait a few minutes after start-up for disk activity to pretty much cease. I also switched off anti-virus software, and closed the Vista sidebar.
Here’s a few highlights. First, XP was fastest on the two dual-boot machines, and that was without any effort optimizing XP for performance. In both cases, the best Vista performance was about 15.5% slower than XP, based on the PassMark performance index. On one machine the worst Vista performance was 28% slower, and on the other 23% slower.
Graphics 2D – Ouch!
The most interesting result was the poor performance of the Graphics 2D tests on Vista. A series of PassMark tests cover drawing lines, rectangles, shapes, font rendering, and common GUI operations like scrolling listboxes, moving windows or filling progress bars. On both dual-boot machines, the best Vista performance was an amazing 70% slower than XP. Put another way, XP was 3 times faster. Since these are common operations, that is worrying.
That said, there are some specific reasons for this poor performance. See here for example:
The Windows Vista graphics system is designed to support a broad range of hardware and usage scenarios to enable new technology while continuing to support existing systems. Existing graphics interfaces, such as GDI, GDI+, and older versions of Direct3D, continue to work on Windows Vista, but are internally remapped where possible. This means that the majority of existing Windows applications will continue to work.
Hmm, re-mapping sounds slow. And here:
GDI primitives like LineTo and Rectangle are now rendered in software rather than video hardware, which greatly simplifies the display drivers. We don’t think this will impact many real-world applications (usually when a GDI application is render bound its because it’s doing something like gradients that was never hardware accelerated), but if you do see problems please let us know.
See also Greg Schechter’s notes here on GDI-rendered windows.
In a nutshell, Vista is optimized for DirectX rendering at the expense of GDI; yet, as Schecter notes:
Today and for the near future, most applications use and will continue to use GDI to render their content.
This I suspect is a large part of the reason why some tests report that XP is twice as fast as Vista. Automating Office could well hit this slow GDI issue. Note that it is not all 2D graphics that are slow – CustomPC got less alarming results for its 2D graphics tests, which by the looks of it are not GDI-based.
Schechter does not cover the scenario where the DWM (Desktop Window Manager) is turned off, but it looks as if some of the factors which make GDI slow still apply.
What’s most effective at speeding up Vista?
I did some experiments where I compared Vista Basic with Vista Aero, or looked at what happened when UAC is switched off. I got inconsistent results. On an older machine, I found that disabling Aero made a significant difference, maybe an 8% speed-up. On another, more recent machine, Aero was actually faster. So much changes when you use DWM that I guess this is to be expected.
UAC? Not a huge difference in these tests – around 2%.
The biggest influence, on the basis of my imperfect testing, came when using Msconfig to switch off all but basic services and start-up programs. This speeds performance by around 10% overall.
In most other tests there were modest differences between Vista and XP. This includes 3D graphics, where Vista actually scored higher than XP on one machine, and CPU, where on one machine there was less than 2.5% difference between best and worst. Vista does come out significantly slower on the PassMark memory test suite, from just over 8% worse to over 20% worse.
Conclusion? First, my informal tests suggest that XP is faster than Vista, but not normally twice as fast. Second, an application written to use DirectX rather than GDI should perform better, other things being equal. WPF (Windows Presentation Foundation) uses DirectX, but unfortunately has its own performance overhead.
If this analysis is right, then Vista is at its worst when rendering the GUI in traditional Windows applications. That will make them feel less snappy, but would not impact the non-visual aspect, like say recalculating a spreadsheet.
Then again, I only tried one test suite, so please don’t take the above too seriously.
I’d be interested in further informed comments.
Hardware makes a difference so for each computer I would be interested to know the hardware details and also the Vista Experience Index score and it’s Component Subscores.
My machine is dual booted x32/x64 and I run XP in a VM because dual booting Vista/XP screws up some Vista features like the ability to System Restore.
This comment I’m making can’t be taken as an objective one, but when I downloaded and ran the Family.Show application the first thing I was impressed with was the speed of the graphics. I can only assume that this is because it’s using WPF.
Another, totally subjective, comparison I made was between Vista and XP when both are run in VM’s on a Vista desktop. The former is noticeably sluggish, the latter quite sprightly.
Sure.
Machine A: Celeron 3.0 Ghz, 1GB RAM, NVIDIA GeForce 6600.
Processor: 3.9
Memory: 4.5
Graphics: 5.3
Gaming graphics: 3.8
Primary hard disk: 4.9
Machine B: Pentum 4 3.0 Ghz, 3GB RAM, NVIDIA GeForce 6800 XP.
Processor: 4.2
Memory: 5.2
Graphics: 5.9
Gaming graphics: 4.6
Primary hard disk: 5.4
Machine C (XP not available): Core Duo 6300 1.8 Ghz, 2GB RAM, NVIDIA 7600 GT
Processor: 5.0
Memory: 5.5
Graphics: 5.9
Gaming graphics: 5.4
Primary hard disk: 5.7
Vista runs like a dream on this machine.
Tim
Here’s my machine’s specs for comparison:
Processor 5.1
Memory 5.5
Graphics 4.8
Gaming graphics 4.2
Primary hard disk 5.9
Vista runs fine on this box as well.
Thanks Tim
Your #3 machine is same hardware except I have EVGA nVidia 680i reference motherboard and my E6300 is OC 1.86 -> 2.86 GHz with FSB OC 266 -> 410 MHz and system HD is a WD 10,000rpm Raptor. This makes a difference in that my ratings are:
Processor: 5.7
Memory: 5.9
Graphics: 5.9
Gaming graphics: 5.1
Primary hard disk: 5.9
I’m very pleased with Vista’s overall performance. I use it for business and app dev using both D-2007(Win32) and VS-2008(.NET 3.5). It does run Halo2 and Crysis albeit a little slow but still fast enough for an old geezer like me. ;>)
It might nevertheless be slower than XP on the same hardware. The question about how much this matters is another thing. People don’t generally care about performance unless something is slow.
Tim
It doesn’t matter to me very much at all. The most irksome feature of Vista I have found lately has been the inability to recognise when updates have been applied: KB941649 has been a bloody nightmare.
Tim – Remeber about a year ago when Vista was released a lot of time was spent on testing and most objective tests found that XP was faster on some tasks and slower on others and overall Vista was more-or-less as fast as XP or at worst slightly (~10%) slower. There were some tasks that were incredibility slow on Vista which will be addressed in SP1 if not already fixed by online updates.
For an excellent user experence IMO Vista hardware should be as least as good as your #3 computer. YMMV…
Hello,
What was your actual score from performace test 6.1?
With my outlook / AVG / desktop sidebar / browsing the internet (several instances) / etc… all running (computer not rebooted) – I scored a 1873.8 – overall – without all that stuff running I scored a 1919.3!! – It is hard to get a feel for how this compares to you and your test results. you simply state percentages – but no actual numbers. – you also do not appear to use any good / new hardware.
@tre roth
My ratings are lowly compared to yours:
Machine A: 292.6
Machine B: 368.1
Machine C: 535.5
(All Vista figures).
That said, they are quite usable. Machine C is only 1 year old – same as Vista.
What about the proposition that Vista’s performance will best XP on high-end hardware? It’s possible. Interesting though to see the PassMark fastest ten machines tested:
http://www.passmark.com/baselines/topten.php
The current record holder (score 2641) is running XP32. All the others in the top 10 are running either XP32, or Vista64.
64-bit – now, there’s another thing.
Tim
Tim:
You can see by my box’s ratings that it’s the graphics performance that lets it down. I’m looking to get a cheap graphics card that performs well. Any recommendations?
Not sure; Toms Hardware and the like would be a good place to look.
Thank you for that pointer: I never knew about that site until you told me just then. I might treat myself to one of those for Xmas.
Windows XP will be even faster with the release of Windows XP SP3. The word on the street is it makes significant performance improvements.
You’ll need to rerun those test when Vista SP1 and XP SP3 comes out ;-).
SP1 for Vista is supposed to make significant improvements to Vista’s speed, as reported by users of the SP1 Beta.
I’ve held off on Vista, but I’m looking forward to seeing where it’s going and plan to see the x64 version on my new desktop later this year.