|
Understanding the impact of Windows Vista
on SPECviewperf performance measurement
by Ian Williams
Author’s note: The OpenGL ARB recently published an article clarifying the facts concerning OpenGL and Vista: http://www.opengl.org/pipeline/article/vol003_9/. The article is a good source for technical information on how OpenGL is implemented on Vista, and can be considered a pre-requisite for this article.
Windows Vista introduces many major new features that greatly improve user experience compared to Windows XP. Some of these features, however, affect performance benchmarking. This document aims to help SPECviewperf users better understand these features and how they impact performance measurement.
Inside Aero and DWM
A key feature of Windows Vista is the Aero user interface. Aero includes features such as transparent and blended windows, animated icons, and application preview. Because Aero is tightly integrated with the GPU, it is essentially a complete 3D application in itself, using much of the GPU’s capabilities and horsepower. Windows Vista’s Desktop Window Manager (DWM) is the underlying architecture and mechanism that enables Aero’s capabilities.
Workstations with Windows Vista pre-installed have the DWM and Aero enabled by default. The figure below from the OpenGL ARB article depicts the OpenGL and DX architectures and how they layer into the DWM and Aero.
As the diagram shows, in order to create the composited desktop, the 3D graphics content of all windows (OpenGL or DX) passes through the DWM. This allows the DWM to blend windows onto other windows and the desktop, as well as facilitate animated icons and other features.
Fundamental differences
Windows Vista architecture is fundamentally different than that for Windows XP, where the content of all 3D graphics windows is managed almost exclusively by the ICD graphics driver and no Desktop Window Manager exists. In XP, a 3D window represents a hole in the desktop as far as the operating system is concerned. The change in the DWM architecture between XP and Vista has two implications when benchmarking performance using SPECviewperf.
The first implication is that Aero, a 3D application itself, uses GPU resources and cycles that compete with those of an application such as SPECviewperf. Aero’s design and the architecture of the DWM tries to minimize this impact as much as possible, but it cannot mitigate the potential for resource and cycle conflict when comparing performance with benchmarks running on Windows XP.
The second implication arises because all XP benchmarks are run with what is frequently referred to as “vsync-disabled.” This means that the 3D content is being updated as soon as it is drawn, rather than waiting for a display blanking region where the new content can be updated without yielding a “tear” arising from seeing one (or more) frames at once. The vsync-disabled mode is desirable for SPECviewperf, since the purpose of the benchmark is to test graphics performance, and without vsync disabled the maximum performance would be the refresh rate of the display.
In Windows Vista, vsync is used with all applications, since otherwise the desktop display would tear and that is unacceptable in terms of user experience and quality. In situations where a window’s content is being generated significantly faster than the refresh rate of the display, a tear-free update of the desktop can be achieved in one of two ways.
The first is to limit the window content update to that of the monitor refresh. This forces vsync to stay on all the time, ensuring that the drawing rate will not exceed the monitor refresh rate. The second method is to display only the last frame created prior to the blanking region and drop all other pending frames. This approach has the advantage that it doesn’t gate the application drawing; clearly, however, not all generated frames are displayed. Windows Vista DWM implements the second of these two methods.
Because of architectural differences, simply turning vsync off on Windows Vista does not have the same effect or implication as it does on XP. The only way to ensure that an application rendering is not gated by the display refresh rate is to disable the DWM, which will also turn off the Aero interface. So when comparing XP and Vista performance on benchmarks that render faster than the monitor refresh rate, it is necessary to turn off the DWM to ensure an apples-to-apples comparison.
Representative benchmarking on Vista
Does this mean benchmarks on Windows Vista should always run with the DWM turned off? Absolutely not.
As mentioned earlier, a typical workstation with Windows Vista pre-installed is configured to have the DWM and Aero enabled by default so that all the new features and quality adjustments are available. As a result, adopting a policy of always turning the DWM and Aero off for benchmarking is not representative of how typical users would run applications on Windows Vista, and is counter to the aims of SPECviewperf.
Since the impact of the DWM and Aero is felt only when windows updates are faster than the display refresh rate, if the window updates are significantly slower than the display refresh rate, the impact of the DWM and Aero is minimal. Even when the window update rate is faster than the display refresh rate, the DWM doesn’t stop the GPU from creating the rendering content of the frames; it only drops multiple pending frames from being displayed. The GPU workload has already been executed; it is only the final display of the pixels that doesn’t occur.
One of the goals of SPECviewperf is to try to ensure that the workload is significantly lower than the display refresh rate, and viewsets are regularly refreshed to ensure this. It’s not possible to ensure that this will be the case at all times, however, since graphics and GPU development is very rapid and sometimes individual viewset subtests will perform faster than the refresh rate. This is why it’s important to understand the impact of Vista’s architecture, and not be tempted into making invalid comparisons between Vista and XP benchmarking results.
In summary...
Vista delivers to Windows users a radical change in features and quality, which is good for the workstation industry in general. Inherent in the new features is an architectural re-design that has two main implications when benchmarking on Vista compared with Windows XP:
- the impact of the Aero interface running concurrently with a benchmarked application; and
- the impact of the architectural design of the Desktop Window Manager in situations when an application window updates faster than the display refresh rate.
In order to provide true apples-to-apples performance comparison between a specific system configuration running under Windows XP and the same configuration running under Windows Vista, both the DWM and Aero should be disabled. Unfortunately, this does not represent the default nor desired user experience of professional workstation users running Windows Vista applications – most users will want the DWM and Aero turned on. Benchmarking in this case is perfectly valid, but these results should not be compared to those generated while running Windows XP, even if testing is done with the same hardware configuration.
###
Ian Williams is Senior Applied Engineer at NVIDIA and chair of the SPECopc project group, developer of SPECviewperf. |
