If you’re wondering why I’ve been quiet the past few weeks it’s because I’ve been devoting most of my free time to finishing off a new benchmark I’m releasing today called GUIMark. GUIMark is kinda like an Acid3 test on speed that’s geared towards RIA technologies. The goal was to figure out how to implement a reference design in different runtimes and then benchmark how smoothly that design could be animated. So far I have implementations in DHTML, Flex, Java, Silverlight 1 and Silverlight 2. All the results and and implementation details can be found under the GUIMark page.
GUIMark shares alot in common with another RIA benchmark Bubblemark. I’ve written a bit about Bubblemark and why I think an alternative is necessary, but I do believe Bubblemark and GUIMark can coexist while serving 2 different purposes. Alexey Gavrilov stated it best in that he sees Bubblemark as a sortof ‘Hello World’ launchpad into comparing different environments and I agree with him. Bubblemark is a *very* accessible test suite and its easy for any kind of developer to jump in and play around with performance techniques. GUIMark takes a different approach by trying to benchmark the types of UI elements common in our Web 2.0 world. This includes things like vector redraws, alpha transparencies, text reflow, bitmap motion, and 9 scale slicing rules. From there I just fill up the render pipeline until it becomes so over-saturated that it becomes easy to visually distinguish which rendering engines are more efficient then others. As a result, the benchmark is more complicated on a visual level and requires a bit more time then Bubblemark to understand the implementation rules. Lastly with GUIMark I’ve tried to get into some of the lower level details behind how rendering engines work and how that’s affected the creation of this project.
I’m hoping that developers and designers will be able to use this test suite to identify any pros or cons to choosing a particular environment when visual transitions are a key element of the experience. I’m also hoping these benchmarks provide a spotlight for the community that we can turn toward the runtime engineers inside Sun or Adobe or Mozilla to demand better performance.
Before creating the reference design I came up with the following rules to influence the overall benchmark. I wanted to make sure that whatever I created, the results would create meaningful data for both end designers and developers as well as platform engineers at Adobe or Sun or Mozilla to be able to use.
The max draw rate for an application is fixed at 60 frames per second. Since mainstream LCDs are fixed at 60 hertz, most applications have no need to render above 60 frames per second and in general, the OS will throw away any draw requests it receives above that rate. While there are a select number of people still using CRTs and TV manufacturers are pushing 120hz technology, it’s a waste of cpu to generally create animations above 50 to 60 fps. With this in mind its important that no benchmark comes close to this cap. Its very difficult to visually distinguish between an animation running at 50 fps and one running at 60 fps. So the closer I can force the framerate down to 10 fps, the more visual difference there will be between the technologies. Unfortunately I underestimated the optimization capabilities of the Flash engineers.
Animations should run on fixed time as opposed to relative time. One of the important distinctions in the GUIMark test case is that at high framerates, the test runs smoother, not faster. The heartbeat-style animation of the test will always take 2 seconds to perform, whether running at 5 fps or at 50 fps. This gives users a good idea of how smooth they can expect their own transitions to animate.
The test case should be hard to optimize. I took a WYSIWYG approach to the benchmark so that there are no clear optimization paths. Each technology is limited in some way by the effects that are required in the reference design.
Code execution should have a negligible impact. The user code in each of the tests has almost zero impact on the test case. This is very evident when observing Silverlight running on javascript vs C#. My prediction was that I’d see no difference between the 2 code paths and it turned out to be right. I wouldn’t expect to see any improvements when porting the Flex project into a pure AS3 project either. This jsut means that the rendering engine is taking up 99% of the cpu time and we get a pretty good comparison between the different environments.
The rendering pipeline should remain saturated. If the fps is below 60 and the CPU isn’t sitting at 100% while running the test then there’s either something wrong with the test or theres something wrong with the runtime.
Understanding 2D Rendering Subsystems
The rendering subsystem is typically very complicated and includes alot of interworking logic to interface with user code, internal display objects and OS apis to optimize the final output. You can generally expect the 3 following pieces to be present in one way or another inside each runtime.
Layout engine: The layout engine is responsible for calculating the location for objects to be displayed. This includes performing matrix transforms like scaling, rotations, and object positioning, as well as positioning text within a flowing text block: It is also responsible for calculating the entire parent child display object chain.
Drawing engine: The drawing engine creates internal bitmaps of each discreet display objects. It applies colors to gradients, determines antialiasing rules, and draws fonts.
Composition engine: The compositing engine takes the results of the layout engine and drawing engine and sends a final render to the OS to draw. The compositing engine determines the z-index order of objects, calculates final pixel values for alpha overlays and applies clipping rules.
Optimizations within the render subsystem usually come from the interactions between the different engines. For instance, the drawing engine has no need to draw outside a clipped area or beneath other z-index objects, and the compositing engine doesn’t need to update areas of the display not changed by the layout engine or draw engine.
Win Browsers: Internet Explorer 7 | Win XP | Average FPS 28.11
IE 7 pulled off the fastest rendering for the HTML test among Windows based browsers. the framerate chart to teh right baffles me more then any other chart listed on this page since the highs and lows are so rapid and don’t appear to correspond to any direct actions within the test.
Win Browsers: Firefox 2 | Win XP | Average FPS 19.35
Firefox pulled off the slowest average fps for the HTML test on Windows. Maybe Mozilla could use a chunk of that money they’re supposedly raking in to hire someone to work on their rendering engine.
Win Browsers: Opera 9.27 | Win XP | Average FPS 21.34
Opera faired slightly better then Firefox on the Windows HTML test, but the graph really shows just how much different the renderers are working in the 2 different browsers.
Mac Browsers: Safari 3 | OS X 10.5 | Average FPS 18.20
On the Mac side, Safari just barely took top spot among the other browsers in the HTML test. I had really hoped to see a better showing out of the Safari team when going head to head with Internet Explorer under Windows on the same hardware. Consider yourself officially “On Notice” Safari.
Mac Browsers: Firefox 2 | OS X 10.5 | Average FPS 12.91
Just like Windows, Firefox 2 came in last in the HTML test for Mac. Not much more to add apart from whats already been said.
Mac Browsers: Opera 9.27 | OS X 10.5 | Average FPS 17.22
Not too shabby there Opera team! The framerate exibited on the HTML test under Mac Opera was so close to Safari that any number of possible test conditions could see Opera pull ahead under different configurations. Not bad considering their not as close to the OS as the Safari team.
Flex 3 | Win XP | Average FPS 46.08
Flex 3 on Windows handily beat out all other runtimes tested on both Windows and Mac. Even though Adobe pretty much rewrote the Flash Player for version 9, it looks like they incorporated the past 10 years worth of rendering research into the latest player.
Flex 3 | OS X 10.5 | Average FPS 8.01
Flex 3 showed the biggest difference in frame rate between Mac and PC. I may be mistaken but its possible that the Flash player for Mac is still optimized for the PowerPC processor and hasn’t been moved over to the Intel base. The big dips on the graph occur when the center text block is at its largest size during the test run. The layout calculations are pretty significant at that point.
Java 5 Swing | Win XP | Average FPS 19.37
I had always read that Java could spank Actionscript when it came to code execution. I’m not sure if thats still true but I do know that Swing doesn’t hold a candle to Flash’s DisplayObject architecture. These numbers had me questioning whether I had written my Swing code correctly. It might not be as optimized as it could be, but I’m sure alot ofthis is out of teh hands of user code.
Java 5 Swing| OS X 10.5 | Average FPS 7.19
Swing has some serious issues running on Mac under Java 5. For those of you able to run the benchmark under Mac, the “Pause” and “Run Test” buttons are really there, you just can’t see them until you click them, so click around the space below the fps timer and you eventually hit them.
Silverlight 1 | Win XP | Average FPS 9.12
Of all the tests built for GUIMark, I was looking forward to building the Silverlight tests the most. I haven’t had alot of reasons to play around with the SDK and this was the perfect opportunity. From a framework perspective, XAML is a worthy competitor to Flex even though the component architecture could use some work. From a rendering perspective, Silverlight has a LOT of growing up to do. In fact, the framerate listed above is a lie. In Silverlight 1 users had to rely on the Storyboard object to fire timed events, but those events would occasionally queue up and fire twice in a row before rendering the output. Thats why on the chart you see those massive spikes in framerate. The “real” framerate is more in the range of 5 to 6 fps.
Silverlight 1 | OS X 10.5 | Average FPS 5.25
One good thing you can say about Silverlight is that rendering performance is closer together on Windows and Mac then the Flash player is.
Silverlight 2 Beta | Win XP | Average FPS 7.59
I felt that Silverlight deserved a fair chance to compete with the other environments and that Silverlight 2 with compiled C# was the truer environment that Microsoft wants to push. It also gave me a chance to test the theory that the GUIMark testcase wasn’t constrained by user code, but instead the rendering engines built in to each runtime. Instead of using the Storyboard object for creating timers, i was able to switch to the TimerDispatcher class. Although you don’t see the massive spikes anymore, Silverlight 2 is still prone to queuing up timer events and dispatching 2 at a time and you’ll notice in the chart where the framerate goes above 60 because of this problem.
Silverlight 2 Beta | OS X 10.5 | Average FPS 5.38
The Silverlight 2 framerate on Mac is practically identical to Silverlight 1, as is the chart. Again this shows just how little of an effect there is between dynamic and compiled user code when it comes to rendering performance.
