Category Archives: Flash

Flash, Flex, Html, Silverlight

Introducing GUIMark – An RIA benchmark for Flex, Silverlight, HTML and more

10 Comments

If you’re wondering why I’ve been quiet the past few weeks it’s because I’ve been devoting most of my free time to finishing off a new benchmark I’m releasing today called GUIMark. GUIMark is kinda like an Acid3 test on speed that’s geared towards RIA technologies. The goal was to figure out how to implement a reference design in different runtimes and then benchmark how smoothly that design could be animated. So far I have implementations in DHTML, Flex, Java, Silverlight 1 and Silverlight 2. All the results and and implementation details can be found under the GUIMark page.

GUIMark shares alot in common with another RIA benchmark Bubblemark. I’ve written a bit about Bubblemark and why I think an alternative is necessary, but I do believe Bubblemark and GUIMark can coexist while serving 2 different purposes. Alexey Gavrilov stated it best in that he sees Bubblemark as a sortof ‘Hello World’ launchpad into comparing different environments and I agree with him. Bubblemark is a *very* accessible test suite and its easy for any kind of developer to jump in and play around with performance techniques. GUIMark takes a different approach by trying to benchmark the types of UI elements common in our Web 2.0 world. This includes things like vector redraws, alpha transparencies, text reflow, bitmap motion, and 9 scale slicing rules. From there I just fill up the render pipeline until it becomes so over-saturated that it becomes easy to visually distinguish which rendering engines are more efficient then others. As a result, the benchmark is more complicated on a visual level and requires a bit more time then Bubblemark to understand the implementation rules. Lastly with GUIMark I’ve tried to get into some of the lower level details behind how rendering engines work and how that’s affected the creation of this project.

I’m hoping that developers and designers will be able to use this test suite to identify any pros or cons to choosing a particular environment when visual transitions are a key element of the experience. I’m also hoping these benchmarks provide a spotlight for the community that we can turn toward the runtime engineers inside Sun or Adobe or Mozilla to demand better performance.

Go to GUIMark home page

Flash, Java, Javascript, Silverlight

GUIMark

99 Comments

 

GUIMark 2 is out now! View the new tests here.

 

Home | Detailed Analysis | Benchmark and Rendering Engine theory

GUIMark is a benchmark test suite designed to compare the rendering systems of several popular UI runtimes. In general it should be able to give designers and developers a good indication of which technologies can draw complex interfaces at a smooth rate of motion. The test mostly addresses RIA technologies like Flash, Silverlight, HTML or Java, but was designed to be easily ported to any 2D GUI environment. The basis for this project was inspired by the Bubblemark animation test, but was designed to heavily saturate the rendering pipeline and determine what kind of visual complexity is achievable in the sub-60 fps realm.

The Test

GUIMark referenceThe reference design was originally created in Flex and then ported to the technologies listed below. All results listed in the matrix as well as the detailed results page were run on the same Macbook Pro running Leopard for OS X, and running Win XP under a Boot Camp partition. Each test case was run 3 times in a new browser instance and the highest framerate observed was recorded. For the HTML test, the fastest performing browser on a each OS was used in the comparison matrix (Internet Explorer 7 and Safari 3 won for their respective OSes). All subsequent plugin based tests for the OS were tested in those browsers.

My hope is to port the benchmark to all the other untested technologies listed below and I fully welcome any optimizations or ports that readers want to contribute. I’m really curious to see if any community experts or platform engineers are able to speed up their technology of choice. Although the code is fairly simple at a glance, there are no easy optimization paths to be found (and no cheating by turning off anti-aliasing).

Results

Results for Win XP running on Macbook Pro Intel Core 2 Duo 2.33 GHz

Tech Base Version Average FPS Source
Browser HTML 28.36 / IE 7 Download
SVG - -
Canvas - -
Flash Flex 3 46.08 Download
Flash 9 - -
Java Java 5 Swing 19.37 Download
Java 6 Swing - -
Processing - -
JavaFX - -
Silverlight Silverlight 1 / Javascript 9.12 Download
Silverlight 2 Beta / C# 7.95 Download

Results for OS X 10.5 running on Macbook Pro Intel Core 2 Duo 2.33 GHz

Tech Base Version Average FPS Source
Browser HTML 18.20 / Safari 3 Download
SVG - -
Canvas - -
Flash Flex 3 8.01 Download
Flash 9 - -
Java Java 5 Swing 7.19 Download
Java 6 Swing - -
Processing - -
JavaFX - -
Silverlight Silverlight 1 / Javascript 5.25 Download
Silverlight 2 Beta / C# 5.38 Download

Findings

I’ve been surprised with the results so far between WinXP and OS X. On the same machine its very clear which vendors take more advantage of the underlying hardware. The results for the different plugin technologies aren’t too surprising since it’s regularly admitted that most companies spend their optimization time on Windows due to its larger install base. This argument doesn’t hold any water though when comparing html rendering on Safari/Mac against IE /Windows where there’s roughly a 1.6 : 1 advantage to the IE team. I can’t help but wonder if the core apis on the Mac platform are creating any unnecessary roadblocks. I’m also extremely surprised at the rendering speed that Flash is able to pull off on Windows. I developed this benchmark under OS X and after compiling the results I’m considered making the testcase more intensive since Flash is running so fast, but for now maybe the really poor Mac performance will give Adobe something to work on.

You can read more about rendering engine theory, the structure of the test case itself, and detailed analysis of the results on the sub pages within the site.

Updates

John Dowdell from Adobe brought up a valid point that plugin vendors are restricted by the browser environments they run in. This is true to an extent, but the limitations enforced on plugins don’t come into play with the GUIMark test. Browsers typically restrict the number of event loops available to a plugin which caps the framerate to around 40 – 50 fps. GUIMark doesn’t come anywhere close to hitting that limit on Mac. There are also no restrictions to the amount of cpu available to a plugin running within the browser which is why all of them peg the cpu to 100%. To illustrate the point, I created an AIR implementation of GUIMark and ran it on my Powerbook and here are the results I got. The Flash players rendering engine performs no differently outside the browser then it does inside the browser. Until plugins start bumping up against the event loop or Beam Sync restrictions, Adobe, Sun, and Microsoft don’t get a free pass for slow performance on Mac.

Cocoa, Flash

From AS3 to Objective-C: Flex vs iPhone development

14 Comments

Recently I’ve been given the opportunity to work full time on commercial iPhone development at EffectiveUI. The most intriguing thing about the platform for me is having access to non traditional user input mechanisms. When I was playing around with Wii remote integration on the desktop, the potential was exciting, but the ubiquity was limiting. In the same way that pc game companies develop for the keyboard and mouse first and then provide hooks for joysticks after the fact, I knew that serious Wii remote integration in a desktop app was limited. Knowing that I can write software for the iPhone that always has access to multitouch and accelerometer data from the outset really allows for unique gestures as a first class citizen within an app.

After working with some code and spending time with the SDK itself, I couldn’t help but naturally compare UI development on the iPhone with GUI frameworks like Flex or Swing, heres the things that stand out so far.

  1. I’m really spoiled by higher level languages. A good high level language like ECMAScript, Ruby, or Java rides the fine line of “Making things as simple as possible, but no simpler”. I’ve never felt constrained by the language features in these technologies, only by the apis exposed. Stepping *back* into Objective-C certainly provides more power and flexibility in the language, but there’s a loss in productivity for me that I just can’t shake. Some of this loss comes from Objective-C’s design itself, and some of it just comes from XCodes introspection ability. For instance, I’m not sure if I’ll ever get to the point where I can read these lines of Objective-C as effeciently as their ActionScript counterparts would probably look. 
    NSString *aString = [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleName"];
UIView *contentView = [[UIView alloc] initWithFrame:[[UIScreen mainScreen] applicationFrame]];
  2. Closed source UI frameworks suck. Most of what I learned about custom UI development in Flex I learned by inspecting the source code for the bundled controls. Ripping open Containers to see how layout rules are determined, or Lists to see how delegates are passed around, or the Image control to see how different display types are handled provides invaluable gems about implementing Flash apis. It has also helped me optimize the interactions of an app knowing the intentions of the developer who created the UI controls. With the iPhone SDK, you’re given documentation for the visual components, but no source to help determine how they work.
  3. Core graphics and animation is really strong. Between Quartz and the OpenGL layer there’s alot of potential for getting easy access to some of the more complex visual hacks. Although I think 3D user interfaces are prone to usability issues, the iPhone is a much better device to explore them on then a standard keyboard and mouse interface.
  4. Data binding, event listeners, and mxml. The Flash Player and Flex model provide features on top of the ActionScript language that arguably optimize UI needs and keep development more declarative. Cocoa development could really benefit from a ‘gui compiler’ that takes Objective-C to a higher level and bakes in features that support common ui design patterns.
  5. Garbage Collection. Objective-C 2.0 provides a unique system that lets you create objects that will be automatically garbage collected, or you can continue to manually manage object allocation/deallocation yourself. The concept sounds cool, but I can imagine allowing both systems to be mixed within the same project is just begging for trouble.

So far I think that Objective-C has alot of power and some really awesome features that outclass GUI features in Flash, but compared to Flex development as a whole, I’d have to say that XCode and the included visual frameworks are not as sophisticated.