Html, Javascript

Blowing up HTML5 video and mapping it into 3D space

305 Comments

I’ve been doing a bit of experimenting with the Canvas and Video tags in HTML5 lately, and found some cool features hiding in plain sight. One of those is the Canvas.drawImage() api call. Here is the description on the draft site.

3.10 Images

To draw images onto the canvas, the drawImage method can be used.
This method can be invoked with three different sets of arguments:

    
	
  •  drawImage(image, dx, dy)
    • 
	
  •  drawImage(image, dx, dy, dw, dh)
    • 
	
  •  drawImage(image, sx, sy, sw, sh, dx, dy, dw, dh)
    • 


Each of those three can take either an HTMLImageElement, an HTMLCanvasElement, or an HTMLVideoElement for the image argument.

The api lets you take the contents of specific HTML elements and draw them into a canvas, and the 3rd element in that list is just begging to be abused. Copying video into a canvas element means opening up the ability to manipulate or process video frames at runtime. Two concepts instantly came to mind that seemed like fun to try and figure out, here they are below.

Blowing apart fragments of video

Click around the video frame to blow up that part of the video, the exploded pieces will continue to play the video inside them. After a while they retract back to their original place. One feature I didn’t have time to figure out was adding depth to the explosion, so pieces that are closest to ground zero fly up into the air as they sail outward. With full shadow effects this could look really cool.

3D Video

This demo in particular runs really well inside webkit based browsers, but not so much in Firefox. Firefox doesn’t appear to have any hardware acceleration for Ogg decoding so I had to drop the video size in half in order to run at acceptable framerates. Even still, Firefox chokes pretty badly on my Macbook Pro.

*Update* – I’ve changed the ogg video to be 640 x 360, prepare to see firefox weep

Lessons learned

There’s a couple hints I found out along the way that are good to know if you want to play around with drawing video. First, you need a bit of hackish code to get this to work effeciently and it flows like this.

[Video playing] -> [Draw Video onto Canvas 1] -> [Draw fragments of Canvas 1 onto Canvas 2]

Don’t ask me why, but copying pixel data out of a video tag is expensive, so expensive that drawing it into a temporary canvas, and then drawing pieces of that temp canvas onto a final canvas is faster then just referencing the video tag repeatedly within the same loop. That’s why you’ll see 2 Canvases in the source code for the demos. I’m sure there’s a technical reason for this duplication process, but it’s a lazy reason.

Secondly, don’t try copying individual pixels around. You can still see the remnants of my first code attempt inside the explosion demo with getPixel() and setPixel(). This turned out to be horribly slow and completely unnecessary. Canvas.drawImage() + matrix transforms on the canvas space is far more efficient then handcrafted pixel pushing. On the other hand, pixel manipulation allows you to do things like runtime chroma keying, get ready for a new wave of “clippy” style videos with full transparency popping over websites to help you out.

Lastly, I’m learning very quickly that not all browsers are created equal when it comes to performance, it’s a crapshoot when it comes to heavy video+image manipulation. Safari and Chrome work well with h.264, Firefox slogs along with Ogg Theora, and Opera is somewhere in the middle.

305 thoughts on “Blowing up HTML5 video and mapping it into 3D space”

  7. Runs well in Chrome Beta on OS X, at least you have alternatives to Safari that run it well…

  11. Yeah I’ve learned throughout the day that Firefox on Windows isn’t too problematic, it’s just Firefox on Mac that is dog slow in the 3D demo.

  13. Incredibly cool! Very well done indeed!

    What was that Adobe said about HTML5 video not being as advanced as Flash video? You just showed us HOW advanced HTML5 is!

  14. Works beautifully in Safari 4.0.5 on OS X 10.5.8 (even on this 1st Gen MacBook Air, notorious for having terrible video performance). I will note though that Safari froze solid for a good twenty seconds while loading each of the pages.

  29. Transcoding the original MP4 to same-resolution Theora the performance is about the same for both in Safari (via XiphQT) and Chrome. Opera plays the full-resolution Theora just fine (earliest core2 mac mini), but doesn’t seem to want anything to do with the H.264 version. Firefox may or may not have problems (I don’t have it installed) but there’s no reason to assume they relate to video decode hardware acceleration.

    Could you provide the same resolution in both formats? Right now you’re penalising Opera artificially.

  32. Chrome 4.1, FF 3.6.3 , Opera 10.50 both examples worked fine, the size of video on Chrome seemed larger and the “blow up” affected a smaller amount of video though, in FF and Opera it blew virtually all of it up with one click.

    Win2003 Server SP2

    Nice job.

  33. Stunning stuff. Sweet in Chrome on Vista. Both fail in IE9 Preview (“Object no found” error at the line in Init() that says “copy = copycanvas.getContext(‘2d’);”). I’ve logged an issue with Microsoft.

  34. This technique opens up a nice venue of video sprites where you can use a single video file to store a whole bunch of animations in various sizes for in games.
    Does the HTML5 video API allow you to set a movie to a specific frame (or time?) Could be a very efficient way of storing animation data that way.

  35. Both examples worked surprisingly well on my “old” Dual G5 PowerMac using Safari. As others had stated though, no go on my iPad (well, the second example does show a rotating shadow).

  38. Works excellently in Opera 10.5. I can’t tell any noticeable difference between how fast Opera handles either compared to Safari on my computer. Opera certainly blows the pieces apart faster, but that could be just because there are less of them. Unsure why the video is smaller in Opera and Firefox, though.

  39. Failed in Windows XP Pro SP3 + Firefox 3.6.3 (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB6 (.NET CLR 3.5.30729))

    Are there certain plug-ins required?

  44. Awesome!

    Ok – so I tried to replicate on my server so I could start playing with your math and start ripping it apart to try and understand it – but I cannot get the .ogv to play. I dl’ed it from you, and uploaded… do I need to re-encode?

  47. michael I’m not sure why it’s wouldn’t work for you, you shouldn’t have to re-encode to play it elsewhere

  48. Excellent example of html5 manipulation – don’t worry to much about speed at the moment, when FF uses direct2D to draw to screen you will see a vast improvement. re: iPad & iPhone – they seemed to have ‘crippled’ html audio and video in safari for these – it only works if the src is streamed. I guess they don’t want anyone making there own iTunes in html :(

  49. I tried this on both Firefox 3.6 and the latest install of chrome. I found that Chrome handled the spinning and shattering of the video better but Firefox did the actually playing of the video much more smoothly. Also the video looping failed on Chrome, so it would play through once, then instead of restarting it would only stop on the first frame. I was surprised, because even without reading that the author expected poor performance on Firefox, I thought Chrome would be better for html5, it seems that this is not universally true. I’m running winxp SP3 with a 3.2 Ghz P4, and 2GB of ram.

    CORRECTION: After revisiting the page in Chrome, I found it was only choppy because ,unlike Firefox, it did not wait when the network fell behind, it would stall then skip whenever the network caught up. It still does not loop properly.

  50. Awesome…I know this is just a demo, but my mind is already racing with possibilities. Great post.

  52. Is this optimized at all or just a proof-of-concept? Because it eats ~70% of one core and ~20% total CPU on my 2.4 GHz Machine.

  54. The slow performance copying out of the video tag is likely because the video is in the YUV color space, copying it to a Canvas will require converting it to the RGB color space (Every pixel needs to be converted individually which is, of course, murder on the CPU. It can be sped up but will require SSE/Neon/Altivec/etc optimisations but only if the blocks being copied are big enough; I’m betting most browsers don’t have those anyway). Rendering directly to a canvas means you get a bulk-transfer-bonus by doing the conversion all at once rather than piecemeal, same optimisation requirement applies though.

    getPixel/setPixel are generally a bad idea for the same reason that accessing random cells out of order in an array is a bad idea, modern CPUs are designed to operate in bulk modes — do large chunks of repetitious work on a slab of memory. Peeling individual pixels apart rather than grinding the lot as a whole will be massively slower. (There may also be GPU bus overhead with pixels flying back and forth between the video card memory and the CPU which is extra-super-slow)

    BTW. I just had a horrible idea of an SVG+JavaScript with embedded HTML5 video tags as some sort of movie palette that zooms in and out on each individual clip when clicked (with grey out and enabling sound on the selected only). I suspect you could probably lock the entire browser UI with something like that.

  58. Very cool!

    Works better for me with Opera 10.52 RC for Mac than with Chrome 5 and Safari 4. Perhaps worthwhile removing the “best viewed in Webkit browsers”?

  61. Andrew, that makes a lot of sense about the YUV to RGB conversion. I was naively assuming that a bitmap is a bitmap is a bitmap, and whether you grab that from a video raster or a canvas raster shouldn’t matter. Thanks for the explanation.

  62. In addition to what andrew said @54, I want to point out that while copying data from main memory *into* GPU memory is slow, copying *back* from GPU to main memory is even slower. If you do bulk operations on canvas data, the browser can (at least in theory) stream compressed video into the GPU memory and do all the manipulation there, which is relatively fast, but individual pixel manipulation has to come back to main memory, which crawls.

    I’m not sure how optimized this stuff is in the current generation of browsers, but making it go as fast as possible is on everyone’s roadmap.

  63. Nicely done, Sean.

    Having said that, both HTML5 video demos run beautifully in Opera v10.52 RC4 for Mac at full-resolution; so the “best viewed…” line is pedestrian. Give due credit to Opera developers. It’s a smokin’ desktop/iPhone browser that makes the agnostic Web better for users & easier for developers (especially with the full Cocoa rewrite on Mac builds now).

  64. Whoa, surely great work.

    Btw, it worked perfectly on a Lenovo IdeaPad s10, running Fedora 12, with latest Google Chrome… so I’d say this HAS to run fine on whatever other hardware ;)

  66. iPad does not have ogg codec, only h.264. Html5 video is a container not a specific codec. That.s what the battle is about.

  68. And you compare this to what you can do with flash? Maybe pick up a flash book. Forget all of the platform problems. (just keep glossing over them). This level of code is beyond the vast majority of web site devs and it doesn’t even work correctly. To say this is somehow a ‘solution’ for a web site build is beyond naive. Are you telling me you would get paid for this? Hilarious. If you’re a hobbyist, great. Have fun. If you are a prfessional, HTML5 is not ready for primetime.

  69. @69

    Are you kidding?
    “This level of code is beyond the vast majority of web site devs and it doesn’t even work correctly.”

    If you are developing websites and you can’t work out this code, you my friend are shortchanging your customers and giving the industry a bad name.

    ” If you are a prfessional, HTML5 is not ready for primetime.”
    shows you are a windows, ie centric fool.
    There is more to computing than “Microsoft”

  70. emphatic, you are reading more into blog post then what I’ve stated. All I’ve done is show some code I’ve been playing around with, nothing more, nothing less.

  71. @thebear: sorry but html5 is really not ready… it’s not working same on all browsers. despite you dont need plugin (like flash), you need ‘right’ browser. what a win? me as flex/flash developer, i dont need to worry what browser you use… get it?
    what about ogg/h246 battle??? flash video playes everywhere same… no OS or codecs problems. :P

    and if you think this is stuff of windows centric fools, you’re wrong… i use Flex4 SDK under linux and solaris using only vim, i am happy with flash, outcome is same on every platform (and i dont care about browser problems) + it’s still easier do any coding in AS3 (few lines of code) then to do programming in Ajax (many many lines of code)…. + i got multitouch, GPU (pixel bender) effects and many other features not possible or too complicated with JavaScript…

  75. Hello, do you mind if I used the 3d video source code on my website? Please e-mail me if you have time. Thanks very much for the demos, I did not know this was possible!

  77. What the hell is the use of that? I don’t see why I’d ever need to blow up my videos or make them rotate like that.

  80. The effect is really awesome!! Congrats!!

    What about the control buttons? Play/Pause/Volume, etc??

    ¿Is it possible to do that over the video on HTML5?

  89. Ok step in the right direction, but how hard is this going to be for us UI designers/developers?

    For instance how many people are still having issues writing css that all browsers can read without having to hack it up a bit. IE, 6 -7 ring alarm bells! Hell talk about throw things back a few years. I know these browsers wont live for ever but there are still people still using these redundant old browsers, half the reason why flash was introduced in the first place – runs on all browses and platforms universally.

    The turn around time for open standards such as html 5 is too slow, and not everyone adopts them the same. At the current rate it’ll be another 10 years before its up to speed with flash, by that time we would have moved on.

    Things need to speed up before we can start using it properly. If you guys are that serious about it then make it work, create tools and flash equivalent applications that help designers use html5 in all its glory.

  92. While it appears to be the smoothest in Chrome (comparing Opera 10.53, Firefox 3.6.3, Chrome 4.1.249.1064 on a Phenom II 955 in Win XP) it also looks the ugliest on Chrome as Opera and Firefox apply anti-aliasing.

    And as Firefox is clearly slower than the others even in the apparently easier blow up page and Opera lags only slightly in the 3D-Version, I declare Opera my personal winner :)

  100. Being able to blow up video files is definitely why we all need to switch over. This is like the cure for cancer. We can all happily blow our video files from now on with our new html5 overlords. Thank goodness me!

Comments are closed.