Video Manipulation using HTML 5 and Javascript

Some funny stuff can be done using HTML 5, canvas elements and the video events API.

In the following example we’re using these techniques to apply graphic effects to a video embedded in a HTML page..

The Goal

Please take a look at the following screencast on YouTube to get an idea of what the final example looks like or just take a look at the demo page.

As you can see a video is rendered and a clone is displayed with a strange swirl effect applied.

The Code

This is the example code – I have ommitted some CSS styles and stuff ..

First we’re loading a video using the HTML 5 Video API. I was too lazy here to deliver multiply sources for the video element to support other browsers than Chrome but in a production scenario we definitely should do that!

The second important element is the canvas. We’re using the canvas to draw and refresh a copy of the video’s image data.

To manipulate this image data I am using Joel Besada’s excellent JSManipulate library here that offers a lot of nice effects.

So what is happening when the video playback is started?

The event handler bound to the video’s play event triggers a function
The video’s data is drawn onto the canvas’ 2d context
We’re fetching this data and we’re using JSManipulate to apply some effects
Afterwards we’re repainting the canvas with the new data
The setTimeout is responsible for updates every 20 ms

<!DOCTYPE html>
<html>
<head>
 <title>hasCode.com HTML Canvas Video Manipulation Example</title>
 <script type="text/javascript" src="jsmanipulate.js"></script>
</head>
<body>
 <video controls id="video">
 <source src="video.mp4" type="video/mp4" />
 </video>
 <canvas id="canvas"></canvas>

 <script type="text/javascript">
 function setup(){
 var video = document.getElementById("video");
 var canvas = document.getElementById("canvas");

 var canvas2DContext = canvas.getContext("2d");

 video.addEventListener("play", function(){
 canvas.height = video.clientHeight;
 canvas.width = video.clientWidth;
 paintCanvas(video, canvas, canvas2DContext);
 }, false);
 }

 function paintCanvas(video, canvas, canvas2DContext){
 if(video.paused || video.ended) return false;

 canvas2DContext.drawImage(video,0,0);
 var data = canvas2DContext.getImageData(0,0,canvas.width, canvas.height);
 JSManipulate.triangleripple.filter(data, {});
 canvas2DContext.putImageData(data,0,0);

 setTimeout(function(){
 paintCanvas(video,canvas,canvas2DContext);
 }, 20);
 }

 setup();
 </script>
</body>
</html>

Running the Example

There are restrictions when opening the index.html from local so if you’ve got Python installed you’re able to start a webserver quick:

cd /path/to/project
python -m SimpleHTTPServer

Afterwards you’re able to watch the demo at http://localhost:8000/

Of course you’re free to use any of the well known web servers here.

Writing your own Effect Filter

It is indeed very easy to create your custom graphic filter – getImageData return an object containing an internal data object with an array of integers that contains the pixel’s data.

The format of this array is as this .. we have four bytes for each pixel .. the first one is the red-value, the second is the green-value, the third is the blue-value and the fourth byte is for the alpha-channel..

Figure 1. The RGB Image Data Format

Now it is easy to write a custom filter – let’s say we just want to invert the video .. how is it done? Just substract the integer RGB value from 255 except for the alpha channel and use the new array to draw the canvas image.

function paintCanvas(video, canvas, canvas2DContext){
  if(video.paused || video.ended) return false;

  canvas2DContext.drawImage(video,0,0);
  var data = canvas2DContext.getImageData(0,0,canvas.width, canvas.height);
  var converted = data;
  for(var i=1;i<=data.data.length;i++){
    if(i % 4 == 0){ // alpha channel
      converted.data[i-1] = 255;
      continue;
    }
    converted.data[i-1] = 255-data.data[i-1]; // invert
  }

  canvas2DContext.putImageData(converted,0,0);

  setTimeout(function(){
    paintCanvas(video,canvas,canvas2DContext);
  }, 20);
}

Thats what the example finally looks like as screencast on YouTube or just take a look at the demo page.

Tutorial Sources

Please feel free to to view and download the complete sources from this tutorial from my GitHub repository – or if you’ve got Mercurial installed just check it out with

git clone https://github.com/hascode/html5-js-video-manipulation.git

Resources

Article Updates

2018-06-01: Embedded YouTube video removed (GDPR/DSGVO).

The Goal#

The Code#

Running the Example#

Writing your own Effect Filter#

Tutorial Sources#

Resources#

Article Updates#