7
\$\begingroup\$

I've been working on an html5 canvas video player with a lot of fun little extras and ui toys, the main one being a chroma key (green screen) effect that allows the user to key out different colors or ranges of color while the video is playing.

I seem to have hit a snag in the recommended chroma key method. It works, but it is a really heavy process and can tax the cpu to the point where the video will look choppy if the user is running other applications in the background.

Here is the stripped down code I'm using for the chroma key effect:

   <video id="sourceVid" controls="true">
        <source src="http://s3.amazonaws.com/dfc_attachments/public/documents/3185373/AAA.webm" />
    </video>
    <canvas id="hCanvas"></canvas>
    <canvas id="dCanvas"></canvas>

    <script type="text/javascript">
        var doc = document;
        var sourceVid = doc.getElementById("sourceVid");
        var hCanvas = doc.getElementById("hCanvas");
        var dCanvas = doc.getElementById("dCanvas");
        var hContext = hCanvas.getContext("2d");
        var dContext = dCanvas.getContext("2d");

                    sourceVid.addEventListener('loadeddata', function() {
                        hCanvas.setAttribute('width', sourceVid.offsetWidth);
                        dCanvas.setAttribute('width', sourceVid.offsetWidth);
                        hCanvas.setAttribute('height', sourceVid.offsetHeight);
                        dCanvas.setAttribute('height', sourceVid.offsetHeight);
                    }, false);

        sourceVid.addEventListener('play', function() {
            runAnalysis();
        });

        var runAnalysis = function() {
            if (sourceVid.paused || sourceVid.ended) {
                return
            }
            frameFix();
            if (window.requestAnimationFrame) {
                requestAnimationFrame(runAnalysis);
            } else {
                setTimeout(runAnalysis, 0);
            }
        };

        var frameFix = function() {

            hContext.drawImage(sourceVid, 0, 0, sourceVid.videoWidth, sourceVid.videoHeight);

            var frame = hContext.getImageData(0, 0, sourceVid.videoWidth, sourceVid.videoHeight);

            var length = frame.data.length;
            for (var i = 0; i < length; i++) {
                var r = frame.data[i * 4 + 0];
                var g = frame.data[i * 4 + 1];
                var b = frame.data[i * 4 + 2];

                if (g >= 0 && g < 100 && r >= 0 && r < 100 && b >= 0 && b < 100) {
                    frame.data[i * 4 + 3] = 0;
                }
            }
            dContext.putImageData(frame, 0, 0);
            return
        };

Live Demo

Hoping to find ways to further optimize this for performance.

\$\endgroup\$
5
+50
\$\begingroup\$

After back and forth with ChrisW and yourself, it seems I found a glaring bug

    var frameFix = function() {

        hContext.drawImage(sourceVid, 0, 0, sourceVid.videoWidth, sourceVid.videoHeight);

        var frame = hContext.getImageData(0, 0, sourceVid.videoWidth, sourceVid.videoHeight);

        var length = frame.data.length;
        for (var i = 0; i < length; i++) {
            var r = frame.data[i * 4 + 0];
            var g = frame.data[i * 4 + 1];
            var b = frame.data[i * 4 + 2];

            if (g >= 0 && g < 100 && r >= 0 && r < 100 && b >= 0 && b < 100) {
                frame.data[i * 4 + 3] = 0;
            }
        }
        dContext.putImageData(frame, 0, 0);
        return
    };

Since in frame.data you have 4 data points per pixel, you only need to go as far as frame.data.length / 4, this should make your code 4 times faster.

This observation with ChrisW's common sense observations makes for:

var frameFix = function() 
{
  hContext.drawImage(sourceVid, 0, 0, sourceVid.videoWidth, sourceVid.videoHeight);

  var frame  = hContext.getImageData(0, 0, sourceVid.videoWidth, sourceVid.videoHeight),
      data   = frame.data,
      length = data.length, i;

  for (i = 0; i < length; i += 4) 
    if (data[i] < 100 && data[i + 1] < 100 && data[i + 2] < 100) 
      data[i + 3] = 0;

  dContext.putImageData(frame, 0, 0);
};
\$\endgroup\$
  • \$\begingroup\$ Your last thought would be wrong; you need to code it as, for (i = 0; i < length; ) { if (data[i++] >= 100 { i += 3; continue } if (data[i++] >= 100 { i += 2; continue } if (data[i++] >= 100 { i += 1; continue } data[i++] = 0; } \$\endgroup\$ – ChrisW Jan 20 '14 at 17:42
  • \$\begingroup\$ Updated and removed last thought. \$\endgroup\$ – konijn Jan 20 '14 at 18:52
4
\$\begingroup\$

Most of your code is API calls. I'm surprised that Javascript is fast enough to run at all (it isn't, on my old laptop, using Chrome browser).

I'll assume that using completely different APIs, and for example writing a plug-in in a different, compiled language (if that's even possible) is out-of-scope for this question.

Therefore I guess that the only performance optimization that's possible/interesting/relevant is of the processing which is implemented in your code, i.e. of the following code fragment:

for (var i = 0; i < length; i++) {
    var r = frame.data[i * 4 + 0];
    var g = frame.data[i * 4 + 1];
    var b = frame.data[i * 4 + 2];

    if (g >= 0 && g < 100 && r >= 0 && r < 100 && b >= 0 && b < 100) {
        frame.data[i * 4 + 3] = 0;
    }
}

Perhaps you can do without the multiply-by-4:

length *= 4;
for (var i = 0; i < length; i += 4) {
    var r = frame.data[i];
    var g = frame.data[i + 1];
    var b = frame.data[i + 2];

    if (g >= 0 && g < 100 && r >= 0 && r < 100 && b >= 0 && b < 100) {
        frame.data[i + 3] = 0;
    }
}

The ImageData contains unsigned numbers, which cannot be negative, so test-for-greater-than-or-equal-to-zero is unnecessary:

length *= 4;
for (var i = 0; i < length; i += 4) {
    var r = frame.data[i];
    var g = frame.data[i + 1];
    var b = frame.data[i + 2];

    if (g < 100 && r < 100 && b < 100) {
        frame.data[i + 3] = 0;
    }
}

If you do without the temporary variables, then you don't need to deference the color value from the array if an earlier color value fails the test, for example:

length *= 4;
for (var i = 0; i < length; i += 4) {
    if (frame.data[i] < 100 && frame.data[i + 1] < 100 && frame.data[i + 2] < 100) {
        frame.data[i + 3] = 0;
    }
}

It may be faster to not dereference from frame each time:

length *= 4;
var data = frame.data;
for (var i = 0; i < length; i += 4) {
    if (data[i] < 100 && data[i + 1] < 100 && data[i + 2] < 100) {
        data[i + 3] = 0;
    }
}

It's possible that these changes will have no effect on performance: depending on your browser, perhaps they're optimizations which its Javascript compiler is already making.

\$\endgroup\$
  • \$\begingroup\$ Thanks for the response but your suggestion seems to break the green screen effect. The video is still rendered to the canvas, but the desired colors aren't being keyed out. Also I'm comparing with 0 so that colors that fall between rgba(0,0,0,1) and rgba(100,100,100,1) will be keyed out. \$\endgroup\$ – apaul Jan 20 '14 at 2:56
  • \$\begingroup\$ This what I'm seeing trying your method: s3.amazonaws.com/dfc_attachments/public/documents/3187063/… \$\endgroup\$ – apaul Jan 20 '14 at 2:59
  • \$\begingroup\$ Sorry, you probably realized why I'm comparing with zero... In the full context both 0 and 100 are variables being controlled by sliders, so I kind of need to have both values. \$\endgroup\$ – apaul Jan 20 '14 at 3:28
  • \$\begingroup\$ Sorry, I don't see anything wrong with my changes, but I'm not a JavaScript expert so I don't know. I also can't run that code (as it said, it doesn't work on my machine). If it changes functionality, try applying the changes one-by-one, starting from the top (as I wrote them) to see which step of the change is wrong. If you need to keep the test against a lower bond, you can of course keep it: if (data[i] >= low && data[i] < high && data[i + 1] >= low && data[i + 1] < high && data[i + 2] >= low && data[i + 2] < high). \$\endgroup\$ – ChrisW Jan 20 '14 at 9:59
  • 1
    \$\begingroup\$ @apaul34208 @ChrisW , length *= 4; is meaningless, just keep var length = frame.data.length; instead. \$\endgroup\$ – konijn Jan 20 '14 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.