1
\$\begingroup\$

I have just finished my first version of a function that returns an array of video screenshots created with canvas. What I am looking for are ways to improve this speed by as much as possible. I want to almost be able to instantly display the results. Yes, this should be a backend thing, no I can't do it in the backend. It needs to be done in the frontend.

One of my biggest performance issues at the moment, or what I at least think it is, is the fact that everytime a new image is pushed inside the array I call the callback function. This is NOT the desired functionality, I wish to return the contents of the result only when the images array is the same length as the given amount parameter. However, this is outside of my ability because I simply do not know how this could be accomplished. If anyone knows a way to fix this, please elaborate.

One more issue I am having is the fact that my renderImage function keeps old images from previous videos. I have no idea why or how I can fix this. Help is appreciated but my biggest priority is to increase the render and display speed of the fake generated gif's (src loops in this case).

// Max Screenshots (Firefox) = 19
// Max Screenshots (Chrome) = Infinite

const generateVideoThumbnail = (
  event,
  callback,
  options = {
    amount: 15,
    step: 2
  },
) => {
  let {
    amount,
    step
  } = options;
  if (amount > 19) {
    amount = 19;
  }
  if (step > 4) {
    step = 4;
  }

  const fileReader = new FileReader();
  const response = {
    status: '',
    images: [],
    firstImage: null
  };
  const videoBlob = event.target.files[0];

  if (videoBlob.type.match('video')) {
    fileReader.onload = () => {
      const blob = new Blob([fileReader.result], {
        type: videoBlob.type
      });
      const url = URL.createObjectURL(blob);
      const video = document.createElement('video');

      const snapImage = () => {
        const canvas = document.createElement('canvas');
        canvas.width = video.videoWidth;
        canvas.height = video.videoHeight;
        canvas
          .getContext('2d')
          .drawImage(video, 0, 0, canvas.width, canvas.height);

        const image = canvas.toDataURL();
        const success = image.length > 100000;

        if (success) {
          response.status = 'Video successfully parsed.';
          response.images.push(image);

          if (!response.firstImage) {
            response.firstImage = image;
          }

          video.currentTime += step;
          URL.revokeObjectURL(url);
          callback(response);
        }

        return success;
      };

      const timeupdate = () => {
        snapImage();
        if (response.images.length < amount) {
          snapImage();
        } else {
          console.log(response);
          video.removeEventListener('timeupdate', timeupdate);
          video.pause();
        }
      };

      video.addEventListener('loadeddata', () => {
        snapImage();
      });

      video.addEventListener('timeupdate', timeupdate);
      video.preload = 'metadata';
      video.src = url;
      video.muted = true;
      video.playsInline = true;
      video.play();
    };
  } else {
    response.status = 'File is not a video.';
    response.images = null;
    return callback(response);
  }

  fileReader.readAsArrayBuffer(videoBlob);
};

const renderImage = (res, amount) => {
  const gif = document.getElementById('gif');

  let imageIndex = 0;
  let timer = undefined;

  const gifAnimation = () => {
    gif.src = res.images[imageIndex];
    imageIndex++;

    if (imageIndex === res.images.length - 1) {
      imageIndex = 0;
    }

    timer = setTimeout(gifAnimation, 225);
  };

  const mouseover = () => {
    gifAnimation();
  };

  const mouseleave = () => {
    clearTimeout(timer);
    gif.src = res.firstImage;
    imageIndex = 0;
  };

  gif.removeEventListener('mouseover', mouseover);
  gif.removeEventListener('mouseleave', mouseleave);

  if (res.images.length === amount) {
    gif.src = res.firstImage;

    gif.addEventListener('mouseover', mouseover);
    gif.addEventListener('mouseleave', mouseleave);
  }
};

document.getElementById('inputFile').addEventListener('change', event =>
  generateVideoThumbnail(event, res => renderImage(res, 15), {
    amount: 15,
    step: 2,
  }),
);
<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <meta http-equiv="X-UA-Compatible" content="ie=edge" />
  <title>Demo 3</title>

  <style>
    body {
      margin: 0;
      padding: 0;
      text-align: center;
    }
    
    img {
      display: block;
      height: 33vh;
      object-fit: cover;
      transition: all .2s ease-in-out;
      width: auto;
    }
    
    #gif {
      margin: 5% auto;
    }
  </style>
</head>

<body>
  <img id="gif" />
  <input type="file" id="inputFile" accept=".mov, .mp4" />
  <script src="./index.js"></script>
</body>

</html>

\$\endgroup\$
  • \$\begingroup\$ There are syntax errors at stacksnippets \$\endgroup\$ – guest271314 Nov 19 '18 at 6:51
  • \$\begingroup\$ What is the purpose of video.currentTime += step? \$\endgroup\$ – guest271314 Nov 19 '18 at 7:02
  • \$\begingroup\$ It is used to skip x frames. So you get bigger differences. \$\endgroup\$ – Vera Perrone Nov 20 '18 at 13:09
  • \$\begingroup\$ Does that fire timeupdate event, creating a race condition? Have you tried using requestAnimationFrame? \$\endgroup\$ – guest271314 Nov 20 '18 at 17:24
2
\$\begingroup\$

Wrong approch

Data URL's are for transport only.

A data URL converts a binary stream to a format that can be transported over networks that handle text files as UTF-8. It does this by EXPANDING the data to fit a safe range of characters (64 of them) storing 6 bits in every byte (UTF-8). That means for every 3 bytes in the source the data URL needs 4 bytes to store.

This is compounded in javascript by the fact that JavaScript strings are 16bit UCS-2 characters. That means that for every 3 bytes in the source the JavaScript dataURL needs 8 bytes to store.

Video

Videos are highly compressed streams, not only is each frame compressed, but that compression is related to previous frames. This allows very high compression rates. For example I picked a random low res video 420 by 270 at 20Fps, running time of 1h 25m, and a file size of 380MB.

A random frame saved as png was 238kb in size.

The data throughput to display is about 10mb a second (and this is low res LOW quality video, full frame HD is about 0.5Gig a second)

That means that running your function on that video the memory needed to complete the function is about 380 * 1024 * 1024 + (1 * 60 * 60 * 20fps + 25 * 60 * 20fps) * 200kb * 1024 * ((8 / 3)dataUrl Storage overhead) which is around 56 gigabytes of RAM (Something only the very top range devices can handle).

If you compare the Video size to the image dataURLs used to store the frames you have expanded the video memory needs to almost 400 times the original size.

Even if you consider better options, store frames as Jpeg Blobs (near one to one of bin size) set at very low compression (eg jpeg quality 0.4) each frame requires ~12kb. The final RAM need to store the frames is (1 * 60 * 60 * 20fps + 25 * 60 * 20fps) * 12kb * 1024 = 1.2Gig (do-able on PC's and laptops if you don't count the IMG element overhead)

Though it will fit memory the processing will no less require a lot of time to render, compress, and store.

The Sad truth

Until browsers come with a way to use a disk based cache for large data needs, video editing like applications will not be able to provide fast random access frame editing like traditional native video editing apps.

What to do?

I did notice that there was a image element id named gif which hints that the application may be a video to gif like thing.

Again this will be slow, but it is do-able if you compress the video frames to gif as you go. Get video frame, quantize color, compress to GIF, delete frame. Use webWorkers to pipeline the quantize, and compress steps. Ensure that unsaved gif data size does not crash the page.

You should also consider options for resolution and frame rate reduction. Also add warnings for long (large) videos. It will be slow, there is nothing you can do, that is the nature of videos,

\$\endgroup\$
  • \$\begingroup\$ Hey, first of all thank you a lot for the very detailed explanation. However, I do think I need to elaborate a bit more. Users can upload a maximum of 10mb, and most videos won’t be longer than 15 seconds. (Only Instagram Videos and Stories). With means that memory will never be a big issue I hope. What I want to do is get an an array with a maximum of x pictures with x seconds between them. This is actually going to be a uploaded video preview. Where users can hover there preview and I will loop trough the array to fake a gif animation. This is only to give users feedback. \$\endgroup\$ – Vera Perrone Nov 16 '18 at 16:47
  • \$\begingroup\$ The same effect you get when you hover a YouTube video, but I need to do this client side. The server will handle everything else and once the upload progress is done, this will be deleted from memory. It is just a feedback mechanism that shows users what they just uploaded. \$\endgroup\$ – Vera Perrone Nov 16 '18 at 16:49
  • \$\begingroup\$ @VeraPerrone videos can play way before they have fully loaded, thus getting frame preview client side will always take longer than to start watching the video. If you have many videos and want to show preview frames for each you would need to load each video completely, something the client will not appreciate. YouTube previews are stored as a single low quality jpeg image containing frames. It is the only way to preview client side. Loading the video to capture frames for a preview, you may as well just play the video directly as a preview and save the bother and memory. \$\endgroup\$ – Blindman67 Nov 16 '18 at 17:16
0
\$\begingroup\$

requestAnimationFrame could be substituted for setTimeout and timeupdate event (which is not executed at a consistent rate) where necessary in the existing code. The same <canvas> element can be used and captured within the snap function.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.