0
\$\begingroup\$

I have an AngularJS code to perform upload of files. At first I thought it works OK but when I've tried to upload bigger files (e.g. 5 files, 10 MB each) then I saw that this code has very poor performance regarding memory handling.

For example: when I attach 5 files (10 MB each = 50 MB in total) then peak browser process memory demand reaches 2 GB! I've tested it in Firefox, Chrome and IE. Additionally, in Chrome when browser process reaches 3 GB of memory committed then the page crashes. In Firefox I can see an error in console when trying to upload more than 80 MB at once: out of memory

I've tried to modify the parts where the actual request is made and removed data: JSON.stringify(formattedData) which was the first bottleneck. And it turned out that stringifying is redundant there.

But the second bottleneck still exists which is the function binArrayToJson which takes the ArrayBuffer and reads the data into the byte array.

var binArrayToJson = function (binArray) {
   var str = [binArray.length];
   for (var i = 0, binLength = binArray.length; i < binLength; i++) {
   str[i] = binArray[i];
   }
   return str;
}

var applyAttachments = function (data1) {
   angular.forEach(data1,
       function (value, key) {
           if (key === "Attachments") {
               for (var i = 0; i < value.length; i++) {
                   if (value[i].file) { //already saved item don't contain 'file' property
                       var reader = new FileReader();

                       reader.onloadend = (function (f) {
                           return function (result) {

                               var arrayBuffer = result.target.result;
                               value[f].fileContent = binArrayToJson(new Uint8Array(arrayBuffer));
                               value[f].isUploaded = true;
                           };
                       })(i);

                       reader.readAsArrayBuffer(value[i].file);
                   }
               }
           }
       });
};


var createMVCModel = function (output) {
   var defaultStringValue = "";
   return {
       id: output.id,
       Name: output.name || defaultStringValue,
       Status: output.status || "Draft",
       Date: output.date || null
       Attachments: output.attachments || []
   };
};

var saveModel = function (data, url) {
   var formattedData = createMVCModel(data);
   var deferred = $q.defer();

   applyAttachments(formattedData);

   var check = function () {
       if (allNewFilesUploaded(formattedData) === true) {
           $http({
                   url: url,
                   method: "POST",
                   data: formattedData,
                   headers: { 'Content-Type': undefined }
               })
               .then(function (result) {
                       deferred.resolve(result);
                   },
                   function (result) {
                       deferred.reject(result);
                   });
       } else {
           setTimeout(check, 1000);
       }

   }
   check();
   return deferred.promise;
};

I omitted the parts where the following requirements are being met:

  • checking number of files (limit is 10)
  • checking size of each file (limit is 10 MB each)
  • checking permitted file extensions
  • sending rest of the (text) data along with files to the server (ASP.NET MVC method)
\$\endgroup\$
  • \$\begingroup\$ Sounds like a question for StackOverflow because the code doesn't work as intended, which makes it off-topic here. \$\endgroup\$ – wOxxOm Aug 27 '17 at 16:12
  • \$\begingroup\$ @wOxxOm If the code works for small files, but fails to scale, then it could be acceptable here on Code Review. \$\endgroup\$ – 200_success Aug 27 '17 at 22:47
1
\$\begingroup\$

Marshaling the file contents as a byte array is obviously problematic.

Browsers can easily upload files through HTML forms, with

<form method="POST" enctype="multipart/form-data">
  <input type="file">
</form>

Adapt the server to handle file uploads using that mechanism, then develop a replacement AJAX client.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.