Is there anything biased in this performance test?

Question

Just wondering if this basic performance test between an untyped array of arrays and a vector (typed array) of arrays in Actionscript 3... is fair and balanced? It's simple, designed to test random access on the collections as well as pushing new items, etc. Here goes:

import flash.events.Event;
import flash.utils.getTimer;

//Performance timer related
var startTime:Number; //ms
//

//Our two container types we're testing IO on
var arrayOfArrays:Array = new Array();
var vectorOfArrays:Vector.<Array> = new Vector.<Array>();
//

//Used to store a bunch of arrays we're going to use to test
var testArrays:Array = new Array();
//

var randomIndex:uint = 0;
var i:uint = 0;
var arr:Array;

//Generate a bunch of arrays of mixed typed content
for(i = 0; i < 100000; ++i) {
    generateTestArray();
}

/*======================================================================================================
***********************************      Array  Tests      *********************************************
*=====================================================================================================*/
//Test push on array of arrays
trace("Testing Array of Arrays push performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    arrayOfArrays.push(testArrays[i]);
}
trace("Total time for 100000 push calls on Array of Arrays: " + (getTimer() - startTime));
trace(" ");
//

//Test random write on array of arrays
trace("Testing Array of Arrays random assignment performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    randomIndex = Math.round(Math.random() * 99999) as uint;
    arrayOfArrays[randomIndex] = testArrays[randomIndex];
}
trace("Total time for 100000 random assignment calls on Array of Arrays: " + (getTimer() - startTime));
trace(" ");
//

//Test sequential read on array of arrays
trace("Testing Array of Arrays sequential read performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    arr = arrayOfArrays[i];
}
trace("Total time for 100000 sequential read calls on Array of Arrays: " + (getTimer() - startTime));
trace(" ");
//

//Test random read on array of arrays
trace("Testing Array of Arrays sequential read performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    randomIndex = Math.round(Math.random() * 99999) as uint;
    arr = arrayOfArrays[randomIndex];
}
trace("Total time for 100000 random read calls on Array of Arrays: " + (getTimer() - startTime));
trace(" ");
//
/*====================================================================================================*/


/*======================================================================================================
***********************************      Vector Tests      *********************************************
*=====================================================================================================*/
//Test push on vector of arrays
trace("Testing Vector of Arrays push performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    vectorOfArrays.push(testArrays[i]);
}
trace("Total time for 100000 push calls on Vector of Arrays: " + (getTimer() - startTime));
trace(" ");
//

//Test random write on vector of arrays
trace("Testing Vector of Arrays random assignment performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    randomIndex = Math.round(Math.random() * 99999) as uint;
    vectorOfArrays[randomIndex] = testArrays[randomIndex];
}
trace("Total time for 100000 random assignment calls on Vector of Arrays: " + (getTimer() - startTime));
trace(" ");
//

//Test sequential read on vector of arrays
trace("Testing Vector of Arrays sequential read performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    arr = vectorOfArrays[i];
}
trace("Total time for 100000 sequential read calls on Vector of Arrays: " + (getTimer() - startTime));
trace(" ");
//

//Test random read on vector of arrays
trace("Testing Vector of Arrays sequential read performance:");
startTime = getTimer();
for(i = 0; i < 100000; ++i) {
    randomIndex = Math.round(Math.random() * 99999) as uint;
    arr = vectorOfArrays[randomIndex];
}
trace("Total time for 100000 random read calls on Vector of Arrays: " + (getTimer() - startTime));
trace(" ");
//
/*====================================================================================================*/

function generateTestArray():void
{
    var newArray:Array = new Array();

    var totalItems:uint = Math.round(Math.random() * 50 + 1);

    var i:uint = 0;

    var dice:uint = 0;

    for(i; i < totalItems; ++i) {

        dice = Math.round(Math.random() * 5);

        switch(dice) {
            case 0:
                newArray.push(new int(Math.random()));
            break;

            case 1:
                newArray.push(new String(Math.random()));
            break;

            case 2:
                newArray.push(new Array());
            break;

            case 3:
                newArray.push(new MovieClip());
            break;

            case 4:
                newArray.push(new Date());
            break;

            case 5:
                newArray.push(new Event(Event.COMPLETE, false, false));
            break;  

        }
    }

    testArrays.push(newArray);
}

Answer 1

Some thoughts

You don't need to use new int (it's very uncommon in actionscript), just cast it to that:

newArray.push(int(Math.random()));

You don't need to use new String (it's very uncommon in actionscript), just use the toString() function

 newArray.push(Math.random().toString());

You could also directly cast to uint, to have it rounded.

 randomIndex = uint(Math.random() * 99999);

You could also benchmark fixed length array/vectors.

In the end results of the benchmark you should add a note that creating/assigning the random value takes some time too.

Answer 2

Math.random()

is a quite slow operator compared to Array access. In this case:

for(i = 0; i < 100000; ++i) {
    randomIndex = Math.round(Math.random() * 99999) as uint;
    arrayOfArrays[randomIndex] = testArrays[randomIndex];
}

i think you'll measure the speed to the Math class, not array assignment speed. (same holds for other examples, where you use do lots of Math calls.) To test random assignment speed, i'd first populate an array with random elements.