5
\$\begingroup\$

Today, I sat down to write a tiny performance test to quickly assess a hunch I had regarding an optimization. The resulting code eventually got rather large for my taste; thus I subsequently tried to abstract it to a degree that allowed me to keep ledgibility, ease of maintenance and extensibility, reduced redundancy and reproduceability.

The basic notion of the code is to run a set of predefined tests for a certain amount of repetitions, and trivially measure the average time needed for each test entry over the whole test conduct.

How, given in the eyes of a professional Java developer, would you have structured the following code?

package com.stackexchange.codereview;

import org.testng.annotations.Test;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;

public class MathTest {
    protected static Random random = new Random();
    protected static Map<TestCase, Double> avg = new HashMap<>();
    protected static Map<TestCase, Integer> count = new HashMap<>();
    protected static Map<TestCase, Long> start = new HashMap<>();
    protected static Map<TestCase, Long> end = new HashMap<>();
    protected static Map<TestCase, ServiceMethodTest> serviceMethodMap = new HashMap<>();
    protected static Double RANGE_LOW = 0.01;
    protected static Double RANGE_HIGH = 0.03;
    protected static int LOOPS = 1000000;
    protected static int REPETITIONS = 5;
    protected enum TestCase {
        POW {
            @Override
            public String toString() {
                return "pow";
            }
        },
        SQUARE {
            @Override
            public String toString() {
                return "square";
            }
        }
    }

    @Test
    public static void PowTest() {
        TestCase testCase;
        int cnt;

        initTestScenario();

        for (int loop = 1; loop <= REPETITIONS; loop++) {
            for (Map.Entry<TestCase, ServiceMethodTest> serviceMethod : serviceMethodMap.entrySet()) {
                testCase = serviceMethod.getKey();
                initTimer(testCase);
                serviceMethod.getValue().execute();
                count.put(testCase, loop);
                calcAvg(testCase, printPerf(testCase, testCase.toString(), true, false).doubleValue());
            }
        }
        System.out.println("Counts     : " + count.entrySet().toString());
        System.out.println("Averages   : " + avg.entrySet().toString());
        System.out.flush();
    }

    public static void calcAvg(TestCase key, Double newval) {
        /* New average = old average * (n-1)/n + new value /n */
        Double oldavg;
        Double newavg;
        Double ratio;
        int n;

        if (avg.get(key) == null) {
            avg.put(key, newval);
        } else {
            n = count.get(key);
            oldavg = avg.get(key);
            ratio = (n - 1.0) / n;
            newavg = oldavg * ratio + newval / n;
            avg.put(key, newavg);
        }
    }

    public static void initTestScenario() {
        serviceMethodMap.put(TestCase.POW, () -> {
            @SuppressWarnings("unused")
            double denom = 0f;
            Double x = randomInRange(RANGE_LOW, RANGE_HIGH);
            Double y = randomInRange(RANGE_LOW, RANGE_HIGH);
            for (int i = 1; i <= LOOPS; i++) {
                denom += Math.pow(x, 2) + Math.pow(y, 2);
            }
        });
        serviceMethodMap.put(TestCase.SQUARE, () -> {
            @SuppressWarnings("unused")
            double denom = 0f;
            Double x = randomInRange(RANGE_LOW, RANGE_HIGH);
            Double y = randomInRange(RANGE_LOW, RANGE_HIGH);
            for (int i = 1; i <= LOOPS; i++) {
                denom += x * x + y * y;
            }
        });
    }

    public static double randomInRange(double min, double max) {
        return (random.nextDouble() * (max - min)) + min;
    }

    public static void initTimer(TestCase key) {
        start.put(key, System.currentTimeMillis());
    }

    public static Long printPerf(TestCase key, String msg, boolean newline, boolean quiet) {
        end.put(key, System.currentTimeMillis());
        Long diff = end.get(key) - start.get(key);
        if (!quiet) {
            System.out.printf("PERFORMANCE: [%s completed in %dms]%s",
                    msg, diff, (newline ? "\n" : ""));
            System.out.flush();
        }
        return diff;
    }
}

And the interface:

package com.stackexchange.codereview;

@FunctionalInterface
public interface ServiceMethodTest {
    void execute();
}

I would like to reiterate that the test cases in the example have purposely been chosen to be brief and simple, since the the question at hand is about the generic pattern or approach one would take writing this code, if he actually had test patterns that do not get eliminated by the JVM.

Here is a better example:

package com.stackexchange.codereview;

import org.testng.annotations.Test;

import java.util.HashMap;
import java.util.Map;
import java.util.Random;

public class MathTest {
    protected static Random random = new Random();
    protected static Map<TestCase, Double> avg = new HashMap<>();
    protected static Map<TestCase, Integer> count = new HashMap<>();
    protected static Map<TestCase, Integer> result = new HashMap<>();
    protected static Map<TestCase, Long> start = new HashMap<>();
    protected static Map<TestCase, Long> end = new HashMap<>();
    protected static Map<TestCase, ServiceMethodTest> serviceMethodMap = new HashMap<>();
    protected static Double RANGE_LOW = 1.214;
    protected static Double RANGE_HIGH = 2.291;
    protected static int LOOPS = 1000000;
    protected static int THRESHOLD = 10;
    protected static int REPETITIONS = 10;
    protected enum TestCase {
        POW {
            @Override
            public String toString() {
                return "pow";
            }
        },
        SQUARE {
            @Override
            public String toString() {
                return "square";
            }
        }
    }

    @Test
    public static void PowTest() {
        TestCase testCase;

        initTestScenario();

        for (int loop = 1; loop <= REPETITIONS; loop++) {
            for (Map.Entry<TestCase, ServiceMethodTest> serviceMethod : serviceMethodMap.entrySet()) {
                testCase = serviceMethod.getKey();
                initTimer(testCase);
                if (result.get(testCase) == null) {
                    result.put(testCase, serviceMethod.getValue().execute());
                } else {
                    result.put(testCase, result.get(testCase) + serviceMethod.getValue().execute());
                }
                count.put(testCase, loop);
                calcAvg(testCase, printPerf(testCase, testCase.toString(), true, true).doubleValue());
            }
        }

        System.out.println("Total runs     : " + REPETITIONS * LOOPS);
        System.out.println("Threshold hits : " + result.entrySet().toString());
        System.out.println("Repetition cnt : " + count.entrySet().toString());
        System.out.println("Averages       : " + avg.entrySet().toString());
        System.out.flush();
    }

    public static void calcAvg(TestCase key, Double newval) {
        /* New average = old average * (n-1)/n + new value /n */
        Double oldavg;
        Double newavg;
        Double ratio;
        int n;

        if (avg.get(key) == null) {
            avg.put(key, newval);
        } else {
            n = count.get(key);
            oldavg = avg.get(key);
            ratio = (n - 1.0) / n;
            newavg = oldavg * ratio + newval / n;
            avg.put(key, newavg);
        }
    }

    public static void initTestScenario() {
        serviceMethodMap.put(TestCase.POW, () -> {
            double denom;
            Double x;
            Double y;
            int overThreshold = 0;

            for (int i = 1; i <= LOOPS; i++) {
                x = randomInRange(RANGE_LOW, RANGE_HIGH);
                y = randomInRange(RANGE_LOW, RANGE_HIGH);
                denom = Math.pow(x, 2) + Math.pow(y, 2);
                if (denom >= THRESHOLD) {
                    overThreshold += 1;
                }
            }
            return overThreshold;
        });
        serviceMethodMap.put(TestCase.SQUARE, () -> {
            double denom;
            Double x;
            Double y;
            int overThreshold = 0;

            for (int i = 1; i <= LOOPS; i++) {
                x = randomInRange(RANGE_LOW, RANGE_HIGH);
                y = randomInRange(RANGE_LOW, RANGE_HIGH);
                denom = x * x + y * y;
                if (denom >= THRESHOLD) {
                    overThreshold += 1;
                }
            }
            return overThreshold;
        });
    }

    public static double randomInRange(double min, double max) {
        return (random.nextDouble() * (max - min)) + min;
    }

    public static void initTimer(TestCase key) {
        start.put(key, System.currentTimeMillis());
    }

    public static Long printPerf(TestCase key, String msg, boolean newline, boolean quiet) {
        end.put(key, System.currentTimeMillis());
        Long diff = end.get(key) - start.get(key);
        if (!quiet) {
            System.out.printf("PERFORMANCE: [%s completed in %dms]%s",
                    msg, diff, (newline ? "\n" : ""));
            System.out.flush();
        }
        return diff;
    }
}

And the adjusted interface:

package com.stackexchange.codereview;

@FunctionalInterface
public interface ServiceMethodTest {
    int execute();
};

I am interested in further useful abstractions, simple patterns I missed, best practise when implementing such ideas in Java, reduction in code size while keeping maintainability and ledgibility, issues with my approach (non-academic ones).

\$\endgroup\$
  • \$\begingroup\$ :) sorry for the misunderstanding I will transform my downvote in upvote as soon as possible, I hope you will get some great reviews \$\endgroup\$ – Caridorc Jul 17 '15 at 19:57
  • 1
    \$\begingroup\$ Sounds like a plan. I've amended the initial post now in hope to avoid future preemptive suboptimal categorization of human language parsing attempts. \$\endgroup\$ – Moreaki Jul 17 '15 at 20:02
1
\$\begingroup\$

best practise when implementing such ideas in Java

The best practice is: Don't do it at all. Especially in Java. Because benchmarking is hard in general and Java benchmarking is even harder. Let me show you why:

    serviceMethodMap.put(TestCase.SQUARE, () -> {
        @SuppressWarnings("unused")
        double denom = 0f;
        Double x = randomInRange(RANGE_LOW, RANGE_HIGH);
        Double y = randomInRange(RANGE_LOW, RANGE_HIGH);
        for (int i = 1; i <= LOOPS; i++) {
            denom += x * x + y * y;
        }
    });

OK, you have a sort of Runnable, which adds some constant expression in a loop. JIT may or may not be smart enough to see that your loop is equivalent to

        if (LOOPS > 0) {
            denom += LOOPS * (x * x + y * y);
        }

It may or may not be smart enough to see that it's actually equivalent to

        // nothing at all

as the result does not get used anywhere and the expression has no side effects. Look how a blackhole gets used in JMH.


Such benchmarking could make sense if the things you're doing are too complicated to be eliminated (and you store the result somewhere where it can't be ignored). But then your

protected static int LOOPS = 1000000;

comes in the way. Sure, you can change it, but this is ugly. It should be no constant (actually, it is no constant and should be written in lowercase) and it should probably be TestCase specific (a member of the enum). I wouldn't call it TestCase, as it's a well-known class (JUnit 3).


You should avoid static nearly everywhere. Having an instance give you a lot of flexibility (imagine e.g., new Benchmark(TestCase.POW).loopCount(1000).repeat(10).gcBetweenRuns().outputFile("whatever.txt").run()).


That all said, your code is not bad, it's just that you're trying to do something what needs a few man-months to get really right. You may get interesting and correct results using it, but you never be sure that they're correct. Actually, even with the best tools I know (caliper and JMH), you can't be really sure when you measure fast code snippets like the one in your example. But the chances the tool measures what you want to know are bigger and they're quite comfortable.

\$\endgroup\$
  • \$\begingroup\$ Java Mission Control is another option, isn't it? \$\endgroup\$ – Gerold Broser Jul 21 '15 at 2:44
  • \$\begingroup\$ @GeroldBroser I guess no. AFAIK, JMC is a profiler (and whatever else), which tells you where your application loses most time. That's a different story. \$\endgroup\$ – maaartinus Jul 21 '15 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.