1
\$\begingroup\$

Everytime a new visitor comes to my webshop I write a cookie (1d) which sets a session entry with mt_rand(1,2) to A or B group.

if(empty($_GET["cltestuser"])) {
    if(empty($_COOKIE['cl_customergroup_test'])) {
        $cookieValue = "cl_use_default";
        if(mt_rand(1, 2) == 1) {
            $cookieValue = "A";
        } else {
            $cookieValue = "B";
        }
        setcookie("cl_customergroup_test", $cookieValue, time() + (86400 * 30)); //1 day
        $_COOKIE['cl_customergroup_test'] = $cookieValue;
    }

    $_SESSION["Customergroup"]->cl_testuser = $_COOKIE['cl_customergroup_test'];
} else {
    $cookieValue = strtoupper($_GET["cltestuser"]);
    setcookie("cl_customergroup_test", $cookieValue, time() + (86400 * 30)); //1 day
    $_COOKIE['cl_customergroup_test'] = $cookieValue;

    $_SESSION["Customergroup"]->cl_testuser = $_COOKIE['cl_customergroup_test'];

}

So the site is the same now for A and B visitors. The testing sample is 15,000 visitors where 150 people bought goods.

(Note: 15,000 times in a for loop with the mt_rand(1,2) is nearly even (7500/7500))

But the conversion for B is about 30% higher then A.

When you have cookies blocked you cannot buy. There are no returning customers.

Is this breaking the law of the great numbers? Where does this come from?

\$\endgroup\$
  • \$\begingroup\$ Welcome to Code Review. Is the code working as expected or not, we can only review working code and can't help debug issues. Please see codereview.stackexchange.com/help/dont-ask and codereview.stackexchange.com/help/how-to-ask. \$\endgroup\$ – pacmaninbw Jun 14 '19 at 16:14
  • 3
    \$\begingroup\$ I don't understand. You did an A-B test and came to a conclusion. Now you're doubting the experiment just because the null hypothesis was rejected? So why do the experiment at all? \$\endgroup\$ – 200_success Jun 14 '19 at 17:51
  • 1
    \$\begingroup\$ @200_success in OP's defense, the null hypothesis ("buying decisions are not influenced by invisible cookie values") appears airtight. @Thomas-z you have a sample size problem. Run mt_rand(100,299) 15000 times. 100 and 200 are conversions. Examine their distribution. \$\endgroup\$ – Oh My Goodness Jun 15 '19 at 8:26
  • \$\begingroup\$ @OhMyGoodness could you explain that a little bit more? Why do I have a sample size problem? Is the 15,000 visitors the problem, it appears enough for me \$\endgroup\$ – Thomas Z. Jun 17 '19 at 5:54
  • \$\begingroup\$ @ThomasZ. you didn't run the mt_rand(100,299) experiment that I described. Suppose you sold 15,000,000 lotto tickets to A and B, and then A wins two jackpots and B wins one — a difference of 100%! How is that possible with a sample size of 15M? Is the sample size actually 15M? \$\endgroup\$ – Oh My Goodness Jun 17 '19 at 9:28

Browse other questions tagged or ask your own question.