Webshop Conversion - Binary testing is not even [closed]

Everytime a new visitor comes to my webshop I write a cookie (1d) which sets a session entry with mt_rand(1,2) to A or B group.

if(empty($_GET["cltestuser"])) { if(empty($_COOKIE['cl_customergroup_test'])) {
$cookieValue = "cl_use_default"; if(mt_rand(1, 2) == 1) {$cookieValue = "A";
} else {
$cookieValue = "B"; } setcookie("cl_customergroup_test",$cookieValue, time() + (86400 * 30)); //1 day
$_COOKIE['cl_customergroup_test'] =$cookieValue;
}

$_SESSION["Customergroup"]->cl_testuser =$_COOKIE['cl_customergroup_test'];
} else {
$cookieValue = strtoupper($_GET["cltestuser"]);
setcookie("cl_customergroup_test", $cookieValue, time() + (86400 * 30)); //1 day$_COOKIE['cl_customergroup_test'] = $cookieValue;$_SESSION["Customergroup"]->cl_testuser = \$_COOKIE['cl_customergroup_test'];

}


So the site is the same now for A and B visitors. The testing sample is 15,000 visitors where 150 people bought goods.

(Note: 15,000 times in a for loop with the mt_rand(1,2) is nearly even (7500/7500))

But the conversion for B is about 30% higher then A.

When you have cookies blocked you cannot buy. There are no returning customers.

Is this breaking the law of the great numbers? Where does this come from?

• Welcome to Code Review. Is the code working as expected or not, we can only review working code and can't help debug issues. Please see codereview.stackexchange.com/help/dont-ask and codereview.stackexchange.com/help/how-to-ask. – pacmaninbw Jun 14 '19 at 16:14
• I don't understand. You did an A-B test and came to a conclusion. Now you're doubting the experiment just because the null hypothesis was rejected? So why do the experiment at all? – 200_success Jun 14 '19 at 17:51
• @200_success in OP's defense, the null hypothesis ("buying decisions are not influenced by invisible cookie values") appears airtight. @Thomas-z you have a sample size problem. Run mt_rand(100,299) 15000 times. 100 and 200 are conversions. Examine their distribution. – Oh My Goodness Jun 15 '19 at 8:26
• @OhMyGoodness could you explain that a little bit more? Why do I have a sample size problem? Is the 15,000 visitors the problem, it appears enough for me – Thomas Z. Jun 17 '19 at 5:54
• @ThomasZ. you didn't run the mt_rand(100,299) experiment that I described. Suppose you sold 15,000,000 lotto tickets to A and B, and then A wins two jackpots and B wins one — a difference of 100%! How is that possible with a sample size of 15M? Is the sample size actually 15M? – Oh My Goodness Jun 17 '19 at 9:28