Everytime a new visitor comes to my webshop I write a cookie (1d) which sets a session entry with mt_rand(1,2) to A or B group.

if(empty($_GET["cltestuser"])) {
    if(empty($_COOKIE['cl_customergroup_test'])) {
        $cookieValue = "cl_use_default";
        if(mt_rand(1, 2) == 1) {
            $cookieValue = "A";
        } else {
            $cookieValue = "B";
        setcookie("cl_customergroup_test", $cookieValue, time() + (86400 * 30)); //1 day
        $_COOKIE['cl_customergroup_test'] = $cookieValue;

    $_SESSION["Customergroup"]->cl_testuser = $_COOKIE['cl_customergroup_test'];
} else {
    $cookieValue = strtoupper($_GET["cltestuser"]);
    setcookie("cl_customergroup_test", $cookieValue, time() + (86400 * 30)); //1 day
    $_COOKIE['cl_customergroup_test'] = $cookieValue;

    $_SESSION["Customergroup"]->cl_testuser = $_COOKIE['cl_customergroup_test'];


So the site is the same now for A and B visitors. The testing sample is 15,000 visitors where 150 people bought goods.

(Note: 15,000 times in a for loop with the mt_rand(1,2) is nearly even (7500/7500))

But the conversion for B is about 30% higher then A.

When you have cookies blocked you cannot buy. There are no returning customers.

Is this breaking the law of the great numbers? Where does this come from?


closed as unclear what you're asking by 200_success, yuri, esote, Toby Speight, Mast Jun 17 at 14:15

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • \$\begingroup\$ Welcome to Code Review. Is the code working as expected or not, we can only review working code and can't help debug issues. Please see codereview.stackexchange.com/help/dont-ask and codereview.stackexchange.com/help/how-to-ask. \$\endgroup\$ – pacmaninbw Jun 14 at 16:14
  • 3
    \$\begingroup\$ I don't understand. You did an A-B test and came to a conclusion. Now you're doubting the experiment just because the null hypothesis was rejected? So why do the experiment at all? \$\endgroup\$ – 200_success Jun 14 at 17:51
  • 1
    \$\begingroup\$ @200_success in OP's defense, the null hypothesis ("buying decisions are not influenced by invisible cookie values") appears airtight. @Thomas-z you have a sample size problem. Run mt_rand(100,299) 15000 times. 100 and 200 are conversions. Examine their distribution. \$\endgroup\$ – Oh My Goodness Jun 15 at 8:26
  • \$\begingroup\$ @OhMyGoodness could you explain that a little bit more? Why do I have a sample size problem? Is the 15,000 visitors the problem, it appears enough for me \$\endgroup\$ – Thomas Z. Jun 17 at 5:54
  • \$\begingroup\$ @ThomasZ. you didn't run the mt_rand(100,299) experiment that I described. Suppose you sold 15,000,000 lotto tickets to A and B, and then A wins two jackpots and B wins one — a difference of 100%! How is that possible with a sample size of 15M? Is the sample size actually 15M? \$\endgroup\$ – Oh My Goodness Jun 17 at 9:28

Browse other questions tagged or ask your own question.