4
\$\begingroup\$

The below I have put together as an experiment, I have profiles on my site that are password protected. I would still like them to be crawled by google.

The script is taken from here and basically does a reverse dns lookup to verify if a googlebot is real or fake. I am then using this logic to allow google entry to my dynamic page by skipping the 'restrict-access.php' include which checks to see if someone is logged in or not.

My question is.. 1. Is this viable? 2. is the code solid/safe?

<?php
function validateGoogleBotIP($ip)
{
    $hostname = gethostbyaddr($ip); //"crawl-66-249-66-1.googlebot.com"
    return preg_match('/\.googlebot\.com$/i', $hostname);
}
if (strpos($_SERVER['HTTP_USER_AGENT'], 'Google') !== false) {
    if (validateGoogleBotIP($_SERVER['REMOTE_ADDR'])) {
        // its google - let them in
    } else {
        exit('You are not authorised to view this page');
    }
} else {
    require_once('php-includes/restrict-access.php');
    if (isset($_GET['user_id'])) {
        if (!ctype_digit($_GET['user_id']))
            exit('You are not authorised to view this page');
    } else {
        exit('You are not authorised to view this page');
    }
}
?>
\$\endgroup\$
  • 1
    \$\begingroup\$ For security, no. I can easily change my user agent with two clicks \$\endgroup\$ – Canadian Luke Apr 4 '15 at 22:53
  • \$\begingroup\$ This does not make sense. Why do you want to hide information for some users but make it public on google? Whatsoever I would display the content that has to be crawled by google on another page that can be accessed by everyone. \$\endgroup\$ – AMartinNo1 Apr 4 '15 at 23:14
  • \$\begingroup\$ I want them to signup to view peoples profiles but have the profiles get indexed to spike curiosity. The script knows if you try and fake the user agent - that's the whole point. \$\endgroup\$ – Jeff Apr 5 '15 at 0:14
  • \$\begingroup\$ Do I understand it correct: Only x of y details should be indexed by google? \$\endgroup\$ – AMartinNo1 Apr 5 '15 at 0:21
  • \$\begingroup\$ Correct, if you are google > you can access the page for index purposes without logging in. \$\endgroup\$ – Jeff Apr 5 '15 at 0:24
2
\$\begingroup\$

It makes more sense to define an isGoogleBotRequest() function, because that is what you want to know, and the IP address verification is just one detail of that. Furthermore, I recommend that you define an getAuthorisedUser() function in restrict-access.php so that you don't have to write two if statements for that.

You have three places where you exit with the "unauthorised" error message. Ideally, there should be one place where you do that.

Furthermore, instead of just dying, your script should output a "403 Forbidden" HTTP header, and exit successfully.

<?php
function isGoogleBotRequest()
{
    // Check User-Agent and verify using reverse-DNS lookup as recommended
    // in https://support.google.com/webmasters/answer/80553
    if (strpos($_SERVER['HTTP_USER_AGENT'], 'Google') !== false) {
        $hostname = gethostbyaddr($_SERVER['REMOTE_ADDR']);
        return preg_match('/\.googlebot\.com$/i', $hostname);
    }
    return false;
}

if (!isGoogleBotRequest()) {
    require_once('php-includes/restrict-access.php');
    if (!getAuthorisedUser()) {
        header('HTTP/1.0 403 Forbidden');
        echo 'You are not authorised to view this page';
        exit;
    }
}
?>
\$\endgroup\$
  • \$\begingroup\$ In terms of code review, this is a nicer presentation - much appreciated \$\endgroup\$ – Jeff Apr 5 '15 at 12:48
3
\$\begingroup\$

Is this viable

Of course it is viable but as I am pretty sure somehow everyone can fake a googlebot without a detection by your script I don't recommend it. Another aspect is mentioned by 200_success. Google policies dislike such behaviour and it is highly recommend to follow their recommendations. At the end of the answer is a sample of my idea of implementation

Is this code solid/safe

General

It looks like you are developing procedural which is not recommend. OOP (object orientend programming) is the better way to go as it is a modular approach of developing software. In oop there are classes and each has specific task to solve. Wikipedia - Object-oriented-programming

Closing php tag

In the embedded source code you close the php tag which is not recommend. It might happen that a space is after a closing php tag and if you receive a headers already sent error it can be time consuming to identifiy the file where a space is after a closing php tag.

My idea of implementation

Independent of the google policies and that only x of y details should be indexed I would not make a google check but display the x of y details always and the z of y only if a function (e.g. userHasAccess($userid); ) returns true.

Sample

<!DOCTYPE HTML>
<html lang="en">
    <head>
        <title>Detail</title>
        <meta name="robots" content="index">
    </head>
    <body>
        <h2>User: John Doe</h2>
        <p>
            A description about John Doe.
        </p>
        <?php
            if (userHasAccess(2))
            {
                echo "<p>Age: 18</p>";
            }
        ?>
    </body>
</html>

Such an implementation allows other search engines to crawl the x of y details while the z of y details are hidden for every search engine. You don't have to implement a googlebot verfication either.

Of course this sample has to be extended.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.