2
\$\begingroup\$

I have been asked by a client to program a bot which searches Google and will show how many no of results I get.

Note: I know about Google Custom Search API and it will not produce the exact output produced by the real google since it is searching only in particular domains

My Code (With GUI):

# SWAMI KARUPPASWAMI THUNNAI

from tkinter import *
from selenium import webdriver
import re
from threading import Thread
from selenium.webdriver.common.keys import Keys
import time
import xlwt
from datetime import datetime
import tkinter as V
from selenium.webdriver.common.proxy import *
from random import randint
import random
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

current = ""
value = 0
val = 0
out = 0

#proxy_chains = [59.148.254.50,]

def START():
    headers = { 'Accept':'*/*',
        'Accept-Encoding':'gzip, deflate, sdch',
        'Accept-Language':'en-US,en;q=0.8',
        'Cache-Control':'max-age=0',
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'
    }

    for key, value in enumerate(headers):
        webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.customHeaders.{}'.format(key)] = value
    xx = 0
    proxy=[]
    chain=[]
    filex = open("proxies.csv")
    for i in filex:
        print(i)
        proxy.append(i)
    for i in proxy:
        x = re.sub("\n","",i)
        chain.append(x)

    print(chain)
    file = open(loc.get(),"r")
    keywords = file.readlines()
    text = Text(root, bg = "white", wrap = WORD, height =8, width = 40)
    text.place(x= 20, y=200)
    for i in keywords:
        keyword = re.sub("\n","",i)
        search =  "allintitle:\""+keyword+"\""
        
        global current
        current = chain[random.randrange(25)]
        service_args = [
        '--proxy='+current+':60099',   # 60099
        '--proxy-auth=username:password',  # It is private I.P
        '--proxy-type=http',
        ]
        text.insert(END,"Scrapping with "+str(current))
        browser = webdriver.PhantomJS("phantomjs.exe",service_args =service_args )
        #now the ip is blocked, it gives captcha -- I spend 5 IPs not more than that ,
        #browser.get("https://www.google.com/#q={}".format(search))
        select_server = random.randrange(3)
        if (select_server==0):
            try:
                browser.get("https://www.proxysite.com")
                text.insert(END,str("\nConncted To Proxy site VPN\n"))
                put = browser.find_element_by_xpath("/html/body/div[1]/main/div[1]/div/div[3]/form/div[1]/input")
                put.send_keys("www.google.com")
                put.send_keys(Keys.ENTER)
                browser.save_screenshot(str(current)+'test.png')
            except:
                browser.save_screenshot(str(current)+'Error.png')
        elif (select_server==1):
            try:
                browser.get("https://www.filterbypass.me/")
                text.insert(END,str("\nConncted To Filter ByPass VPN\n"))
                put = browser.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/input[1]")
                put.send_keys("www.google.com")
                put.send_keys(Keys.ENTER)
                browser.save_screenshot(str(current)+'test.png')
            except:
                browser.save_screenshot(str(current)+'Error.png')
                
        else:
            try:
                print(select_server)
                browser.get("https://hide.me/en/proxy")
                text.insert(END,str("\nConncted To Hide me VPN\n"))
                browser.save_screenshot(str(current)+'test.png')
                put = browser.find_element_by_xpath("/html/body/main/div[2]/form/fieldset/div[1]/div/input")
                put.send_keys("www.google.com")
                put.send_keys(Keys.ENTER)
                browser.save_screenshot(str(current)+'test.png')
            except:
                browser.save_screenshot(str(current)+'Error.png')
            
        
        # It redirects to .co.in  open that
        time.sleep(5)
        put = browser.find_element_by_name('q')
        put.send_keys(search)
        put.send_keys(Keys.ENTER)
        browser.save_screenshot(str(current)+'test1.png')
        print(search)
        rectify = ""
        create = open("output.csv","a")
        browser.save_screenshot(str(current)+'test.png')
        try:
            result = browser.find_element_by_id("resultStats")
            rectify= re.sub("result","",str(result.text))
            rectify = re.sub(r",", "", rectify)
            print(rectify)
            create.write("\n")
            create.write(search+","+rectify)
            create.close()
            browser.quit()
            text.insert(END,str(search+"\n"+rectify+"\n"))
            x = randint(0,15)
            time.sleep(x)
            text.insert(END,str("Sleeping for"+x))
            global out
            
        
        except:
            global current
            current = chain[random.randrange(25)]
            text.insert(END,"Rotating Proxies")
            text.insert(END,"Changed To"+str(current))
            browser.quit()
        
        

def start():
    child = Thread(target=START)
    child.daemon =True
    child.start()

if __name__ =="__main__":
    root = Tk()
    root.geometry("400x400")
    root.title("Fake Google Search Engine with proxy chains")
    root.resizable(0,0)
    title = Label(text=" Competition Analyzer ",fg="white",bg="black",font=("Helvetica",30)).pack()
    title1 = Label(text="  PASTE THE FILE LOCATION  ",fg="green",bg="yellow",font=("Helvetica",20)).pack()
    loc = StringVar()
    location = Entry(textvariable=loc).place(x=20,y=130)
    button = Button(text="Start Scrapping",fg="white",bg="green",command=start).place(x=190,y=125)
    info0 = Label(text="Entries scrapped",fg="blue").place(x=20,y=180)
    root.mainloop()

What Google does for web-scrapers: With a maximum of 4 hits, Google will block the specific I.P and won't allow further scraping.

What My Bot does Input keywords needed to be entered and proxy-list will be taken from the default path

It will collect the list of proxies into a list. list = [proxy1,proxy2,proxy3,...proxy(n)]

The bot has three VPN servers

A variable is taken where variable = list of proxies[random value] so Random I.P will bee set for connecting to the V.P.N the V.P.N servers will also be selected randomly.

So I.P = Random Proxy -> Random VPN ->Google

By this user's I.P is masked and fetching Google data is completely randomized.

Neither the VPN and Nor the Google will have access on the same I.P since the I.P is made to be rotating. However I need some suggestions to make this bot work perfectly.

So that I can make a robust application for my client. I may use Google API but it will not give the same results (search results) . BS4 and requests may not be the best option since many Javascript things have to be bypassed. Tor comes to mind but we are on windows machine

\$\endgroup\$
2
  • \$\begingroup\$ The Tor Browser lets you use Tor on Windows - Straight from the Tor website. So I believe that you can use Tor on windows \$\endgroup\$ Sep 12 '16 at 14:26
  • \$\begingroup\$ @Mr.goosberry I am in fear I have seen many dark web videos and peoples getting killed and FBI arrested Tor users my hands are chivering in fear to use tor \$\endgroup\$ Sep 12 '16 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.