I just started learning Python and I wrote my first useful script for work. I did a bunch of the basic tutorials and really enjoyed learning Python so far.

I am looking for any advice on how to make things more pythonic. What areas can I improve going forward. I want to make this script better/move on to my new project but I don't want to build off bad fundamentals.

My script works and I am using it.

My script goes to a set of websites using the selenium webdriver to pull all the links down to a list. I then delete the duplicates. Then I use the requests module to verify a 200 response code.

I incorporated multiprocessing into it because the first version took way to long, 5+ hours to scan on 7300 links. I got the script time down to about an hour.

I just started learning Python and I wrote my first useful script for work. I did a bunch of the basic tutorials and really enjoyed learning Python so far.

I am looking for any advice on how to make things more pythonic. What areas can I improve going forward? I want to make this script better/move on to my new project but I don't want to build off bad fundamentals.

My script works and I am using it. My script goes to a set of websites using the selenium webdriver to pull all the links down to a list. I then delete the duplicates. Then I use the requests module to verify a 200 response code.

I incorporated multiprocessing into it because the first version took way too long, 5+ hours to scan on 7300 links. I got the script time down to about an hour.

I am sure I am opening myself up a bunch of criticism but I am doing this to get better.

SO my question is this? I am looking for any advice on how to make things more pythonic. (at least that seems to be the term I keep reading about) What areas can I improve going forward. I want to make this script better/move on to my new project but I don't want to build off bad fundamentals.

Below I have pasted the code. My script works and I am using it. So any advice would be greatly appreciated.

Basically, my script goes to a set of websites uses the selenium webdriver to pull all the links down to a list. I then delete the duplicates. Then I use the requests module to verify a 200 response code.

I incorporated multiprocessing into it becuase the first version took way to long. 5+ hours to scan on 7300 links. I got the script time down to about an hour.

Thank you to everyone that takes the time out of their day to help me get the fundamentals down better.

CustomFunctions.py

import requests
from selenium import webdriver
import time
import multiprocessing

def get_links(x):
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--disable-application-cache')
    driver = webdriver.Chrome('/Desktop/project/SiteCheck/LinkCheckV03/chromedriver', chrome_options=chrome_options)
    driver.get(x)
    links = driver.find_elements_by_xpath('//*[@href]')
    time.sleep(4)
    return links
   
def check_links(links):
    try:
        r = requests.get(links)
        rc = r.status_code
        strRc = str(rc)
        result = links, strRc
        return result
    except Exception as e:
        logz = open('exception.log', 'w')
        logz.write(str(e) + '\n')
    
def main(func, mlist):
    pool = multiprocessing.Pool(4)
    results = pool.map(func, mlist)

    pool.close()
    pool.join()
    
    return results

LinkCheck.Py

from CustomFunctions import get_links, check_links, main
import fileinput
import sys

#redircting stdout to fie
old_stdout = sys.stdout
log_file = open("output.log","w")
sys.stdout = log_file


#Gateways
home = 'http://www.bonton.com'
#brands = 'http://www.bonton.com/sc1/brands/'
women = 'http://www.bonton.com/sc1/women/'
shoes = 'http://www.bonton.com/sc1/shoes/'
handbagsAccessories = 'http://www.bonton.com/sc1/handbags-accessories/'
jewelryWatches = 'http://www.bonton.com/sc1/jewelry-watches/'
beautyFragrance = 'http://www.bonton.com/sc1/beauty-fragrance/'
juniors = 'http://www.bonton.com/sc1/juniors/'
men = 'http://www.bonton.com/sc1/men/'
babyKids = 'http://www.bonton.com/sc1/baby-kids/'
bedBath = 'http://www.bonton.com/sc1/bed-bath/'
homegw = 'http://www.bonton.com/sc1/home/'
furniture = 'http://www.bonton.com/sc1/furniture/'
#clearence = 'http://www.bonton.com/clearence/'

#Master List of Links
masterList =[]

#### Fetch Links from host

#Home GW Populate Links
Homelist = get_links(home)

for ii in Homelist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Homepage: ', len(Homelist))
    
#Women GW Populate List
Womenlist = get_links(women)

for ii in Womenlist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Womens GW: ', len(Womenlist))

#Shoes GW Populate List    
Shoeslist = get_links(shoes)

for ii in Shoeslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Shoes GW: ', len(Shoeslist))

#HandbagandAcc. GW Populate List
handbagsAccessorieslist = get_links(handbagsAccessories)

for ii in handbagsAccessorieslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Handbags&Accessories GW: ', len(handbagsAccessorieslist))
    
#jewelryWatches GW Populate List
jewelryWatcheslist = get_links(jewelryWatches)

for ii in jewelryWatcheslist:
    masterList.append(ii.get_attribute('href')) 
    
print('Total number of links on Jewelry&Watches GW: ', len(jewelryWatcheslist))

#beautyFragrance GW Populate List
beautyFragrancelist = get_links(beautyFragrance)

for ii in beautyFragrancelist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Beauty&Fragrance GW: ', len(beautyFragrancelist))
    
#juniors GW Populate List
juniorslist = get_links(juniors)

for ii in juniorslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Juniors GW: ', len(juniorslist))
    
#men GW Populate List
menlist = get_links(men)

for ii in menlist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Mens GW: ', len(menlist))
    
#babyKids GW Populate List
babyKidslist = get_links(babyKids)

for ii in babyKidslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Baby&Kids GW: ', len(babyKidslist))
    
#bedBath GW Populate List
bedBathlist = get_links(bedBath)

for ii in bedBathlist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Bed&Bath GW: ', len(bedBathlist))
    
#homegw GW Populate List
homegwlist = get_links(homegw)

for ii in homegwlist:
    masterList.append(ii.get_attribute('href'))

print('Total number of links on Home GW: ', len(homegwlist))

#furniture GW Populate List
furniturelist = get_links(furniture)

for ii in furniturelist:
    masterList.append(ii.get_attribute('href'))

# Print link totals and get rid of dupelicates    
print('Total number of links on Furniture GW: ', len(furniturelist))
print('Total number of links before duplicates are removed:', len(masterList))
MasterListNoDupes = list(set(masterList))
print('Total number of links after duplicates are removed: ', len(MasterListNoDupes))

#removes empty elements in MasterListNoDupes
#MasterListNoDupes.remove('None')

#exextue the check_links commands with multiprocesing
if __name__ == '__main__':
    logd = open('linklist.log', 'w')
    line = main(check_links, MasterListNoDupes)
    for items in line:
        logd.write(str(items) + '\n')
    logd.close()
       
#Only print links that do now have a 200 response code 
with open('linklist.log', 'r') as searchfile:
    for line in searchfile:
        if '200' in line:
            pass            
        else:
            print(line)

#ending stdout logging to file            
sys.stdout = old_stdout

log_file.close()

I am looking for any advice on how to make things more pythonic. What areas can I improve going forward. I want to make this script better/move on to my new project but I don't want to build off bad fundamentals.

My script works and I am using it.

My script goes to a set of websites using the selenium webdriver to pull all the links down to a list. I then delete the duplicates. Then I use the requests module to verify a 200 response code.

I incorporated multiprocessing into it because the first version took way to long, 5+ hours to scan on 7300 links. I got the script time down to about an hour.

CustomFunctions.py

import requests
from selenium import webdriver
import time
import multiprocessing

def get_links(x):
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--disable-application-cache')
    driver = webdriver.Chrome('/Desktop/project/SiteCheck/LinkCheckV03/chromedriver', chrome_options=chrome_options)
    driver.get(x)
    links = driver.find_elements_by_xpath('//*[@href]')
    time.sleep(4)
    return links
   
def check_links(links):
    try:
        r = requests.get(links)
        rc = r.status_code
        strRc = str(rc)
        result = links, strRc
        return result
    except Exception as e:
        logz = open('exception.log', 'w')
        logz.write(str(e) + '\n')
    
def main(func, mlist):
    pool = multiprocessing.Pool(4)
    results = pool.map(func, mlist)

    pool.close()
    pool.join()
    
    return results

from CustomFunctions import get_links, check_links, main
import fileinput
import sys

#redircting stdout to fie
old_stdout = sys.stdout
log_file = open("output.log","w")
sys.stdout = log_file


#Gateways
home = 'http://www.bonton.com'
#brands = 'http://www.bonton.com/sc1/brands/'
women = 'http://www.bonton.com/sc1/women/'
shoes = 'http://www.bonton.com/sc1/shoes/'
handbagsAccessories = 'http://www.bonton.com/sc1/handbags-accessories/'
jewelryWatches = 'http://www.bonton.com/sc1/jewelry-watches/'
beautyFragrance = 'http://www.bonton.com/sc1/beauty-fragrance/'
juniors = 'http://www.bonton.com/sc1/juniors/'
men = 'http://www.bonton.com/sc1/men/'
babyKids = 'http://www.bonton.com/sc1/baby-kids/'
bedBath = 'http://www.bonton.com/sc1/bed-bath/'
homegw = 'http://www.bonton.com/sc1/home/'
furniture = 'http://www.bonton.com/sc1/furniture/'
#clearence = 'http://www.bonton.com/clearence/'

#Master List of Links
masterList =[]

#### Fetch Links from host

#Home GW Populate Links
Homelist = get_links(home)

for ii in Homelist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Homepage: ', len(Homelist))
    
#Women GW Populate List
Womenlist = get_links(women)

for ii in Womenlist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Womens GW: ', len(Womenlist))

#Shoes GW Populate List    
Shoeslist = get_links(shoes)

for ii in Shoeslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Shoes GW: ', len(Shoeslist))

#HandbagandAcc. GW Populate List
handbagsAccessorieslist = get_links(handbagsAccessories)

for ii in handbagsAccessorieslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Handbags&Accessories GW: ', len(handbagsAccessorieslist))
    
#jewelryWatches GW Populate List
jewelryWatcheslist = get_links(jewelryWatches)

for ii in jewelryWatcheslist:
    masterList.append(ii.get_attribute('href')) 
    
print('Total number of links on Jewelry&Watches GW: ', len(jewelryWatcheslist))

#beautyFragrance GW Populate List
beautyFragrancelist = get_links(beautyFragrance)

for ii in beautyFragrancelist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Beauty&Fragrance GW: ', len(beautyFragrancelist))
    
#juniors GW Populate List
juniorslist = get_links(juniors)

for ii in juniorslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Juniors GW: ', len(juniorslist))
    
#men GW Populate List
menlist = get_links(men)

for ii in menlist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Mens GW: ', len(menlist))
    
#babyKids GW Populate List
babyKidslist = get_links(babyKids)

for ii in babyKidslist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Baby&Kids GW: ', len(babyKidslist))
    
#bedBath GW Populate List
bedBathlist = get_links(bedBath)

for ii in bedBathlist:
    masterList.append(ii.get_attribute('href'))
    
print('Total number of links on Bed&Bath GW: ', len(bedBathlist))
    
#homegw GW Populate List
homegwlist = get_links(homegw)

for ii in homegwlist:
    masterList.append(ii.get_attribute('href'))

print('Total number of links on Home GW: ', len(homegwlist))

#furniture GW Populate List
furniturelist = get_links(furniture)

for ii in furniturelist:
    masterList.append(ii.get_attribute('href'))

# Print link totals and get rid of dupelicates    
print('Total number of links on Furniture GW: ', len(furniturelist))
print('Total number of links before duplicates are removed:', len(masterList))
MasterListNoDupes = list(set(masterList))
print('Total number of links after duplicates are removed: ', len(MasterListNoDupes))

#removes empty elements in MasterListNoDupes
#MasterListNoDupes.remove('None')

#exextue the check_links commands with multiprocesing
if __name__ == '__main__':
    logd = open('linklist.log', 'w')
    line = main(check_links, MasterListNoDupes)
    for items in line:
        logd.write(str(items) + '\n')
    logd.close()
       
#Only print links that do now have a 200 response code 
with open('linklist.log', 'r') as searchfile:
    for line in searchfile:
        if '200' in line:
            pass            
        else:
            print(line)

#ending stdout logging to file            
sys.stdout = old_stdout

log_file.close()