3
\$\begingroup\$

I made the switch from JavaScript to Python lately. This is my first Python project and my first web-scraper. Do I have good practices? And could this be optimized?

from bs4 import BeautifulSoup
import requests


urlHalf = 'codingbat.com'

print("what is your email")
inputOne = input(">")
print("password")
inputTwo = input(">")

post_params = {
    "uname": inputOne,
    "pw": inputTwo,
}

# urls to use
base_url = "https://codingbat.com"
login_url = "https://codingbat.com/login"
java_url = "https://codingbat.com/java/"


def get_href():
    print("Getting directories")
    href_array = []
    initial = requests.get(base_url)
    soup = BeautifulSoup(initial.content, "lxml")
    tab_data = soup.find(class_="floatleft")
    hrefs = tab_data.find_all(class_="h2")
    for href in hrefs:
        href_array.append(href.text)
    return(href_array)


def gather_links():
    directories = get_href()
    print("Getting problem links")
    link_array = []
    for directory in directories:
        r = requests.get(java_url + directory)
        # get links to problems
        content = BeautifulSoup(r.content, "lxml")
        links = content.find_all(
            "a", href=lambda href: href and "prob" in href)
        link_array.append(links)
    return(link_array)


def join_links():
    linkList = gather_links()
    new_link_list = {}
    print("Joining links")
    for link in linkList:
        for each in link:
            text = each.text
            backUrl = each.get("href")
            realUrl = "http://" + urlHalf + backUrl
            new_link_list.update({text: realUrl})
    return new_link_list


def scrape_data():
    f = open('answers/answers.java', 'w')
    linkList = join_links()
    with requests.Session() as session:
        post = session.post(login_url, data=post_params)
    for i in linkList:
        print("Scraping: " + i)
        response = session.get(linkList[i])
        content = BeautifulSoup(response.content, "lxml")
        indentDiv = content.find(class_="indent")
        table = indentDiv.find("form", {"name": "codeform"})
        aceDiv = table.find("div", id="ace_div")
        f.write("//" + i + ": " + linkList[i] + "\n\n")
        f.write(aceDiv.text + "\n")
    print("done")


if __name__ == "__main__":
    scrape_data()

\$\endgroup\$
2
\$\begingroup\$

Global nomenclature

urlHalf should be URL_HALF. That said, a better name would be DOMAIN, since that's what that string actually is.

Further, after declaring this you don't use it correctly; these:

base_url = "https://codingbat.com"
login_url = "https://codingbat.com/login"
java_url = "https://codingbat.com/java/"

could be

BASE_URL = 'https://' + DOMAIN
LOGIN_URL = BASE_URL = '/login'
JAVA_URL = BASE_URL + '/java'

Input

print("what is your email")
inputOne = input(">")
print("password")
inputTwo = input(">")

should not occur at global scope; it should occur in a function. Also, the use of input for passwords is insecure; use getpass instead.

Return parens

These parens:

return(href_array)

are redundant.

Generator

link_array = []
for directory in directories:
    # ...
    link_array.append(links)
return(link_array)

can simply be

for directory in directories:
    # ...
    yield links

or, if you want a "flattened" output that does not have nested lists, use yield from links.

File extensions

Does this file:

f = open('answers/answers.java', 'w')

actually receive Java code? It seems not, in which case - use just .txt.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.