New answers tagged web-scraping
0
I'll go out on a limb and say that there is nothing clearly wrong with this script. As a casual batch file, the only improvement I would suggest using the "Main Method" pattern. I've yet to find a clear, focused explanation of both what the pattern is and why one should use it, but this is a good start.
Oh, and while it's good that you're calling csv_file....
1
Great answers, as always.
Just a few remarks from me:
There is not enough validation in your project.
You are scraping a website that could change at any time, and your script is expecting DOM elements that may not be there. So you need to check each of them.
From the doc (emphasis is mine):
If find_all() can’t find anything, it returns an empty list. ...
0
Two things about the way you handle files:
Since you open the file and only close it at the end, if an exception disrupts your program in between, the file will not be properly closed. Instead use a with context manager:
with open("bundesliga_table.csv", "w") as csv_file:
...
This automatically closes the file for you when leaving the block, whether ...
2
What you are doing sounds reasonable overall.
It's good that you are using F-strings:
nmoPag = input(f"There are {noPag} number of pages. If you want to scrape all of them write 'Max' else write number of pages you wish to scrape: ")
But there are many other places where you don't eg:
print('Your job in '+jobL+' as a '+ jobT.text.strip()+
...
3
Duplicated statements in an if-block
if find_name.startswith('https'):
self._search_name = urllib.parse.urlparse(find_name).path.split('/')[1]
return self._search_name
else:
return self._search_name
should just be
if find_name.startswith('https'):
self._search_name = urllib.parse.urlparse(find_name).path.split('/...
1
Type hints
def __init__(self, username, password, folder, search_name):
can (probably) be
def __init__(self, username: str, password: str, folder: Path, search_name: str):
Also, since these lists are initialized without a direct reference to the args, they should be type-declared:
self.links: List[str] = []
self.pictures: List[str] = []
self....
1
Requests makes things easy
session_base.get(urllib.parse.urljoin(link, "?__a=1"))
should be
session_base.get(link, params={__a: 1})
Exception types
raise Exception("[!] Account is private")
The use of the base Exception should be replaced by a custom exception of yours. They're easy to make, and using them makes it so that upstream code can more ...
4
Requests raising
This pattern:
if search.ok:
...
else:
search.raise_for_status()
is redundant. Just call raise_for_status(), which will be a no-op if the response is OK.
check_availability still has a confused return. If you're returning False, is there ever a point where it could return True? If not, why return a value at all? This boils down to ...
2
Error management
This:
if search.status_code == 404:
return "Sorry, this page isn't available."
elif search.json()["graphql"]["user"]["is_private"] is True:
return "This Account is Private"
else:
return True
is problematic. First of all, you're mixing return types (boolean and string). More ...
5
Seems like you have as many filing_dates as you have URLs, so you should have those together and handle them similarly.
Your problem seems to come from the fact that you're losing the intel of which row comes from which URL, and so your only option becomes to set one date for the full dataframe.
Here's an updated version saving the dates at the same time ...
4
Wikipedia has an API available. You can make calls to e.g. "https://en.wikipedia.org/w/api.php?action=query&prop=links&titles={title}&pllimit=500" and get back a list of titles the Wikipedia page title links to, without all the internal links (but with non-existing pages and things like categories).
Incidentally, there is even a python package ...
Top 50 recent answers are included
