As @scnerd already discussed avoiding the use of a bare try/except
clause, I'll focus on removing it entirely.
After inspecting website source, you will notice that elements of class row-fluid
are containers for each lawyer. In each of those elements, there is two .span5 clearfix
elements, one containing lawyer's name and firm, other their specialization. Since you don't seem to be interested in the latter, we can skip that element entirely and move onto the next one.
for element in soup.select(".span5 b"):
name = element.text.strip()
if name.startswith('Area'):
continue
title = element.next_sibling.next_sibling.strip()
print('{}: {}'.format(name, title))
You will notice I left out row-fluid
container from the selector, as we are iterating only over the span5
elements that are contained within them, however if you wanted to keep that it, you can chain the CSS selector as such: soup.select(".row-fluid .span5 b")
. If there was any elements of the span5
class outside of the row-fluid
containers, it would be better to chain the CSS, making it more explicit.
get_names
is a fairly ambiguous function name that also suggests it will return an iterable with names. What you are doing in this function is printing names of lawyers, together with the firm they're working for. I'd suggest renaming it to print_lawyer_information
, or better yet get_lawyer_information
and return a dictionary containing the name
as key and title
as value.
lawyer_information = {}
for element in soup.select(".span5 b"):
name = element.text.strip()
if name.startswith('Area'):
continue
title = element.next_sibling.next_sibling.strip()
lawyer_information[name] = title
return lawyer_information
As you can see above, we're creating an empty dictionary and then, as we iterate over the elements, we append records to it. Once that is done, you just return the dictionary, which then you can print or do more things with. This can be done with a much neater dictionary comprehension.
return {row.text.strip(): row.next_sibling.next_sibling.strip()
for row in soup.select('.span5 b')
if not row.text.strip().startswith('Area')}
A couple other nitpicks would involve some PEP-8 (Python Style Guide) violations, such as lack of 2 lines before function definition and multiple statements on one line. Run your code through something like http://pep8online.com and you'll get a better idea of how to make your code easier to read and follow. You'll thank yourself in a years time when you look at your code again.
import requests
from bs4 import BeautifulSoup
def get_lawyer_information(url):
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')
return {element.text.strip(): element.next_sibling.next_sibling.strip()
for element in soup.select('.span5 b')
if not element.text.strip().startswith('Area')}
if __name__ == '__main__':
url = "https://www.otla.com/index.cfm?&m_firstname=a&seed=566485&fs_match=s&pg=publicdirectory&diraction=SearchResults&memPageNum=1"
lawyer_information = get_lawyer_information(url)
for name, title in lawyer_information.items():
print('{} | {}'.format(name, title))
try:except:continue
block is what I meant by that. However, i didn't know that a working script can also get downvote here. \$\endgroup\$