4
\$\begingroup\$

I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry.
Now, each job entry page is visited to grab info not present in the xml feed (company logo URL, salary info, etc.) through bs4 with couple functions in the get_so_extras module.

parse_feed.py:

from datetime import datetime as dt
from xml.etree import ElementTree as etree
from get_so_extras import get_company_logo, get_so_salary
import urllib3
import json
import sys


def get_so_listings(url):
    """
    Get remote job listings from stackoverflow.com xml feed
    """

    http = urllib3.PoolManager()

    try:
        so_file = http.request("GET", url).data
    except Exception as e:
        print(e.args)
        sys.exit()

    so_root = etree.fromstring(so_file)
    items = so_root.findall("channel/item")

    # Parse SO xml feed in dict
    so_listings = []
    for e in items:
        job_dict = {
            "so_id": e.findtext("guid"),
            "title": e.findtext("title"),
            "so_url": e.findtext("link"),
            "author": e.findtext(
                """{http://www.w3.org/2005/Atom}author/{http://www.w3.org/2005/Atom}name"""
            ),
            "description": e.findtext("description"),
            "published_at": str(
                dt.strptime(
                    e.findtext("pubDate").strip(), "%a, %d %b %Y %H:%M:%S Z"
                    )
            ),
            "updated_at": str(
                dt.strptime(
                    e.findtext("{http://www.w3.org/2005/Atom}updated"),
                    "%Y-%m-%dT%H:%M:%SZ",
                    )
            ),
            "company_logo_url": get_company_logo(e.findtext("link")),
            "categories": [c.text for c in e.findall("category")],
        }

        # @TODO: get_company_logo(job_dict["so_url"])
        so_listings.append(job_dict)

    print("Jobs imported: {}".format(len(so_listings)))

    # Helper - export to json, ideally pushed to db
    with open('output1.json', 'w', encoding='utf-8') as f:
        json.dump(so_listings, f, indent=4)

    return True

And here the helper function used to grab company logo URLs from get_so_extras module.

get_so_extras.py:

from bs4 import BeautifulSoup
import requests
import time


def get_company_logo(job_url):
    """
    Get company logo from stackoverflow.com job listing page
    """

    try:
        page = requests.get(job_url)
        soup = BeautifulSoup(page.text, 'html.parser')
        logo = soup.find("div", attrs={'class': 's-avatar s-avatar__lg mr8 bg-white fl-shrink0'}).\
            img["src"]
    except Exception as e:
        print(e)
        pass

    time.sleep(5) # Be kind

    return logo

It'd be great to understand whether I made any obvious mistake or bad choice along the way. Obviously calling get_company_logo within the XML parsing loop slows things down, what would be the best approach to get those fields?
Probably checking if we're looking at a new company/listing from db before visiting the page to scrape the logo would be more efficient.

edit: also, given job listings have pretty much similar attributes, could it make sense to create a class listing with attribute and a method to push it to db (with de-duplication logic perhaps)?

Be ruthless, thanks in advance.

\$\endgroup\$
5
  • \$\begingroup\$ I'm trying to familiarize myself with the feed etc., can't even get lxml to parse it --' Do you know if it's documented anywhere? \$\endgroup\$
    – AMC
    Dec 13 '19 at 0:36
  • \$\begingroup\$ Haven't found any docs either. The structure is pretty straightforward though: results are stored in <item> tags inside <channel>. Each listing has just a handful of fields, you can eyeball them to decide which you want to parse. \$\endgroup\$
    – anddt
    Dec 13 '19 at 9:23
  • \$\begingroup\$ I figured out the issue in the end. As you said, the rest should be straightforward, it’s rather simple format. \$\endgroup\$
    – AMC
    Dec 13 '19 at 9:53
  • \$\begingroup\$ What does your output look like, particularly for the description? \$\endgroup\$
    – AMC
    Dec 14 '19 at 0:22
  • \$\begingroup\$ My output is a list of dictionaries (so_listings) where a dict represents a job entry. I've ditched the JSON output in favour of pushing the entry to a MySQL db through SQLAlchemy after checking for duplicates though. \$\endgroup\$
    – anddt
    Dec 14 '19 at 13:16
1
\$\begingroup\$

I've been experimenting with the best data structure for this. Here is what I have for now. It is definitely a work in progress, I hope to keep updating it constantly.

The CSV output was unexpected, ultimately a result of my decision to use a namedtuple. I chose namedtuples because, like you, I thought the data fit that tabular style quite well. You wrote given job listings have pretty much similar attributes, could it make sense to create a class listing with attribute and a method to push it to db, and I mostly agree, I just didn't feel that the data itself warranted a whole class.

import collections as colls
import csv
import datetime as dt

import requests
from lxml import etree

Job = colls.namedtuple('Job', ['guid', 'title', 'url', 'author', 'logo', 'description', 'pub_date', 'update_date',
                               'categories'])

html_parser = etree.HTMLParser()

req_session = requests.Session()
req = req_session.get(url_1)

root = etree.fromstring(req.content)
ns_map = root.nsmap

job_elems = root.xpath('/rss/channel/item')[:10]


def parse_job_item(job_item):
    guid = job_item.findtext('guid')
    title = job_item.findtext('title')
    link = job_item.findtext('link')
    author = job_item.findtext('a10:author/a10:name', namespaces=ns_map)
    description = job_item.findtext('description')
    pub_date = job_item.findtext('pubDate')
    if pub_date:
        pub_date = dt.datetime.strptime(pub_date, "%a, %d %b %Y %H:%M:%S Z")
    update_date = job_item.findtext('a10:updated', namespaces=ns_map)
    if update_date:
        update_date = dt.datetime.strptime(update_date, "%Y-%m-%dT%H:%M:%SZ")
    categories = [elem.text for elem in job_item.findall('category')]

    job_page_elem = etree.fromstring(req_session.get(link).content, parser=html_parser)
    # "//body/div[@class='container']/div[@id='content']/header/div/a/img/@src"
    # contains(concat(' ', normalize-space(@class), ' '), ' s-avatar ')
    company_logo = job_page_elem.xpath(
        "/html/body/div[@class='container']/div[@id='content']/header/div[contains(concat(' ', normalize-space(@class), ' '), ' s-avatar ')]//img/@src")
    if company_logo:
        company_logo = company_logo[0]
    res_item = Job(guid, title, link, author, company_logo, description, pub_date, update_date, categories)
    return res_item


job_data_list = (parse_job_item(curr_elem) for curr_elem in job_elems)

with open('../out/jobs_out.csv', 'w', newline='') as out_file:
    writer = csv.writer(out_file)
    writer.writerow(Job._fields)
    writer.writerows(job_data_list)
\$\endgroup\$
4
  • \$\begingroup\$ All in all both approaches look quite similar, except for the namedtuple. The only reason I would consider setting up a class would be to structure things a bit more if starting to get job entries from additional sources. \$\endgroup\$
    – anddt
    Dec 14 '19 at 21:00
  • 1
    \$\begingroup\$ @AndreaDodet Yeah, that would make sense. I'm currently trying to to think of the best way to integrate some form of concurrency/parallelism. \$\endgroup\$
    – AMC
    Dec 14 '19 at 23:20
  • \$\begingroup\$ I guess my approach should change to add parallelism: 1. Parse entire xml. 2. Check if new entries are present (threaded). 3. Gather data for new entries 4. Push to db. I realize I am checkig for duplicates way too late, it basically just spares the push_to_db()call where the real advantage would be to cut the website requests for additional info. \$\endgroup\$
    – anddt
    Dec 16 '19 at 8:22
  • \$\begingroup\$ @AndreaDodet Yes, check for duplicates as early as possible. I'm going to go see what multiprocessing has to say about sharing data like this. \$\endgroup\$
    – AMC
    Dec 16 '19 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.