My name is Ethan and I am trying to build an API for scaping technical papers to use for developers. Right now it only works for ArXiV but I would greatly appreciate some mentoring or a code review of my repo. I am a new developer and want to get my code to professional quality.

Repo: https://github.com/evader110/ArXivPully

Source provided as well:

from falcon import API
from urllib import request
from bs4 import BeautifulSoup

class ArXivPully:
    # Removes rogue newline characters from the title and abstract
    def cleanText(self,text):
        return ' '.join(text.split('\n'))

    def pullFromArXiv(self,search_query, num_results=10):
        # Fix Input if it has spaces in it
        split_query = search_query.split(' ')
        if(len(split_query) > 1):
            search_query = '%20'.join(split_query)
        url = 'https://export.arxiv.org/api/query?search_query=all:'+search_query+'&start=0&max_results='+str(num_results)
        data = request.urlopen(url).read()
        output = []
        soup = BeautifulSoup(data, 'html.parser')
        titles = soup.find_all('title')

        # ArXiv populates the first title value as the search query

        bodies = soup.find_all('summary')
        links = soup.find_all('link', title='pdf')
        for i in range(len(titles)):
            title = self.cleanText(titles[i].text.strip())
            body = self.cleanText(bodies[i].text.strip())
            pdf_link = links[i]['href']
            output.append([pdf_link, title, body])
        return output

    def on_get(self, req, resp):
        """Handles GET requests"""
        output = []
        for item in req.params.items():
        resp.media = output

api = API()
api.add_route('/api/query', ArXivPully())

Some design explanations. I run this API through Google Cloud Platform using Falcon API because both options are free for me and were the simplest to implement. Some known issues are already posted in the repo but I want to better understand Software Development skills, best practices, etc. I greatly appreciate any tips big or small and I look forward to drastically changing this source code to make it more robust.


1 Answer 1

  • Python has an official style-guide, PEP8. It recommends using lower_case for variables, functions and methods.

  • I would encourage you to use requests.get instead of urllib.request. It can take care of urlencoding the parameters for you.

  • You can make pull_from_arxiv a generator to save a few lines.

  • BeautifulSoup can be sped up using the lxml parser.

  • on_get can be simplified a bit using a list comprehension.

  • Not sure if your cleanText is really needed. Anyways, I would use str.replace instead of str.split and str.join.

import requests
from bs4 import BeautifulSoup

class ArXivPully:
    def pull_from_arxiv(self, search_query, num_results=10):
        url = "https://export.arxiv.org/api/query"
        params = {"search_query": f"all:{search_query}",
                  "start": 0,
                  "max_results": num_results}
        data = requests.get(url, params=params).text
        soup = BeautifulSoup(data, 'lxml')
        # ArXiv populates the first title value as the search query
        titles = soup.find_all('title')[1:] 
        bodies = soup.find_all('summary')
        links = soup.find_all('link', title='pdf')

        for title, body, link in zip(titles, bodies, links):
            yield (link['href'],
                   title.text.strip().replace("\n", " "),
                   body.text.strip().replace("\n", " "))

    def on_get(self, req, resp):
        """Handles GET requests"""
        resp.media = [list(self.pull_from_arxiv(*item))
                      for item in req.params.items()]

Side note: Using this returns completely different results compared to entering the search string on the arxiv website search field. Not sure why. Same is true for your queries, though (the only difference is that gets encoded as + and : as %3a by requests.get).

  • \$\begingroup\$ I forgot about PEP8 I'll get to reading that. So clean text exists only because I want to add more logic for cleaning LaTeX and other formats. It's not needed in its current form but as a placeholder it will act as a point to build off of. Is it better to yield in this instance? Should it be an option for the developers to ask for batches or articles 1 at time? \$\endgroup\$
    – evader110
    Apr 15, 2019 at 18:24
  • 1
    \$\begingroup\$ @evader110: A generator has the nice feature that it is entirely up to the user of the generator how many objects they get. If they want ten, they can do itertools.islice(generator_object, 10). If they want only one, they can do next(generator_object). If they want all, they can do for x in generator_object or list(generator_object). You could even make the method an infinite generator ( well, infinite until the end of the search results...), that requests the result in batches (of 10 or whatever) and yields them one by one to the user. \$\endgroup\$
    – Graipher
    Apr 15, 2019 at 21:05
  • \$\begingroup\$ Oh snap that makes a lot of sense actually \$\endgroup\$
    – evader110
    Apr 16, 2019 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.