ArXiV Technical Paper API Github Repo

Question

My name is Ethan and I am trying to build an API for scaping technical papers to use for developers. Right now it only works for ArXiV but I would greatly appreciate some mentoring or a code review of my repo. I am a new developer and want to get my code to professional quality.

Repo: https://github.com/evader110/ArXivPully

Source provided as well:

from falcon import API
from urllib import request
from bs4 import BeautifulSoup

class ArXivPully:
    # Removes rogue newline characters from the title and abstract
    def cleanText(self,text):
        return ' '.join(text.split('\n'))

    def pullFromArXiv(self,search_query, num_results=10):
        # Fix Input if it has spaces in it
        split_query = search_query.split(' ')
        if(len(split_query) > 1):
            search_query = '%20'.join(split_query)
        url = 'https://export.arxiv.org/api/query?search_query=all:'+search_query+'&start=0&max_results='+str(num_results)
        data = request.urlopen(url).read()
        output = []
        soup = BeautifulSoup(data, 'html.parser')
        titles = soup.find_all('title')

        # ArXiv populates the first title value as the search query
        titles.pop(0)

        bodies = soup.find_all('summary')
        links = soup.find_all('link', title='pdf')
        for i in range(len(titles)):
            title = self.cleanText(titles[i].text.strip())
            body = self.cleanText(bodies[i].text.strip())
            pdf_link = links[i]['href']
            output.append([pdf_link, title, body])
        return output

    def on_get(self, req, resp):
        """Handles GET requests"""
        output = []
        for item in req.params.items():
            output.append(self.pullFromArXiv(item[0],item[1]))
        resp.media = output

api = API()
api.add_route('/api/query', ArXivPully())

Some design explanations. I run this API through Google Cloud Platform using Falcon API because both options are free for me and were the simplest to implement. Some known issues are already posted in the repo but I want to better understand Software Development skills, best practices, etc. I greatly appreciate any tips big or small and I look forward to drastically changing this source code to make it more robust.

Answer 1

• Python has an official style-guide, PEP8. It recommends using lower_case for variables, functions and methods.

• I would encourage you to use requests.get instead of urllib.request. It can take care of urlencoding the parameters for you.

• You can make pull_from_arxiv a generator to save a few lines.

• BeautifulSoup can be sped up using the lxml parser.

• on_get can be simplified a bit using a list comprehension.

• Not sure if your cleanText is really needed. Anyways, I would use str.replace instead of str.split and str.join.

import requests
from bs4 import BeautifulSoup

class ArXivPully:
    def pull_from_arxiv(self, search_query, num_results=10):
        url = "https://export.arxiv.org/api/query"
        params = {"search_query": f"all:{search_query}",
                  "start": 0,
                  "max_results": num_results}
        data = requests.get(url, params=params).text
        soup = BeautifulSoup(data, 'lxml')
        # ArXiv populates the first title value as the search query
        titles = soup.find_all('title')[1:] 
        bodies = soup.find_all('summary')
        links = soup.find_all('link', title='pdf')

        for title, body, link in zip(titles, bodies, links):
            yield (link['href'],
                   title.text.strip().replace("\n", " "),
                   body.text.strip().replace("\n", " "))

    def on_get(self, req, resp):
        """Handles GET requests"""
        resp.media = [list(self.pull_from_arxiv(*item))
                      for item in req.params.items()]

Side note: Using this returns completely different results compared to entering the search string on the arxiv website search field. Not sure why. Same is true for your queries, though (the only difference is that gets encoded as + and : as %3a by requests.get).