My name is Ethan and I am trying to build an API for scaping technical papers to use for developers. Right now it only works for ArXiV but I would greatly appreciate some mentoring or a code review of my repo. I am a new developer and want to get my code to professional quality.
Source provided as well:
from falcon import API from urllib import request from bs4 import BeautifulSoup class ArXivPully: # Removes rogue newline characters from the title and abstract def cleanText(self,text): return ' '.join(text.split('\n')) def pullFromArXiv(self,search_query, num_results=10): # Fix Input if it has spaces in it split_query = search_query.split(' ') if(len(split_query) > 1): search_query = '%20'.join(split_query) url = 'https://export.arxiv.org/api/query?search_query=all:'+search_query+'&start=0&max_results='+str(num_results) data = request.urlopen(url).read() output =  soup = BeautifulSoup(data, 'html.parser') titles = soup.find_all('title') # ArXiv populates the first title value as the search query titles.pop(0) bodies = soup.find_all('summary') links = soup.find_all('link', title='pdf') for i in range(len(titles)): title = self.cleanText(titles[i].text.strip()) body = self.cleanText(bodies[i].text.strip()) pdf_link = links[i]['href'] output.append([pdf_link, title, body]) return output def on_get(self, req, resp): """Handles GET requests""" output =  for item in req.params.items(): output.append(self.pullFromArXiv(item,item)) resp.media = output api = API() api.add_route('/api/query', ArXivPully())
Some design explanations. I run this API through Google Cloud Platform using Falcon API because both options are free for me and were the simplest to implement. Some known issues are already posted in the repo but I want to better understand Software Development skills, best practices, etc. I greatly appreciate any tips big or small and I look forward to drastically changing this source code to make it more robust.