It's been nearly a year I've been working with C++ and now I am diving in Python and its libraries and frameworks. I am currently creating a website for my portfolio that gathers news articles from a wide range of news websites. I am using Django framework for web development and the Beautiful Soup library for web scraping.
This is currently my Django views.py file, where all the web scraping happens:
from django.shortcuts import render
import urllib.request
from bs4 import BeautifulSoup
import re
# More websites will be added below.
URLS = ["https://www.theguardian.com/uk/rss", "http://rss.cnn.com/rss/edition.rss"];
class Brick:
"""
Class to represent each article in a news website. 1 brick = 1 news article.
"""
def __init__(self, title="Untitled", link="#", description="No description",
img_url="#", creator="No creator", date="No Date"):
self.title = title;
self.link = link;
self.description = description;
self.img_url = img_url;
self.creator = creator;
self.date = date;
def strip_html(raw_html):
"""
Strips any HTML tags within a string.
(I know that this is not safe, but I would like to practice regular expressions).
"""
pattern = re.compile(r"<.*?>");
return pattern.sub("", raw_html);
def shorten(cleaned_html):
"""
If the description of an article is too long, let's cut it
to only 50 characters.
"""
new_string = "";
index = 0;
for c in cleaned_html:
if index < 100:
new_string += cleaned_html[index];
index += 1;
new_string += "...";
return new_string;
# Create your views here.
def news_list(request):
"""
View that will redirect to the home template that shows the full list
of articles.
"""
unformatted_xmls = [];
formatted_xmls = [];
group_of_items = [];
for URL in URLS:
with urllib.request.urlopen(URL) as file_object:
unformatted_xmls.append(file_object.read());
for unf_xml in unformatted_xmls:
formatted_xmls.append(BeautifulSoup(unf_xml, "xml"));
for form_xml in formatted_xmls:
group_of_items.append(form_xml.find_all("item"));
bricks = [];
for item_group in group_of_items:
for item in item_group:
b = Brick();
b.title = item.find("title").get_text();
b.link = item.find("link").get_text();
b.description = shorten(strip_html(item.find("description").get_text())) if item.find("description") is not None else "No description.";
b.img_url = item.find("media:content").get("url") if item.find("media:content") is not None else "No media.";
b.creator = item.find("dc:creator").get_text() if item.find("dc:creator") is not None else "No creator.";
b.date = item.find("dc:date").get_text() if item.find("dc:date") is not None else "No date.";
bricks.append(b);
return render(request, "news/list.html", {"bricks": bricks});
As I am new to web scraping and web development in general, would you mind giving me your tips and opinions about this code? Would you recommend any good practices? Are these libraries and frameworks strongly recommended for this purpose?