Simple RSS to HTML converter

Question

Is this code pythonic, easy to read, stable, or missing something?

# -*- coding: utf-8 -*-
""" Simple rss to html converter """

__version__ = "0.0.1"
__author__ = "Ricky L Wilson"

import StringIO
from feedparser import parse as parsefeed
from bs4 import BeautifulSoup as bs
from unicodedata import normalize


def flatten_unicode_keys(dic):
    '''pass unicode keywords to **kwargs '''
    for key in dic.keys():
        if isinstance(key, unicode):
            value = dic[key]
            del dic[key]
            dic[normalize('NFKD',key).encode('ascii','ignore')] = value


def entry2html(**kwargs):
    """ Format feedparser entry """
    flatten_unicode_keys(kwargs)
    title = kwargs['title']
    link = kwargs['link']
    description = kwargs['description']
    template = u"""
    <h2 class='title'>{title}</h2>
    <a class='link' href='{link}'>{title}</a>
    <span class='description'>{description}</span>
    """
    return template.format(title=title, link=link, description=description).encode('utf-8')


def convert_feed(**kwargs):
    """ Main loop """
    flatten_unicode_keys(kwargs)
    out = StringIO.StringIO("")
    for entry in parsefeed(kwargs['url']).entries:
        title = entry['title']
        link = entry['link']
        description = entry['description']
        print >> out, entry2html(title=title, link=link, description=description)
    return bs(out.getvalue(), 'lxml').prettify()


def save_file(url, fname):
    ''' Save data to disc'''
    with open(fname, 'w') as file_object:
        file_object.write(convert_feed(url=url).encode('utf-8'))

save_file('http://stackoverflow.com/feeds', 'index.html')

Answer 1

The small improvement I would like to suggest you is related to performance in terms of speed.

In flatten_unicode_keys(dic), you use for key in dic.keys(): which is less fast than for key in dic:. This is mentioned in PEP 234:

• Dictionaries implement a tp_iter slot that returns an efficient iterator that iterates over the keys of the dictionary. [...] This means that we can write

  for k in dict: ...
  

  which is equivalent to, but much faster than

  for k in dict.keys(): ...
  

An other side note: it is a better coding habit not to encode the container type into the name. So instead, for example, of typing dic, use a name which actually better describes what that dictionary consists of.

Answer 2

Overall, the code is readable and Pythonic. Some minor improvements:

• you can switch from stringIO to cStringIO to gain some speed improvements:

  try:
      from cStringIO import StringIO
  except ImportError:
      from StringIO import StringIO
  

• I would define template as a high-level proper constant outside of the function:

  TEMPLATE = u"""
  <h2 class='title'>{title}</h2>
  <a class='link' href='{link}'>{title}</a>
  <span class='description'>{description}</span>
  """
  

• when you loop over the entries, you don't have to define separate title, link and description variables - you can unpack the entry into the entry2html arguments:

  for entry in parsefeed(kwargs['url']).entries:
      print >> out, entry2html(**entry)
  

• similarly, you can pass the kwargs directly to the .format() call:

  return template.format(**kwargs).encode('utf-8')
  

• organize imports according to PEP8 guide

• define docstrings according to the PEP8 guide - triple quotes, starts with a capital letter, ends with a dot
• dic is probably not the best variable name

Here is the modified code with the above and other improvements applied:

# -*- coding: utf-8 -*-
"""Simple RSS to HTML converter."""

__version__ = "0.0.1"
__author__ = "Ricky L Wilson"

from unicodedata import normalize

from bs4 import BeautifulSoup
from feedparser import parse as parse_feed


TEMPLATE = u"""
<h2 class='title'>{title}</h2>
<a class='link' href='{link}'>{title}</a>
<span class='description'>{summary}</span>
"""


def flatten_unicode_keys(entry_properties):
    """Ensures passing unicode keywords to **kwargs."""
    for key in entry_properties:
        if isinstance(key, unicode):
            value = entry_properties[key]
            del entry_properties[key]
            entry_properties[normalize('NFKD', key).encode('ascii', 'ignore')] = value


def entry_to_html(**kwargs):
    """Formats feedparser entry."""
    flatten_unicode_keys(kwargs)
    return TEMPLATE.format(**kwargs).encode('utf-8')


def convert_feed(url):
    """Main loop."""
    html_fragments = [entry_to_html(**entry) for entry in parse_feed(url).entries]
    return BeautifulSoup("\n".join(html_fragments), 'lxml').prettify()


def save_file(url, filename):
    """Saves data to disc."""
    with open(filename, 'w') as file_object:
        file_object.write(convert_feed(url).encode('utf-8'))


if __name__ == '__main__':
    save_file('http://stackoverflow.com/feeds', 'index.html')

Note that I had to replace {description} placeholder with {summary} to test that it is working.

Note that I've avoided using an output buffer altogether - just collected feed HTML fragments into a list and then joined.

---

Overall, there are some other things I would also look into - for instance, it really looks like you can make use of a template engine like mako or jinja2 - define a template with a loop over the feeds and then render it. Example using mako:

from feedparser import parse as parse_feed

from mako.template import Template


def convert_feed(url, filename):
    """Convert feed to an HTML."""
    with open(filename, 'w') as file_object:
        feeds = parse_feed(url).entries
        html_content = Template(filename='template.html', output_encoding='utf-8').render(feeds=feeds)
        file_object.write(html_content)


if __name__ == '__main__':
    convert_feed('http://stackoverflow.com/feeds', 'index.html')

where template.html contains:

<html>
    <body>
        % for feed in feeds:
            <div>
                <h2 class='title'>${feed.title}</h2>
                <a class='link' href='${feed.link}'>${feed.title}</a>
                <span class='description'>${feed.summary}</span>
            </div>
        % endfor
    </body>
</html>

Answer 3

Well here is the new script. I eliminated the flatten_unicode_keys() function and made most the improvements that alecxe and Billal BEGUERADJ suggested i make. i couldn't figure out how to integrate mako into the script and i would really like to.

# -*- coding: utf-8 -*-
"""Simple RSS to HTML converter."""

__version__ = "0.0.2"
__author__ = "Ricky L Wilson"

from bs4 import BeautifulSoup
from feedparser import parse as parse_feed


TEMPLATE = u"""
<h2 class='title'>{title}</h2>
<a class='link' href='{link}'>{title}</a>
<span class='description'>{summary}</span>
"""

def entry_to_html(**kwargs):
    """Formats feedparser entry."""
    return TEMPLATE.format(**kwargs).encode('utf-8')


def convert_feed(url):
    """Main loop."""
    html_fragments = [entry_to_html(**entry) for entry in parse_feed(url).entries]
    return BeautifulSoup("\n".join(html_fragments), 'lxml').prettify()


def save_file(url, filename):
    """Saves data to disc."""
    with open(filename, 'w') as file_object:
        file_object.write(convert_feed(url).encode('utf-8'))


if __name__ == '__main__':
    save_file('http://stackoverflow.com/feeds', 'index.html')
    with open('index.html') as fobj:
        print fobj.read()