This is a project I came up just as an exercise with to familiarize myself with Python syntax and data types. I am just learning how to code and would like to avoid establishing bad habits. On the other hand, I wanted this program to function so when I found something that worked, I left it as is, even if I suspected that there might be a better way. Any suggestions on how to improve the code would be greatly appreciated.

import feedparser
import nltk
import random

# idea is to scrape headlines from RSS feed and make new headlines 
# using words from original headlines and mimicking the syntax of original headlines 

#scrape titles into scrptitles list

data = feedparser.parse('http://rss.cnn.com/rss/cnn_topstories.rss')
for entry in data.entries:
   data = entry.title

#tokenize and tag

toks = [(nltk.pos_tag(nltk.word_tokenize(title))) for title in scrptitles]

# create list of dictionaries from list of tuples
# [{'Word':'TAG',...headline structure...},]

d_list = [dict(toks[1]) for toks[1] in toks]

# remove unwanted characters from d_list ['``',"''",'>','.',',','']

nonetype_list = ['``',"''","'",'>',',','.','|','(',')','-   ','_','=','+','<','@','#','$',"%",'^','*','[',']','{','}',":",";"]

for dicts in d_list:
    for item in nonetype_list:
        if item in dicts:
            del dicts[item]    

    # create list of tags perserving headline structure

tags_list =  [dict.values() for dict in d_list]

    # get a clean version of the tags list

tags_list = [[element for element in dict_values] for dict_values in tags_list]

    # reverse dictionaries inside of list of dictionaries 
    # inv_map = {TAG: [list of words]}

inv_map = {}
for dct in d_list:
    for k, v in dct.items():
        inv_map[v] = inv_map.get(v,[])

    # grab a headline_pattern, choose random words from dictionary values, 
    # print out new headline

headline_pat = (random.choice(tags_list))

new_headline = [random.choice(inv_map[item]) for item in headline_pat]

for item in new_headline:
   print(item.upper(), end=' ')

  • 1
    \$\begingroup\$ This looks like a bug. Is it? for entry in data.entries: ; data = entry.title ; scrptitles.append(data) (the scrptitles is not indented in your code) \$\endgroup\$ – Austin Hastings Feb 19 '19 at 23:20
  • \$\begingroup\$ Correct, that should be indented. \$\endgroup\$ – GCM Feb 19 '19 at 23:48
  • 1
    \$\begingroup\$ I don't think [dict(toks[1]) for toks[1] in toks] does what you think it does. You first two entries of the resulting list are actually the same. If you want to make every entry into a dictionary, use [dict(t) for t in toks] or list(map(dict, toks)). \$\endgroup\$ – Graipher Feb 20 '19 at 10:14
  • 1
    \$\begingroup\$ Yes, I had not noticed that. The expression as is duplicates the first entry and eliminates the second. Your suggestions work much better. Thank you. \$\endgroup\$ – GCM Feb 20 '19 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.