# Process text file of fulltext articles from news publications

I'm relatively new to programming and have started writing some code for work. This script is something I wrote that I'd like to have someone comment on. Am I doing anything silly, stupid or totally unnecessary? Am I making things more complicated than they need to be? This script takes a plain text file downloaded from a publication database (example file below) and processes it to create a csv of the metadata from all the articles and a separate text file for the full text of each article.

from sys import argv
import time
import csv

timestamp = time.strftime('%H%M-%Y%m%d')

class Article(object):
fulltxt = ""
Title = ""
Publication = ""
Date = ""
Year = ""
Author = ""
PQID = ""

fields = ['ID', 'Title', 'Author', 'Publication', 'Date', 'Year']
csvFile = open(csvName, 'wb')
csvwriter = csv.DictWriter(csvFile, delimiter=',', fieldnames=fields)
csvwriter.writerow(dict((fn, fn) for fn in fields))

def abbreviate(pub):
#a publication abbreviation is included in the filename of the textfile
pubs = {
'USA TODAY (pre-1997 Fulltext)' : 'USAT',
'USA TODAY' : 'USAT',
'New York Times' : 'NYT',
'Wall Street Journal' : 'WSJ'
}
return pubs.get(pub, "X")

#write the field values of the current doc to the csv file
if doc.fulltxt != "":
writefields = dict((field, getattr(doc, field)) for field in vars(doc) if not field.startswith('__') and field != 'fulltxt')
csvwriter.writerow(writefields)

def writeArticle(doc):
name = abbreviate(doc.Publication)+'_'+doc.Year+'_'+doc.PQID+'.txt'
txtFile = open(name, 'a')
txtFile.write(doc.Title)
txtFile.write(doc.fulltxt)
txtFile.close()

script, filename = argv
docs = open(filename)

for line in docs:
line = line.strip()
if line == '____________________________________________________________':
#write out the previous doc
try: doc
except NameError: doc = None
else:
writeArticle(doc)
#start new doc
doc = Article()
continue
if line.startswith("Full text:"):
doc.fulltxt = line[10:]
continue
if line.startswith("Illustration") or line == "" :
continue
else:
doc.fulltxt = doc.fulltxt + "\n" +line
elif line.startswith("Title:"):
doc.Title = line[6:]
elif line.startswith("Publication title:"):
doc.Publication = line[19:]
elif line.startswith("Author:"):
doc.Author = line[8:]
elif line.startswith("Publication date:"):
doc.Date = line[18:]
elif line.startswith("ProQuest document ID:"):
doc.PQID = line[22:]
elif line.startswith("Publication year:"):
doc.Year = line[18:]
elif not line or line.isspace() :
continue
else:
continue

csvFile.close()
docs.close()


example text file:

____________________________________________________________

Report Information from ProQuest
August 21 2015 10:55
____________________________________________________________

____________________________________________________________

1. Obama plan: 'Pitiful' or 'compelling'?

2. Feeling bullied at school? Call Charlie's angel

3. Clinton proposes setting national standards for schools

4. A NATION'S HUNGER FOR THE TRUTH Irish confront legacy of the potato famine

5. Minnesota's muscle drawing stronger praise Foes intimidated by No. 11 Gophers

____________________________________________________________

Document 1 of 5

Obama plan: 'Pitiful' or 'compelling'?

Author: Anonymous

http://search.proquest.com/docview/862079852?accountid=7379

Abstract: The New York Times, in an editorial: (Obama) used his budget speech to clearly distance himself from Republican plans to heap tax benefits on the rich while casting adrift the nation's poor, elderly and unemployed. Karl Rove, former adviser to President Bush, in The Wall Street Journal: The goal Obama laid out -- 'a balanced approach to achieve \$4 trillion in deficit reduction over 12 years' -- was missing but one thing: a concrete blueprint. ...

Full text: On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
Illustration PHOTO, B/W, Carolyn Kaster, AP; Caption:

Subject: Studies; Tax increases; Fiscal policy; Government spending; Federal budget; Editorials

Location: United States--US

People: Obama, Barack

Title: Obama plan: 'Pitiful' or 'compelling'?

Publication title: USA TODAY

Pages: A.9

Publication year: 2011

Publication date: Apr 15, 2011

Year: 2011

Section: NEWS

Publisher: USA Today, a division of Gannett Satellite Information Network, Inc.

Place of publication: McLean, Va.

Country of publication: United States

Publication subject: General Interest Periodicals--United States

ISSN: 07347456

Source type: Newspapers

Language of publication: English

Document type: Feature

ProQuest document ID: 862079852

Document URL: http://search.proquest.com/docview/862079852?accountid=7379

Last updated: 2011-05-04

Database: National Newspapers Expanded

____________________________________________________________

Document 2 of 5

Feeling bullied at school? Call Charlie's angel

Author: Parker, Kathleen

http://search.proquest.com/docview/306803544?accountid=7379

Abstract: Some days nothing goes right. I was feeling particularly homicidal recently following a full day of rude encounters with my fellow woman. In one instance, a teen driver dodged in front of me, snatching the last parking space at the grocery store. In another, an elaborately coifed woman leaned on her horn as I waited for oncoming traffic to permit a sane left turn. And in yet another, get this, a woman butted in line in the emergency room (!) where I'd taken by kid for stitches on his split chin. "Attorney General Condon's No Bully Line," said an angel's voice with a perfect blend of authority and compassion. I began to tell her about my urgent need to kill mean people when she asked me the name of my school. School? I'm a grown-up, a parent. I don't go to school; I go to grocery stores! Now about these bullies. She stopped me, sweetly. The bully hot line, she explained, was for playground bullies, not horn-blowers and line-butters. Attorney General Charles "Charlie" Condon, weary of crime statistics, apparently had decided to take the bullies by the horn. Well, why not? when the president of the United States cares (deeply) about women recovering from labor and delivery, why shouldn't an attorney general sift the sandbox?

Full text: Not available.

Title: Feeling bullied at school? Call Charlie's angel

Publication title: USA TODAY (pre-1997 Fulltext)

Pages: 11.A

Number of pages: 0

Publication year: 1997

Publication date: Feb 12, 1997

Year: 1997

Section: NEWS

Publisher: USA Today, a division of Gannett Satellite Information Network, Inc.

Place of publication: McLean, Va.

Country of publication: United States

Publication subject: General Interest Periodicals--United States

ISSN: 07347456

Source type: Newspapers

Language of publication: English

Document type: NEWSPAPER

ProQuest document ID: 306803544

Document URL: http://search.proquest.com/docview/306803544?accountid=7379

Last updated: 2010-08-27

Database: National Newspapers Expanded

____________________________________________________________

Document 3 of 5

Clinton proposes setting national standards for schools

Author: Kelly, Dennis

http://search.proquest.com/docview/306802529?accountid=7379

Abstract: The national tests that President Clinton proposed Tuesday night would give states and local school districts a new yardstick to see how their students stack up against the rest of the country and the world. Clinton is proposing that states adopt newly developed, voluntary national tests for fourth-graders in reading and for eighth-graders in math. The tests would be administered begining in 1999. The reading test would be based on the National Assessment of Educational Progress (NAEP), a federal project that has used a random sample of U.S. students to gauge skills in math, reading and other subjects for 27 years.

Full text: On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
Illustration

Title: Clinton proposes setting national standards for schools:   [FINAL Edition]

Publication title: USA TODAY (pre-1997 Fulltext)

Pages: 03.A

Number of pages: 0

Publication year: 1997

Publication date: Feb 5, 1997

Year: 1997

Section: NEWS

Publisher: USA Today, a division of Gannett Satellite Information Network, Inc.

Place of publication: McLean, Va.

Country of publication: United States

Publication subject: General Interest Periodicals--United States

ISSN: 07347456

Source type: Newspapers

Language of publication: English

Document type: NEWSPAPER

ProQuest document ID: 306802529

Document URL: http://search.proquest.com/docview/306802529?accountid=7379

Last updated: 2010-08-20

Database: National Newspapers Expanded

____________________________________________________________

Document 4 of 5

A NATION'S HUNGER FOR THE TRUTH Irish confront legacy of the potato famine

Author: Lynch, David J

http://search.proquest.com/docview/306802422?accountid=7379

Abstract: LONGFORD, Ireland -- For years, the hilltop cemetery known as Bully's Acre was a debris-strewn ruin. Windswept weeds grew waist high. Dead dogs and empty beer kegs littered the ground. But that was before Sister Calasanctius Duffy came along and before Ireland decided to confront one of the most painful episodes in its long, anguished history: the 1845-1847 famine that killed or forced into exile one-quarter of its 8 million residents. This year, 60,000 people, up one-third from 1996, are expected to visit the Famine Museum in Strokestown, where the tragedy's grim statistics are detailed in words and pictures. Sites like Bully's Acre are being rehabilitated. New research is under way by scholars, with more than 50 books on the famine published since 1994.

Full text: On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
Illustration PHOTOS, Color,Eileen Blass,USA TODAY (3)

Title: A NATION'S HUNGER FOR THE TRUTH Irish confront legacy of the potato famine:   [FINAL Edition]

Publication title: USA TODAY (pre-1997 Fulltext)

Pages: 01.D

Number of pages: 0

Publication year: 1997

Publication date: Jan 15, 1997

Year: 1997

Section: LIFE

Publisher: USA Today, a division of Gannett Satellite Information Network, Inc.

Place of publication: McLean, Va.

Country of publication: United States

Publication subject: General Interest Periodicals--United States

ISSN: 07347456

Source type: Newspapers

Language of publication: English

Document type: COVER STORY

ProQuest document ID: 306802422

Document URL: http://search.proquest.com/docview/306802422?accountid=7379

Last updated: 2010-07-30

Database: National Newspapers Expanded

____________________________________________________________

Document 5 of 5

Minnesota's muscle drawing stronger praise Foes intimidated by No. 11 Gophers

Author: Blauvelt, Harry

http://search.proquest.com/docview/306790419?accountid=7379

Abstract: Make room for brawny Minnesota, the newest bully on the Big Ten basketball block. The No. 11 Golden Gophers (13-1) haven't broken from the starting gate this quickly since 1976-77, when Kevin McHale, Mychal Thompson and Flip Saunders triggered a 16-1 run. In a conference known for its muscle, Minnesota can mix it up with anyone. Center John Thomas, 6-9, 275, a tight end in high school who squats 600 pounds, and power forward Courtney James, 6-8, 275, punish opponents in the paint.

Full text: On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains
Illustration PHOTO, B/W, Dale Atkins, AP

Title: Minnesota's muscle drawing stronger praise Foes intimidated by No. 11 Gophers:   [FINAL Edition]

Publication title: USA TODAY (pre-1997 Fulltext)

Pages: 09.C

Number of pages: 0

Publication year: 1997

Publication date: Jan 8, 1997

Year: 1997

Section: SPORTS

Publisher: USA Today, a division of Gannett Satellite Information Network, Inc.

Place of publication: McLean, Va.

Country of publication: United States

Publication subject: General Interest Periodicals--United States

ISSN: 07347456

Source type: Newspapers

Language of publication: English

Document type: NEWSPAPER

ProQuest document ID: 306790419

Document URL: http://search.proquest.com/docview/306790419?accountid=7379

Last updated: 2010-08-21

Database: National Newspapers Expanded

____________________________________________________________

Terms and Conditions: http://search.proquest.com/info/termsAndConditions


Constants are CAPITALas a convention, so:

TIMESTAMP = time.strftime('%H%M-%Y%m%d')
CSV_NAME = 'Metadata_' + timestamp + '.csv' # <- added some spaces
FIELDS = ['ID', 'Title', 'Author', 'Publication', 'Date', 'Year']


elif line.startswith("Title:"):
doc.Title = line[6:]
elif line.startswith("Publication title:"):
doc.Publication = line[19:]
elif line.startswith("Author:"):
doc.Author = line[8:]
elif line.startswith("Publication date:"):
doc.Date = line[18:]
elif line.startswith("ProQuest document ID:"):
doc.PQID = line[22:]
elif line.startswith("Publication year:"):
doc.Year = line[18:]


^^ That was a real puzzler, I know understood that the slicing is there to avoid saving things like "ProQuest document ID:" or "Publication year:", I suggest just using split and [1], like this:

elif line.startswith("Publication year:"):
doc.Year = line.split("Publication year:")[1]


The same result of what you did but more intuitive

In fact you should use a loop:

for salient_part in ("Title", "Publication Title:" ...):
if line.startswith(salient_part):
doc[salient_part] = line.split(salient_part)[1]


^ The loop above may need other modifications to your code before working properly (not tested).

Possibly, you may stop using the

class Article(object):
fulltxt = ""
Title = ""
Publication = ""
Date = ""
Year = ""
Author = ""
ID = ""


And just go for a plain dict, that sure would make the loop simple to implement.

Use with

txtFile = open(name, 'a')
txtFile.write(Article.Title)
txtFile.write(Article.fulltxt)
txtFile.close()


Should become

with open(name, 'a') as f:
f.write(Article.Title)
f.write(Article.fulltxt)


As it automatically closes the file.

Boring == possibly wrong

if line == '____________________________________________________________':


^^ Counting that the above has the correct number of underscores is boring so I will not do it (which is the correct number anyway?) you should do

if line == '_' * NUMBER_OF_UNDERSCORES_NEEDED:


Why are you using fn as a name when writing your first row? I'm guessing it's fieldname, but I read it as short for function initially. You could just do name, because then it'd be

csvwriter.writerow(dict((name, name) for name in fields))


That reads well, name in fields implies that name refers to a fieldname. It's more helpful to think about what the user will read than about what name technically makes sense. You do this with other abbreviations too, like pub but you at least then comment with the full word so that it can be intuited.

You're writing comments for your functions which is good, but you should make them docstrings. That way they're programmatically accessible and parseable by users/programs. All you need to do is wrap the comment in triple quotes instead of using #. See here for more on docstrings.

def writeMetadata(doc):
"""Write the field values of the current doc to the csv file"""


I don't like this line either:

if line == '____________________________________________________________':


But I think there are better tests than even using '_' * num. You could just strip out underscores and see if there's any text left in line. That seems to satisfy the test based on your sample document.

if line.strip('_') == '':


Note that this is after doing .strip() anyway, so you wont get caught out by whitespace.

As a last note, you have some style quirks that are a bit off for Python. Variable and function names should be in snake_case, all operators should have consistent spacing around them, ideally one each (ie. var = 12 + (45 * 10)). You can read about this in more detail in the style guide, PEP0008. It's very helpful to make scripts cleaner and easier to read for yourself and others, well worth reading!