4
\$\begingroup\$

Requirements:

Given a long section of text, where the only indication that a paragraph has ended is a shorter line, make a guess about the first paragraph. The lines are hardwrapped, and the wrapping is consistent for the entire text.

The code below assumes that a paragraph ends with a line that is shorter than the average of all of the other lines. It also checks to see whether the line is shorter merely because of word wrapping by looking at the word in the next line and seeing whether that would have made the line extend beyond the "maximum" width for the paragraph.

def get_first_paragraph(source_text):
    lines = source_text.splitlines()
    lens = [len(line) for line in lines]
    avglen = sum(lens)/len(lens)
    maxlen = max(lens)
    newlines = []
    for line_idx, line in enumerate(lines):
        newlines.append(line)
        try:
            word_in_next_line = lines[line_idx+1].split()[0]
        except IndexError:
            break # we've reached the last line
        if len(line) < avglen and len(line) + 1 + len(word_in_next_line) < maxlen: # 1 is for space between words
            break
    return '\n'.join(newlines)

Sample #1

Input:

This is a sample paragaraph. It goes on and on for several sentences. 
Many OF These Remarkable Sentences are Considerable in Length.
It has a variety of words with different lengths, and there is not a
consistent line length, although it appears to hover 
supercalifragilisticexpialidociously around the 70 character mark.
Ideally the code should recognize that one line is much shorter than
the rest, and is shorter not because of a much longer word following
it which has wrapped the line, but because we have reached the end of
a paragraph.
This is the next paragraph, and continues onwards for
more and more sentences.

Output:

This is a sample paragaraph. It goes on and on for several sentences.
Many OF These Remarkable Sentences are Considerable in Length.
It has a variety of words with different lengths, and there is not a
consistent line length, although it appears to hover
supercalifragilisticexpialidociously around the 70 character mark.
Ideally the code should recognize that one line is much shorter than
the rest, and is shorter not because of a much longer word following
it which has wrapped the line, but because we have reached the end of
a paragraph.

Using other sample inputs I see there are a few issues, particularly if the text features a short paragraph or if there is more than one paragraph in the source text (leading to the trailing shorter lines reducing the overall average).

\$\endgroup\$
  • \$\begingroup\$ Can you post example inputs and expected outputs? \$\endgroup\$ – strager Mar 27 '13 at 16:46
  • \$\begingroup\$ Why do you consider the average length at all? if len(line) + len(word_in_next_line) < maxlen: would seem better to me. \$\endgroup\$ – Janne Karila Mar 28 '13 at 8:31
  • \$\begingroup\$ I've updated the example to make it clearer. The inputs are originally text from PDFs, and the fonts used are proportional. As a result, the number of characters per line is not constant but rather based on the number of narrow vs wide characters. The sentence Many OF These Remarkable Sentences are Considerable in Length. is just 63 characters even though it takes up the whole width. Without the check for average length, it would be falsely identified as a new paragraph. \$\endgroup\$ – Jordan Reiter Sep 15 '15 at 20:44
6
\$\begingroup\$

Stating Your Requirements

It's important to have a clear defintion of what you want to achieve before you start writing code, even though there are many different ways of achieving that, for instance by practicing Test-Driven Development or writing a formal specification.

The important part is that without a clear definition, you can't validate whether you're done. In your case, the question contains a description that is completely different than the code and quite unclear.

The above is vital even if you're only writing the code as an exercise for personal use or learning.

Testing and Edge Cases

In the following code:

word_in_next_line = lines[li+1].split()[0]

Why are you assuming that

  • there will be a next line? What if the text consists of only one paragraph?
  • the next line will not be empty?

These assumptions are unreasonable and when I first tried out your code on some text, it immediately threw an exception.

Naming

  • Be careful with historically significant terms such as ss (Google it if you don't know what I mean).
  • Expressive names are better than abbreviations! Replace:

    • ss with source_text
    • ll with line (this looks like the number 11!)
    • lens with line_lengths
    • avglen with average_length
    • maxlen with maximum_length
    • in the for loop, li with index and ll with line

Conclusion

Without a clear explanation of what you are trying to accomplish, how the input data looks and how you define a paragraph, it's impossible to show you a better way of solving the problem.

\$\endgroup\$
  • \$\begingroup\$ Good point about word_in_next_line. I now check for an IndexError and stop if it's reached. If the next line is empty then the check will still succeed and I'll have identified the first paragraph. \$\endgroup\$ – Jordan Reiter Mar 27 '13 at 20:38
1
\$\begingroup\$

Below is a small hash-up for this problem. I have modified the assumptions slightly, in that:

  • Considering the MODE of the line lengths (i.e. taken the most common line length as the 'average' length)

  • Tested the length of each line + the next word against the MODE (this is loose, I am assuming that the end of paragraph line is SIGNIFICANTLY shorter than the MODE - but I think you could refine this :) )

So here we go:

source_lines = source_text.split('\n')

# add the length of each line to a dictionary, incrementing common lengths
line_lengths = {}
for each_line in source_lines:
    line_size = len(each_line)

    # increment a common line length:
    if line_size in line_lengths:
        line_lengths[line_size] += 1

    # or add a new line length to dictionary
    else:
        line_lengths[line_size] = 1

# find the most common length (mode)
mode_length = max(line_lengths, key=line_lengths.get)

# loop through all lines, and test length against MODE
for index in range(0, len(source_lines) -1):
    try:
        # length = this line + next word
        length = len( source_lines[index] + source_lines[index + 1].split()[0] )
    except IndexError:
        # ie. end of file
        length - len( source_lines[index] )

    # test length against MODE
    if length < mode_length:
        # is end of paragraph
        print('\n'.join( source_lines[:index + 1] ) )
        # break loop once first paragraph identified
        break

There is more than likely a more elegant way to implement this using list comprehension. But conceptually, does this work for your needs?

\$\endgroup\$
  • \$\begingroup\$ Revised: the mode_length is not actually the mode here! max(line_lengths) is actually the longest line. Still think this is appropriate however. I will revise and calculate the mode correctly \$\endgroup\$ – Nick Burns Mar 27 '13 at 22:16
  • \$\begingroup\$ Will use mode_length = max(line_lengths, key=line_lengths.get) to get an accurate value for MODE (c.f. just the longest line) \$\endgroup\$ – Nick Burns Mar 27 '13 at 23:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.