Replace badly formatted questionnaires in a document using regex

Question

I have rather badly formatted questionnaires (i.e. ordered lists) in a bunch of documents that I want to clean up and replace the current version with the cleaned-up version.

Example text

STUDY: A trial of Passy-Muir valve was completed to allow the patient to achieve hands-free voicing and also to improve his secretion management. A clinical swallow evaluation was not completed due to the severity of the patient's mucus and lack of saliva control.

The patient's laryngeal area was palpated during a dry swallow and he does have significantly reduced laryngeal elevation and radiation fibrosis. The further evaluate of his swallowing function is safety; a modified barium swallow study needs to be concluded to objectively evaluate his swallow safety, and to rule out aspiration. A trial of neuromuscular electrical stimulation therapy was completed to determine if this therapy protocol will be beneficial and improving the patient's swallowing function and safety.\n01 Do you have preexisting      No\nconditions?\n02 Within the past 12 months I worried about          Never True\nmy health would get worse.\n03 Within the past 12 months I have had         Never True\nhigh blood pressure.\n04 What is your housing situation today?   I have housing\n05 How many times have you moved in the past 12        Zero (I did not move)\nmonths?\n06 Are you worried that in the next 2 months, you may not    No\nhave your own housing to live in?\n07 Do you have trouble paying your heating or electricity    No\nbill?\n08 Do you have trouble paying for medicines?                 No\n09 Are you currently unemployed and looking for work?        No\n10 Are you interested in more education?                     Yes\n\nFor his neuromuscular electrical stimulation therapy, the type was BMR with a single mode cycle time is 4 seconds and 12 seconds off with frequency was 60 __________ with a ramp of 2 seconds, phase duration was 220 with an output of 99 milliamps. Electrodes were placed on the suprahyoid/submandibular triangle with an upright body position, trial length was 10 minutes.\n On a pain scale, the patient reported no pain with the electrical stimulation therapy.

I got some help from @Amadan to write this implementation which takes the questionnaire, splits it into individual questions, and cleans them up separately. I have made the code to identify the questionnaire (well, w/o the last question which I have no idea how to include it since I dunno what comes after it) and replace it with the cleaned-up version. Also, questionnaires may have varying numbers of questions but here I just explicitly set the number to 10.

The code

question_break_re = re.compile("\n(?=\\d{2} )")
answer_re = re.compile("\\s{2,}([^\n]+)")
whitespace_re = re.compile("\\s+")
end_of_question_mark_re = re.compile(r"(?:\?|\.)?$")

def tidy_up_question(question):
    answer = None
    match = answer_re.search(question)
    if match:
        answer = match.group(1)
        question = question[:match.start(0)] + question[match.end(0):]
    question = whitespace_re.sub(' ', question).strip()
    if answer is not None:
        question = end_of_question_mark_re.sub(f": {answer}", question, count=1)
    return question+"\n"

q_n_a = re.findall(r"\n01[\s\S]*\n(?=10)", text)[0]
qlist = [
    tidy_up_question(question)
    for question in question_break_re.split(q_n_a)
    if question.strip()
]

print(text.replace(q_n_a, '\n'.join(qlist)))

Output

STUDY: A trial of Passy-Muir valve was completed to allow the patient to achieve hands-free voicing and also to improve his secretion management. A clinical swallow evaluation was not completed due to the severity of the patient's mucus and lack of saliva control.

The patient's laryngeal area was palpated during a dry swallow and he does have significantly reduced laryngeal elevation and radiation fibrosis. The further evaluate of his swallowing function is safety; a modified barium swallow study needs to be concluded to objectively evaluate his swallow safety, and to rule out aspiration. A trial of neuromuscular electrical stimulation therapy was completed to determine if this therapy protocol will be beneficial and improving the patient's swallowing function and safety.01 Do you have preexisting conditions: No

02 Within the past 12 months I worried about my health would get worse: Never True

03 Within the past 12 months I have had high blood pressure: Never True

04 What is your housing situation today: I have housing

05 How many times have you moved in the past 12 months: Zero (I did not move)

06 Are you worried that in the next 2 months, you may not have your own housing to live in: No

07 Do you have trouble paying your heating or electricity bill: No

08 Do you have trouble paying for medicines: No

09 Are you currently unemployed and looking for work: No 10 Are you interested in more education? Yes

For his neuromuscular electrical stimulation therapy, the type was BMR with a single mode cycle time is 4 seconds and 12 seconds off with frequency was 60 __________ with a ramp of 2 seconds, phase duration was 220 with an output of 99 milliamps. Electrodes were placed on the suprahyoid/submandibular triangle with an upright body position, trial length was 10 minutes. On a pain scale, the patient reported no pain with the electrical stimulation therapy.

This is a success! However, I feel the implementation has too many steps and is perhaps not efficient enough. I am wondering if I can use re.sub() to identify each questionnaire item/question and replace it with the cleaned version. Something like re.sub(r"\\n(\d{2} ).*\\n(?=\d{2} )", lambda m: tidy_up_question(m.group()), text) but of course this doesn't work yet. Is this possible?

Questions

1. Can I identify and replace each question (or rather each item in an ordered list) with a single re.sub or other function?

2. Can I do so in the larger text efficiently?

3. Any other possible improvements to make it faster and perhaps can identify questionnaires with a different number of questions?

Answer 1

Based on the Example Text, it looks like each questions starts with 2 digits at the beginning of a line and ends just before the next question or at a blank line. This can be captured by a regex pattern like this:

question_re = re.compile(r"""
    ^(?P<number>\d\d)  # two digits, but only at start of a line
    \s
    (?P<question>.*?)      # match anything until 
    (?=\n\d\d|\n\n)    #   the next question or a blank line
    """,
    re.VERBOSE | re.DOTALL)

Then use Pattern.sub(repl, string), where repl is a function that returns the replacement string. In this case, repl will be a function that rearranges the text of a question. For example:

def rearrange(match):
    """Presumes that the question and answer are on one or more lines. The first
    part of the question and the answer are on the first line separated by a run
    of 2+ spaces. The rest of the question, if any, follows in succeding lines."""
    question, answer = re.split(r"\s{2,}", match['question'], maxsplit=1)
    answer, *rest = answer.split('\n')
    return F"\n{match['number']} {question} {' '.join(rest)}  {answer}."

Python 3.6 added the __getitem__() method to MatchObject, so you can write match['question'] instead of match.group('question').

With those two pieces, fixing up the text is a one-liner

reformatted_text = question_re.sub(rearrange, text)