# Regex to match several slightly different strings

I work with a charity that does a number of jobs, more than 100, less than 1000 a year. Jobs are identified with a year, and an up to 3 digit serial number, so 2021#123 for instance.

I'm writing a Python report program to pull jobs out of our database, and I'd like be be able to specify ranges of jobs rather like a print dialogue can specify ranges of pages, so a dash separated range 2021#123-45, or a comma separated list 2021#003,5,07,89,131 as well. I want to be able to parse all of these representations out of text strings.

I'm only interested in the syntax at the moment, the semantics of whether a range is valid is handled later. Even though I have very limited experience of regexes, I felt a one-liner would beat a whole bunch of python ifs.

The valid strings I want to match are therefore of the form

• 20y#n OR
• 20y#n,q,p,r OR
• 20y#n-m

where y is any two digits, and m,n,p,q,r are 1 to 3 digits.

This is a question about the regex. The only python consideration is the use of r'…' for a raw string.

I struggled and failed to write a regex that captures any of these forms by trying to match the year#ddd, and then OR the different endings. I think I fell foul of the single job form not having an ending at all. I've therefore ended up with this rather agricultural effort, that works, but looks ugly. I didn't put the comments in just for this post; I put them in knowing that if I needed to change things next month, I'd need them.

job_patt = r'((?:20\d\d#\d{1,3}-\d{1,3})|(?:20\d\d#\d{1,3}(?:,\d{1,3})+)|(?:20\d\d#\d{1,3}))'
# cap group  (                                                                             )
# non-caps    (?:                      ) (?:                           ) (?:              )
# OR the non-caps, in this L->R order   |                               |
# get year#      20\d\d#                    20\d\d#                         20\d\d#
# get 1 to 3 digits     \d{1,3}                    \d{1,3}                         \d{1,3}
# get one instance of range    -\d{1,3}
# get one or more repeats of list         ',ddd'          (?:,\d{1,3})+


What could be done to shorten or tidy up this regex?

• I changed the title so that it's a bit more specific. Please check that I haven't misrepresented your code, and correct it if I have. – Toby Speight May 27 at 9:51
• @TobySpeight Yes, I'd changed it just a minute before you. Maybe yours matches my exact problem closely, but it could be more general without losing the flavour of the regex problem - matching several similar strings - as it refers to years and jobs, which aren't relevant to the issue. I'll try to revert it. The fundamental problem that remains unaddressed is that one of the options is null, which I need to think about more. – Neil_UK May 27 at 9:56
• Yes, it was supposed to be specific. Did you intend to revert all my other fixes, too? – Toby Speight May 27 at 15:36
• @TobySpeight No, didn't spot those, fixed. I prefer the title as more general. – Neil_UK May 27 at 16:10

This isn't so much a Python question as it is a RegEx question. The Python tag is almost irrelevant.

# '+' -vs- '*'

You regex includes two alternates:

• (?:20\d\d#\d{1,3}(?:,\d{1,3})+)
• (?:20\d\d#\d{1,3})

The difference in these expressions is the first one include one or more repeats of ,\d{1,3}, where as the second does not include any repeats of that ...

... in other words, you have 20\d\d#\d{1,3} followed by 0 or more repeats of ,\d{1,3}.

So we can replace the + modifier with the * modifier, and the pair of alternates becomes simply

(?:20\d\d#\d{1,3}(?:,\d{1,3})*)

# Common prefix

All three (or rather, both now that the last two have been combined into one) regex's start with 20\d\d#. You can factor this out of both patterns.

20\d\d# (?: (?: \d{1,3}-\d{1,3} | \d{1,3}(?:,\d{1,3})* ) )

# Capture Groups

There is no point creating a capture group for the entire regex expression. The entire match is always captured, as group 0.

# Reworked Regex & Example

The regex can be rewritten as:

r"20\d\d#(?:(?:\d{1,3}-\d{1,3}|\d{1,3}(?:,\d{1,3})*))"

which is shorter, but is still quite unreadable. It is better for future programmers at the charity (as well as for yourself) to document the regex using the re.VERBOSE option. I find it clearer than comments below the regular expression that try to document the parts of the regular expression through vertically lined up notes.

Note: Since comments follow the octothorpe (#) character in re.VERBOSE mode, we need to escape it (\#) to indicate that it is actually a character we want matched, and not the start of a comment.

import re

JOB_PATTERN = re.compile(r"""
20\d\d                     # A 4-digit year
\#                         # followed by a hash mark
(?:                        # followed by either ...
(?:\d{1,3}-\d{1,3})        # a range of serial #'s
|                          # or
(?:\d{1,3}(?:,\d{1,3})*)   # a list of serial #'s
)""", re.VERBOSE)

test_cases = ["junk2021#123-45bar",
"baz2021#003,5,07,89,131foo"]

for test_case in test_cases:
m = JOB_PATTERN.search(test_case)
if m:
print("Found:", m[0])


Output:

Found: 2021#123-45
Found: 2021#003,5,07,89,131


# Update

Based on the OP's response to RootTwo's comment, named capturing groups are actually desired. Here is an updated example with RootTwo's enhancement:

import re

JOB_PATTERN = re.compile(
r"""(?P<year>              # Capturedas "year" ...
20\d\d                 # a 4-digit year
)
\#                     # followed by a hash mark
(?:                    # followed by either ...
(?P<range>             # captured as "range" ...
\d{1,3}-\d{1,3}        # a range of serial #'s
)
|                      # or
(?P<list>              # captured as "list" ...
\d{1,3}               # a serial number
(?:,\d{1,3})*         # or list of comma-seperated serial #'s
)
)
""", re.VERBOSE)

test_cases = ["junk2021#123-45bar",
"baz2021#003,5,07,89,131foo"]

for test_case in test_cases:
m = JOB_PATTERN.search(test_case)
if m:
if m.lastgroup == "range":
start, end = m['range'].split('-')
print("Year:", m['year'], " from", start, "to", end)
else:
serial_numbers = m['list'].split(',')
print("Year:", m['year'], " serial #':", serial_numbers)


Output:

Year: 2021  from 123 to 45
Year: 2021  serial #': ['003', '5', '07', '89', '131']

• I came up with an identical pattern, except I used named capture groups so that one can use Match.lastgroup() to determine if it matched the range or list of jobs. – RootTwo May 22 at 18:31
• @RootTwo That would be a useful improvement. The OP was simply capturing the entire expression, and explicitly using non-capturing groups for the individual details, so I didn't change that in my regex reworkings. – AJNeufeld May 22 at 19:51
• @RootTwo could I trouble you to write an answer with named capture groups please. Just because I explicitly used non-capturing groups doesn't mean it's the best thing to do, and I don't like the what appears to be excessive (?: syntax anyway. Maybe it wouldn't shorten it, verbose certain doesn't, but it does make it better. – Neil_UK May 23 at 5:38
• @Neil_UK, it is literally the same as AJNeufeld's answer, except in the year part I used (?P<year>20\d\d). in the range part I used (?P<range> instead of (?:, and in the list part I used (?<list> instead of the first (?:. – RootTwo May 23 at 6:55
• Note: @RootTwo obviously meant (?P<list>, not (?<list>. I've added an updated example with RootTwo's capture group and demonstration of the the Match.lastgroup property. – AJNeufeld May 23 at 16:28