# Check values in datatables, with validation rules that vary according to the data source

Background

I receive dataframes from several sources (below I call these sources s1, s2 and s3). I first want to check whether the provided data meets certain requirements and if so, the data will be further processed; below I only provide the code for the check whether the provided data is correct.

Approach

For each column in each dataframe I create an own function that returns a Boolean variable depending on the evaluation of a regular expression. The regular expression depends on the data source and the column name.

Input - output

An example could look like this (coming from source s1):

                         ID     content
x      fo_stuff.s1orig_0987  ABBBCCCCBB
y           cool_abc.s1orig  CCBBAABBTY
z  er_something.illegal.foo    BABCACCB


In ID the regex evaluates to True if an element consists of

(2 letters)_(arbitrary number of letters/numbers/underscores).(s1orig)(optional: underscore and four digits)


So, in this case only the entry in x is correct.

In content the regular expression evaluates to True if the string consists of either only A, B, C or a, b, c (arbitrary many), so x and z are correct, y is not as it contains a T and a Y.

If I run the code from below, I then get my expected outcome:

test = ProcessDataTables(df, "s1")
test.get_invalid_entries()
print(test.invalid_entries)
# {'ID': ['y', 'z'], 'content': ['y']}


So, I get a dictionary that has as keys the column headers and as values a list of indices that correspond to invalid data.

Questions

1. Is there a better way of organizing the code? Now it kind of violates the DRY principle. On the other hand, having individual function for each data source and column is quite flexible if regular expressions have to change/additional columns need to be considered.

2. Is there are a better way of applying the is_valid_*** functions to the dataframe columns? Using __get_mapping_dict works but looks rather strange to pass functions around like this.

3. Is there a straightforward way to have all the is_valid_*** functions inside the class?

This is the entire code:

import re
import pandas as pd

class ProcessDataTables(object):

def __init__(self, data_table, data_source):

self.data = data_table

if data_source in {'s1', 's2', 's3'}:
self.source = data_source
else:
raise ValueError(f"{data_source} is not a valid data source.")

self._mapping_dict = self.__get_mapping_dict()
self.invalid_entries = None

if not set(self._mapping_dict).issubset(self.data.columns):

def get_invalid_entries(self):

# apply the passed functions to the respective columns and filter all
# values that are False (meaning that they have a wrong format)
temp = {ci: self.data.index[~self.data[ci].apply(fi)].tolist()
for ci, fi in self._mapping_dict.items()}
self.invalid_entries = temp

def __get_mapping_dict(self):

if self.source == "s1":
return {'ID': is_valid_name_s1,
'content': is_valid_sequence}

elif self.source == "s2":
return {'ID': is_valid_name_s2,
'content': is_valid_sequence}

elif self.source == "s3":
return {'ID': is_valid_name_s3,
'content': is_valid_sequence}

def is_valid_name_s1(name):

"""
This function checks whether *name* matches the following pattern:

(2letters)_(arbitrary number of letters/numbers/underscores).
(s1orig)(optional: underscore and for digits)

*name*: a string
return: True if *name matches the pattern, False if not
"""

regex_name = re.compile(r'^[A-Za-z]{2}_([A-Za-z0-9_]+)\.(s1orig)(_\d{4})?$') if regex_name.match(name) is not None: return True return False def is_valid_name_s2(name): """ This function checks whether *name* matches the following pattern: (2 or 4 letters)_(arbitrary number of letters/numbers/underscores). (s2orig)(optional: underscore and for digits) *name*: a string return: True if *name matches the pattern, False if not """ regex_name = re.compile(r'^([A-Za-z]{2}|[A-Za-z]{4})_([A-Za-z0-9_]+)\.(s2orig)(_\d{4})?$')
if regex_name.match(name) is not None:
return True
return False

def is_valid_name_s3(name):

"""
This function checks whether *name* matches the following pattern:

(3 letters)_(arbitrary number of letters/numbers/underscores).
(s3orig)(optional: underscore and for digits)

*name*: a string
return: True if *name matches the pattern, False if not
"""

regex_name = re.compile(r'^[A-Za-z]{3}_([A-Za-z0-9_]+)\.(s3orig)(_\d{4})?$') if regex_name.match(name) is not None: return True return False def is_valid_sequence(sequence): """ True if string consists of either only A, B, C or a, b, c """ regex_seq = re.compile(r'^([ABC]+|[abc]+)$')

if regex_seq.match(sequence) is not None:
return True
return False

if __name__ == '__main__':

df = pd.DataFrame({'ID': ["fo_stuff.s1orig_0987",
"cool_abc.s1orig",
"er_something.illegal.foo"],
'content': ['ABBBCCCCBB',
'CCBBAABBTY',
'BABCACCB']},
index=list('xyz'))

test = ProcessDataTables(df, "s1")
test.get_invalid_entries()
print(test.invalid_entries)


## Object-oriented design

This usage is awkward:

test = ProcessDataTables(df, "s1")
test.get_invalid_entries()
print(test.invalid_entries)


Specifically:

• "ProcessDataTables" is a vague name that does not convey the object's purpose. Furthermore, classes should be named as nouns, not verbs. Also, why do you call it a "DataTable" instead of a "DataFrame", which is the PANDAS terminology?
• "s1" is a cryptic shorthand for a set of validation rules.
• .get_invalid_entries() doesn't actually "get" anything, as its name implies. Rather, it sets test.invalid_entries as a side-effect, which is really weird.

I'd expect something more like this:

s1_validator = DataFrameValidator(
ID=re.compile(r'^[A-Za-z]{2}_[A-Za-z0-9_]+\.s1orig(?:_\d{4})?$'), content=re.compile(r'^(?:[ABC]+|[abc]+)$'),
)
print(s1_validator.invalid_entries(df))


Also, you defined the class using class ProcessDataTables(object): …, but this code is clearly intended for Python ≥ 3.6, since you used f-strings. In Python 3, the class does not need to explicitly inherit from object.

## Implementation

PANDAS supports regex matching using series.str.match(regex), so you don't need to write the is_valid_…() adapter functions.

Your regexes contain some superfluous capture groups.

import re
import pandas as pd

class DataFrameValidator:
def __init__(self, regexes={}, **kwargs):
"""
Validator for PANDAS dataframes.  Requirements are specified as a
dictionary (column names as keys, compiled regexes as values), or
as named arguments.
"""
self.regexes = dict(regexes, **kwargs)

def invalid_entries(self, df):
return {
col: df.index[~df[col].str.match(regex)].tolist()
for col, regex in self.regexes.items()
}


Usage is as in the example above.

• Great, that looks far more sane than what I had to work with. Where would you store the regular expressions that belong to the different data sources? In an extra file which is then imported or would there also be a nice option to store them inside the DataFrameValidator class?
– Cleb
Oct 3, 2018 at 6:12
• If there are just three of them, I'd hard-code the regexes at the point where you instantiate the validators. Maybe define a content_regex = re.compile('^(?:[ABC]+|[abc]+)\$') first since multiple validators share it. Oct 3, 2018 at 6:31