I'm looking to irreversibly hash valid email addresses which have been parsed from mail log data as part of a machine learning project. I need to ensure that the processed data has been scrubbed of personally identifiable information (specific users/client domain etc).

A quick google-fu led me to pyffx and the inbuilt secrets package.

I'm looking to scrub the email addresses while retain the formatting and beginning character sequence:

#!/usr/bin/env python
# coding: utf-8

import pyffx, secrets

def ffx_encrypt(email,secret):
    raw_user, raw_domain = email.split('@')
    #retaining first few characters to test entropy of bulk sender lists
    user_chars = raw_user[:3]
    user_rem = raw_user[3:]
    #get unique characters for each string to retain entropy
    uniq_user_chars = ''.join(set(raw_user))
    uniq_dom_chars = ''.join(set(raw_domain))

    e_user = pyffx.String(secret,alphabet=uniq_user_chars,length=len(user_rem))
    e_dom = pyffx.String(secret,alphabet=uniq_dom_chars,length=len(raw_domain))

    user_encrypt = e_user.encrypt(user_rem)
    dom_encrypt = e_dom.encrypt(raw_domain)

    return user_chars + user_encrypt + '@' + dom_encrypt;

#To be generated at runtime
secret = secrets.token_hex(32).encode()


##Sample run results

At the moment I'm not focused on performance, elegance or robustness, it's more about avoiding stepping on a landmine if there's an obvious flaw with my implementation which could make the post-processed email addresses/domains reversible.

Feedback would be greatly appreciated.

EDIT: Clarified that input will be valid email addresses.


Test cases are very limited. You might want to normalise equivalent addresses before hashing; for example, these addresses are all equivalent:

  • user@example.org
  • user@Example.Org
  • "user"@example.org

It's probably desirable that they hash to the same value.

Simply splitting on @ is naive - it's better to split on unquoted @, or more simply, just on the last @, given that DNS names don't contain @.

It's probably worth reading I Knew How To Validate An Email Address Until I Read The RFC. After that, start looking for an email address parsing library for Python; I haven't used it, but it appears that Flanker will handle the parsing much more robustly than this code.

  • \$\begingroup\$ Whilst they may be equivalent from an MTA perspective, such deviations would be of interest from a ML standpoint as it would suggest different processes were responsible in generating the addresses. Parsing is out of scope for this as it'll be handled separately, probably with perl as the dataset is quite large. I'll update the question to make it clear the input will be valid email addresses. Thank you for the @ split advice, I'll make the suggested change. \$\endgroup\$ – Prassein Jun 11 '19 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.