3
\$\begingroup\$

I've create a function that rewrites the key or "path" of an object in s3.

By default, Amazon Web Services Firehose writes to s3 in the format of YYYY/MM/DD/HH/foo.json. We have a AWS Lambda function listening for putObjects on l1/source/event_type/fh/, and when a new file is added to s3, the Lambda is invoked and the key or 'path' to that file is rewritten to a flat structure of l1/source/event_type/daily/dt=YYYY-MM-DD/foo.json - yes, I purposefully left-off the fh and HH paths.

input key: l1/source/event_type/fh/YYYY/MM/DD/HH/foo.json

output key: l1/source/event_type/daily/dt=YYYY-MM-DD/foo.json

def create_date_parition_from_key(key):
    '''creates new date parition prefix
    '''

    try:
        key_split = re.split(r'(/\d{4})', key)
        start_path = (key_split[0].split('/')[0], key_split[0].split('/')[1])
        remove_fh_path = '/'.join(start_path)
        default = key_split[2].split('/')
        year = key_split[1][1:]

        s3_prefix = remove_fh_path + '/'# /l1/foo/bar/baz/
        date_partion = ('daily/dt=' + 
                        '-'.join([year, default[1], default[2]]) + 
                        '/') # dt=YYYY-MM-DD/
        file_name = default[-1] # foo.json

        new_key = s3_prefix + date_partion + file_name

        print ('New partition key created: {}.'.format(new_key))

        return new_key
    except Exception as ex:
        print(ex)
        print('Error paritioning key {}.'.format(key))
        raise ex

I'm newer to python and am just looking for ways to improve my code as it seems fragile.

EDIT: The input key can vary in its number of paths:

  • l1/source/event_type/fh/YYYY/MM/DD/HH/foo.json
  • l1/app/source/event_type/fh/YYYY/MM/DD/HH/foo.json
  • l1/event_type/fh/YYYY/MM/DD/HH/foo.json
\$\endgroup\$
4
\$\begingroup\$

Simplicity

Tuple unpacking and str.format can simplify the function so much:

def create_date_parition_from_key(key):
    a,b,c,_,year, month, day, _, name = key.split('/')
    return "{}/{}/{}/daily/dt={}-{}-{}/{}".format(\
            a, b, c, year, month, day, name)

If input key can change in the number of paths:

def create_date_parition_from_key(key):
    *_, year, month, day, _, name = key.split('/')
    return "{}/daily/dt={}-{}-{}/{}".format(\
            key[:key.index("/fh/")], year, month, day, name)

Exception handling

try:
    # Things
except Exception as ex:
    print(ex)
    print('Error paritioning key {}.'.format(key))
    raise ex

This is against re-use. I cannot re-use this function because it inevitably prints to the console on error, I have no way of working around it throwing an exception with try - except.

This kind of error notification should be done in the main function that has exactly the job of communicating errors / successes of the other functions to the end user.

Finally Exception is too vague and too much code is inside the try block. Please reduce the code inside try as much as possible and specify a precise Exception kind.

I just removed all exception handling in my version as in my opinion it was just making the code worse while there was no need for it.

\$\endgroup\$
  • \$\begingroup\$ Thanks, however, I forgot to include that the 'path' can vary in the amount of folders it has prior to /fh. See EDIT in question. \$\endgroup\$ – Blexy Sep 23 '16 at 17:58
  • \$\begingroup\$ @Blexy Ok, code edited for the new requirements. Please remember to add more examples / cite possible differences in input in your next question. \$\endgroup\$ – Caridorc Sep 23 '16 at 18:09
  • \$\begingroup\$ you bet. I'm trying to enter your code in python interpretor and I keep getting error when I enter-in *_ - invalid syntax. \$\endgroup\$ – Blexy Sep 23 '16 at 18:23
  • \$\begingroup\$ @Blexy which Python version do you use? \$\endgroup\$ – Caridorc Sep 23 '16 at 18:25
  • 2
    \$\begingroup\$ @Blexy My code works in 3.4.3 . In Python 2 use slicing year, month, day, _, name = key[-5:] instead of my variable length tuple unpacking. In the future also specify your python version with the appropriate tag in the question. \$\endgroup\$ – Caridorc Sep 23 '16 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.