1
\$\begingroup\$

I'm looking for optimized code with faster execution time. This writes CSV to parquet with the max size of 64 MB chunks.

we are trying to concatenate the csv file contents being written as an parquet file until the parquet file size reaches 64MB

The loop would continue to run until reads all the csv files content being concatenated being written as parquet format

import pandas as pd
import uuid
import boto3

s3 = boto3.client('s3')
out_path='s3://ahshan/parquet-out/'
ssize=64*1024*1024


def get_s3_keys(bucket, prefix):

    """Get a list of keys in an S3 bucket."""
    resp = s3.list_objects_v2(Bucket=bucket, Prefix=prefix)
    files=[]
    for obj in resp['Contents']:
      str_key = obj['Key']
      files.append(str_key.split('\n'))
#      print(files)
    return files


def lambda_handler(event,context):
    counter=0
    count=0
    cond=False
    extension = 'csv'
    all_filenames = get_s3_keys('ahshan','csvfiles/csvtemp/')
#    print (all_filenames)  
    df = []
    sizedf =0
    cond=False
    flat_list = [item for sublist in all_filenames for item in sublist]
    flatl= [ item for item in flat_list if '.csv' in item]
    print(' shiva pre :', flatl)
    for f in flatl:
        ipd = pd.read_csv('s3://ahshan/'+f)
        sizedf += ipd.size
        print(sizedf, ":", ssize)
        if sizedf > ssize:
            pd.concat(df).to_parquet(out_path+str(uuid.uuid1())+'.parq', compression='gzip')
            sizedf = 0
            df = []
            cond=True
        else:
            df.append(ipd)
            cond=False

    if not cond:
        print('hello')
        pd.concat(df).to_parquet(out_path+str(uuid.uuid1())+'.parq', compression='gzip')
```
\$\endgroup\$
  • 1
    \$\begingroup\$ With this condition if sizedf > ssize: all csv files whose records size is greater than 64Mb will be skipped from export. Is that intended? \$\endgroup\$ – RomanPerekhrest Nov 28 '19 at 7:12
  • \$\begingroup\$ @RomanPerekhrest - you are right , we are not expecting csv files greater than 1MB \$\endgroup\$ – user1510139 Nov 28 '19 at 9:20
  • 2
    \$\begingroup\$ This question lacks any indication of what the code is intended to achieve. To help reviewers give you better answers, please add sufficient context to your question, including a title that summarises the purpose of the code. We want to know why much more than how. The more you tell us about what your code is for, the easier it will be for reviewers to help you. The title needs an edit to simply state the task. \$\endgroup\$ – Toby Speight Nov 28 '19 at 9:32
  • 1
    \$\begingroup\$ Goal is to concatenate multiple csv files of small size and write it as parquet file not exceeding 64MB \$\endgroup\$ – user1510139 Nov 28 '19 at 10:24
  • 1
    \$\begingroup\$ @user1510139, the current approach does different than a conception you are describing. It does not look workable in terms the conditions you mentioned and does not look ready to review \$\endgroup\$ – RomanPerekhrest Nov 28 '19 at 10:33