Checking whether fund prices changed over multiple CSV files

Question

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes.

• 1st download - 2 Fund houses updated their prices
• 2nd download - 8 Fund houses updated their prices
• 3rd download - 5 Fund houses updated their prices
• and so on …

I download all the prices and stored them into CSV files with the filename in the format of [date_time].csv.

For example, ['03112019_18:00:05.csv','03112019_18:30:02.csv','03112019_19:00:08.csv',....]

I kept all the files in a directory. I want to check whether are the prices of the Mutual Funds changed from the early time during the subsequent updates. Simply, I want to analyse whether they are updating only new prices of Funds in their every update or If there is a modification of the price that has already been updated.

To do this, I write the following code in python and pandas. Please review it and give me your valuable feedback. Also please tell me an alternative or better way If there is.

# my routine to extract all files of csv.
def listofFiles(dirPath, extension=""):
    if not extension:
        return os.listdir(dirPath)

    return [file for file in os.listdir(dirPath) if file.endswith("." + extension)]

# setting up directory path
dirpath = '/home/user/fund/{}/'.format(arguments[0])

# List out csv file names from the directory to process
# my routine to extract all files of csv.
list_of_files = listofFiles(dirpath, "csv")

# Store date of the filename into a variable
if list_of_files:
    date = list_of_files[0].rsplit('.')[0].rsplit('_')[0]

# Extract times from the files and make list of times and sort it
time_list = set()
for path in list_of_files:
    if path:
        if "_" in path:
            time = path.rsplit('.')[0].rsplit('_')[1]
            time_list.add(dt.strptime(time, "%H:%M:%S"))

time_list = sorted(time_list)

# Get earliest file and load into pandas Data Frame
time_s = dt.strftime(time_list[0], "%H:%M:%S")
file = "{}_{}.csv".format(date, time_s)
merged_df = pd.read_csv(dirpath + file)
# Filter only needed column
merged_df = start_df = merged_df[['Scheme Name', 'pri']]
# here merged_df for generating resulting data frame
# start_df for comparing data of new one with earliest data frame

# Rename the name of the column 'pri' with 'pri_[time_of_the_file]'

start_suffix = dt.strftime(time_list[1], "_%H:%M")
merged_df = merged_df.rename(columns={'pri': 'pri{}'.format(start_suffix)})

# Start Iterating with next time file
for time in time_list[1:]:

    time_s = dt.strftime(time, "%H:%M:%S")  # for making filename
    # for making columns as per filename
    end_prefix = dt.strftime(time, "_%H:%M")
    file = "{}_{}.csv".format(date, time_s)  # Set file name
    frame = pd.read_csv(dirpath + file)  # Read csv

    frame = frame[['Scheme Name', 'pri']]

    # prepare Intersected list with previous time file
    inter_df = pd.merge(start_df, frame, on='Scheme Name', how='inner',
                        suffixes=[start_suffix, end_prefix])

    # Append the current time price column for resulting data frame
    merged_df = pd.merge(merged_df, inter_df[[
                            'Scheme Name', 'pri'+end_prefix]], on='Scheme Name', how='right')

    start_df = frame  # Make the current data frame as previous
    start_suffix = end_prefix  # Change the previous time suffix to current

# print the result
print(merged_df.head())

# Check the pair of price columns from earliest to newest If there is a price change for the funds.
start = dt.strftime(time_list[0], "%H:%M")
for time in time_list[1:]:
    end = dt.strftime(time, "%H:%M")
    print("Comparing prices consistency between {} and {}".format(start, end))
    print(merged_df.loc[merged_df['pri_'+start]
                        != merged_df['pri_'+end]].dropna())
    print("---------------------------------------------------------------------")
    start = end

My Input:

03112019 directory contains the following CSV files.
    03112019_18:00:05.csv
    03112019_18:30:02.csv
    03112019_19:00:03.csv
    03112019_19:30:05.csv
    03112019_20:00:08.csv

contents of 03112019_18:00:05.csv
                                         Scheme Name              pri
0  Franklin India Banking & PSU Debt Fund - Direc...          10.7959
1  Franklin India Banking & PSU Debt Fund - Direc...          15.0045
2  Franklin India Banking & PSU Debt Fund - Dividend          10.5216
3    Franklin India Banking & PSU Debt Fund - Growth          14.6659
4  SBI BANKING & PSU FUND - Direct Plan  - Weekly...        1016.8984
.....
.....
---------------------------------------------------------------------
contents of 03112019_18:30:02.csv
                                         Scheme Name              pri
0  Aditya Birla Sun Life Banking & PSU Debt Fund ...         152.1524
1  Aditya Birla Sun Life Banking & PSU Debt Fund ...         107.1248
2  Aditya Birla Sun Life Banking & PSU Debt Fund ...         105.7569
3  Aditya Birla Sun Life Banking & PSU Debt Fund ...         159.7587
4  Aditya Birla Sun Life Banking & PSU Debt Fund ...         235.8929
.....
.....
---------------------------------------------------------------------
contents of 03112019_19:00:03.csv
                                         Scheme Name              pri
0  Aditya Birla Sun Life Banking & PSU Debt Fund ...         152.1524
1  Aditya Birla Sun Life Banking & PSU Debt Fund ...         107.1248
2  Aditya Birla Sun Life Banking & PSU Debt Fund ...         105.7569
3  Aditya Birla Sun Life Banking & PSU Debt Fund ...         159.7587
4  Aditya Birla Sun Life Banking & PSU Debt Fund ...         235.8929
.....
.....

My executable command is,

python3 checkconsistency.py 03112019

and You have to define dirpath. I set this from my configuration file.

We could also send the column names to be check the price consistency as an argument. Here I just hardcoded it.

Answer 1

This is an interesting piece of code. You have clearly tried to make it look better. Well done.

---

• I recently read in Refactoring book following about comments. Thick comments means code is bad.

Comments aren’t a bad smell; indeed they are a sweet smell. The reason we mention comments here is that comments are often used as a deodorant. It’s surprising how often you look at thickly commented code and notice that the comments are there because the code is bad.

-- Refactoring: Improving Design of Existing Code by Martin Fowler

• Do not specify what your code is doing.
  • Anyone who want's to read your code will know python or will know how to learn it.
  • You are not writing a tutorial.
  • Explain your intent.
  • Avoid obvious comments.
• Specify why you decide to do it that way.

def listofFiles(dirPath

• Please follow PEP-8 and make method names and parameters snake_case