3
\$\begingroup\$

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes.

  • 1st download - 2 Fund houses updated their prices
  • 2nd download - 8 Fund houses updated their prices
  • 3rd download - 5 Fund houses updated their prices
  • and so on …

I download all the prices and stored them into CSV files with the filename in the format of [date_time].csv.

For example, ['03112019_18:00:05.csv','03112019_18:30:02.csv','03112019_19:00:08.csv',....]

I kept all the files in a directory. I want to check whether are the prices of the Mutual Funds changed from the early time during the subsequent updates. Simply, I want to analyse whether they are updating only new prices of Funds in their every update or If there is a modification of the price that has already been updated.

To do this, I write the following code in python and pandas. Please review it and give me your valuable feedback. Also please tell me an alternative or better way If there is.

# my routine to extract all files of csv.
def listofFiles(dirPath, extension=""):
    if not extension:
        return os.listdir(dirPath)

    return [file for file in os.listdir(dirPath) if file.endswith("." + extension)]

# setting up directory path
dirpath = '/home/user/fund/{}/'.format(arguments[0])

# List out csv file names from the directory to process
# my routine to extract all files of csv.
list_of_files = listofFiles(dirpath, "csv")

# Store date of the filename into a variable
if list_of_files:
    date = list_of_files[0].rsplit('.')[0].rsplit('_')[0]

# Extract times from the files and make list of times and sort it
time_list = set()
for path in list_of_files:
    if path:
        if "_" in path:
            time = path.rsplit('.')[0].rsplit('_')[1]
            time_list.add(dt.strptime(time, "%H:%M:%S"))

time_list = sorted(time_list)

# Get earliest file and load into pandas Data Frame
time_s = dt.strftime(time_list[0], "%H:%M:%S")
file = "{}_{}.csv".format(date, time_s)
merged_df = pd.read_csv(dirpath + file)
# Filter only needed column
merged_df = start_df = merged_df[['Scheme Name', 'pri']]
# here merged_df for generating resulting data frame
# start_df for comparing data of new one with earliest data frame

# Rename the name of the column 'pri' with 'pri_[time_of_the_file]'

start_suffix = dt.strftime(time_list[1], "_%H:%M")
merged_df = merged_df.rename(columns={'pri': 'pri{}'.format(start_suffix)})

# Start Iterating with next time file
for time in time_list[1:]:

    time_s = dt.strftime(time, "%H:%M:%S")  # for making filename
    # for making columns as per filename
    end_prefix = dt.strftime(time, "_%H:%M")
    file = "{}_{}.csv".format(date, time_s)  # Set file name
    frame = pd.read_csv(dirpath + file)  # Read csv

    frame = frame[['Scheme Name', 'pri']]

    # prepare Intersected list with previous time file
    inter_df = pd.merge(start_df, frame, on='Scheme Name', how='inner',
                        suffixes=[start_suffix, end_prefix])

    # Append the current time price column for resulting data frame
    merged_df = pd.merge(merged_df, inter_df[[
                            'Scheme Name', 'pri'+end_prefix]], on='Scheme Name', how='right')

    start_df = frame  # Make the current data frame as previous
    start_suffix = end_prefix  # Change the previous time suffix to current

# print the result
print(merged_df.head())

# Check the pair of price columns from earliest to newest If there is a price change for the funds.
start = dt.strftime(time_list[0], "%H:%M")
for time in time_list[1:]:
    end = dt.strftime(time, "%H:%M")
    print("Comparing prices consistency between {} and {}".format(start, end))
    print(merged_df.loc[merged_df['pri_'+start]
                        != merged_df['pri_'+end]].dropna())
    print("---------------------------------------------------------------------")
    start = end

My Input:

03112019 directory contains the following CSV files.
    03112019_18:00:05.csv
    03112019_18:30:02.csv
    03112019_19:00:03.csv
    03112019_19:30:05.csv
    03112019_20:00:08.csv

contents of 03112019_18:00:05.csv
                                         Scheme Name              pri
0  Franklin India Banking & PSU Debt Fund - Direc...          10.7959
1  Franklin India Banking & PSU Debt Fund - Direc...          15.0045
2  Franklin India Banking & PSU Debt Fund - Dividend          10.5216
3    Franklin India Banking & PSU Debt Fund - Growth          14.6659
4  SBI BANKING & PSU FUND - Direct Plan  - Weekly...        1016.8984
.....
.....
---------------------------------------------------------------------
contents of 03112019_18:30:02.csv
                                         Scheme Name              pri
0  Aditya Birla Sun Life Banking & PSU Debt Fund ...         152.1524
1  Aditya Birla Sun Life Banking & PSU Debt Fund ...         107.1248
2  Aditya Birla Sun Life Banking & PSU Debt Fund ...         105.7569
3  Aditya Birla Sun Life Banking & PSU Debt Fund ...         159.7587
4  Aditya Birla Sun Life Banking & PSU Debt Fund ...         235.8929
.....
.....
---------------------------------------------------------------------
contents of 03112019_19:00:03.csv
                                         Scheme Name              pri
0  Aditya Birla Sun Life Banking & PSU Debt Fund ...         152.1524
1  Aditya Birla Sun Life Banking & PSU Debt Fund ...         107.1248
2  Aditya Birla Sun Life Banking & PSU Debt Fund ...         105.7569
3  Aditya Birla Sun Life Banking & PSU Debt Fund ...         159.7587
4  Aditya Birla Sun Life Banking & PSU Debt Fund ...         235.8929
.....
.....

My executable command is,

python3 checkconsistency.py 03112019

and You have to define dirpath. I set this from my configuration file.

We could also send the column names to be check the price consistency as an argument. Here I just hardcoded it.

\$\endgroup\$
  • \$\begingroup\$ Could you add minimal examples of the CSV files so that we could run the program? Also, what is arguments? It is never defined in your code. \$\endgroup\$ – Georgy Mar 14 at 13:22
  • \$\begingroup\$ I don't know how to add examples here. Could I add those examples as CSV files here? \$\endgroup\$ – Smith Dwayne Mar 15 at 4:58
  • \$\begingroup\$ You can't add files, but you can add text like, for example, here. I think a small example of 3 files, each of them not more than 10 lines would be enough. \$\endgroup\$ – Georgy Mar 15 at 8:46
  • \$\begingroup\$ @Georgy: I edited the post as per your request. Is that enough? If not, Please help me to give the better view. \$\endgroup\$ – Smith Dwayne Mar 16 at 5:29
  • \$\begingroup\$ Better, but the contents of the CSV files are still unusable, and the relation between them is still unclear for me. What happens to the lines from the first CSV in the second CSV file? Do they disappear, or shift down? Try to provide such example so that we could just copy and paste the CSV data, and be able to run your program without guessing what missing data could look like. I tried to guess but my examples won't pass the merging part. \$\endgroup\$ – Georgy Mar 16 at 9:45
3
\$\begingroup\$

This is an interesting piece of code. You have clearly tried to make it look better. Well done.


  • I recently read in Refactoring book following about comments. Thick comments means code is bad.

Comments aren’t a bad smell; indeed they are a sweet smell. The reason we mention comments here is that comments are often used as a deodorant. It’s surprising how often you look at thickly commented code and notice that the comments are there because the code is bad.

-- Refactoring: Improving Design of Existing Code by Martin Fowler

  • Do not specify what your code is doing.
    • Anyone who want's to read your code will know python or will know how to learn it.
    • You are not writing a tutorial.
    • Explain your intent.
    • Avoid obvious comments.
  • Specify why you decide to do it that way.
def listofFiles(dirPath
  • Please follow PEP-8 and make method names and parameters snake_case
\$\endgroup\$
  • 1
    \$\begingroup\$ Thank you for your appreciation and for your valuable time review my code. Thank you for the two points you mention to what to do in my code and your other feedbacks. Your feedback is highly appreciated and will help me to improve my ability to write my code well clear. \$\endgroup\$ – Smith Dwayne Mar 13 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.