# Reading and processing a file using Pandas

I am trying to read a file using pandas and then process it. For opening the file I use the following function:

import os
import pandas as pd

files = map(lambda x: os.path.join(data_folder, x), os.listdir(data_folder))
if base_file in files:
try:
df = pd.read_csv(base_file, na_values=["", " ", "-"])
except Exception, e:
print e
df = pd.DataFrame()

else:
df = pd.DataFrame()
return df


My main concerns are the if statement and what I should return if there is an error.

Generator expression

I advice using a generator expression instead of map:

map(lambda x: os.path.join(data_folder, x), os.listdir(data_folder))


should become:

(os.path.join(data_folder, x) for x  in os.listdir(data_folder))


Also x should be renamed to something more expressive.

Separation of concerns

You both print and return values, if the printing is for debugging purposes, use logger.log

Specific Exception

If you write:

except Exception, e:


any Exception will be caught, I suggest IOException.

• Should I let try/except handle if the file exists or not ? – GiannisIordanou Jul 22 '15 at 19:43
• @evil_inside sure, just remove the printing. IOError will catch file not existing or unreadable. – Caridorc Jul 22 '15 at 19:46

Ideally you should ask forgiveness, not permission.

the check if base_file is in datafolders is not helping. If the file is not in data folder then you get an error trying to return df before defining it. If you mean to check if the file is in that folder and not in another you can do this with an assertion. In this case you are simply asserting that the data folder's path is included in the file_name (you're not stiching folder+file together anywhere...) so you can achieve with a check like: assert 'abc' in 'abcde' That will ensure your base_file isn't coming from the wrong folder.

The assignment of df = pd.DataFrame() is also redundant, since you don't do anything with the df object before returning it, and you seem to default to returning an empty dataframe. Something like this could do the trick:

import pandas as pd