# How to better process csv file with pandas and further dealing with set

I have below working code with pandas and python, i'm looking if there is an improvement or simplification which can be done.

Can we Just wrap this up into a definition.

\$ cat getcbk_srvlist_1.py
#!/python/v3.6.1/bin/python3
from __future__ import print_function
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
import pandas as pd
import os
##### Python pandas, widen output display to see more columns. ####
pd.set_option('display.height', None)
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('expand_frame_repr', True)
##################### END OF THE Display Settings ###################

################# PANDAS Extraction ###########
hostData = df_csv[df_csv['Platform ID'].str.startswith("CDS-Unix")]['Target system address']
hostData.to_csv('host_file1', header=None, index=None, sep=' ', mode='a')

with open('host_file1') as f1, open('host_file2') as f2:
dataset1 = set(f1)
dataset2 = set(f2)

for i, item in enumerate(sorted(dataset2 - dataset1)):
print(str(item).strip())



The above code just compares the two files one is processed through pandas ie host_file1 and another is already existing host_file2.

# main guard

It is common to put the code you want to run behind an if __name__ == "__main__":, so you can later import the functions that might be reused in a different module

# naming

You use both snake_case and CamelCase. Try to stick to 1 naming convention. PEP-8 advised snake_case for variables and functions, CamelCase for classes

# functions

split the code in logical parts

## pandas settings

def settings_pandas():
pd.set_option("display.height", None)
pd.set_option("display.max_rows", None)
pd.set_option("display.max_columns", None)
pd.set_option("display.width", None)
pd.set_option("expand_frame_repr", True)


## filename input

The way you ask the filename is very fragile. A more robust way would be to ask the filename in a different function, and then validate it

from pathlib import Path
"""
Asks the user for a filename.
If validate is True, it checks whether the file exists and it is a file
"""
while True:
file = Path(input("Please input the CSV File Name: (CTRL+C to abort)"))
if validate:
if not file.exists() and file.is_file():
print("Filename is invalid")
continue
return file


# IO

def read_host_data(filename):
"""reads filename, filters the unix platforms, and returns the Target system address"""
unix_platforms = df['Platform ID'].str.startswith("CDS-Unix")


There is no need to save the intermediary data to a file. You could use a io.StringIO. An alternative if you need a temporary file is tempfile.

But in this case, where you just need the set of the values of a pd.Series, you can do just set(host_data), without the intermediary file.

# putting it together:

if __name__ == "__main__":
settings_pandas()  # needed?

since the i is not used, I dropped the enumerate. Since host_data2 is directly read from a file, there are no conversions, and it are all strs, so the conversion to str is dropped too.
Since I don't see any printing of pandas data, This part can be dropped apparently.