I have a text file which is tab delimited:
1 324 2344
1 8372 1234
2 62 12
2 872 12111
2 1211 28736
3 87636 198272
The first "column" contains the id value and the rest containing data. I would like a csv file to be created for each id value so in this example, there would be 3 csv files. And for each csv file, I would like it to contain all rows of data with the same id. So when id = 1
, the csv file will contain the 2 rows of data; when id = 2
, the csv file will contain 3 rows etc.
I have a script which achieves this but I don't think it's very efficient as it stores all unique id values in a list, iterates through this list, for each iteration it reads the text file line by line and if the first element in the line matches the id value, it creates a csv and writes all rows of data with the same id. But if there's 20 id values in the list, it will read the text file 20 times. For small text files, it's not a problem but for larger files, I'm wondering if it would be more efficient to only read the text file once and then create the relevant csv files according to id.
So could this script be improved?
import csv
id_list = []
mylines = []
source_file = 'path/to/source_file.txt'
target_directory = 'save/to/target_directory/'
with open (source_file, 'rt') as myfile:
# Get and store ids
for line in myfile:
data = line.split()
if data[0] in id_list:
pass
else:
id_list.append(data[0])
for ids in id_list:
with open(target_directory + ids + '.csv', 'w', newline='', encoding='utf-8') as csvfile:
fieldnames = ['id', 'data_column_1', 'data_column_2']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
with open (source_file, 'rt') as myfile:
for line in myfile:
data = line.split()
if data[0] == ids:
writer.writerow({'id': data[0], \
'data_column_1': data[1], \
'data_column_2': data[2]})