Your code does loop 700 000 multiplied by 65 000 multiplied by the number of elements in each cluster. That is a lot of iterations, and not very useful.
The better approach would be to read the smaller file into memory, and then read the larger file line by line. In addition as you iterate over each row in the smaller file, matching each of the keys, it makes sense to switch from a dict with cluster
as key, and the different keys as values, to actually using the keys as keys, and list all the clusters it belongs to.
This approach would leave the lower memory footprint, but should be rather efficient to work with. Here is some code to start you off with. You might need to adjust a little related to splitting on space or tabs, but I get your wanted output using this.
from collections import defaultdict
def build_cluster_dict(filename):
"""Return a dict of keys with all the cluster the key exists in."""
result = defaultdict(list)
with open(filename) as infile:
for line in infile:
elements = line.strip().split()
cluster = elements[0]
for key in elements[1:]:
result[key].append(cluster)
return result
def build_output(cluster_dict, filename, output_filename):
with open(filename) as infile, open(output_filename, 'w') as outfile:
for line in infile:
key, text = line.strip().split()
if key in cluster_dict:
for cluster in cluster_dict[key]:
outfile.write('{}\t{}\t{}\n'.format(cluster, key, text))
def main():
cluster_dict = build_cluster_dict("cluster.txt")
print (cluster_dict)
build_output(cluster_dict, "file2.txt", "output.txt")
if __name__ == '__main__':
main()
Note that I've left out the use of numpy, as I don't see the need for it in this context. I've also used a double with
statement to open both the in- and out-file at the same time in the same context. I left the print (cluster_dict)
in there just to see the intermediate list it generates. For your test files this gave the following output (somewhat formatted):
defaultdict(<type 'list'>,
{'AA': ['CL1', 'CL2'],
'SS': ['CL1'],
'YY': ['CL1'],
'XX': ['CL1'],
'3_b': ['CL2'],
'ZZ': ['CL1']})
Addendum: Locate erroneous input line
In comments OP said their was a problem in the key, text
line, and to detect this these lines:
for line in infile:
key, text = line.strip().split()
can be replaced with:
for line_number, line in enumerate(infile):
try:
key, text = line.strip().split()
except ValueError:
print("Error on line {}: {}".format(line_number, line))
## Option a) Use empty text
#key = line.strip()
#text = ""
# Option b) Continue with next line
continue
This code will catch the error situation, and as it stands it will display an error message with the offending line. I've set it to use option b), that is continue with next line. If you want to use an empty string and write the output to file file, uncomment option a), and comment out option b).