Each line of the JSON has a structure looks as the following
|- Sentence | |-- WordPosition: | |-- |-- Candidate | |-- |-- |-- index: | |-- |-- |-- retrievals
def output_conll_retrievals_to_json(output_file, conll_mcs,conll_sentences, num_retrieval): num_sentence = len(conll_sentences) with open(output_file, 'w') as f: for i in range(num_sentence): num_position = len(conll_sentences[i]) level2 = dict() level1 = dict() for j in range(num_position): level0 = dict() for mc in conll_mcs[i][j]: mc_str = ' '.join(mc.mention_string) level0[mc_str] = dict() level0[mc_str]['index'] = mc.mention level0[mc_str]['retrievals'] = self.get_one_mc_retrievals(mc, num_retrieval) level1[conll_sentences[i][j]] = level0 level2[' '.join(conll_sentences[i])] = level1 json.dump(level2, f) f.write('\n')
self.get_one_mc_retrievals is implemented in parallel, accessing a memory map and performs a hierarchical search. Executing the snippet of code, status showing that CPUs are under-used (like 10-20% for each core as writing is serial programming) and the whole JSON output takes 8 days (approximately 3000 sentences with
num_retrievals of 100) to generate. Would it be possible to boost the speed (i.e., Cython, parallel or something else)?
** Edited **
This just comes off my head, would it be possible to utilize multiprocessing to speed up JSON writing? Right now, my code is kind of serially doing a process that retrieves a dict from reading
conll_mcs[i] and writes a line to the JSON file. If I could simultaneously perform many of such a process, speed will very likely improve, however I am still trying to find how to put such an idea into practice. Any help would be greatly appreciated.