The code below works pretty well - it takes about 4 seconds (this seems too slow for me) to find a group that's located in the tail area of the text file I'm searching. The text file has ≃ 250,000 lines with 20 elements per line. I cut my teeth on other programming languages and picked up Python just for this current project I'm working on, so I really am a neophyte when it comes to python efficiency.

with open(file) as infile:
    datadictionary = csv.DictReader(infile, dialect='excel-tab', quoting=csv.QUOTE_NONE)
    for key, group in itertools.groupby(datadictionary, key=lambda x:x[patient_number_field_header] == patient_id):
        if key:
            super_list = group
  1. 'patient_id' is a string of digits
  2. file is a text file

I'm wondering what you think - how can I make this more efficient? Am I "doing it wrong"?

  • \$\begingroup\$ the code is probably I/O bound. Difficult to speed it up. I would have not used DictReader but a standard reader & the index of the column to speed up grouping by providing a faster key. But that's not going to save much \$\endgroup\$ – Jean-François Fabre Apr 27 '18 at 20:16
  • \$\begingroup\$ I/O bound seems very plausible to me. That makes it a case of thinking about the environment that the program is running in. For example, moving the file to a RAM disk would both help confirm the suspicion, and possibly help solve it. (Although moving it to the RAM disk is of course not instantaneous) \$\endgroup\$ – Josiah Apr 27 '18 at 21:50
  • \$\begingroup\$ Might look at Pandas -- it is super optimized. You can read from csv or excel directly then manipulate your data in declarative ways. You will likely be hard pressed to beat anything they are already doing. pandas.pydata.org \$\endgroup\$ – SteveJ May 10 '18 at 3:47

I'm assuming your code does some background processing, because otherwise the patient data would be stored in a database, not in a text file. In that scenario, 4 seconds is probably ok.

Instead of grouping the records, you could simply filter them. That saves two lines of code, but will probably not be much faster.

| improve this answer | |
  • \$\begingroup\$ I'm transferring all the data from one medical system that's being taken offline to a new one. No real processing needs doing, just generation of a textfile with a header. Its ten years worth of data. I'm just ballparking how long it will take. It has to be grouped because the files are slightly unordered. It looks like I'll probably be using this method. Thanks for your input \$\endgroup\$ – Integration Apr 30 '18 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.