-1
\$\begingroup\$

Task: I'm importing a .CSV files as a pandas dataframes. I'm writing one column of that dataframe to a .txt file. Importantly, each row of the column must only be written as one row in the text file, never more (hence stripping /n)!

I have a very large dataframe (2 million rows), and this loop is naturally very slow given the I/O overhead. Any suggestions for improvements?

for i in tqdm(data['column'].head(500)):
        f = open("Questions.txt","a", newline="\n",encoding='utf-8')
        f.write(i.strip("/n"))
        f.write("\n")
        f.close()
\$\endgroup\$
  • 2
    \$\begingroup\$ You could try to open/close the file only once and call write only once per iteration. \$\endgroup\$ – SylvainD Jun 20 '18 at 16:37
  • 1
    \$\begingroup\$ Where comes the magic 500 from? What does your data look like? \$\endgroup\$ – Mast Jun 20 '18 at 18:32
0
\$\begingroup\$

and this loop is naturally very slow given the I/O overhead

Most of the overhead is probably produced from the way you are opening/closing the file with each iteration. That could be easily fixed by moving these operations out of the loop:

f = open("Questions.txt","a", newline="\n",encoding='utf-8')
for i in tqdm(data['column'].head(500)):
        f.write(i.strip("/n"))
        f.write("\n")
f.close()
| improve this answer | |
\$\endgroup\$
  • 2
    \$\begingroup\$ Wouldn't with open(...) be more idiomatic? \$\endgroup\$ – Phrancis Jun 20 '18 at 17:30
  • \$\begingroup\$ @Phrancis I'm no python expert and can't really tell, but with code blocks are the most horrible and obfuscating things with any programming language I've seen. \$\endgroup\$ – πάντα ῥεῖ Jun 20 '18 at 17:34
  • \$\begingroup\$ @Phrancis Such with code blocks are a strong indication you should just split up these parts, and move it into a separate function. \$\endgroup\$ – πάντα ῥεῖ Jun 20 '18 at 17:39
  • \$\begingroup\$ @πάνταῥεῖ think of with blocks in python as the equivalent of using blocks in C# or try-with-resources in java. They're a resource handling construct, nothing like the With blocks you may know from Basic dialects \$\endgroup\$ – Vogel612 Jun 20 '18 at 17:43
  • \$\begingroup\$ @Vogel612 As mentioned, I just find such constructs just horrible and defeating the readability, well structuring and refactoring flexibility of code. It's quite similar in any language. \$\endgroup\$ – πάντα ῥεῖ Jun 20 '18 at 17:47
1
\$\begingroup\$

Long story short: you don't deal with the I/O and call into numpy.savetext instead. Consider the following code:

import numpy as np

np.savetext("Questions.txt", data['column'].map(strip_newlines).head(500), newline="\n", encoding="utf-8")

This makes abundantly clear that you only care about a newline-stripped representation of the column 'column' in your dataframe. Note that I removed the progress bar from this. I expect this code to be blazingly fast in comparison to yours, because it does two things:

  1. expensive I/O operations (open and close) are only done once
  2. I/O is pushed from python into C++
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ That's probably what separates the theorists from the experts. Great answer! \$\endgroup\$ – πάντα ῥεῖ Jun 20 '18 at 18:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.