-2
\$\begingroup\$

The code adds approximately 1,720,000 different combinations of numbers to a file, some_other_file.txt. The numbers range from 2 digits long to 16 digits long.

I add 500 numbers at a time to some_file.txt and upon hitting the 500th I move all the numbers to the end of some_other_file.txt.

My code works. But it takes ages to run and uses a lot of ram. How can I improve on them?

while True:
    for u in range(1,501):
      with open('some_file.txt', "a") as file:
        for g in range(1,500):
          if (g % 100)==0:# speeds up the code by only running one out of 100 times
            os.system("clear")
            print("adding to file...")
            print ("sections loaded: "+str(u-1)+"/100")
            print("total loaded: "+str((g+((u-1)*500))*chunks)+"/"+str(length3))
            sleep(0.0000001)
          if ((g+((u-1)*500))*chunks)<=length3: # length3 is the length of the list I'm adding to file (1720000 long aprox) and chunks is how many times while True loop has run through
            file.write("\n"+update[((g-1)+((u-1)*500))*chunks])
          else:
            print("error")
            break
      with open('some_file.txt', "r") as file:
        l=file.read()
      with open('some_file.txt', "w") as file: # clears file
        file.write("")
      with open('some_other_file.txt', "a") as file:
        file.write("\n"+l)
      l=""
    if int((g+((u-1)*5))*chunks)>=int(length3): # chunks is how many times the while True loop has run through
      break
\$\endgroup\$
1
  • 7
    \$\begingroup\$ "I will update this post as I improve the code" please do not do that, that will make reviewing your code hard as it'll be forever shifting. Additionally upon getting an edit you are no longer allowed to edit the code in the question. \$\endgroup\$ – Peilonrayz Sep 26 '20 at 23:38
2
\$\begingroup\$
  1. We should remove the file information from the loops. We should interact with the files and the output outside the loop.

    Note: Code changed to follow a standard Python style

    def generate_numbers(update, length):
        chunks = 1
        while True:
            for u in range(1, 501):
                for g in range(1, 500):
                    if chunks * (g + (u - 1) * 500) <= length3:
                        file.write("\n"+update[(g - 1 + (u - 1) * 500) * chunks])
                    else:
                        print("error")
                        break
            if int((g + (u - 1) * 5) * chunks) >= int(length3):
              break
            chunks += 1
    
  2. We can see that u is always used as u-1 so we should change the range to range(500).

  3. We should change file.write to yield and move the "\n"+ outside the function.

  4. Your control flow with print("error") is really odd. Let's run a quick whitebox test.

    (u = 497, g = 1, length3 = 500, chunks = 1) # Part way through iterating
    > if chunks * (g + u   * 500) <= length3:
    >         1 * (1 + 497 * 500) <= 500
    >                      248501 <= 500
    > print("error")
    > break
    (u = 498)
    > if chunks * (g + u   * 500) <= length3:
    >         1 * (1 + 498 * 500) <= 500
    >                      249001 <= 500
    > print("error")
    > break
    (u = 499)
    > if chunks * (g + u   * 500) <= length3:
    >         1 * (1 + 499 * 500) <= 500
    >                      249501 <= 500
    > print("error")
    > break
    > if int((g +   u * 5) * chunks) >= int(500):
    > if int((1 + 499 * 5) *      1) >= int(500):
    > if                        2496 >= int(500):
    > break
    

    We should notice that there are 2 oddities here.

    • Your last if looks wrong. (g+((u-1)*5))*chunks should probably be (g+((u-1)*500))*chunks.
    • You should use a try and remove the if as length3 looks like the bound of update.
def generate_numbers(update):
    chunks = 1
    try:
        while True:
            for u in range(500):
                for g in range(1, 500):
                    yield update[(g + u * 500 - 1) * chunks]
            chunks += 1
    except IndexError:
        pass
  1. Since we've removed the file manipulation we can merge the loop to make u and g together. This is because they together make the range 500*500.
  2. We can change the while True to a for that calls itertools.count.
import itertools

def generate_values(update):
    try:
        for chunks in itertools.count(1):
            for index in range(250000):
                yield update[index * chunks]
    except IndexError:
        pass

Let's look at how you interact with the files.

while True:
    for u in range(1,501):
        with open('some_file.txt', "a") as file:
            file.write(...)
        with open('some_file.txt', "r") as file:
            l=file.read()
        with open('some_file.txt', "w") as file:
            file.write("")
        with open('some_other_file.txt', "a") as file:
            file.write("\n"+l)
        l=""

It doesn't make sense to not just write to some_other_file.txt in append mode.

with open('some_other_file.txt', "a") as file:
    for item in generate_values(update):
        file.write("\n" + item)

If you want to display the output then you can just use itertools.islice. And you can use enumerate to build the index.

def chunk_iter(it, amount):
    while True:
        chunk = list(itertools.islice(it, amount))
        if not chunk:
            break
        yield chunk


with open('some_other_file.txt', "a") as file:
    for i, chunk in enumerate(chunk_iter(generate_values(update), 100)):
        print("adding to file...")
        print(f"total loaded: {i * 100 + len(chunk)}/{len(update)}")
        for item in chunk:
            file.write("\n" + item)
\$\endgroup\$
2
  • \$\begingroup\$ I can't use intertools, I tried getting the code for the intertool functions off the official docs, but that wasn't working either (said num of arguments in generators weren't supported), what would be an alternative to that? \$\endgroup\$ – Hackernoob666 Sep 29 '20 at 7:16
  • \$\begingroup\$ nvm, I'm going to code my own alternative to it. \$\endgroup\$ – Hackernoob666 Oct 2 '20 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.