Adding 1 million combinations of numbers to a file

Question

The code adds approximately 1,720,000 different combinations of numbers to a file, some_other_file.txt. The numbers range from 2 digits long to 16 digits long.

I add 500 numbers at a time to some_file.txt and upon hitting the 500th I move all the numbers to the end of some_other_file.txt.

My code works. But it takes ages to run and uses a lot of ram. How can I improve on them?

while True:
    for u in range(1,501):
      with open('some_file.txt', "a") as file:
        for g in range(1,500):
          if (g % 100)==0:# speeds up the code by only running one out of 100 times
            os.system("clear")
            print("adding to file...")
            print ("sections loaded: "+str(u-1)+"/100")
            print("total loaded: "+str((g+((u-1)*500))*chunks)+"/"+str(length3))
            sleep(0.0000001)
          if ((g+((u-1)*500))*chunks)<=length3: # length3 is the length of the list I'm adding to file (1720000 long aprox) and chunks is how many times while True loop has run through
            file.write("\n"+update[((g-1)+((u-1)*500))*chunks])
          else:
            print("error")
            break
      with open('some_file.txt', "r") as file:
        l=file.read()
      with open('some_file.txt', "w") as file: # clears file
        file.write("")
      with open('some_other_file.txt', "a") as file:
        file.write("\n"+l)
      l=""
    if int((g+((u-1)*5))*chunks)>=int(length3): # chunks is how many times the while True loop has run through
      break

Answer 1

1. We should remove the file information from the loops. We should interact with the files and the output outside the loop.

   Note: Code changed to follow a standard Python style

   def generate_numbers(update, length):
       chunks = 1
       while True:
           for u in range(1, 501):
               for g in range(1, 500):
                   if chunks * (g + (u - 1) * 500) <= length3:
                       file.write("\n"+update[(g - 1 + (u - 1) * 500) * chunks])
                   else:
                       print("error")
                       break
           if int((g + (u - 1) * 5) * chunks) >= int(length3):
             break
           chunks += 1
   

2. We can see that u is always used as u-1 so we should change the range to range(500).

3. We should change file.write to yield and move the "\n"+ outside the function.

4. Your control flow with print("error") is really odd. Let's run a quick whitebox test.

   (u = 497, g = 1, length3 = 500, chunks = 1) # Part way through iterating
   > if chunks * (g + u   * 500) <= length3:
   >         1 * (1 + 497 * 500) <= 500
   >                      248501 <= 500
   > print("error")
   > break
   (u = 498)
   > if chunks * (g + u   * 500) <= length3:
   >         1 * (1 + 498 * 500) <= 500
   >                      249001 <= 500
   > print("error")
   > break
   (u = 499)
   > if chunks * (g + u   * 500) <= length3:
   >         1 * (1 + 499 * 500) <= 500
   >                      249501 <= 500
   > print("error")
   > break
   > if int((g +   u * 5) * chunks) >= int(500):
   > if int((1 + 499 * 5) *      1) >= int(500):
   > if                        2496 >= int(500):
   > break
   

   We should notice that there are 2 oddities here.

   • Your last if looks wrong. (g+((u-1)*5))*chunks should probably be (g+((u-1)*500))*chunks.
   • You should use a try and remove the if as length3 looks like the bound of update.

def generate_numbers(update):
    chunks = 1
    try:
        while True:
            for u in range(500):
                for g in range(1, 500):
                    yield update[(g + u * 500 - 1) * chunks]
            chunks += 1
    except IndexError:
        pass

5. Since we've removed the file manipulation we can merge the loop to make u and g together. This is because they together make the range 500*500.
6. We can change the while True to a for that calls itertools.count.

import itertools

def generate_values(update):
    try:
        for chunks in itertools.count(1):
            for index in range(250000):
                yield update[index * chunks]
    except IndexError:
        pass

---

Let's look at how you interact with the files.

while True:
    for u in range(1,501):
        with open('some_file.txt', "a") as file:
            file.write(...)
        with open('some_file.txt', "r") as file:
            l=file.read()
        with open('some_file.txt', "w") as file:
            file.write("")
        with open('some_other_file.txt', "a") as file:
            file.write("\n"+l)
        l=""

It doesn't make sense to not just write to some_other_file.txt in append mode.

with open('some_other_file.txt', "a") as file:
    for item in generate_values(update):
        file.write("\n" + item)

If you want to display the output then you can just use itertools.islice. And you can use enumerate to build the index.

def chunk_iter(it, amount):
    while True:
        chunk = list(itertools.islice(it, amount))
        if not chunk:
            break
        yield chunk


with open('some_other_file.txt', "a") as file:
    for i, chunk in enumerate(chunk_iter(generate_values(update), 100)):
        print("adding to file...")
        print(f"total loaded: {i * 100 + len(chunk)}/{len(update)}")
        for item in chunk:
            file.write("\n" + item)