I have written a program to optimize a point cloud in dependency of their distances to each other. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I have 300000 points in the point cloud. The code is still running after almost 30 hours.

I am sure there is a pythonic way to optimze the code. But I do not know how can I reduce the calculation time and make the performance better. I have read alot about multithreding, chunk size etc. But I think there is no need for this size of dat. I think, I am using the memory in a very bad way. Am I right? Here is the code:

import pickle
import math
import itertools
import gc

def dist(p0, p1):
    return math.sqrt((p0[0] - p1[0])**2 + (p0[1] - p1[1])**2)

def join_pair(points, r):
    for p, q in itertools.combinations(points, 2):
        if dist(p, q) < r:
            points.remove(q)
            return True
    return False


if __name__ == "__main__":
    filename = 'points.txt'
    filename2 = 'points_modified.txt'
    mynumbers = []
    with open(filename) as f:
        for line in f:
            mynumbers.append([float(n) for n in line.strip().split(' ')])
        mynumbers = sorted(mynumbers, key=lambda tup: (tup[1], tup[0]))
    gc.collect()    
    while join_pair(mynumbers, 3.6):
        pass
    
    with open (filename2,"w")as fp2:
        for j in range(len(mynumbers)):
            fp2.write(' '.join(map(repr, mynumbers[j]))+'\n')


        

and the input data is like the following:

661234.58 5763766.03 72.63
661254.81 5763765.08 75.04
661244.86 5763764.14 74.99
661234.90 5763763.21 74.94
661225.13 5763762.29 74.89

I have written a program to optimize a point cloud in dependency of their distances to each other. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I have 300000 points in the point cloud. The code is still running after almost 30 hours.

I am sure there is a pythonic way to optimze the code. But I do not know how can I reduce the calculation time and make the performance better. I have read alot about multithreding, chunk size etc. But I think there is no need for this size of data. I think, I am using the memory in a very bad way. Am I right? Here is the code:

import pickle
import math
import itertools
import gc

def dist(p0, p1):
    return math.sqrt((p0[0] - p1[0])**2 + (p0[1] - p1[1])**2)

def join_pair(points, r):
    for p, q in itertools.combinations(points, 2):
        if dist(p, q) < r:
            points.remove(q)
            return True
    return False


if __name__ == "__main__":
    filename = 'points.txt'
    filename2 = 'points_modified.txt'
    mynumbers = []
    with open(filename) as f:
        for line in f:
            mynumbers.append([float(n) for n in line.strip().split(' ')])
        mynumbers = sorted(mynumbers, key=lambda tup: (tup[1], tup[0]))
    gc.collect()    
    while join_pair(mynumbers, 3.6):
        pass
    
    with open (filename2,"w")as fp2:
        for j in range(len(mynumbers)):
            fp2.write(' '.join(map(repr, mynumbers[j]))+'\n')


        

and the input data is like the following:

661234.58 5763766.03 72.63
661254.81 5763765.08 75.04
661244.86 5763764.14 74.99
661234.90 5763763.21 74.94
661225.13 5763762.29 74.89

I have written a code to optimize a point cloud in dependency of their distances to each other. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I have 300000 points in the point cloud. The code is still running after almost 30 hours.

I am sure there is a pythonic way to optimze the code. But I do not know how can I reduce the calculation time and make the performance better. I have read alot about multithreding, chunk size etc. But I think there is no need for this size of dat. I think, I am using the memory in a very bad way. Am I write? Here is the code  :

import pickle
import math
import itertools
import gc

def dist(p0, p1):
    return math.sqrt((p0[0] - p1[0])**2 + (p0[1] - p1[1])**2)

def join_pair(points, r):
    for p, q in itertools.combinations(points, 2):
        if dist(p, q) < r:
            points.remove(q)
            return True
    return False


if __name__ == "__main__":
    filename = 'points.txt'
    filename2 = 'points_modified.txt'
    mynumbers = []
    with open(filename) as f:
        for line in f:
            mynumbers.append([float(n) for n in line.strip().split(' ')])
        mynumbers = sorted(mynumbers, key=lambda tup: (tup[1], tup[0]))
    gc.collect()    
    while join_pair(mynumbers, 3.6):
        pass
    
    with open (filename2,"w")as fp2:
        for j in range(len(mynumbers)):
            fp2.write(' '.join(map(repr, mynumbers[j]))+'\n')


        

and the input data is like the following:

661234.58 5763766.03 72.63
661254.81 5763765.08 75.04
661244.86 5763764.14 74.99
661234.90 5763763.21 74.94
661225.13 5763762.29 74.89

I have written a program to optimize a point cloud in dependency of their distances to each other. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I have 300000 points in the point cloud. The code is still running after almost 30 hours.

I am sure there is a pythonic way to optimze the code. But I do not know how can I reduce the calculation time and make the performance better. I have read alot about multithreding, chunk size etc. But I think there is no need for this size of dat. I think, I am using the memory in a very bad way. Am I right? Here is the code:

import pickle
import math
import itertools
import gc

def dist(p0, p1):
    return math.sqrt((p0[0] - p1[0])**2 + (p0[1] - p1[1])**2)

def join_pair(points, r):
    for p, q in itertools.combinations(points, 2):
        if dist(p, q) < r:
            points.remove(q)
            return True
    return False


if __name__ == "__main__":
    filename = 'points.txt'
    filename2 = 'points_modified.txt'
    mynumbers = []
    with open(filename) as f:
        for line in f:
            mynumbers.append([float(n) for n in line.strip().split(' ')])
        mynumbers = sorted(mynumbers, key=lambda tup: (tup[1], tup[0]))
    gc.collect()    
    while join_pair(mynumbers, 3.6):
        pass
    
    with open (filename2,"w")as fp2:
        for j in range(len(mynumbers)):
            fp2.write(' '.join(map(repr, mynumbers[j]))+'\n')


        

and the input data is like the following:

661234.58 5763766.03 72.63
661254.81 5763765.08 75.04
661244.86 5763764.14 74.99
661234.90 5763763.21 74.94
661225.13 5763762.29 74.89

I have written a code to optimize a point cloud in dependency of their distances to eachother. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I have 300000 points in the point cloud. The code is still running after almost 30 hours.

I am sure there is a pythonic way to optimze the code. But I do not know how can I reduce the calculation time and make the performance better. I have read alot about multithreding, chunk size etc. But I think there is no need for this size of dat. I think, I am using the memory in a very bad way. Am I write?Here is the code :

    import pickle
    import math
    import itertools
    import gc
    
    def dist(p0, p1):
        return math.sqrt((p0[0] - p1[0])**2 + (p0[1] - p1[1])**2)
    
    def join_pair(points, r):
        for p, q in itertools.combinations(points, 2):
            if dist(p, q) < r:
                points.remove(q)
                return True
        return False
    
    
    if __name__ == "__main__":
        filename = 'points.txt'
        filename2 = 'points_modified.txt'
        mynumbers = []
        with open(filename) as f:
            for line in f:
                mynumbers.append([float(n) for n in line.strip().split(' ')])
            mynumbers = sorted(mynumbers, key=lambda tup: (tup[1], tup[0]))
        gc.collect()    
        while join_pair(mynumbers, 3.6):
            pass
        
        with open (filename2,"w")as fp2:
            for j in range(len(mynumbers)):
                fp2.write(' '.join(map(repr, mynumbers[j]))+'\n')
        

and the input data is like the following

    661234.58 5763766.03 72.63
    661254.81 5763765.08 75.04
    661244.86 5763764.14 74.99
    661234.90 5763763.21 74.94
    661225.13 5763762.29 74.89

I have written a code to optimize a point cloud in dependency of their distances to each other. The code works very well for smaller number of points. For 1700 points it takes ca. 6 minutes. But I have 300000 points in the point cloud. The code is still running after almost 30 hours.

I am sure there is a pythonic way to optimze the code. But I do not know how can I reduce the calculation time and make the performance better. I have read alot about multithreding, chunk size etc. But I think there is no need for this size of dat. I think, I am using the memory in a very bad way. Am I write? Here is the code :

import pickle
import math
import itertools
import gc

def dist(p0, p1):
    return math.sqrt((p0[0] - p1[0])**2 + (p0[1] - p1[1])**2)

def join_pair(points, r):
    for p, q in itertools.combinations(points, 2):
        if dist(p, q) < r:
            points.remove(q)
            return True
    return False


if __name__ == "__main__":
    filename = 'points.txt'
    filename2 = 'points_modified.txt'
    mynumbers = []
    with open(filename) as f:
        for line in f:
            mynumbers.append([float(n) for n in line.strip().split(' ')])
        mynumbers = sorted(mynumbers, key=lambda tup: (tup[1], tup[0]))
    gc.collect()    
    while join_pair(mynumbers, 3.6):
        pass
    
    with open (filename2,"w")as fp2:
        for j in range(len(mynumbers)):
            fp2.write(' '.join(map(repr, mynumbers[j]))+'\n') 


        

and the input data is like the following:

661234.58 5763766.03 72.63
661254.81 5763765.08 75.04
661244.86 5763764.14 74.99
661234.90 5763763.21 74.94
661225.13 5763762.29 74.89