1
\$\begingroup\$

I have a loop that's updating a dictionary's values a large amount of time (by adding the new value to the old. Each value for each dictionary key is actually a list of 3 number, and so for each dictionary value the data structures that I thought of using are:

  • Python Lists:
    The problem is that lists are inefficient at modifying numbers inside of them, and require indexing.

  • Numpy Arrays:
    The problem is that numpy arrays are immutable and are taking a huge amount of time to do this.

  • Python tuples:
    The worst of each world

My code looks like this:

class ModelErrors():

    def __init__(self):
        self.fwhm = 0.7 #arcsec
        self.sigma = self.fwhm/(2*np.sqrt(2*np.log(2)))
        self.TrM = 2*self.sigma**2
        self.PSF.TrM = self.TrM
        self.STAR.TrM = self.TrM
        self.survey_area = (np.radians(200),np.radians(50)) # in radians
        stars_X = np.random.rand(int(2000))*self.survey_area[0] 
        Y = 0.77*np.random.rand(int(2000))/2 + 0.5
        stars_Y = np.arccos(2*Y-1) - np.pi/2 # random points on a sphere
        stars = np.array((stars_X,stars_Y))
        self.stars = stars.swapaxes(1,0)
        self.DELTA.M = defaultdict(list)
        self.STAR.M = defaultdict(list)
        self.PSF.M = defaultdict(list)
        self.n = defaultdict(int)

    def getModels(self, positions):
        '''calls one of the model methods to create an overly simplified model, 
        or (N/A yet) imports one
        '''
        for i in range(len(positions)):
            self.radial_pattern(position_num=i)

    def radial_pattern(self,position_num):
        ''' method to create a radial pattern (one of the overly simplified models),
            e = 0.05* distance from origin
        '''
        from angles import r2d, r2arcs, d2arcs
        center = self.positions[position_num]
        stars = self.stars[np.where(((self.stars[:,0]-center[0])**2+(self.stars[:,1]-center[1])**2)<np.radians(1.2))[0]] 
        rel_X, rel_Y = stars[:,0] - center[0], stars[:,1] - center[1]
        r = np.sqrt(rel_X**2+rel_Y**2)
        theta = np.arctan((rel_Y)/(rel_X))
        stare1 = r*np.cos(2*theta)*5
        stare2 = r*np.sin(2*theta)*5
        psfe1 = stare1/1.1
        psfe2 = stare2/1.1
        starMxx = 0.5*self.STAR.TrM*(stare1+1)
        starMxy = 0.5*self.STAR.TrM*stare2
        starMyy = 0.5*self.STAR.TrM*(-stare1+1)
        psfMxx = 0.5*self.PSF.TrM*(psfe1+1)
        psfMxy = 0.5*self.PSF.TrM*psfe2
        psfMyy = 0.5*self.PSF.TrM*(-psfe1+1)

        # populate new defaultdict with star/psf M[pos]=(Mxx,Mxy,Myy)
        for ind in range(len(psfMxx)):
            if (stars[ind][0],stars[ind][1]) in self.STAR.M.keys():
                self.n[(stars[ind][0],stars[ind][1])]+=1
            self.STAR.M[(stars[ind][0],stars[ind][1])]+=[starMxx[ind],starMxy[ind],starMyy[ind]]
            self.PSF.M[(stars[ind][0],stars[ind][1])]+=[psfMxx[ind],psfMxy[ind],psfMyy[ind]]

and then I call:

instance = ModelErrors()
instance.getModels(positions)

Is there a better data structure/more efficient way to do this?

\$\endgroup\$
  • 1
    \$\begingroup\$ I'd be interested on your source for "lists are inefficient at modifying numbers inside of them". Also the code is a stub, and I wouldn't be able to review it on the most part. \$\endgroup\$ – Peilonrayz Nov 16 '17 at 14:45
  • \$\begingroup\$ Thanks @Ludisposed I updated it to include my actual code. \$\endgroup\$ – hsnee Nov 16 '17 at 15:43
  • \$\begingroup\$ @Peilonrayz lists do not support addition, so one cannot do lst+=1, it would just add '1' to the list rather than to each element. To add it to each element one would have to update each element by it's index, and doing this inside a for-loop in python is inefficient. \$\endgroup\$ – hsnee Nov 16 '17 at 15:45
  • \$\begingroup\$ @hsnee You can subclass list and modify the behaviour of __add__ to archieve this behaviour. \$\endgroup\$ – Richard Neumann Nov 16 '17 at 16:41
  • \$\begingroup\$ @RichardNeumann the only way I'm thinking about doing that is by moving the indexing into the __add__ method, which wouldn't make this more efficient, would it? Is there a better way to do this? \$\endgroup\$ – hsnee Nov 16 '17 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.