0
\$\begingroup\$

I have a set of code that calculates how well a templates fits a users details. It's kind of long, so I've cut out all the code that does it's job well and only included the bottle neck and surrounding code. My target here is speed!

A couple of notes on the variables:

Calculated at run time (changes each user):

  • TopSecHead - list of list of string - different sub list lengths
  • allSecScoreDicP1 - dict of list of float - always same length
  • allSecScoreDicP2 - dict of list of float - always same length
  • allSecReducerDicP1 - dict of list of float - always same length
  • allSecReducerDicP2 - dict of list of float - always same length

The last 4 above are all the same dimentions as each other

Calculated once and save/loaded (doesn't change per user):

  • docSecSizesFull - list of list of list of float - sub lists are not same length

  • shortSecSizesFull - list of list of list of float - sub lists are not same length

  • cutPointsFull - list of list of list of float - sub lists are not same length

  • tmplNumFull - list of list of list of float - sub lists are not same length

  • AllDocSplitsFull - list of list of list of float - sub lists are not same length

All five above are same dimentions as each other.

for x in range(4,8,1):

    docSecSizes = docSecSizesFull[x-4]
    shortSecSizes = shortSecSizesFull[x-4]

    cutPoints = cutPointsFull[x-4]
    tmpltNum = tmplNumFull[x-4]

    layoutNums = 0

    numTemps = len(docSecSizes)
    tmpsplits = []
    tmpsplits = [AllDocSplitsFull[x-4][z] for z in range(numTemps)]
    alltmplIds = [tmplNumFull[x-4][z] for z in range(numTemps)]

    for y in list(itertools.permutations(TopSecHead[x-4][1:])):

        tmpHeadSec = []
        tmpHeadSec.append('BasicInfo')

        headingIDs = []
        headingIDs.append(str(0))

        for z in y:
            tmpHeadSec.append(z)          
            headingIDs.append(str(headingLookups.index(z)))

        SectionIDs = ','.join(headingIDs)

        tmpvals = []


        tmpArray = []
        for key in allSecScoreDicP1:
            tmpArray.append(allSecScoreDicP1[key])
        nparr = np.array(tmpArray)
        print(nparr.transpose())

        for z in range(numTemps):

            docScore = 0
            docScoreReducer = 1

            for q in range(len(shortSecSizes[z])):

                if q < cutPoints[z]:
                    indexVal = shortSecSizes[z][q]
                    docScore+= allSecScoreDicP1[tmpHeadSec[q]][indexVal]
                    docScoreReducer *= allSecReducerDicP1[tmpHeadSec[q]][indexVal]
                else:
                    indexVal = shortSecSizes[z][q]
                    docScore+= allSecScoreDicP2[tmpHeadSec[q]][indexVal]
                    docScoreReducer *= allSecReducerDicP2[tmpHeadSec[q]][indexVal]
            docScore = docScore * docScoreReducer
            tmpvals.append(docScore)

        numTemplate = len(tmpvals)
        totaldocs += numTemplate
        sectionNum = [x] * numTemplate
        layoutNumIterable = [layoutNums] * numTemplate
        SectionIDsIterable = [SectionIDs] * numTemplate
        scoredTemplates.append(pd.DataFrame(list(zip(sectionNum,alltmplIds,layoutNumIterable,tmpvals,SectionIDsIterable,tmpsplits)),columns = ['#Sections','TemplateID','LayoutID','Score','SectionIDs','Splits']))
        layoutNums +=1


allScoredTemplates = pd.concat(scoredTemplates,ignore_index=True)

The problem code is this bit:

for z in range(numTemps):

    docScore = 0
    docScoreReducer = 1

    for q in range(len(shortSecSizes[z])):

        if q < cutPoints[z]:
            indexVal = shortSecSizes[z][q]
            docScore+= allSecScoreDicP1[tmpHeadSec[q]][indexVal]
            docScoreReducer *= allSecReducerDicP1[tmpHeadSec[q]][indexVal]
        else:
            indexVal = shortSecSizes[z][q]
            docScore+= allSecScoreDicP2[tmpHeadSec[q]][indexVal]
            docScoreReducer *= allSecReducerDicP2[tmpHeadSec[q]][indexVal]
    docScore = docScore * docScoreReducer
    tmpvals.append(docScore)

I've tried changing it to list comprehension but it was slower:

        docScore = [sum([allSecScoreDicP1[tmpHeadSec[q]][shortSecSizes[z][q]] if q < cutPoints[z] else allSecScoreDicP2[tmpHeadSec[q]][shortSecSizes[z][q]] for q in range(len(shortSecSizes[z]))]) for z in range(numTemps)]
        docReducer = [np.prod([allSecReducerDicP1[tmpHeadSec[q]][shortSecSizes[z][q]] if q < cutPoints[z] else allSecReducerDicP2[tmpHeadSec[q]][shortSecSizes[z][q]] for q in range(len(shortSecSizes[z]))]) for z in range(numTemps)]
        tmpvals = [docScore[x] * docReducer[x] for x in range(len(docScore))]

Any suggestions on optimisation methods would be massively appreciated. I have also attempted to convert the code to cython, I got it coverted and working, but it was about 10 times slower!

\$\endgroup\$
  • \$\begingroup\$ Welcome to CodeReview@SE. With performance concerns, it would be useful to include size of input and output as well as as many measurements as you see fit. A test input generator would help others gauge modifications. I've cut out all the code that does it's job well and only included the bottle neck and surrounding code fine if it did work out well, the ideal being real code from real projects. Can you provide a hyperlink to the full code? \$\endgroup\$ – greybeard Jan 25 at 8:31
  • \$\begingroup\$ @greybeard thanks for the input. I could provide the rest of the code, but the majority of the data is pulled from a mysql server so I can't really share that online (can I?). This is why I haven't shared the data, it's too big to be easily shareable and I can't create a simple function to replicate it \$\endgroup\$ – user6916458 Jan 25 at 18:28
  • 1
    \$\begingroup\$ If you have improved code, please post a new question with that code. Again, don't vandalize the current post. Putting in links in both this and the new question pointing towards each other is fine. We have a FAQ on how to handle iterative reviews indicating not to invalidate answers. \$\endgroup\$ – Mast Jan 26 at 8:21
3
\$\begingroup\$

Probably the most important thing is to reduce the amount of nested loops. It looks like it's currently n^4, so for every 1,000 items it'll loop around 1,000,000,000,000 times. I solved a similar problem once by having an initial loop which mapped the data structure into a less deeply nested form, and then I didn't need to do as many nested loops.

Also, as a side note, it would be good to name your variables rather than just using letters of the alphabet. Even just small changes like replacing x with num or number would improve the readability quite a bit.

| improve this answer | |
\$\endgroup\$
1
\$\begingroup\$

If "My target here is speed!" (and assuming you mean speed of execution) then the best advice is to move it into a compiled language. There are many, many benefits to Python but execution speed is seldom one of them.

The first step, as Andre O suggested, is to get a good algorithm. Python can be very helpful with that. The standard profile module can help you find where the code is spending its time and you can focus your optimization on that part of the code.

As an intermediate step to a fully compiled language you can take a Python program and move it to Cython which compiles the Python to machine language.

If your profiling finds certain portions are the most heavily used and slowest then you can code just that portion in C and call it from Python.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.