3
\$\begingroup\$

The code below matches a list of features to a large corpus and returns the sub-query match with a score above 80. The challenge is the list of features on the full data-set is > 5,000 and comparing to multiple documents. Therefore is taking too long to work using the fuzzywuzzy package.

Per the Spyder profiler, the bottlenecks are at:

if(fuzz.ratio(wordtocompare,feature.lower())> match) and _find_and_load_unlocked

Would vectorizing the code in its current form help or is there a faster approximate matcher that accounts for matching a sub-query (information extraction) of text compared to a defined list? Has anyone had success using polyleven and port the results back into Python?

import pandas as pd
from fuzzywuzzy import fuzz
import re

document = """If you're shopping within the Toyota family, the Highlander offers appreciably more space than the RAV4, both in terms of cargo capacity and its extra row of seats. It also has a deeper, more accessible space than what's in the 4Runner.
That said, the Highlander is one of the smallest three-row crossovers available. Apart from the Kia Sorento and maybe the Mazda CX-9, you're going to find more cargo capacity and passenger space in the Highlander's competitors. That's especially true in the third row. The second row slides a bit more to grant extra legroom now, but the third row remains awfully close to the floor, and it won't be long before your growing kids will feel cramped and claustrophobic in the way-back. Full-size teens and adults will be flat-out grumpy.
That said, the Highlander's smaller size might be just right for many buyers who appreciate its more manageable dimensions when parking or maneuvering in tight spots. Plus, if you only need that third row for occasional use and just a little more space than what a RAV4 provides, it really won't matter that the Highlander can't match its competitors' jumbo size.
We expect pricing for the 2020 Highlander to be announced closer to its on-sale date in December 2019, with the Hybrid arriving in February 2020. Specifically, it should correspond with our first test drive opportunity, likely in November. We do have a pretty comprehensive features breakdown, however, which you can see below.
Standard equipment on the Highlander L includes 18-inch alloy wheels, three-zone automatic climate control, accident avoidance tech features (see safety section below), full-speed adaptive cruise control, LED headlights, rear privacy glass, proximity entry and push-button start, an eight-way power driver seat and the 8-inch touchscreen. The LE additions include a power liftgate, blind-spot warning, LED foglamps, and a leather-wrapped steering wheel.
The XLE additions include automatic headlights, roof rails, a sunroof, heated front seats, driver power lumbar, a four-way power passenger seat, SofTex vinyl upholstery, second-row sunshades and an auto-dimming rearview mirror.
The Limited additions include 20-inch wheels, a handsfree power liftgate, upgraded LED headlights, a cargo cover, driver memory settings, ventilated front seats, leather upholstery, integrated navigation and a JBL sound system upgrade.
The Platinum additions include adaptive and self-leveling headlights, automatic wipers, a panoramic sunroof bird's-eye parking camera, a head-up display, a digital rearview mirror camera, perforated leather upholstery, heated second-row seats and a 12.3-inch touchscreen.
"""
features =["steering","touch screen","LED headlight"]

def findcarfeatures(features, document, match=80):
    result=[]
    for feature in features:
        lenfeature = len(feature.split(" "))
        word_tokens = nltk.word_tokenize(document)
        #filterd_word_tokens = [w for w in word_tokens if not w in stop_words]
        for i in range (len(word_tokens)-lenfeature+1):
            wordtocompare = ""
            j=0
            for j in range(i, i+lenfeature):
                if re.search(r'[,!?{}\[\]\"\"\'\']',word_tokens[j]):
                    break
                wordtocompare = wordtocompare+" "+word_tokens[j].lower()
            wordtocompare.strip()
            if not wordtocompare=="":
                if(fuzz.ratio(wordtocompare,feature.lower())> match):
                    result.append([wordtocompare,feature,i,j])
    return result

findcarfeatures(features,document)

Out[90]: 
[[' steering', 'steering', 353, 353],
 [' touchscreen .', 'touch screen', 334, 335],
 [' touchscreen .', 'touch screen', 474, 475],
 [' led headlights', 'LED headlight', 313, 314],
 [' headlights', 'LED headlight', 314, 315],
 [' headlights', 'LED headlight', 361, 362],
 [' led headlights', 'LED headlight', 408, 409],
 [' headlights', 'LED headlight', 409, 410],
 [' headlights', 'LED headlight', 442, 443]]
\$\endgroup\$
6
  • \$\begingroup\$ Have you profiled the code and do you know where your bottlenecks are? \$\endgroup\$ – pacmaninbw Apr 15 '20 at 20:02
  • \$\begingroup\$ @pacmaninbw hi, the if(fuzz.ratio(wordtocompare,feature.lower())> match): portion is where it's taking a long time to process. \$\endgroup\$ – Rtimeseries Apr 15 '20 at 20:04
  • \$\begingroup\$ Please add that information to the question body. \$\endgroup\$ – pacmaninbw Apr 15 '20 at 20:08
  • \$\begingroup\$ Welcome to Code Review! Please see What to do when someone answers. I have rolled back Rev 5 → 4 \$\endgroup\$ – Sᴀᴍ Onᴇᴌᴀ Apr 16 '20 at 16:49
  • \$\begingroup\$ (Just winced - anyone remember Neuro-Linguistic Programming? \$\endgroup\$ – greybeard Apr 20 '20 at 15:25
3
\$\begingroup\$

Minor perf improvements

These are unlikely to impact your performance in a material way, but they are performance improvements nonetheless:

re.search(r'[,!?{}\[\]\"\"\'\']',word_tokens[j])

recompiles the regex every time. re.compile() outside of your loops so that this does not happen.

Repeated concatenation such as this:

wordtocompare = wordtocompare+" "+word_tokens[j].lower()

can be a problem; strings in Python are immutable, so this is recreating a new string instance every time the concatenation is done. To avoid this, consider using StringIO or join a generator.

Other improvements

if not wordtocompare=="":

should be

if word_to_compare != "":

Also, wordtocompare.strip() is not being assigned to anything so it does not have any effect, currently.

\$\endgroup\$
8
  • \$\begingroup\$ Comparing word_to_compare against an empty string could just be if word_to_compare. String concatenation could be done simply by collecting elements into a list, then calling " ".join(list). \$\endgroup\$ – Alex Povel Apr 28 '20 at 9:05
  • 1
    \$\begingroup\$ Re. comparing: you could do that, but it would not be strictly equivalent (it would also include None). \$\endgroup\$ – Reinderien Apr 28 '20 at 13:55
  • 1
    \$\begingroup\$ @HansLollo Re. concatenation: sometimes join will be worth it. What you do want to do is join from a generator. What you don't want to do is join from a list. \$\endgroup\$ – Reinderien Apr 28 '20 at 13:56
  • \$\begingroup\$ Yes, the comparison would no longer be equivalent. It should only be preferred if the strictness of comparing against an empty string is not needed. What do you mean with join from a list? Joining a generator is possibly the most efficient, but shouldn't joining a list still be more efficient than string concatenation? \$\endgroup\$ – Alex Povel Apr 28 '20 at 14:07
  • 1
    \$\begingroup\$ Joining will be more efficient than "naive" concatenation, but not necessarily more efficient that StringIO. That depends on a number of factors, including Python version and input size. See github.com/reinderien/py-cat-times \$\endgroup\$ – Reinderien Apr 28 '20 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.