1
\$\begingroup\$

I've just picked coding back up for the first time in a long time, so I understand if your eyes bleed looking at this code. It all works, but I'd be grateful for any tips (how to improve the python code, how to use Spacy's semantic similarity vector feature better, how to make it work faster, etc).

The main product of the code is the list_of_matches list, which shows how I got from one page to another. Clearly this example goes from "frog" to "crystal", but theoretically those could be anything.

import spacy
import wikipedia
import en_core_web_lg
nlp = en_core_web_lg.load()

word_1 = 'frog'
word_2 = 'crystal'
count = 0
closest_match = word_1
list_of_matches = []

link_list_1 = [nlp(word) for word in(wikipedia.page("Frog", auto_suggest=False).links)]

for item in link_list_1:
    print(item)

while float(item.similarity(nlp(word_2))) != 1.0 and count != 20:
    if closest_match.lower() == word_2:
        list_of_matches.append(word_2)
        break

    list_of_matches.append(closest_match)
    try:
        link_list = [nlp(word) for word in wikipedia.page(closest_match, auto_suggest=False).links]
    except wikipedia.exceptions.DisambiguationError:
        pass
    count += 1
    highest_match = 0
    for item in link_list:
        if(item.vector_norm):
            if float(item.similarity(nlp(word_2))) != 0.0:
                similarity = (item.text, word_2, item.similarity(nlp(word_2)))
                #print(similarity)
                link, word_2, vector = similarity
                if vector > highest_match and link not in list_of_matches:
                    highest_match = vector
                    closest_match = link
                    print(closest_match, highest_match)

print(list_of_matches)
\$\endgroup\$
2
  • 1
    \$\begingroup\$ Is the indentation correct? Or should the while loop be indented further (in the body of the for item in link_list_1 loop? \$\endgroup\$
    – RootTwo
    Mar 15, 2021 at 5:43
  • \$\begingroup\$ @RootTwo No, it's meant to be outside of the while loop. While I'm sure there's a better way of doing this, I couldn't think of it at the moment. The for item in link_list_1 loop is simply there to assign the 'item' variable so it could be used in the condition of the while loop \$\endgroup\$
    – Will
    Mar 15, 2021 at 17:35

1 Answer 1

2
\$\begingroup\$

First a few observations, then we'll try to improve things.

  • "Frog" in the first list comprehension should be closest_match (which is initialized to 'frog').
  • The two list comprehensions are identical, which suggests the code could be rearranged to eliminate the redundancy.
  • In the while loop, item.similarity() already returns a float, so the float() call is redundant. Also, it seems fishy that the test involves whatever happend to be the last item in link_list_1 or linked_list.
  • Avoid hard coding "magic numbers" like 1.0, 0.0, and 20 in the code. Use a defined constants with meaningful names to make the code clearer. For example, while item.similarity(nlp(word_2))) != PERFECT_MATCH.
  • I suspect calling nlp() is computationally expensive, but the code calls nlp(word_2) repeatedly.
  • word is a poor variable name, particularly when the page titles or links often have multiple words.
  • If a DisambiguationError is raised, the error is ignored. But, closest_match isn't changed, so the error will repeat the next time through the loop...until the loop count limit is reached.

Now let's try to improve the code.

It looks like the basic idea behind your code is to try to get from one Wikipedia page to a target page by following links. The strategy for picking which link to follow is to pick the one with text that is most similar to the target page name. There doesn't appear to be any kind of breadth first search or back tracking or anything.

You got the code to grab the links on a page. Right now, the code runs nlp() on each link and check if it is similar to the target before checking if it's already been visited.

raw_links = wikipedia.page(current_page, auto_suggest=False).links
new_links = (link for link in raw_links if link not in path)

According to the spacy documentation it is more efficient to use nlp.pipe(), to process a sequence of texts. nlp.pipe() is a generator that yields a spacy.Doc for each text in the input.

next_pages = nlp.pipe(new_links)

But we want the page with the highest similarity to the target page. The Python builtin max() should work with an approriate key function:

target_page = nlp(target)  # this is done once outside the loop

next_page = max(nlp.pipe(new_links), key=target_page.similarity)

Of course, package it all up in a function:

def find_path(start, target, limit=20):
    start = start.lower()
    target = target.lower()
    
    try:    
        target_page = nlp(target)
    
        current = start
        path = [start]

        for _ in range(limit):
            raw_links = wikipedia.page(current, auto_suggest=False).links
            new_links = (link.lower() for link in raw_links if link not in path)   
            best_page = max(nlp.pipe(new_links), key=target_page.similarity)

            current = best_page.text
            path.append(current)
            
            if current == target:
                return path
                
    except wikipedia.exceptions.DisambiguationError as e:
        print(e)
    
    return []
    
START = "frog"
TARGET = "crystal"    

print(find_path(START, TARGET))

Note: I don't have spacy so I couldn't test this code. It may have some bugs, but based on the docs, I think it should point you in the right direction.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.