-4
\$\begingroup\$

Here is a piece of code for acquiring URLS

import re
import requests
from bs4 import BeautifulSoup

for i in range(7):
    r = requests.get(("http://www.yellowpages.com/atlanta-ga/tae-kwon-do?page={}").format(i))
    soup = BeautifulSoup(r.content , "html.parser")
    for link in soup.find_all("a",href=re.compile('/atlanta-ga/mip/')):
        if 'yp-rating' not in link.get('href'):
           if 'geo_location' not in link.get('href'):
              link = (link.get('href'))
              i = set(link)
              for link in i:
                links = (("http://www.yellowpages.com{}").format(link))
                print(links)

Have tried many variations and all not getting rid of duplicates.

Here was the original that will give the full list of urls

import re

import requests
from bs4 import BeautifulSoup

for i in range(7):
   r = requests.get(("http://www.yellowpages.com/atlanta-ga/tae-kwon-do?page={}").format(i))
   soup = BeautifulSoup(r.content , "html.parser")
   for link in soup.find_all("a",href=re.compile('/atlanta-ga/mip/')):
      if 'yp-rating' not in link.get('href'):
        if 'geo_location' not in link.get('href'):
            link = (link.get('href'))
            links = (("http://www.yellowpages.com{}").format(link))
            print(links)

Please correct me on how to remove the list of duplicates here.

\$\endgroup\$

1 Answer 1

0
\$\begingroup\$

You are calling set too early. Create a unique set of links before you enter the first for loop:

allLinks = set()
for i in range(7):
    ...

Then instead of print(link) at each iteration, just add link to that set; this ensures that it is unique. Or if you want to print it right away, do something like this instead of the print(link)

if link not in allLinks:
    print(link)
allLinks.add(link)
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.