3
\$\begingroup\$

The following code asynchronously extract title from URLs (saved in book-urls.txt).

from bs4 import BeautifulSoup
import grequests

links = list()

file1 = open("book-urls.txt", "r")
Lines = file1.readlines()
for line in Lines:
    links.append(line)
file1.close()

reqs = (grequests.get(link) for link in links)
resp = grequests.map(reqs)

for r in resp:
    try:
        soup = BeautifulSoup(r.text)
        print(soup.title.string)
    except:
        pass

The input is:

https://learning.oreilly.com/library/view/a-reviewers-handbook/9781118025635/
https://learning.oreilly.com/library/view/accounting-all-in-one-for/9781119453895/
https://learning.oreilly.com/library/view/business-valuation-for/9780470344019/
https://learning.oreilly.com/library/view/the-business-of/9780470444481/

The output is:

A Reviewer's Handbook to Business Valuation: Practical Guidance to the Use and Abuse of a Business Appraisal [Book]
Accounting All-in-One For Dummies, with Online Practice, 2nd Edition [Book]
Business Valuation For Dummies [Book]
The Business of Value Investing: Six Essential Elements to Buying Companies Like Warren Buffett [Book]

Is grequests the best option or should I use something else?

Please give some suggestion on how I can make this code better.

\$\endgroup\$
1
  • 3
    \$\begingroup\$ The second paragraph of the grequests README advises you select a different tool. That's usually a pretty strong indicator. \$\endgroup\$ – FMc Sep 19 '20 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.