Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
If you were paying attention, the website adds \url\default\url\video\default-N.htm at the end of the url depending on the set of pages you click in the bottom of the page. There's a total of 13 urls.
If you were paying attention, the website adds \url\default-N.htm at the end of the url depending on the set of pages you click in the bottom of the page. There's a total of 13 urls.
If you were paying attention, the website adds \url\video\default-N.htm at the end of the url depending on the set of pages you click in the bottom of the page. There's a total of 13 urls.
it's not necessary to create a new session in all of your functions. It's preferred to create only one session for the same website as it can give a boost of speed to your scrapper. More, the session object allows you to persist certain parameters across requests (if needed);
you only use got_already in your last function, so define it there. Or better yet, use a Set()
instead of if "author" not in title and "year" not in title you can use all()
PAGE_LINK is irrelevant and confusing. You can build your full video links using just BLINK.
last but not least, I'd use beautifulsoup for html parsing: but, here I'd like to add that The downside of using this parser is that it is much slower than the HTML parser of lxml. So if performance matters, you might want to consider using soupparser only as a fallback for certain cases.
it's not necessary to create a new session in all of your functions. It's preferred to create only one session for the same website as it can give a boost of speed to your scrapper. More, the session object allows you to persist certain parameters across requests (if needed);
you only use got_already in your last function, so define it there. Or better yet, use a Set()
instead of if "author" not in title and "year" not in title you can use all()
PAGE_LINK is irrelevant and confusing. You can build your full video links using just BLINK.
last but not least, I'd use beautifulsoup for html parsing
it's not necessary to create a new session in all of your functions. It's preferred to create only one session for the same website as it can give a boost of speed to your scrapper. More, the session object allows you to persist certain parameters across requests (if needed);
you only use got_already in your last function, so define it there. Or better yet, use a Set()
instead of if "author" not in title and "year" not in title you can use all()
PAGE_LINK is irrelevant and confusing. You can build your full video links using just BLINK.
last but not least, I'd use beautifulsoup for html parsing: but, here I'd like to add that The downside of using this parser is that it is much slower than the HTML parser of lxml. So if performance matters, you might want to consider using soupparser only as a fallback for certain cases.
