Questions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
465
questions
0
votes
0answers
28 views
bs4 HTML cleaner function with repetitive nested if clauses,
[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
4
votes
1answer
89 views
Scraping next page using BeautifulSoup
I have created a script for article scraping - it finds title, subtitle, href-link, and the time of publication. Once retrieved, information is converted to a pandas dataframe, and the link for the ...
2
votes
2answers
73 views
CeWL written in Python
This is my implementation of CeWL, a ruby application that returns a list of words which can be used for password crackers.
My main concern is performance, as it takes a couple seconds to run. I ...
4
votes
1answer
89 views
XML parser + bs4 combo
I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry.
Now, each job entry page is visited to grab info not present in the xml feed (...
9
votes
2answers
969 views
DeliciousSoda: A simple parser for robots.txt files
This is my first big project that I've decided to release publically. It's available on my Github, and through pip3: ...
2
votes
1answer
65 views
Scrape company data from Google search
I have written this code for crawling and scraping search results from Google. But it's super slow when I run it.
What it does: read a pandas column, search values in Google and take the first result ...
3
votes
1answer
63 views
Scraping IMDB and exporting to JSON or/and CSV
This code is scraping the data from IMDB and displays it in JSON or CSV format.
I plan to extend it beyond the top rated movies. How can I improve the quality of my code? I am using Python 3.7.3.
<...
1
vote
0answers
78 views
Homemade web-scraper that feeds perceptron with hidden layer [closed]
I’d like to start by apologising for the mess that is my code. I know that it’s really hard to parse, inefficient, and probably not idiomatic. I just don’t know how to fix it which is why I'm here.
...
5
votes
1answer
199 views
Ultra fast Amazon scraper multi-threaded
This is a follow up to the code here: Web scraper that extracts urls from Amazon and eBay
A multi-threaded modification to the previous version that is Amazon focused and most of the necessary ...
13
votes
3answers
2k views
AniPop - The anime downloader
Note: The topics of performance and Selenium/BS4 have not yet been addressed,
so this question can still receive a better answer!
Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
6
votes
1answer
154 views
Web scraper that extracts urls from Amazon and eBay
Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case
of previously saved files, the files will be read and no ...
4
votes
0answers
121 views
The anime downloader [duplicate]
NOTE: Here's the latest version of this program, since this question idled out.
This is a recreational script made to update my home server w/ the latest anime from HorribleSubs. I'd like to know if ...
3
votes
2answers
90 views
Get a notification when questions get posted
I like to help people on StackOverflow, matter of fact nowadays it depends on how quick you are when answering question.
With that being said, I can't always keep a tab and an eye to refresh the ...
2
votes
1answer
110 views
High Speed Python Web Scraper Optimised
I needed a lot of data for a tensorflow project so I made a web scraper to get all of the text and links off of websites then to repeat the process at all of those links.
I left it on overnight and ...
5
votes
3answers
131 views
Effective fetching data algorithm
I'm fetching some data by using bs4, and want to know why access speed gets slower as my program progresses.
I'm using thread, and I know threading isn't so fast because of GIL.
My problem is that ...
3
votes
1answer
72 views
Script to download Annual Reports (~70,000 in total) from a Website
It's only 38 lines of code and I haven't worked with web scraping that much before.
How heavy would my code be on their server?
It's for Deep Learning purposes and I haven't ran it yet but ...
3
votes
1answer
43 views
Monitor a web page for when a desired price is met
I have a code that keeps checking a webpage with a price on it and when the price meets a number I set it to, the thing is purchased. I am looking for ways to increase the speed of it.
I have added ...
32
votes
7answers
9k views
Tiny image scraper for xkcd.com
This came out on the spur of the moment, as a quick and dirty tool to have the job done.
This is a simple image scraper for immensely popular and legendary comic website inspired by a Python Easter ...
7
votes
2answers
332 views
Network helper class with retry logic on failure
I wrote an helper class which allow me to request the body content from a site that is created by AJAX, for doing so I'm using <...
3
votes
1answer
41 views
Wiki Link Mapper — A webcrawler which crawls links recursively and prints them into a graph
I would like some suggestions on how to make it better and also if any features that i should add to this Script. I am using beautifulsoup,requests for scraping and networkx for plotting the graph.
...
2
votes
2answers
84 views
Jsoup connection to URL
I have simple class that I want to ask if is there any possible to improve it? I mean, for me it looks poor. Is there any way to use here try-with-resources, stream or ...
3
votes
2answers
162 views
Web-scraping investment fund data into Pandas
Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
4
votes
3answers
198 views
Why is my Python web scraper taking so long?
I am trying to create a program to download a long list of websites using Python 3.7. By using multiprocessing for each request, my code runs much faster than when it's run synchronously.
I've added ...
2
votes
1answer
223 views
Python Script to download images and videos from a user's profile on Instagram
The script downloads images and videos from a user's profile page on Instagram. It works fine, but I'd like to know if there are ways to improve it. All ideas and suggestions are welcome.
...
3
votes
1answer
73 views
the fox is black - a simple image scraper
I wrote a simple image scraper for the Desktop Wallpaper project by The Fox is Black. The site is a fantastic (yet, apparently dead) blog full of fabulous images. However, there's no way to download ...
4
votes
0answers
52 views
Scraping a hiring website using python's requests and BeautifulSoup
I'm designing a scraping application using python, requests and BeautifulSoup4.
I decided to divide the logic into two classes:
Spider : gets the base url ...
5
votes
2answers
68 views
E-commerce scraper needs shortening
How would I go about shortening this code?
I'm sure there's a smarter way to write the code.
It's working fine, it's just ugly.
It is a way to scrape the number of items on an e-commerce site.
<...
2
votes
0answers
37 views
Scraping Forum Tables w/ Links Using Beautiful Soup
This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
4
votes
1answer
61 views
Webscraper code to download a manga series
I wrote a program to download a manga series fron www.mangapanda.com
Here it is:
...
3
votes
1answer
245 views
Sticking to a working proxy generated by a rotating proxy script
I've created a script in python to make proxied requests by picking working proxies from a list of proxies scraped from a free proxy site. The bot traverses few links to parse the urls of the ...
1
vote
1answer
45 views
Python classes to grab HTTP response from website entities
I am trying to implement a parent WebContent class and child classes to grab HTTP responses from the actual website entities. There are highlevel codes below and I ...
1
vote
1answer
67 views
Python program that gets my hackerrank code and scores using requests [closed]
I'm trying to extend my coding skills to other languages by doing projects. (learned Java in school) and since python is very popular right now thats what I ended up choosing for this project, though ...
10
votes
1answer
115 views
Strip Dilbert Comics
A friend of mine is a huge fan of a well-known comic website and he recently said that it'd be great to have a tool that downloads comic strips for a given date range, instead of clicking through the ...
6
votes
0answers
81 views
Get a movie directory, rename it and save imdb info as a webpage
I have my movie folders formatted:
~: no sub included
{}: already watched
nothing: sub included, ready to watch
the program asks for the movie directory; extracts the name and searches IMDb for the ...
3
votes
1answer
843 views
Python script to download adult comics from 8muses
I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the ...
2
votes
2answers
82 views
Simple Web Crawler for Small Sites
This script is my web crawlerfor small sites. Can you possibly review it for best coding practices?
Code
...
7
votes
1answer
61 views
ArXiV Technical Paper API Github Repo
My name is Ethan and I am trying to build an API for scaping technical papers to use for developers. Right now it only works for ArXiV but I would greatly appreciate some mentoring or a code review of ...
4
votes
0answers
134 views
Return different types from web scraper method
Application Logic
The software take odds data from an internet site and store them inside my own database, this application is basically a scraper.
There are different types of odds, eg:
FullTime
...
3
votes
1answer
88 views
Web scraping job postings from Facebook
I built a web scraper to pull a list of jobs on Facebook and other websites but want to break up the code into functions that I can reuse for other websites. This structure is working but I think it ...
3
votes
1answer
74 views
Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)
The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
4
votes
1answer
86 views
Batch retrieve web titles from URLs
This working script reads a list of URLs from one or more text files,
and for each, retrieves the page title from the Internet
and appends the results in the format
...
5
votes
2answers
136 views
Web scraper for football data with three very similar methods
Application Logic
I wrote a web scraper that store some football data inside my database. Those data need to be update if 7 days are passed since the last update.
I have three types of record:
Team
...
2
votes
1answer
74 views
Python webscraper to get property info from Tripadvisor
I'm very much a beginner at programming, and hoping to get some advice!
I'm having some trouble with a tripadvisor scraper being slow, and have identified the part of my code that is taking a while. ...
7
votes
1answer
218 views
Scraper to grab a Stack Overflow profile name
I've created a script in Python to log into stackoverflow.com using credentials and fetch the profilename once logged in. I've tried to do it using class. I created the methods within that class in ...
0
votes
0answers
101 views
Data Scraping Using VBA
I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance. Thanks In Advance
Note-I haven't posted ...
7
votes
1answer
105 views
Web-scraping C.R. Laurence product catalog
In the last couple of days I've tried to play around with some scraping using XPATHs and Python to consolidate my knowledge (since it's been some time). I did this on a simple website which doesn't ...
6
votes
1answer
103 views
NodeJs crawler for recipes
Here is my try on a crawler made in nodeJs with cheerio, I made it with the idea in mind to use it in a future project I wanna make. Here is the git link: https://github.com/Just4lol/CookCrawler
If ...
4
votes
2answers
211 views
PHP script that writes a JSON file with iextrading API data
iextrading has two APIs (v1, v2) which provide financial market data. Following script records a JSON file with their equity information (using a CRON job).
Would you be kind and review it for any ...
1
vote
0answers
229 views
Basic Kibana/Elasticsearch Search - Powershell
I am traditionally a Java/Backend Developer, and have recently been spending a lot of time dealing with issues that higher tiers of support should be able to handle themselves. To assist them with ...
4
votes
1answer
637 views
NFL data scraper
I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
