Questions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
483
questions
3
votes
0answers
15 views
Command line dictionary tool, using webscraping
This is my first proper Go program, after completing Go by Example and https://tour.golang.org. I have a background in Python.
This program scrapes definitions from Wordnik, then prints them nicely ...
2
votes
1answer
30 views
Instagram Scraping Posts Using Selenium
Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Log in in instragram using selenium and navigate to ...
3
votes
1answer
27 views
Dynamic web scraper in Python
The other week I faced the struggle of scraping dynamically generated content. So I used the selenium library with combination of requests & bs4, the thing is I am unsure of the quality of the ...
1
vote
3answers
47 views
Scraping the Bundesliga table and saving it to CSV
Wrote my first mini-script to get data from the german football league table and save the data to a csv file. Do you like the approach or could I be more efficient? Thanks in advance!
...
5
votes
1answer
59 views
Web Scraping with Python
My target website is Indeed. I tried to implement the scraper such that it is convenient for the end user. I am introducing my code in the readme file on my github repo.
I am somewhat a beginner in ...
2
votes
1answer
51 views
Instagram Scraping Using Selenium
Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Log in in instragram using selenium and navigate to ...
1
vote
1answer
122 views
Instagram Bot, selenium, web scraping
I made some changes in my code from the previous post.
The changes that I made:
I put all the functions to the class
All the global arrays I moved them to class too
Created ...
2
votes
1answer
77 views
Web scraping using selenium, multiprocessing, InstagramBot
An Instagram Bot which downloads the posts from profile
I have to mention my previous posts:
Instagram scraper Posts (Videos and Photos)
Scraping Instagram with selenium, extract URLs, download ...
3
votes
1answer
92 views
Scraping Instagram with selenium, extract URLs, download posts
I made a very simple Instagram Bot that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Creating directory for saving ...
4
votes
1answer
125 views
Instagram scraper Posts (Videos and Photos)
I wrote this code which has the ability to download images and videos from a specific Instagram profile.
Using multiprocessing and threading I managed to speed up the extraction of data.
My goal is ...
4
votes
1answer
77 views
Scraping Data From Multiple URLs into Single Dataframe
I have a Class that:
goes to a url
grabs a link from that page, and a date (filing_date)
navigates to the link, and
writes the table from that page to a ...
6
votes
1answer
60 views
Reach certain Wiki page through random links
I just read about Wiki game on Wikipedia and wanted to implement it as a fun project for myself using python.
Task:To reach Mathematics Wikipedia page using wikipedia hyperlinks randomly.
Using ...
1
vote
0answers
74 views
Web scraping dynamic content
Hi I'm fairly new to coding and would appreciate some feedback on the code. This is for a site that has a dynamic login page and infinite scrolling.
I wanted to use scrapy not necessarily because it ...
14
votes
2answers
2k views
Web crawler in Rust
I am learning Rust. I have written a web crawler that would scrape all the pages from my own blog (which is running on Ghost) and would generate a static version of it. Because of this, I'm not ...
2
votes
1answer
70 views
Tentative to transform this scrape script into a class object program
From the last few days I have been studying scrapping webpages and for further development I challenge myself to transform the script I have created into a class which will receive inputs from a user.
...
-1
votes
1answer
113 views
Instagram Scraper running to slow
Can someone help me how to speed up this scraping process getting all the usernames from the # 'cats' on instagram. My goal is to make this as fast as possible because currently the process is kinda ...
1
vote
1answer
91 views
Premier League data scraper
I have started a project where I am scraping JSON-data from an API. How the scraping is done right now is done in a very repetitive way where the keys of interest are specified and scraped. The data-...
4
votes
1answer
61 views
Beginner web scraper for Nagios
I am attempting to learn Python. It was suggested to me to try a web scraper, so I thought to get myself to look at multiple Nagios instances. I have not programmed in Python before, but learned from ...
1
vote
0answers
24 views
Trouble returning web scraping output as python dictionary [closed]
So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
1
vote
0answers
39 views
bs4 HTML cleaner function with repetitive nested if clauses,
[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
4
votes
1answer
111 views
Scraping next page using BeautifulSoup
I have created a script for article scraping - it finds title, subtitle, href-link, and the time of publication. Once retrieved, information is converted to a pandas dataframe, and the link for the ...
2
votes
2answers
82 views
CeWL written in Python
This is my implementation of CeWL, a ruby application that returns a list of words which can be used for password crackers.
My main concern is performance, as it takes a couple seconds to run. I ...
4
votes
1answer
97 views
XML parser + bs4 combo
I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry.
Now, each job entry page is visited to grab info not present in the xml feed (...
9
votes
2answers
981 views
DeliciousSoda: A simple parser for robots.txt files
This is my first big project that I've decided to release publically. It's available on my Github, and through pip3: ...
2
votes
1answer
68 views
Scrape company data from Google search
I have written this code for crawling and scraping search results from Google. But it's super slow when I run it.
What it does: read a pandas column, search values in Google and take the first result ...
3
votes
1answer
84 views
Scraping IMDB and exporting to JSON or/and CSV
This code is scraping the data from IMDB and displays it in JSON or CSV format.
I plan to extend it beyond the top rated movies. How can I improve the quality of my code? I am using Python 3.7.3.
<...
1
vote
0answers
78 views
Homemade web-scraper that feeds perceptron with hidden layer [closed]
I’d like to start by apologising for the mess that is my code. I know that it’s really hard to parse, inefficient, and probably not idiomatic. I just don’t know how to fix it which is why I'm here.
...
5
votes
1answer
231 views
Ultra fast Amazon scraper multi-threaded
This is a follow up to the code here: Web scraper that extracts urls from Amazon and eBay
A multi-threaded modification to the previous version that is Amazon focused and most of the necessary ...
13
votes
3answers
2k views
AniPop - The anime downloader
Note: The topics of performance and Selenium/BS4 have not yet been addressed,
so this question can still receive a better answer!
Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
6
votes
1answer
174 views
Web scraper that extracts urls from Amazon and eBay
Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case
of previously saved files, the files will be read and no ...
4
votes
0answers
122 views
The anime downloader [duplicate]
NOTE: Here's the latest version of this program, since this question idled out.
This is a recreational script made to update my home server w/ the latest anime from HorribleSubs. I'd like to know if ...
3
votes
2answers
95 views
Get a notification when questions get posted
I like to help people on StackOverflow, matter of fact nowadays it depends on how quick you are when answering question.
With that being said, I can't always keep a tab and an eye to refresh the ...
2
votes
1answer
150 views
High Speed Python Web Scraper Optimised
I needed a lot of data for a tensorflow project so I made a web scraper to get all of the text and links off of websites then to repeat the process at all of those links.
I left it on overnight and ...
5
votes
3answers
136 views
Effective fetching data algorithm
I'm fetching some data by using bs4, and want to know why access speed gets slower as my program progresses.
I'm using thread, and I know threading isn't so fast because of GIL.
My problem is that ...
3
votes
1answer
75 views
Script to download Annual Reports (~70,000 in total) from a Website
It's only 38 lines of code and I haven't worked with web scraping that much before.
How heavy would my code be on their server?
It's for Deep Learning purposes and I haven't ran it yet but ...
3
votes
1answer
43 views
Monitor a web page for when a desired price is met
I have a code that keeps checking a webpage with a price on it and when the price meets a number I set it to, the thing is purchased. I am looking for ways to increase the speed of it.
I have added ...
32
votes
7answers
9k views
Tiny image scraper for xkcd.com
This came out on the spur of the moment, as a quick and dirty tool to have the job done.
This is a simple image scraper for immensely popular and legendary comic website inspired by a Python Easter ...
7
votes
2answers
361 views
Network helper class with retry logic on failure
I wrote an helper class which allow me to request the body content from a site that is created by AJAX, for doing so I'm using <...
3
votes
1answer
41 views
Wiki Link Mapper — A webcrawler which crawls links recursively and prints them into a graph
I would like some suggestions on how to make it better and also if any features that i should add to this Script. I am using beautifulsoup,requests for scraping and networkx for plotting the graph.
...
2
votes
2answers
109 views
Jsoup connection to URL
I have simple class that I want to ask if is there any possible to improve it? I mean, for me it looks poor. Is there any way to use here try-with-resources, stream or ...
3
votes
2answers
245 views
Web-scraping investment fund data into Pandas
Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
4
votes
3answers
380 views
Why is my Python web scraper taking so long?
I am trying to create a program to download a long list of websites using Python 3.7. By using multiprocessing for each request, my code runs much faster than when it's run synchronously.
I've added ...
2
votes
1answer
590 views
Python Script to download images and videos from a user's profile on Instagram
The script downloads images and videos from a user's profile page on Instagram. It works fine, but I'd like to know if there are ways to improve it. All ideas and suggestions are welcome.
...
3
votes
1answer
75 views
the fox is black - a simple image scraper
I wrote a simple image scraper for the Desktop Wallpaper project by The Fox is Black. The site is a fantastic (yet, apparently dead) blog full of fabulous images. However, there's no way to download ...
4
votes
0answers
54 views
Scraping a hiring website using python's requests and BeautifulSoup
I'm designing a scraping application using python, requests and BeautifulSoup4.
I decided to divide the logic into two classes:
Spider : gets the base url ...
5
votes
2answers
68 views
E-commerce scraper needs shortening
How would I go about shortening this code?
I'm sure there's a smarter way to write the code.
It's working fine, it's just ugly.
It is a way to scrape the number of items on an e-commerce site.
<...
2
votes
0answers
37 views
Scraping Forum Tables w/ Links Using Beautiful Soup
This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
4
votes
1answer
64 views
Webscraper code to download a manga series
I wrote a program to download a manga series fron www.mangapanda.com
Here it is:
...
3
votes
1answer
269 views
Sticking to a working proxy generated by a rotating proxy script
I've created a script in python to make proxied requests by picking working proxies from a list of proxies scraped from a free proxy site. The bot traverses few links to parse the urls of the ...
1
vote
1answer
45 views
Python classes to grab HTTP response from website entities
I am trying to implement a parent WebContent class and child classes to grab HTTP responses from the actual website entities. There are highlevel codes below and I ...
