Questions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
589
questions
0
votes
0
answers
39
views
1
vote
0
answers
30
views
Web scraper for pokemon prices using Selenium
I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
- 11
1
vote
1
answer
29
views
Selenium web scraper to scrape Apple tweets of the last 3 years
I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
- 11
2
votes
0
answers
48
views
A simple Python script that crawls information about Youtube playlists and your watch history
You will need to follow this guide.
This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
- 1,490
1
vote
1
answer
34
views
Iterating over tables with nested <th> tags with BS4
The following code iterates over a table with nested th headings.
The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
- 321
3
votes
1
answer
51
views
Acupuncture database builder
The following code builds a rudimentary acupuncture database by collecting data from the web.
I would like to hear suggestions about improvements to the database structure, code organization, web-...
- 321
0
votes
0
answers
26
views
Python scraper that gets product information from e-commerce website
I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
- 101
0
votes
0
answers
30
views
Web Scraping - Python & BeautifulSoup
I'm new to Python, and well, I had to develop web scraping code & generate data CSV file.
Here is what I have written so far:
...
3
votes
0
answers
65
views
Scraping housing dataset row by row from Estate Agent's website
I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame.
...
- 31
4
votes
2
answers
92
views
Scraping campsite availability from a webpage using vba with selenium
I wrote some code to extract the information from a table, but it takes an extremely long time.
The table is in the format of a calendar.
I need the information on an Excel sheet with column 1 as the ...
- 41
8
votes
1
answer
101
views
Scrape a URL for articles to save
Below is an exercise project I was doing on a certain educational site.
It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
0
votes
0
answers
44
views
A Python program that scrapes questions from an online Stanford-Binet Test website
This is a simple Python program that scrapes questions from https://stanfordbinettest.com/, it has a version of Stanford-Binet IQ Test which randomly selects 100 questions from a pool of ...
- 1,490
1
vote
1
answer
26
views
Extracting information from HTML with XSLT 3.0 when data is grouped visually as siblings in a td separated by blank lines
I have a work-in-progress where I'm using XSLT 3 to extract information from some preprocessed archaic HTML. I'd like to produce JSON showing the relationships between the various entities for further ...
- 135
1
vote
1
answer
70
views
Web-scrape bonds information from nasdaqomxnordic
I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated!
...
- 13
2
votes
1
answer
48
views
Using XSLT 3.0 to extract information from real-world HTML and produce JSON
For work, I extract information from HTML and XML sources to save in databases. The objective is to generate JSON representing the source document's information and its relationships in order to 1) ...
- 135
4
votes
0
answers
34
views
Shell script to download Project Euler problems and combine to PDF
This is a script I created that downloads Project Euler webpages and combines them to PDF. The script also downloads animated files.
...
- 798
4
votes
2
answers
393
views
Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python
I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths.
The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
- 55
2
votes
0
answers
79
views
Web scraping articles using asyncio
I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages.
I would like to improve ...
- 387
3
votes
1
answer
137
views
Web scraping international trade data
I'm trying to scrape arrival data from this website. My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process?
Here's my script:
...
- 51
2
votes
0
answers
104
views
My first vba-selenium code
I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company.
The asset value is not available other than on ...
- 121
2
votes
2
answers
62
views
Web scraping playlist from website and writing to CSV
I have very recently learned some Python and found out about web scraping.
This is my first bash at writing some code to get a playlist from a site I had previously been copying and pasting. I would ...
2
votes
1
answer
76
views
golang http.Client wrapper with rate and concurrency limiting
My goal is to safely consume an API without getting 429 Too Many Requests error, activating DDoS protection or going over system limit of maximum open file ...
1
vote
1
answer
95
views
Scrape from Craigslist
I'm new to Python and just started to learn about scraping and pandas library. Here is a little scraper I wrote. I'd like to know what's a professional code for this would look like. I have a sense my ...
- 13
2
votes
0
answers
115
views
How to speed up the VBA scraping process
I want to scrape 25000 addresses using Column A cell values as search strings. I'm using the following code, it is working fine but it's too slow to extract 25000 results. Is there any way to speed up ...
- 21
2
votes
1
answer
61
views
Output webscraping results to BibTex file
Here is my attempt to output webscraping results to a bibtex file.
I would like to use this format as input to the Zotero database.
As BetterBibtex already has a ...
- 321
0
votes
1
answer
38
views
Adding page iteration capability to Requests scraper
I am trying to build on @Reinderien's answer to my previous question over here to add page iteration functionality to the code:
...
- 321
3
votes
1
answer
425
views
Scraping a dynamic website with Scrapy (or Requests) and Selenium
I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here.
Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
- 321
2
votes
1
answer
325
views
Yahoo finance option chain scraper
This is my 1st Python program I've actually written and I have little to no background in the language. I just figured I could learn and do something that interested me at the same time.
I'm not a ...
- 61
2
votes
1
answer
184
views
Extract flight arrival data from web
I'm trying to scrape arrival data from Flight Radar 24
My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process?
Here's my script:
...
- 51
1
vote
1
answer
65
views
Python script that get a web API
I've written a program that gets the web API endpoint for some functions on a website. I'm just wondering if you guys see anything I could do to improve the code at all.
...
- 45
1
vote
0
answers
20
views
Organizing things together to form a minimum viable Scraper App (part 2)
This is a follow-up of my question over here.
Response to @Reinderien's answer:
I have corrected the more trivial issues highlighted in @Reinderien's answer below as follows. ...
- 321
1
vote
0
answers
58
views
Optimising this web-scraping code
A member of SO who I immensely respect just told me that the code below makes him uncomfortable.
...
- 169
3
votes
1
answer
64
views
2
votes
0
answers
811
views
Python script to scrape google maps without API
I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance.
It uses selenium ...
- 21
1
vote
1
answer
109
views
Organizing things together to form a minimum viable Scraper App
This is a follow-up of my group of scraper questions starting from here.
I have thus far, with the help of @Reinderien, written 4 separate "modules" that expose a ...
- 321
2
votes
1
answer
75
views
Looping through search results in Selenium
This is a follow-up of my question over here.
I am extending the functionality of @Reinderien's suggested code to enable the automatic looping through of pages in the search result.
I've mainly ...
- 321
4
votes
0
answers
438
views
Scraping OddsPortal with requests only
This is a scraper written to do most of what had been attempted by another user in this question:
How can I optimise this webscraping code
I did the rewrite because I felt bad that the new user didn't ...
- 47.7k
2
votes
1
answer
74
views
Simple Scraper_Downloader based on Youtube-dl
Any help with my code. I think it's kind of hard code where I want to extract links from a page and download them.
It's going to scrape some links from websites (but the ...
- 183
2
votes
0
answers
69
views
requests vs selenium vs scrapy
This is a follow-up of my question over here.
I have been working on my web-scraping techniques by trying out different approaches and rewriting code for a handful of online databases.
I've recently ...
- 321
4
votes
1
answer
72
views
Use of dataclass when scraping with Requests
This is a follow-up of my last question over here.
Following @Reinderien's suggestion in that previous post, I've managed to furnish my web-scraper code as follows:
fudan.py
...
- 321
11
votes
3
answers
489
views
Asynchronously scrape fabric information
This is my first asyncio/aiohttp web scraper. I am trying to wrap my head around Python's asyncio/aiohttp libs these days and I am not sure if I fully understand it or not yet. So I'd like have some ...
- 111
3
votes
2
answers
262
views
Simple Python Recursive Web Scraper
I tried to make a simple recursive web scraper using Python. My idea was to grab all the links, titles and tag names.
Website: https://lifebridgecapital.com/podcast/
Course of Action:
Grab all the ...
- 39
3
votes
0
answers
117
views
Speed Up API Requests & Overall Python Code
I'm not asking for help solving a problem but rather asking for help for possible ways to improve the speed of my program.
Essentially what this does is:
Tracks market data by pulling the data from ...
- 31
1
vote
1
answer
82
views
Simple download manager based on Youtube-dl
There is a personal program of mine which logs in, scrape links and then download them. The downloader part is based on Youtube-dl and I think it's a smell code. Is ...
- 183
3
votes
1
answer
51
views
Refactor Web Scraper
I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
2
votes
1
answer
106
views
Take information from a webpage and compare to previous request
After I have been doing some improvements from my Previous code review. I have taken the knowledge to upgrade and be a better coder but now im here again asking for Code review where I think it could ...
- 503
2
votes
1
answer
192
views
Check if new dict values has been added
I have been working on creating a scraper that auto detects if there is any changes on a website as example as there is a release date or a increas/decrease of size value.
I have created couple of ...
- 503
1
vote
1
answer
100
views
Web scraping cars
Is there any way to lower the lines of this code and make it better in terms of being more Pythonic? The program collects data from truecar.com and then puts them ...
- 183
6
votes
1
answer
1k
views
Async download of files
The following code is an async beginner exploration into asynchronous file downloads, to try and improve the download time of files from a specific website.
Tasks:
The tasks are as follows:
Visit ...
- 375
2
votes
2
answers
87
views
Web scraping and design pattern lifting
I would like to get some feed back on my code; the goal is to get all agencies address of banks. I wrote a pretty simple brute force algorithm.
I was wondering if you would have any advice to improve ...
- 273