Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

Filter by
Sorted by
Tagged with
0 votes
0 answers
39 views

Web-scraping favorites from a website and saving them to an Excel sheet

...
user avatar
1 vote
0 answers
30 views

Web scraper for pokemon prices using Selenium

I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
user avatar
1 vote
1 answer
29 views

Selenium web scraper to scrape Apple tweets of the last 3 years

I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
user avatar
  • 11
2 votes
0 answers
48 views

A simple Python script that crawls information about Youtube playlists and your watch history

You will need to follow this guide. This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
user avatar
1 vote
1 answer
34 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
user avatar
  • 321
3 votes
1 answer
51 views

Acupuncture database builder

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
user avatar
  • 321
0 votes
0 answers
26 views

Python scraper that gets product information from e-commerce website

I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
user avatar
0 votes
0 answers
30 views

Web Scraping - Python & BeautifulSoup

I'm new to Python, and well, I had to develop web scraping code & generate data CSV file. Here is what I have written so far: ...
user avatar
3 votes
0 answers
65 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
user avatar
4 votes
2 answers
92 views

Scraping campsite availability from a webpage using vba with selenium

I wrote some code to extract the information from a table, but it takes an extremely long time. The table is in the format of a calendar. I need the information on an Excel sheet with column 1 as the ...
user avatar
  • 41
8 votes
1 answer
101 views

Scrape a URL for articles to save

Below is an exercise project I was doing on a certain educational site. It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
user avatar
0 votes
0 answers
44 views

A Python program that scrapes questions from an online Stanford-Binet Test website

This is a simple Python program that scrapes questions from https://stanfordbinettest.com/, it has a version of Stanford-Binet IQ Test which randomly selects 100 questions from a pool of ...
user avatar
1 vote
1 answer
26 views

Extracting information from HTML with XSLT 3.0 when data is grouped visually as siblings in a td separated by blank lines

I have a work-in-progress where I'm using XSLT 3 to extract information from some preprocessed archaic HTML. I'd like to produce JSON showing the relationships between the various entities for further ...
user avatar
1 vote
1 answer
70 views

Web-scrape bonds information from nasdaqomxnordic

I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated! ...
user avatar
  • 13
2 votes
1 answer
48 views

Using XSLT 3.0 to extract information from real-world HTML and produce JSON

For work, I extract information from HTML and XML sources to save in databases. The objective is to generate JSON representing the source document's information and its relationships in order to 1) ...
user avatar
4 votes
0 answers
34 views

Shell script to download Project Euler problems and combine to PDF

This is a script I created that downloads Project Euler webpages and combines them to PDF. The script also downloads animated files. ...
user avatar
  • 798
4 votes
2 answers
393 views

Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python

I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths. The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
user avatar
  • 55
2 votes
0 answers
79 views

Web scraping articles using asyncio

I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages. I would like to improve ...
user avatar
  • 387
3 votes
1 answer
137 views

Web scraping international trade data

I'm trying to scrape arrival data from this website. My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process? Here's my script: ...
user avatar
  • 51
2 votes
0 answers
104 views

My first vba-selenium code

I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company. The asset value is not available other than on ...
user avatar
2 votes
2 answers
62 views

Web scraping playlist from website and writing to CSV

I have very recently learned some Python and found out about web scraping. This is my first bash at writing some code to get a playlist from a site I had previously been copying and pasting. I would ...
user avatar
2 votes
1 answer
76 views

golang http.Client wrapper with rate and concurrency limiting

My goal is to safely consume an API without getting 429 Too Many Requests error, activating DDoS protection or going over system limit of maximum open file ...
user avatar
1 vote
1 answer
95 views

Scrape from Craigslist

I'm new to Python and just started to learn about scraping and pandas library. Here is a little scraper I wrote. I'd like to know what's a professional code for this would look like. I have a sense my ...
user avatar
2 votes
0 answers
115 views

How to speed up the VBA scraping process

I want to scrape 25000 addresses using Column A cell values as search strings. I'm using the following code, it is working fine but it's too slow to extract 25000 results. Is there any way to speed up ...
user avatar
2 votes
1 answer
61 views

Output webscraping results to BibTex file

Here is my attempt to output webscraping results to a bibtex file. I would like to use this format as input to the Zotero database. As BetterBibtex already has a ...
user avatar
  • 321
0 votes
1 answer
38 views

Adding page iteration capability to Requests scraper

I am trying to build on @Reinderien's answer to my previous question over here to add page iteration functionality to the code: ...
user avatar
  • 321
3 votes
1 answer
425 views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
user avatar
  • 321
2 votes
1 answer
325 views

Yahoo finance option chain scraper

This is my 1st Python program I've actually written and I have little to no background in the language. I just figured I could learn and do something that interested me at the same time. I'm not a ...
user avatar
  • 61
2 votes
1 answer
184 views

Extract flight arrival data from web

I'm trying to scrape arrival data from Flight Radar 24 My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process? Here's my script: ...
user avatar
  • 51
1 vote
1 answer
65 views

Python script that get a web API

I've written a program that gets the web API endpoint for some functions on a website. I'm just wondering if you guys see anything I could do to improve the code at all. ...
user avatar
1 vote
0 answers
20 views

Organizing things together to form a minimum viable Scraper App (part 2)

This is a follow-up of my question over here. Response to @Reinderien's answer: I have corrected the more trivial issues highlighted in @Reinderien's answer below as follows. ...
user avatar
  • 321
1 vote
0 answers
58 views

Optimising this web-scraping code

A member of SO who I immensely respect just told me that the code below makes him uncomfortable. ...
user avatar
  • 169
3 votes
1 answer
64 views

Collect Pokémon from a URL and store them in a dataframe

...
user avatar
  • 31
2 votes
0 answers
811 views

Python script to scrape google maps without API

I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance. It uses selenium ...
user avatar
1 vote
1 answer
109 views

Organizing things together to form a minimum viable Scraper App

This is a follow-up of my group of scraper questions starting from here. I have thus far, with the help of @Reinderien, written 4 separate "modules" that expose a ...
user avatar
  • 321
2 votes
1 answer
75 views

Looping through search results in Selenium

This is a follow-up of my question over here. I am extending the functionality of @Reinderien's suggested code to enable the automatic looping through of pages in the search result. I've mainly ...
user avatar
  • 321
4 votes
0 answers
438 views

Scraping OddsPortal with requests only

This is a scraper written to do most of what had been attempted by another user in this question: How can I optimise this webscraping code I did the rewrite because I felt bad that the new user didn't ...
user avatar
  • 47.7k
2 votes
1 answer
74 views

Simple Scraper_Downloader based on Youtube-dl

Any help with my code. I think it's kind of hard code where I want to extract links from a page and download them. It's going to scrape some links from websites (but the ...
user avatar
  • 183
2 votes
0 answers
69 views

requests vs selenium vs scrapy

This is a follow-up of my question over here. I have been working on my web-scraping techniques by trying out different approaches and rewriting code for a handful of online databases. I've recently ...
user avatar
  • 321
4 votes
1 answer
72 views

Use of dataclass when scraping with Requests

This is a follow-up of my last question over here. Following @Reinderien's suggestion in that previous post, I've managed to furnish my web-scraper code as follows: fudan.py ...
user avatar
  • 321
11 votes
3 answers
489 views

Asynchronously scrape fabric information

This is my first asyncio/aiohttp web scraper. I am trying to wrap my head around Python's asyncio/aiohttp libs these days and I am not sure if I fully understand it or not yet. So I'd like have some ...
user avatar
  • 111
3 votes
2 answers
262 views

Simple Python Recursive Web Scraper

I tried to make a simple recursive web scraper using Python. My idea was to grab all the links, titles and tag names. Website: https://lifebridgecapital.com/podcast/ Course of Action: Grab all the ...
user avatar
  • 39
3 votes
0 answers
117 views

Speed Up API Requests & Overall Python Code

I'm not asking for help solving a problem but rather asking for help for possible ways to improve the speed of my program. Essentially what this does is: Tracks market data by pulling the data from ...
user avatar
1 vote
1 answer
82 views

Simple download manager based on Youtube-dl

There is a personal program of mine which logs in, scrape links and then download them. The downloader part is based on Youtube-dl and I think it's a smell code. Is ...
user avatar
  • 183
3 votes
1 answer
51 views

Refactor Web Scraper

I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
user avatar
2 votes
1 answer
106 views

Take information from a webpage and compare to previous request

After I have been doing some improvements from my Previous code review. I have taken the knowledge to upgrade and be a better coder but now im here again asking for Code review where I think it could ...
user avatar
2 votes
1 answer
192 views

Check if new dict values has been added

I have been working on creating a scraper that auto detects if there is any changes on a website as example as there is a release date or a increas/decrease of size value. I have created couple of ...
user avatar
1 vote
1 answer
100 views

Web scraping cars

Is there any way to lower the lines of this code and make it better in terms of being more Pythonic? The program collects data from truecar.com and then puts them ...
user avatar
  • 183
6 votes
1 answer
1k views

Async download of files

The following code is an async beginner exploration into asynchronous file downloads, to try and improve the download time of files from a specific website. Tasks: The tasks are as follows: Visit ...
user avatar
  • 375
2 votes
2 answers
87 views

Web scraping and design pattern lifting

I would like to get some feed back on my code; the goal is to get all agencies address of banks. I wrote a pretty simple brute force algorithm. I was wondering if you would have any advice to improve ...
user avatar
  • 273

1
2 3 4 5
12