Skip to main content

Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

Filter by
Sorted by
Tagged with
3 votes
1 answer
70 views

Scraping website with Python and Selenium to collect data from dynamic website

Summary: The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
sangharsh's user avatar
  • 279
-1 votes
2 answers
123 views

Drayage Webscraper: Limited to table structure

This is my first working scraper. I'm sure a lot can be improved. My biggest question is how can I better specify what data to pull? All the data I'm currently grabbing is needed, but I couldn't ...
wigglesthe3rd's user avatar
2 votes
1 answer
45 views

A selenium web scraper to package NBA data

I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
BluffShove's user avatar
4 votes
1 answer
81 views

Java classes for downloading all in-coming/out-going links of an article in the Wikipedia article graph

(The entire project is in GitHub.) Introduction This project provides facilities for generating in-coming or out-going links in a given Wikipedia page. Code ...
coderodde's user avatar
  • 28.3k
0 votes
0 answers
84 views

Why is my scraping code so slow?

I'm a beginner with scraping with Selenium. And I wrote a couple of scrips. And when the script is used for a website with 10-15 pages, it works as it should. However, when I use it on a larger ...
hemoglobin's user avatar
3 votes
1 answer
156 views

Scraping the Divar.ir

I've wrote a code to scrape the Divar, which is an equivalent of Ebay in Iran. I have a few questions: Am I doing the error handling and logging ok? Is there a better way to optimize this code? (note ...
Amirhossein Rezaei's user avatar
0 votes
2 answers
143 views

Web scraping spider

I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
Max's user avatar
  • 17
3 votes
2 answers
174 views

Enum to deserialize HTML sizes from JSON with serde

I added an enum for my webscraper to deserialize data from a JSON field that represents an HTML image size, which can either be an unsigned int like 1080 or a ...
Richard Neumann's user avatar
1 vote
0 answers
66 views

Web scraper to retrieve articles from HTML embedded in JavaScript

I wrote a web scraper library to parse articles from HTML documents where they are encoded in a JSON object in JavaScript element. I wrote the example HTML file and ...
Richard Neumann's user avatar
3 votes
0 answers
39 views

Automatically extract useful cars from car site

I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well. I tried to ...
Mah Neh's user avatar
  • 31
2 votes
0 answers
57 views

Simplified HTML parsing for LEGO features

The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
alvas's user avatar
  • 705
0 votes
1 answer
99 views

Poetry Web Scraping in Python [closed]

I have a script that obtains urls that lead to a specific poem. This code current works and uses multiprocessing pools. I currently am getting restricted or blocked by some way from the website that I ...
watrgoat's user avatar
0 votes
1 answer
46 views

HTTP scraper for Python Package

I'm trying to make my first Python package as a learning experience. There's a lot of things that I suspect I am doing poorly, but this post is specifically about my HttpRequest class. I made this ...
jtb's user avatar
  • 185
2 votes
1 answer
54 views

URL link scraper and analyser

I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on). Essentially,...
Jessica's user avatar
  • 818
4 votes
2 answers
237 views

Test generator I made for practice

Made this generator to practice using imports from other modules and better readability for coding. What could I have done better and what did I do wrong? File called test_generator.py ...
Beginner's user avatar
  • 189
1 vote
0 answers
63 views

Search Stack Overflow and GitHub for code in a specified language

This code is designed to scrape Stack Overflow and GitHub, pulling information based on a user-specified programming language and processing the data into a format for AI learning. It uses a number of ...
Robert3737's user avatar
3 votes
1 answer
191 views

A simple web scraper for nature.com news articles

I have created a simple web scraper that fetches news article previews from nature.com and saves each article to a file containing the article preview text. I am learning independently, so I would ...
razzleDazzle's user avatar
1 vote
0 answers
1k views

Code scraping oddsportal using beautifulsoup

This code scrapes the website www.oddsportal.com for betting odds and historical games. code: ...
Rander's user avatar
  • 11
2 votes
1 answer
177 views

Beginner Python Web scraping

I'm a newbie in programming, I chose Python. I'm learning on my own. Currently I'm preparing code for a portfolio on github. I will be grateful for any code review, especially in the subject of OOP: ...
Magda's user avatar
  • 29
3 votes
0 answers
61 views

Gather product information from the Apotheek Dedeyne web site

I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/ This project is for personal learning & ...
Seraph776's user avatar
  • 191
6 votes
1 answer
764 views

Python web scraper for Amazon customer reviews

I'm new to web scraping and tried building a web scraper for Amazon customer reviews. The program works fine as is but I wanted to improve the design and get some feedback. The basic idea was to ...
nightstand's user avatar
0 votes
1 answer
181 views

Selenium/Beautifulsoup web scrape of Grainger

This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
cahilltyler's user avatar
3 votes
2 answers
185 views

Saving Scraped Data to a File

When scraping and saving data into a file, Which method is more efficient when saving scraped data to a file? open the file first, scrape, and save the data all ...
Seraph776's user avatar
  • 191
5 votes
1 answer
165 views

Web scraper for data sources from Statistics Canada

I've written a parser to scrape data from Canadian Statistics Bureau. ...
alphamu's user avatar
  • 153
5 votes
1 answer
324 views

Grabbing youtube thumbnails from urls

I made some C# code to grab youtube thumbnails from urls, I originally made this in python took some time converting it to C#. I am VERY new to C# and did this with minimal help. ...
Joshua Bursill's user avatar
3 votes
1 answer
37 views

request data and print results

On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
Luca's user avatar
  • 57
1 vote
0 answers
66 views

Web-scraping bountied questions from Stack Exchange sites

I recently built my first web scraper in python, and decided to use SO and SE as test websites. Code ...
DialFrost's user avatar
  • 123
2 votes
1 answer
51 views

scraping pollution data tables from government website

i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved: ...
Omar Morales Rivera's user avatar
1 vote
1 answer
115 views

Parsing: Сollecting data from a book site

A task: Сollect data from the site in the following format: book; user; book_rating; comment_rating; publication_date; comment For one book at once several pages of ...
Pom Mop's user avatar
  • 57
3 votes
1 answer
120 views

A Language Bot: For creating noun declension or verb conjugation tables

I'm a freshman student. What does my program solve? This is actually a bot for reddit, and there are language learning subreddits on that website, sometimes when discussing something with people, we ...
user9560064's user avatar
3 votes
1 answer
188 views

Scrape PokeDex and display in tkinter

Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
Salah Zahran's user avatar
3 votes
1 answer
75 views

Get the 64 most popular items from a list of tree species using webscraping

I made this python program to find the 64 most common elements of a list stored in a text file. Since this works using binomial names of trees I decided to use the number of mentions in PubMed Central ...
Omar Morales Rivera's user avatar
9 votes
2 answers
2k views

Python script to scrape and parse the Stanford Encyclopedia of Philosophy

I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
user avatar
1 vote
0 answers
57 views

How to make memory and cpu efficient a code

I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
Learner's user avatar
  • 111
1 vote
0 answers
44 views

Iterate tables from table id from href links until no table with specific table id is found

I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html What I am trying to achieve is to parse the ...
Gescof's user avatar
  • 11
3 votes
1 answer
181 views

Crawler/scraper for soccer match results

I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
Olgierd Wiśniewski's user avatar
1 vote
1 answer
124 views

Design Pattern: Builder - BeautifulSoup directory navigation and scraping

I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory. There are a couple navigation ...
Jason Leaver's user avatar
3 votes
1 answer
224 views

web scraping with pandas and apscheduler

Below is a script to scrape data from the National Severe Storms Laboratory Probsevere dataset. Their data is available at a 2 minute interval and is only available for 24 hours before moving to the ...
Jason Leaver's user avatar
1 vote
0 answers
228 views

Web scraper for pokemon prices using Selenium

I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
Moorgort's user avatar
1 vote
1 answer
370 views

Selenium web scraper to scrape Apple tweets of the last 3 years

I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
VaKt's user avatar
  • 11
3 votes
0 answers
761 views

A simple Python script that crawls information about Youtube playlists and your watch history

You will need to follow this guide. This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
Ξένη Γήινος's user avatar
1 vote
1 answer
60 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
Sati's user avatar
  • 417
3 votes
1 answer
98 views

Acupuncture database builder

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
Sati's user avatar
  • 417
0 votes
0 answers
44 views

Python scraper that gets product information from e-commerce website

I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
dovexz12323's user avatar
3 votes
0 answers
86 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
berkersenol's user avatar
4 votes
2 answers
323 views

Scraping campsite availability from a webpage using vba with selenium

I wrote some code to extract the information from a table, but it takes an extremely long time. The table is in the format of a calendar. I need the information on an Excel sheet with column 1 as the ...
Nelj's user avatar
  • 41
8 votes
1 answer
146 views

Scrape a URL for articles to save

Below is an exercise project I was doing on a certain educational site. It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
Gavin Greenhorn's user avatar
1 vote
1 answer
37 views

Extracting information from HTML with XSLT 3.0 when data is grouped visually as siblings in a td separated by blank lines

I have a work-in-progress where I'm using XSLT 3 to extract information from some preprocessed archaic HTML. I'd like to produce JSON showing the relationships between the various entities for further ...
Forensic_07's user avatar
1 vote
1 answer
358 views

Web-scrape bonds information from nasdaqomxnordic

I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated! ...
yogi06's user avatar
  • 13
2 votes
1 answer
468 views

Using XSLT 3.0 to extract information from real-world HTML and produce JSON

For work, I extract information from HTML and XML sources to save in databases. The objective is to generate JSON representing the source document's information and its relationships in order to 1) ...
Forensic_07's user avatar

1
2 3 4 5
13