Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

62 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
85 views

Get a movie directory, rename it and save imdb info as a webpage

I have my movie folders formatted: ~: no sub included {}: already watched nothing: sub included, ready to watch the program asks for the movie directory; extracts the name and searches IMDb for the ...
5
votes
0answers
585 views

Parsing different categories using scrapy from a webpage

I've written a script in python scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach the target page to scrape the ...
5
votes
0answers
247 views

Perl crypto-scraper

I created a crypto-scraper script to scrape crypto currencies from Coingecko. I have used the library LWP::UserAgent and now I want to know if you could give me ...
4
votes
0answers
52 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
4
votes
0answers
136 views

Return different types from web scraper method

Application Logic The software take odds data from an internet site and store them inside my own database, this application is basically a scraper. There are different types of odds, eg: FullTime ...
4
votes
0answers
48 views

Class based Kjijii Webscraper

Using PowerShell I have made a class based Kijiji webscraper that records progress in a MariaDB table (for use in another project). I had made this as a module with cmdlets months ago but I wanted to ...
4
votes
0answers
81 views

List all links in a website

I wrote this code as part of a project I'm working on. It's supposed to get all links from a website and then perform black-box tests on them. How can I improve this code to be faster and more ...
4
votes
0answers
108 views

Web scrape data for a list of stocks

I wrote a script that will web scrape data for a list of stocks. The scraper has to get the data from 2 separate pages so each stock symbol must scrape 2 different pages. If I run the process on a ...
4
votes
0answers
789 views

Speeding web-scraping up Python 3

I want to get information from two websites and display it on 'real-time' in the console. To get the information from the website I am using BeautifulSoup 4. I have read, that the bottleneck of ...
4
votes
0answers
200 views

Web scraping with Nokogiri

At work we have a need to know what printers are getting dangerously low on their toner, and paper consumption, etc.. So I've created a program that pulls the printer information off the websites the ...
4
votes
0answers
2k views

Web scraping from the Google Play store

I am using this R function to web scrape data from the google play store. Is there a way to increase its efficiency using R? This code takes about 4 seconds for 14 urls with my machine/internet ...
4
votes
0answers
437 views

Crawling and parsing meteorological data from the web into R

I am interested in collecting directly into R data published by the Mexican Met-office. The data pieces are spread through several URLs, but one can start here. There I can get the names and ...
3
votes
2answers
208 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
3
votes
0answers
248 views

Python web scraper

This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused. However, ...
3
votes
0answers
245 views

Scraping StackOverflow's questions with Node.js

I'm new to node and web scraping in general but with some research I've written this working code. My goal is to get all questions from X Stack Overflow's pages sorted by most rated and save that ...
3
votes
0answers
508 views

Scraping website and print them via routers

I've been working on a personal project for a while which simply extracts some information from a website and save them as JSON files. Then I've decided to move those scraping functionalities to the ...
3
votes
0answers
4k views

Fetching data from a website using “POST” request

I've written some code in vba for the purpose of making twofold "POST" requests to get to the destination page and harvest name and address from there. There are two types of structures within which ...
3
votes
0answers
313 views

Scraper Class with Regex

I already posted a not (yet) complete version of this class, but that question got closed, because it contained not passing tests. Here is the full version, all 5 tests pass. Originally there was ...
3
votes
0answers
177 views

Scrape a stock value and add it to a csv file along with other data. [Python 3]

Any and all suggestions are welcome, mainly looking for suggestions on clean up, and making it more readable as I feel that it is inefficient and not very clean. Some specific things: A better way ...
3
votes
0answers
1k views

Scraping table data from websites

I've written a script in vba to parse table from any website if the html elements are not encrypted in javascript. There is a table index in my script in which inputting the right number it is ...
3
votes
0answers
43 views

Regularly watch recent posts of a blog for specific words with HTML scraping

Task I want to watch the "Recent Posts" section of a blog for changes/new posts but only for specific posts containing a pre-defined word. Afterwards a list should be outputted in the console with ...
3
votes
0answers
51 views

VIM colors downloader in Python, using multiprocessing

I recently posted this script: VIM colors downloader in Python But since I'm not allowed to update the code there, I wanted to get an idea on this version, that uses multiprocessing: ...
3
votes
0answers
394 views

Scraping links from the first page of Google using Kivy

I'm making a scraper/web crawler using Kivy when I run the code it works but I'm not sure if what I'm doing is Pythonic because all the language I can find is about using the Kivy library. I'm unsure ...
3
votes
0answers
119 views

Using Nokogiri to scrape Oscars winners from Wikipedia

I am scraping a Wikipedia page, getting info from that page and instantiating a new object with the information collected: ...
3
votes
0answers
548 views

Rails app that scrapes forum using Nokogiri gem

I've built a website that scrapes a guitar forum's pages and populates Rails model. I'm using rake task along with heroku scheduler to run background scrapes every hour. On the homepage, the forum ads ...
2
votes
0answers
37 views

Scraping Forum Tables w/ Links Using Beautiful Soup

This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
2
votes
0answers
214 views

Wrapping a web scraper in a RESTful API

The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
2
votes
0answers
645 views

Grabbing information traversing multiple pages

I've written a script in python in combination with selenium to parse different information from a webpage and store the collected data in a csv file. The data are rightly coming through. The email ...
2
votes
0answers
646 views

Recursively scrape links from web pages and check them

I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles. The overall idea ...
2
votes
0answers
129 views

Handling IndexError using lambda function within scrapy

I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
2
votes
0answers
133 views

Football Web Scraper Part 2

A revised version of the code in this question. Things I have done so far: Adjusted some formatting things like constant & variable naming and indentation Wrapped most of the functions into a ...
2
votes
0answers
122 views

Scraping data from json response using twofold request

I've written a script which is able to get javascript encrypted links from a webpage and then using those newly produced links by making another request It can reach the target page and going there it ...
2
votes
0answers
51 views

River Flood Warning System v2.1 - Those Pesky NoneTypes

Below is my River Flood Warning System version 2 build 1. After following the help and advice given for version 1 the whole program is looking and behaving much better. The original code would only ...
2
votes
0answers
1k views

Google Searching Bot with Proxy support

I have been asked by a client to program a bot which searches Google and will show how many no of results I get. Note: I know about Google Custom Search API and it will not produce the exact output ...
2
votes
0answers
912 views

Movie torrent-site web scraper with IMDb info and streaming

I'm completely new to Javascript and NodeJs and functional programming in general. The code below scrapes a torrent-website containing movies, gets info about the movie from the OMDb API and lets a ...
2
votes
0answers
2k views

A web crawler for scraping images from stock photo websites

I created a web crawler that uses beautiful soup to crawl images from a website and scrape them to a database. in order to use it you have to create a class that inherits from Crawler and implements 4 ...
2
votes
0answers
162 views

OOP Web scraper using regex to grab tag contents

I'm about learning about implementing the solid principle in PHP. I want to create simple content crawler/grabber from some websites. This crawler will grab the content from the website url. Since we ...
2
votes
0answers
260 views

BeautifulSoup web spider for driver links

The following spider will grab some driver links, OS version, and the name. All the info is in a table class, but some pages might be a little different in the location and Number of cells in each row....
2
votes
0answers
128 views

Compressing a blog into a preview using tumblr_api_read

Here is what I have currently working. I would like to make it look more aesthetically pleasing, so not finish words in mid word. Also not have the two previews be so much larger than the other. ...
2
votes
0answers
1k views

Scraping HTML using PHP

Because a website I need data from doesn't have any API or RSS feed for their service status, I use a Web Scraper I built using PHP to grab the data I need and structure it as JSON. However I want to ...
2
votes
0answers
326 views

Prototype spider for indexing RSS feeds

This code is super slow. I'm looking for advice on how to improve its performance. ...
1
vote
0answers
38 views

bs4 HTML cleaner function with repetitive nested if clauses,

[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
1
vote
0answers
286 views

Basic Kibana/Elasticsearch Search - Powershell

I am traditionally a Java/Backend Developer, and have recently been spending a lot of time dealing with issues that higher tiers of support should be able to handle themselves. To assist them with ...
1
vote
0answers
501 views

Downloading multiple urls with aiohttp Python 3

I am trying to use aiohttp library in python to download information from url. I have about 300 000 urls. They are saved in file "my_file.txt". When I get web page, I extract pairs of a question and ...
1
vote
0answers
104 views

Scraping and printing titles from Craigslist

I've written a very tiny script using class to scrape some titles of products from craigslist. My intention is to make use of __str__() method so that my script can ...
1
vote
0answers
210 views

Parse news content with PHP

I'm using PHP Readability to extract contents from news to get title, description and body like so. ...
1
vote
0answers
53 views

Python3.x Download(async) + Process(bs4) + Save(EPUB)

I had a simple webscraper with Beautiful Soup 4 which downloaded novel chapters from a website and converted them to an EPUB file. It was straight and simple imperative programming. Then I thought, ...
1
vote
0answers
17 views

JSON API Scraper v2: Limited concurrency AJAX from a shared queue (ES2017)

Essentially the AJAX component of this question, reimplemented in ES2017 style async/await. The code connects to a remote JSON API and scrapes it using a limited number of streams to avoid triggering ...
1
vote
0answers
135 views

Extracting data from a used car sales site

I am developing code for extracting data from a used car sales site. There are 4 sites in total. In 3 of them I use requests and beautifulsoup. The time taken to extract data from these sites was ...
1
vote
0answers
64 views

Webscraping with Cheerio

I would like to get a code review about my Cheerio code. I'm trying to scrape a dutch book store with Cheerio. The web scraping takes up between 1,5 and 3,5 seconds per book. I would like the ...