Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

1
vote
0answers
50 views

Scraping and printing titles from Craigslist

I've written a very tiny script using class to scrape some titles of products from craigslist. My intention is to make use of __str__() method so that my script can ...
1
vote
1answer
81 views

Making use of a decorator within a python script

I've written a script in python which is able to collect links of posts and then fetch the title of each post by going one layer deep from the target page. I've applied @get_links decorator which ...
6
votes
2answers
240 views

Implementing lock within a Python script

I've written a script in Python using multiprocessing to handle multiple process at the same time and make the scraping process faster. I've used locking within it to prevent two processes from ...
4
votes
1answer
57 views

Web scraping google trends in python

Question Recently I have tried to create a web scraping program to get data from google trends. It uses an RSS feed to do so. My question is as follows: How can I improve my code such that it is ...
1
vote
0answers
27 views

Parse news content with PHP

I'm using PHP Readability to extract contents from news to get title, description and body like so. ...
4
votes
3answers
63 views

Creating a python dataframe by parsing JSON API response

In this SO question the OP is unable to scrape a table from a dynamically loaded website. In monitoring the web traffic, via Chrome dev tools, I found that there is an API request made that returns a ...
2
votes
1answer
62 views

Data scraper in Python 3

This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...
2
votes
2answers
70 views

Extract name, address and phone number from some web pages using multiprocessing

I've written a script in Python using the multiprocessing module to scrape values from web pages (one page per subprocess). As I'm very new to ...
2
votes
1answer
49 views

Crawling pages to obtain .htm and .txt files from sec.gov

I am pretty much a python newbie and was looking at this question on StackOverflow. Whereas the OP was interested in downloading the .htm |.txt files, I was simply ...
3
votes
1answer
60 views

Python webscrape Amazon Music, generate tracklist

So I've been working on a tracklist generator that scrapes from Amazon Music using a url using Python for albums with one artist. I've made uhhh this, I'm really new to this requests and ...
4
votes
1answer
56 views

Python Selenium webscraper for job listing website

I'm a newbie getting into web scrapers. I've made something that works, but it takes hours and hours to get everything I need. I read something about using parallel processes to process the URLs but I ...
2
votes
0answers
53 views

Wrapping a web scraper in a RESTful API (Docker + Flask Restplus + Beautiful Soup)

Learning to code whilst learning French! Faire d’une pierre deux coups! The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from ...
1
vote
1answer
126 views

Sourcing content from a webpage in an organized manner

I've written a script in python to grab different title and address from different pages of a website. Firstly the script will ...
1
vote
1answer
81 views

Scrape data from a webpage and store them in a database

I've written a script in python to parse different names ,bubbles and reviews of various ...
3
votes
0answers
56 views

Python web scraper

This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused. However, ...
1
vote
0answers
36 views

Python3.x Download(async) + Process(bs4) + Save(EPUB)

I had a simple webscraper with Beautiful Soup 4 which downloaded novel chapters from a website and converted them to an EPUB file. It was straight and simple imperative programming. Then I thought, ...
0
votes
0answers
13 views

Oscillate between NextPage and PrevPage to parse a list brutely

There's a well-defensed job-hunting website in China, I refactored my code which could retrieve data unlimited and effectively ...
4
votes
2answers
89 views

Scrape data from website into dataframe(s) using Split function

Outline: This code uses the Split function to extract specific information from the following website: https://www.webscraper.io/test-sites/tables. The required ...
2
votes
1answer
288 views

Store nested json repsonses in relational database

The goal is to collect market data from a free rest api. The response is in json and size per response is above 1MB. I want to get at least updated data once per minute which means about 24h * 60min * ...
4
votes
1answer
142 views

Scraping urls asynchronous including exception handling

I'm trying to understand how to work with aiohttp and asyncio. The code below retrieves all websites in ...
7
votes
1answer
99 views

Web scraping and downloading manga

I'm new to Python and decided to work on a personal project and make a program that web scrapes and downloads manga. It works as expected, however, I am sure that there is a better way to structure my ...
1
vote
1answer
219 views

Parsing scraped data from html table

I've written a simple python web scraper that parses text from an html table and stores the scraped data in List of dictionaries. The code works and doesn't seem to have any glaring issues performance-...
1
vote
1answer
71 views

Scraping content from yellowpage and write them to a csv file

I've written a script in python to scrape names and phone numbers of a certain search from yellowpage traversing multiple pages. The scraper has the ability to collect all the links connected to all ...
2
votes
1answer
45 views

Finding a swimmer's competing position from a website

This is a code that allows a swimmer to know his position with respect to other competitors. It gets the swimmer's time from the meet's website, finds other competitors' times and shows the better ...
0
votes
0answers
83 views

Scraping from Chemspider

I am learning web scraping, so I wrote code to scrape some information from Chemspider. The code is really slow, about 40 seconds. Is Scrapy faster than BeautifulSoup? ...
1
vote
0answers
15 views

JSON API Scraper v2: Limited concurrency AJAX from a shared queue (ES2017)

Essentially the AJAX component of this question, reimplemented in ES2017 style async/await. The code connects to a remote JSON API and scrapes it using a limited number of streams to avoid triggering ...
5
votes
3answers
962 views

Writing to a csv file in a customized way using scrapy

I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
5
votes
1answer
493 views

Python scraping tables

I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
2
votes
0answers
107 views

Grabbing information traversing multiple pages

I've written a script in python in combination with selenium to parse different information from a webpage and store the collected data in a csv file. The data are rightly coming through. The email ...
3
votes
0answers
112 views

Scraping StackOverflow's questions with Node.js

I'm new to node and web scraping in general but with some research I've written this working code. My goal is to get all questions from X Stack Overflow's pages sorted by most rated and save that ...
4
votes
0answers
28 views

Class based Kjijii Webscraper

Using PowerShell I have made a class based Kijiji webscraper that records progress in a MariaDB table (for use in another project). I had made this as a module with cmdlets months ago but I wanted to ...
6
votes
1answer
92 views

Chart of Percentage questions Answered

Every day, Duga reports the number of unanswered questions and percentage of answered questions (after she calls the API endpoint /info) in The 2nd Monitor (so we ...
5
votes
1answer
81 views

Web page downloader that supports redirects

I wrote a class that downloads the HTML of a page from the internet. This class also manages redirects. This works well but I'm not very satisfied about time management for slower servers. In fact, if ...
4
votes
1answer
212 views

Python yelp scraper

I've written a script in python to parse different names and phone numbers of various restaurant names from yelp.com. The scraper is doing its job just fine. The most important feature of this scraper ...
0
votes
0answers
41 views

Web scrape Appointment Workbook to a text file

This is a Web scraping script I wrote to get data from my companies Appointment workbook into a text file so I can review each appointment that is made online through our website. The context is that ...
2
votes
1answer
672 views

Web-scraping through a rotating proxy script

I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops ...
11
votes
3answers
1k views

Web-scraping a list of lawyers

I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
2
votes
1answer
239 views

Web scraping news articles with Beautiful Soup

It's been nearly a year I've been working with C++ and now I am diving in Python and its libraries and frameworks. I am currently creating a website for my portfolio that gathers news articles from a ...
4
votes
1answer
124 views

Scraping web data with Python 3

Below is a portion of the code i have written to scrape the bikesales.com.au website for details of bikes for sales (The full code is here). This finds all the 'href' attributes on each search page ...
1
vote
0answers
59 views

Extracting data from a used car sales site

I am developing code for extracting data from a used car sales site. There are 4 sites in total. In 3 of them I use requests and beautifulsoup. The time taken to extract data from these sites was ...
6
votes
1answer
782 views

Using rotation of proxies within a Python script

I've written a script in Python using rotation of proxies which is able to fetch different titles from a website traversing different pages. I've tried to write this scraper in such a way so that it ...
1
vote
1answer
66 views

Handle redirect manually

I built a class which uses an HttpClient instance for downloading the content of a page. This is working pretty well but I'm not really satisfied about my ...
2
votes
1answer
35 views

Program to automate skipping users on a chat website until the username meets certain conditions (v.2.0)

This is a python program that automates skipping through users on a chat website called talkwithstrangers using the Selenium Webdriver and BeautifulSoup modules of Python. The first part uses ...
6
votes
0answers
200 views

Scraping Clinical Studies database (via ClinicalTrials.Gov)

I have written the following script to scrape data from the U.S. National Library of Medicine website ClinicalTrials.Gov based on an NCTID. ...
4
votes
3answers
2k views

Webscraper for www.coinmarketcap.com

I built a webscraper to take all the historical data from CoinMarketCap.com. A site which records historical data for cryptocurrencies which unfortunately does not have a feature to download all of ...
3
votes
1answer
42 views

Finding the comic ID of the last XKCD comic published

I decided to sidetrack and create a XKCD viewer. For certain functionality, I needed to be able to find the ID of the last comic published. This was my attempt. I'm using Enlive here to parse the page ...
2
votes
0answers
340 views

Recursively scrape links from web pages and check them

I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles. The overall idea ...
5
votes
1answer
109 views

Wikipedia Random Page in Category Bot

I recently wrote a Python script to generate a random page within a Wikipedia category and its subcategories: ...
6
votes
2answers
193 views

Scraping NHL Individual stats

With the idea of gathering the biggest hockey's individual stats dataset ever, I've started with the big league, the NHL. Using Selenium, I've been able to scrape this NHL's statistics page. To ...
3
votes
0answers
163 views

Scraping website and print them via routers

I've been working on a personal project for a while which simply extracts some information from a website and save them as JSON files. Then I've decided to move those scraping functionalities to the ...