Questions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
-1
votes
0answers
17 views
Generating PHP code for web requests
I have a program that generates PHP code for web requests using a template and its output is like the code below.
I was trying to make it a function that could be easily pasted and called in every ...
1
vote
0answers
9 views
Basic Kibana/Elasticsearch Search - Powershell
I am traditionally a Java/Backend Developer, and have recently been spending a lot of time dealing with issues that higher tiers of support should be able to handle themselves. To assist them with ...
-1
votes
0answers
15 views
When scraping webpage XPATH variable returns Unfinished Literal [on hold]
I am trying to scrape Amazon for ASIN Best Seller Rank and when I run the script in python it returns an unfinished literal error
Everything works except for the Best Sellers Rank.
I would really ...
4
votes
1answer
48 views
NFL data scraper
I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
1
vote
0answers
35 views
Downloading multiple urls with aiohttp Python 3
I am trying to use aiohttp library in python to download information from url. I have about 300 000 urls. They are saved in file "my_file.txt". When I get web page, I extract pairs of a question and ...
1
vote
1answer
39 views
Handle connection time outs when loading a web page in a c# console app
The aim of the program is to read the yahoo finance website in order to find potentially profitable stocks based on their past performance.
I have some code that works but I think it is of poor ...
1
vote
1answer
38 views
BTC price checker with web scraping, regex and Tkinter GUI
The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...
3
votes
1answer
28 views
Pulls href tags using BeautifulSoup with Python
As if this isn't extremely obvious, I am a new coder.
My code works, but is quite far from ideal. I'm also not sure if there are unnecessary bits, since I've been trying to get this to work for a ...
4
votes
1answer
62 views
Cleaner way of appending data to List in BeautifulSoup
So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scrapper to collect data such as <...
2
votes
1answer
36 views
Web scraping JSON-data from API put into Dataframe
The web-scraping process currently takes quite a bit of time, and I wonder if I can structure the code otherwise or improve it in any way.
Code looks like this:
...
2
votes
2answers
42 views
A bot for SaltyBet that watches and records matches
I've been working on a bot that records the result of matches on SaltyBet.com. It uses that data to calculate the probability of winning for fighters of a given match. A fighters probability is based ...
7
votes
1answer
126 views
Scraping Instagram for Hashtag data
Over the past few months, I've been actively using python and I have made a few scripts to scrape #hashtag data from Instagram.
It all started with some basic script I had made early 2017 and I have ...
18
votes
4answers
3k views
Web scraping the titles and descriptions of trending YouTube videos
This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?
...
2
votes
2answers
91 views
Creating a csv file using scrapy
I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
2
votes
1answer
62 views
A simple web scraper
I've written a simple web scraper in Python. As it is my first venture in Python, I'd like to ask for a code review from more experienced coders. All input is welcome.
A little bit of background:
I'...
1
vote
0answers
91 views
Scraping and printing titles from Craigslist
I've written a very tiny script using class to scrape some titles of products from craigslist. My intention is to make use of __str__() method so that my script can ...
1
vote
1answer
104 views
Making use of a decorator within a python script
I've written a script in python which is able to collect links of posts and then fetch the title of each post by going one layer deep from the target page.
I've applied @get_links decorator which ...
5
votes
2answers
245 views
Implementing lock within a Python script
I've written a script in Python using multiprocessing to handle multiple process at the same time and make the scraping process faster. I've used locking within it to prevent two processes from ...
5
votes
1answer
146 views
Web scraping google trends in python
Question
Recently I have tried to create a web scraping program to get data from google trends. It uses an RSS feed to do so. My question is as follows:
How can I improve my code such that it is ...
1
vote
0answers
69 views
Parse news content with PHP
I'm using PHP Readability to extract contents from news to get title, description and body like so.
...
4
votes
3answers
121 views
Creating a python dataframe by parsing JSON API response
In this SO question the OP is unable to scrape a table from a dynamically loaded website. In monitoring the web traffic, via Chrome dev tools, I found that there is an API request made that returns a ...
2
votes
1answer
63 views
Data scraper in Python 3
This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...
2
votes
2answers
88 views
Extract name, address and phone number from some web pages using multiprocessing
I've written a script in Python using the multiprocessing module to scrape values from web pages (one page per subprocess). As I'm very new to ...
2
votes
1answer
49 views
Crawling pages to obtain .htm and .txt files from sec.gov
I am pretty much a python newbie and was looking at this question on StackOverflow.
Whereas the OP was interested in downloading the .htm |.txt files, I was simply ...
3
votes
1answer
63 views
Python webscrape Amazon Music, generate tracklist
So I've been working on a tracklist generator that scrapes from Amazon Music using a url using Python for albums with one artist. I've made uhhh this, I'm really new to this requests and ...
4
votes
1answer
67 views
Python Selenium webscraper for job listing website
I'm a newbie getting into web scrapers. I've made something that works, but it takes hours and hours to get everything I need. I read something about using parallel processes to process the URLs but I ...
2
votes
0answers
90 views
Wrapping a web scraper in a RESTful API
The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
1
vote
1answer
128 views
Sourcing content from a webpage in an organized manner
I've written a script in python to grab different title and address from different pages of a website. Firstly the script will ...
1
vote
1answer
250 views
Scrape data from a webpage and store them in a database
I've written a script in python to parse different names ,bubbles and reviews of various ...
3
votes
0answers
73 views
Python web scraper
This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused.
However, ...
1
vote
0answers
38 views
Python3.x Download(async) + Process(bs4) + Save(EPUB)
I had a simple webscraper with Beautiful Soup 4 which downloaded novel chapters from a website and converted them to an EPUB file. It was straight and simple imperative programming.
Then I thought, ...
0
votes
0answers
17 views
Oscillate between NextPage and PrevPage to parse a list brutely
There's a well-defensed job-hunting website in China,
I refactored my code which could retrieve data unlimited and effectively
...
4
votes
2answers
94 views
Scrape data from website into dataframe(s) using Split function
Outline:
This code uses the Split function to extract specific information from the following website: https://www.webscraper.io/test-sites/tables.
The required ...
2
votes
1answer
446 views
Store nested json repsonses in relational database
The goal is to collect market data from a free rest api. The response is in json and size per response is above 1MB. I want to get at least updated data once per minute which means about 24h * 60min * ...
4
votes
1answer
235 views
Scraping urls asynchronous including exception handling
I'm trying to understand how to work with aiohttp and asyncio. The code below retrieves all websites in ...
7
votes
1answer
165 views
Web scraping and downloading manga
I'm new to Python and decided to work on a personal project and make a program that web scrapes and downloads manga. It works as expected, however, I am sure that there is a better way to structure my ...
1
vote
1answer
382 views
Parsing scraped data from html table
I've written a simple python web scraper that parses text from an html table and stores the scraped data in List of dictionaries. The code works and doesn't seem to have any glaring issues performance-...
1
vote
1answer
92 views
Scraping content from yellowpage and write them to a csv file
I've written a script in python to scrape names and phone numbers of a certain search from yellowpage traversing multiple pages. The scraper has the ability to collect all the links connected to all ...
2
votes
1answer
45 views
Finding a swimmer's competing position from a website
This is a code that allows a swimmer to know his position with respect to other competitors. It gets the swimmer's time from the meet's website, finds other competitors' times and shows the better ...
1
vote
0answers
17 views
JSON API Scraper v2: Limited concurrency AJAX from a shared queue (ES2017)
Essentially the AJAX component of this question, reimplemented in ES2017 style async/await. The code connects to a remote JSON API and scrapes it using a limited number of streams to avoid triggering ...
5
votes
3answers
2k views
Writing to a csv file in a customized way using scrapy
I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
5
votes
1answer
822 views
Python scraping tables
I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
2
votes
0answers
179 views
Grabbing information traversing multiple pages
I've written a script in python in combination with selenium to parse different information from a webpage and store the collected data in a csv file. The data are rightly coming through. The email ...
3
votes
0answers
141 views
Scraping StackOverflow's questions with Node.js
I'm new to node and web scraping in general but with some research I've written this working code.
My goal is to get all questions from X Stack Overflow's pages sorted by most rated and save that ...
4
votes
0answers
35 views
Class based Kjijii Webscraper
Using PowerShell I have made a class based Kijiji webscraper that records progress in a MariaDB table (for use in another project). I had made this as a module with cmdlets months ago but I wanted to ...
6
votes
1answer
104 views
Chart of Percentage questions Answered
Every day, Duga reports the number of unanswered questions and percentage of answered questions (after she calls the API endpoint /info) in The 2nd Monitor (so we ...
5
votes
1answer
83 views
Web page downloader that supports redirects
I wrote a class that downloads the HTML of a page from the internet. This class also manages redirects. This works well but I'm not very satisfied about time management for slower servers. In fact, if ...
4
votes
1answer
298 views
Python yelp scraper
I've written a script in python to parse different names and phone numbers of various restaurant names from yelp.com. The scraper is doing its job just fine. The most important feature of this scraper ...
-1
votes
0answers
51 views
Web scrape Appointment Workbook to a text file
This is a Web scraping script I wrote to get data from my companies Appointment workbook into a text file so I can review each appointment that is made online through our website. The context is that ...
2
votes
1answer
1k views
Web-scraping through a rotating proxy script
I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops ...
