Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

FInding all the CURL requests from a given document [closed]

I have a 12 page Google Docx file which is an editable text file. In the document, there is some information in passage format and between those passages, there are curl requests in (text format) ...
3
votes
1answer
35 views

Wiki Link Mapper — A webcrawler which crawls links recursively and prints them into a graph

I would like some suggestions on how to make it better and also if any features that i should add to this Script. I am using beautifulsoup,requests for scraping and networkx for plotting the graph. ...
2
votes
2answers
59 views

Jsoup connection to URL

I have simple class that I want to ask if is there any possible to improve it? I mean, for me it looks poor. Is there any way to use here try-with-resources, stream or ...
3
votes
2answers
43 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
4
votes
3answers
121 views

Why is my Python web scraper taking so long?

I am trying to create a program to download a long list of websites using Python 3.7. By using multiprocessing for each request, my code runs much faster than when it's run synchronously. I've added ...
2
votes
1answer
52 views

Python Script to download images and videos from a user's profile on Instagram

The script downloads images and videos from a user's profile page on Instagram. It works fine, but I'd like to know if there are ways to improve it. All ideas and suggestions are welcome. ...
3
votes
1answer
49 views

the fox is black - a simple image scraper

I wrote a simple image scraper for the Desktop Wallpaper project by The Fox is Black. The site is a fantastic (yet, apparently dead) blog full of fabulous images. However, there's no way to download ...
4
votes
0answers
42 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
5
votes
2answers
66 views

E-commerce scraper needs shortening

How would I go about shortening this code? I'm sure there's a smarter way to write the code. It's working fine, it's just ugly. It is a way to scrape the number of items on an e-commerce site. <...
2
votes
0answers
24 views

Scraping Forum Tables w/ Links Using Beautiful Soup

This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
4
votes
1answer
55 views

Webscraper code to download a manga series

I wrote a program to download a manga series fron www.mangapanda.com Here it is: ...
3
votes
1answer
212 views

Sticking to a working proxy generated by a rotating proxy script

I've created a script in python to make proxied requests by picking working proxies from a list of proxies scraped from a free proxy site. The bot traverses few links to parse the urls of the ...
0
votes
1answer
29 views

Python classes to grab HTTP response from website entities

I am trying to implement a parent WebContent class and child classes to grab HTTP responses from the actual website entities. There are highlevel codes below and I ...
1
vote
1answer
45 views

Python program that gets my hackerrank code and scores using requests [closed]

I'm trying to extend my coding skills to other languages by doing projects. (learned Java in school) and since python is very popular right now thats what I ended up choosing for this project, though ...
10
votes
1answer
93 views

Strip Dilbert Comics

A friend of mine is a huge fan of a well-known comic website and he recently said that it'd be great to have a tool that downloads comic strips for a given date range, instead of clicking through the ...
5
votes
0answers
54 views

Get a movie directory, rename it and save imdb info as a webpage

I have my movie folders formatted: ~: no sub included {}: already watched nothing: sub included, ready to watch the program asks for the movie directory; extracts the name and searches IMDb for the ...
3
votes
1answer
426 views

Python script to download adult comics from 8muses

I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the ...
2
votes
2answers
76 views

Simple Web Crawler for Small Sites

This script is my web crawlerfor small sites. Can you possibly review it for best coding practices? Code ...
7
votes
1answer
52 views

ArXiV Technical Paper API Github Repo

My name is Ethan and I am trying to build an API for scaping technical papers to use for developers. Right now it only works for ArXiV but I would greatly appreciate some mentoring or a code review of ...
4
votes
0answers
126 views

Return different types from web scraper method

Application Logic The software take odds data from an internet site and store them inside my own database, this application is basically a scraper. There are different types of odds, eg: FullTime ...
3
votes
1answer
67 views

Web scraping job postings from Facebook

I built a web scraper to pull a list of jobs on Facebook and other websites but want to break up the code into functions that I can reuse for other websites. This structure is working but I think it ...
3
votes
1answer
57 views

Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)

The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
4
votes
1answer
61 views

Batch retrieve web titles from URLs

This working script reads a list of URLs from one or more text files, and for each, retrieves the page title from the Internet and appends the results in the format ...
5
votes
2answers
133 views

Web scraper for football data with three very similar methods

Application Logic I wrote a web scraper that store some football data inside my database. Those data need to be update if 7 days are passed since the last update. I have three types of record: Team ...
2
votes
1answer
46 views

Python webscraper to get property info from Tripadvisor

I'm very much a beginner at programming, and hoping to get some advice! I'm having some trouble with a tripadvisor scraper being slow, and have identified the part of my code that is taking a while. ...
7
votes
1answer
207 views

Scraper to grab a Stack Overflow profile name

I've created a script in Python to log into stackoverflow.com using credentials and fetch the profilename once logged in. I've tried to do it using class. I created the methods within that class in ...
0
votes
0answers
73 views

Data Scraping Using VBA

I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance. Thanks In Advance Note-I haven't posted ...
7
votes
1answer
98 views

Web-scraping C.R. Laurence product catalog

In the last couple of days I've tried to play around with some scraping using XPATHs and Python to consolidate my knowledge (since it's been some time). I did this on a simple website which doesn't ...
5
votes
0answers
81 views

NodeJs crawler for recipes

Here is my try on a crawler made in nodeJs with cheerio, I made it with the idea in mind to use it in a future project I wanna make. Here is the git link: https://github.com/Just4lol/CookCrawler If ...
3
votes
2answers
158 views

PHP script that writes a JSON file with iextrading API data

iextrading has two APIs (v1, v2) which provide financial market data. Following script records a JSON file with their equity information (using a CRON job). Would you be kind and review it for any ...
1
vote
0answers
100 views

Basic Kibana/Elasticsearch Search - Powershell

I am traditionally a Java/Backend Developer, and have recently been spending a lot of time dealing with issues that higher tiers of support should be able to handle themselves. To assist them with ...
4
votes
1answer
267 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
1
vote
0answers
291 views

Downloading multiple urls with aiohttp Python 3

I am trying to use aiohttp library in python to download information from url. I have about 300 000 urls. They are saved in file "my_file.txt". When I get web page, I extract pairs of a question and ...
1
vote
1answer
41 views

Handle connection time outs when loading a web page in a c# console app

The aim of the program is to read the yahoo finance website in order to find potentially profitable stocks based on their past performance. I have some code that works but I think it is of poor ...
1
vote
1answer
49 views

BTC price checker with web scraping, regex and Tkinter GUI

The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...
3
votes
1answer
73 views

Pulls href tags using BeautifulSoup with Python

As if this isn't extremely obvious, I am a new coder. My code works, but is quite far from ideal. I'm also not sure if there are unnecessary bits, since I've been trying to get this to work for a ...
4
votes
1answer
112 views

Cleaner way of appending data to List in BeautifulSoup

So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scrapper to collect data such as <...
2
votes
1answer
83 views

Web scraping JSON-data from API put into Dataframe

The web-scraping process currently takes quite a bit of time, and I wonder if I can structure the code otherwise or improve it in any way. Code looks like this: ...
2
votes
2answers
109 views

A bot for SaltyBet that watches and records matches

I've been working on a bot that records the result of matches on SaltyBet.com. It uses that data to calculate the probability of winning for fighters of a given match. A fighters probability is based ...
8
votes
1answer
2k views

Scraping Instagram for Hashtag data

Over the past few months, I've been actively using python and I have made a few scripts to scrape #hashtag data from Instagram. It all started with some basic script I had made early 2017 and I have ...
19
votes
4answers
5k views

Web scraping the titles and descriptions of trending YouTube videos

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make? ...
2
votes
2answers
323 views

Creating a csv file using scrapy

I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
2
votes
1answer
76 views

A simple web scraper

I've written a simple web scraper in Python. As it is my first venture in Python, I'd like to ask for a code review from more experienced coders. All input is welcome. A little bit of background: I'...
1
vote
0answers
97 views

Scraping and printing titles from Craigslist

I've written a very tiny script using class to scrape some titles of products from craigslist. My intention is to make use of __str__() method so that my script can ...
1
vote
1answer
107 views

Making use of a decorator within a python script

I've written a script in python which is able to collect links of posts and then fetch the title of each post by going one layer deep from the target page. I've applied @get_links decorator which ...
5
votes
2answers
259 views

Implementing lock within a Python script

I've written a script in Python using multiprocessing to handle multiple process at the same time and make the scraping process faster. I've used locking within it to prevent two processes from ...
5
votes
1answer
491 views

Web scraping google trends in python

Question Recently I have tried to create a web scraping program to get data from google trends. It uses an RSS feed to do so. My question is as follows: How can I improve my code such that it is ...
1
vote
0answers
161 views

Parse news content with PHP

I'm using PHP Readability to extract contents from news to get title, description and body like so. ...
4
votes
3answers
2k views

Creating a python dataframe by parsing JSON API response

In this SO question the OP is unable to scrape a table from a dynamically loaded website. In monitoring the web traffic, via Chrome dev tools, I found that there is an API request made that returns a ...
2
votes
1answer
69 views

Data scraper in Python 3

This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...