Questions tagged [python-requests]
USE ONLY FOR THE PYTHON REQUESTS LIBRARY. Requests is a full-featured Python HTTP library with an easy-to-use, logical API.
52
questions
1
vote
0
answers
488
views
Streaming Parquet file in chunks for write operation
I am taking beginner steps into DE and was tinkering with writing an ingestion script which does the following tasks:
Reads data from a source (in this case a remote parquet file)
Writes it to local ...
- 111
2
votes
1
answer
43
views
URL link scraper and analyser
I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on).
Essentially,...
- 818
4
votes
1
answer
129
views
A simple stock exchange app
I've struggled to make a small stock project for almost 5 hours (biggest one so far).
The project is simple: I've chosen a stock (Tesla in my case) and I've decided to get a message on my phone with ...
1
vote
1
answer
96
views
Excluding attributes entirely vs. setting them to None
I am making a program that parses profiles from a website. Some profiles are "public" and others are "private" with limited information. Public profiles contain a lot more data and ...
- 35
5
votes
3
answers
1k
views
Python: Handling different HTTP status codes using the requests library
I am using the requests library to make requests to an API. However, I am not sure how to handle different status codes. I am particularly interested in handling ...
1
vote
0
answers
306
views
Calling PagerDuty REST API to recursively fetch oncall info for a team
I am a beginnner and have put up a code that talks to pagerduty API and then it fethces oncall info for a team (get team id, get escalation policy id, then finally get on call info). response is ...
- 121
2
votes
1
answer
63
views
Create URL's, then download and print each RSS feed text
Looking for a more efficient way of achieving this. In particular the combine_all_rss_lists so I don't need all those params. Feedparser module doesn't accept lists which is why I've had to initially ...
- 57
3
votes
1
answer
35
views
request data and print results
On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
- 57
1
vote
1
answer
86
views
Single API call with authentication in Python
I've written a small script for Zabbix to check the status of our ESXi Hosts. The request needs to be authenticated, so I went for this process:
Read auth token from file
If file does not exist, ...
- 123
2
votes
1
answer
198
views
python: requests large.zip -> unzip -> fix -> filter ->gunzip
I wrote a function to download a large zipfile 5-7gb from Iowa State MRMS data archive.
The zip files appear to be malformed and results in a BadZipFileError hence ...
- 628
2
votes
1
answer
90
views
Monitoring webpages to be notified
I have been writing a monitoring script where I check for whenever there is changes on a webpage. When a change has happend I want to be notified by printing out that there is a difference.
I have ...
- 609
2
votes
1
answer
95
views
Compare between two dictionaries
Hello beautiful people!
I have currently worked on a small script that I would of course continue to work with whenever I get a good feedback from the best code reviewer in here <3 - I have worked ...
- 609
1
vote
1
answer
2k
views
A python script to download video from a website
I wrote this script for gaining experience, this script takes in a website link (NSFW) and download the video on that site. It uses regular expressions to parse the website source code to extract urls ...
- 201
9
votes
2
answers
2k
views
Python script to scrape and parse the Stanford Encyclopedia of Philosophy
I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback.
There is no function but I didn't think it was worth adding. Also there is no test ...
user260644
1
vote
0
answers
54
views
How to make memory and cpu efficient a code
I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
- 111
3
votes
1
answer
297
views
Python script to POST requests to a networking device
Request to review my code please and see if I am following the best practices.
This is sending a POST operation via a Python script using the requests library to a networking cloud orchestration ...
1
vote
1
answer
119
views
Printing a JSON/HTTP response from a Cisco endpoint
Please check over my Python code for a HTTP GET operation using the requests library, and provide any potential pointers for improvement.
...
1
vote
1
answer
53
views
Iterating over tables with nested <th> tags with BS4
The following code iterates over a table with nested th headings.
The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
- 417
3
votes
1
answer
88
views
Acupuncture database builder
The following code builds a rudimentary acupuncture database by collecting data from the web.
I would like to hear suggestions about improvements to the database structure, code organization, web-...
- 417
0
votes
0
answers
41
views
Python scraper that gets product information from e-commerce website
I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
- 101
4
votes
2
answers
69
views
Search for providers of TV Shows
All my code is below. It uses an API to search for a show and then outputs the providers (like breaking bad is available on Netflix).
The main aim of this application was to practice using APIs and ...
- 41
0
votes
2
answers
138
views
Python script for Web scraping
I wrote a Python script for Web scraping of a Website. Please review my code and suggest me any changes or make me aware of my blunders/mistakes?.
I wrote the almost same script for other websites ...
0
votes
1
answer
186
views
Config Variables Initialization | Python
I found interesting the way Python's requests library does the status_code data structure initialization (see code here). I am reusing it in a hobby project, but ...
- 301
3
votes
1
answer
70
views
Refactor Web Scraper
I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
3
votes
1
answer
178
views
benchmarking requests to localhost
I am writing a benchmarking tool from scratch in Python.
However I can't get the performance of other benchmarking tools like wrk or ...
2
votes
2
answers
175
views
Scrape webelements
I have been working with requests where I as easy as it is. Scraping the webpage and add it into a dict and print the payload if we find a new value or not.
...
- 609
2
votes
1
answer
2k
views
Run python File With get request flask API
Code is Working Review and Recommend What Are Best Practice
What I am trying to achieve.
API with Flask that runs any python file from the current directory.
Run the file and get output in JSON
...
2
votes
2
answers
467
views
Web scraping and design pattern lifting
I would like to get some feed back on my code; the goal is to get all agencies address of banks. I wrote a pretty simple brute force algorithm.
I was wondering if you would have any advice to improve ...
- 293
2
votes
1
answer
62
views
Scraping webpage elements using Python
I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
- 609
2
votes
0
answers
147
views
Python REST API Data Extraction with requests library
Is there a better way to write this piece of code to extract data from a REST API with the requests library? It currently takes about 10 minutes to finish extracting the data from the REST API. I ...
- 21
2
votes
1
answer
94
views
Beautifulsoup scrape posts that has the word python in it with no python tag
This is my first time web scraping, and here is the code I whipped up:
...
- 948
2
votes
1
answer
264
views
Do I need to use inheritance and classes for my OOP webscraper? [closed]
I am currently writing python code that scrapes information from the web.
I have to scrape several sites, but there a two types of procedures:
Directly scrape from the website
Download pdf and scrape ...
- 41
5
votes
2
answers
295
views
Python Web scraping
I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any.
What do I want to achieve?
I want to scrape ...
1
vote
0
answers
27
views
Trouble returning web scraping output as python dictionary [closed]
So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
- 11
3
votes
0
answers
809
views
Python Async Request using Tor and User-Agent Rotation
The code below will scrape data from three APIs. The APIs are for property listings. So, for each listing, I will have one request. There will be around 20000 requests. That's why I use ayncio to make ...
- 131
5
votes
0
answers
88
views
Scraping a hiring website using python's requests and BeautifulSoup
I'm designing a scraping application using python, requests and BeautifulSoup4.
I decided to divide the logic into two classes:
Spider : gets the base url ...
- 151
1
vote
1
answer
319
views
Python program that gets my hackerrank code and scores using requests [closed]
I'm trying to extend my coding skills to other languages by doing projects. (learned Java in school) and since python is very popular right now thats what I ended up choosing for this project, though ...
- 11
8
votes
1
answer
159
views
Finding unimplemented tasks on Rosetta Code using Requests
This is a Find unimplemented tasks task from Rosetta Code:
Given the name of a language on Rosetta Code, find all tasks which are
not implemented in that language.
There is already a solution ...
- 1,938
6
votes
1
answer
2k
views
Web scraping Google Trends in Python
Question
Recently I have tried to create a web scraping program to get data from Google Trends. It uses an RSS feed to do so. My question is as follows:
How can I improve my code such that it is more ...
1
vote
1
answer
4k
views
Python requests downloading big files in slower speed than wget
This is a simple toy downloader using python's requests library.
I’ve monitored the download process is slower on an ethernet connected box. The realtime speed is ...
- 11
4
votes
1
answer
189
views
Python program that gets basic information on a Github user
I've created a basic Python CLI script, which takes a Github username, and returns some basic information associated with said username. I use the requests for HTTP requests (I'm new to this area) and ...
- 1,098
5
votes
1
answer
5k
views
Python scraping tables
I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
5
votes
1
answer
407
views
Scraping web data with Python 3
Below is a portion of the code i have written to scrape the bikesales.com.au website for details of bikes for sales (The full code is here). This finds all the 'href' attributes on each search page ...
- 255
7
votes
2
answers
1k
views
Repeatable HTTP requests in Python
I have a simple python script that's repeating a HTTP response/request. I would like to know how I could have constructed this better, particularly if there's a way I could have incorporated the slice ...
- 1,641
3
votes
1
answer
976
views
PyCryptoScraper to scrape currency rates
I created a PyCryptoScraper script to scrape crypto-currencies from Coingecko. Now I want to know if you could give me some hints or improvements for my script, which works fine.
The whole code also ...
- 491
3
votes
1
answer
3k
views
Downloading files using Python-requests
I wrote a Python script to download files using multiple (source) IP addresses -- kindly suggest any improvements.
...
- 155
2
votes
1
answer
1k
views
Object-oriented URL loading and HTTP client with requests in Python
I have an API which I want to consume in Python 3.x using the requests module.
Right now, my solution is to create Requests objects for each URL like the ...
- 121
8
votes
1
answer
4k
views
Web Scraping with Python + asyncio
I've been working at speeding up my web scraping with the asyncio library. I have a working solution, but am unsure as to how pythonic it is or if I am properly ...
- 83
7
votes
1
answer
10k
views
Python auth using requests
I am trying to implement a custom ApiClient in Python, using requests.
The way it does authentication is by:
login(username, password) -> get back a token if valid, http error code if not
set the ...
- 215
5
votes
0
answers
371
views
Choose the Transport Layer Security protocol - requests python
I am using using the following code to choose the transport layer security protocol while issuing a request to server. Could someone please review it.
This is in reference to the following question:
...
- 151