Questions tagged [python-requests]

USE ONLY FOR THE PYTHON REQUESTS LIBRARY. Requests is a full-featured Python HTTP library with an easy-to-use, logical API.

Filter by
Sorted by
Tagged with
0 votes
2 answers
92 views

Mapping an API's setting IDs to setting names with class methods, enabling clearer utilization

I'm trying to access an API using the requests library in as clean a manner as possible. The API is accessing various settings of various devices. I'd like to keep my code as readable and ...
Floriancitt's user avatar
4 votes
2 answers
934 views

Python network manager class using exec()

I wanted to make a small self-integrated API for my PyQt5 application. This class is a tiny chunk of the whole project: ...
KhodeNima's user avatar
  • 349
1 vote
0 answers
1k views

Streaming Parquet file in chunks for write operation

I am taking beginner steps into DE and was tinkering with writing an ingestion script which does the following tasks: Reads data from a source (in this case a remote parquet file) Writes it to local ...
av abhishiek's user avatar
2 votes
1 answer
53 views

URL link scraper and analyser

I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on). Essentially,...
Jessica's user avatar
  • 818
4 votes
1 answer
130 views

A simple stock exchange app

I've struggled to make a small stock project for almost 5 hours (biggest one so far). The project is simple: I've chosen a stock (Tesla in my case) and I've decided to get a message on my phone with ...
biscuit_delicious's user avatar
1 vote
1 answer
137 views

Excluding attributes entirely vs. setting them to None

I am making a program that parses profiles from a website. Some profiles are "public" and others are "private" with limited information. Public profiles contain a lot more data and ...
angel's user avatar
  • 35
5 votes
3 answers
2k views

Python: Handling different HTTP status codes using the requests library

I am using the requests library to make requests to an API. However, I am not sure how to handle different status codes. I am particularly interested in handling ...
heuristic_type's user avatar
1 vote
0 answers
415 views

Calling PagerDuty REST API to recursively fetch oncall info for a team

I am a beginnner and have put up a code that talks to pagerduty API and then it fethces oncall info for a team (get team id, get escalation policy id, then finally get on call info). response is ...
AhmFM's user avatar
  • 121
2 votes
1 answer
69 views

Create URL's, then download and print each RSS feed text

Looking for a more efficient way of achieving this. In particular the combine_all_rss_lists so I don't need all those params. Feedparser module doesn't accept lists which is why I've had to initially ...
Luca's user avatar
  • 57
3 votes
1 answer
37 views

request data and print results

On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
Luca's user avatar
  • 57
1 vote
1 answer
115 views

Single API call with authentication in Python

I've written a small script for Zabbix to check the status of our ESXi Hosts. The request needs to be authenticated, so I went for this process: Read auth token from file If file does not exist, ...
Gerald Schneider's user avatar
2 votes
1 answer
275 views

python: requests large.zip -> unzip -> fix -> filter ->gunzip

I wrote a function to download a large zipfile 5-7gb from Iowa State MRMS data archive. The zip files appear to be malformed and results in a BadZipFileError hence ...
Jason Leaver's user avatar
2 votes
1 answer
95 views

Monitoring webpages to be notified

I have been writing a monitoring script where I check for whenever there is changes on a webpage. When a change has happend I want to be notified by printing out that there is a difference. I have ...
PythonNewbie's user avatar
2 votes
1 answer
97 views

Compare between two dictionaries

Hello beautiful people! I have currently worked on a small script that I would of course continue to work with whenever I get a good feedback from the best code reviewer in here <3 - I have worked ...
PythonNewbie's user avatar
1 vote
1 answer
4k views

A python script to download video from a website

I wrote this script for gaining experience, this script takes in a website link (NSFW) and download the video on that site. It uses regular expressions to parse the website source code to extract urls ...
User's user avatar
  • 201
9 votes
2 answers
2k views

Python script to scrape and parse the Stanford Encyclopedia of Philosophy

I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
user avatar
1 vote
0 answers
56 views

How to make memory and cpu efficient a code

I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
Learner's user avatar
  • 111
3 votes
1 answer
313 views

Python script to POST requests to a networking device

Request to review my code please and see if I am following the best practices. This is sending a POST operation via a Python script using the requests library to a networking cloud orchestration ...
pythontestuser's user avatar
1 vote
1 answer
126 views

Printing a JSON/HTTP response from a Cisco endpoint

Please check over my Python code for a HTTP GET operation using the requests library, and provide any potential pointers for improvement. ...
pythontestuser's user avatar
1 vote
1 answer
60 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
Sati's user avatar
  • 417
3 votes
1 answer
97 views

Acupuncture database builder

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
Sati's user avatar
  • 417
0 votes
0 answers
44 views

Python scraper that gets product information from e-commerce website

I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
dovexz12323's user avatar
4 votes
2 answers
72 views

Search for providers of TV Shows

All my code is below. It uses an API to search for a show and then outputs the providers (like breaking bad is available on Netflix). The main aim of this application was to practice using APIs and ...
Twam's user avatar
  • 41
0 votes
2 answers
144 views

Python script for Web scraping

I wrote a Python script for Web scraping of a Website. Please review my code and suggest me any changes or make me aware of my blunders/mistakes?. I wrote the almost same script for other websites ...
Muhammad Fahim's user avatar
0 votes
1 answer
218 views

Config Variables Initialization | Python

I found interesting the way Python's requests library does the status_code data structure initialization (see code here). I am reusing it in a hobby project, but ...
Nestor's user avatar
  • 291
3 votes
1 answer
76 views

Refactor Web Scraper

I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
Tauqeer Abbas's user avatar
3 votes
1 answer
239 views

benchmarking requests to localhost

I am writing a benchmarking tool from scratch in Python. However I can't get the performance of other benchmarking tools like wrk or ...
Masudul Hasan Masud's user avatar
2 votes
2 answers
182 views

Scrape webelements

I have been working with requests where I as easy as it is. Scraping the webpage and add it into a dict and print the payload if we find a new value or not. ...
PythonNewbie's user avatar
2 votes
1 answer
2k views

Run python File With get request flask API

Code is Working Review and Recommend What Are Best Practice What I am trying to achieve. API with Flask that runs any python file from the current directory. Run the file and get output in JSON ...
Qureshi Owais's user avatar
2 votes
2 answers
583 views

Web scraping and design pattern lifting

I would like to get some feed back on my code; the goal is to get all agencies address of banks. I wrote a pretty simple brute force algorithm. I was wondering if you would have any advice to improve ...
curious's user avatar
  • 293
2 votes
1 answer
62 views

Scraping webpage elements using Python

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
PythonNewbie's user avatar
2 votes
0 answers
210 views

Python REST API Data Extraction with requests library

Is there a better way to write this piece of code to extract data from a REST API with the requests library? It currently takes about 10 minutes to finish extracting the data from the REST API. I ...
Benb27's user avatar
  • 21
2 votes
1 answer
94 views

Beautifulsoup scrape posts that has the word python in it with no python tag

This is my first time web scraping, and here is the code I whipped up: ...
Chocolate's user avatar
  • 948
2 votes
1 answer
279 views

Do I need to use inheritance and classes for my OOP webscraper? [closed]

I am currently writing python code that scrapes information from the web. I have to scrape several sites, but there a two types of procedures: Directly scrape from the website Download pdf and scrape ...
FarFetched's user avatar
5 votes
2 answers
299 views

Python Web scraping

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any. What do I want to achieve? I want to scrape ...
Faheem Akhtar's user avatar
1 vote
0 answers
27 views

Trouble returning web scraping output as python dictionary [closed]

So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
Zack's user avatar
  • 11
3 votes
0 answers
854 views

Python Async Request using Tor and User-Agent Rotation

The code below will scrape data from three APIs. The APIs are for property listings. So, for each listing, I will have one request. There will be around 20000 requests. That's why I use ayncio to make ...
JOHN's user avatar
  • 131
5 votes
0 answers
92 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
Iduoad's user avatar
  • 151
1 vote
1 answer
354 views

Python program that gets my hackerrank code and scores using requests [closed]

I'm trying to extend my coding skills to other languages by doing projects. (learned Java in school) and since python is very popular right now thats what I ended up choosing for this project, though ...
jbbouz's user avatar
  • 11
8 votes
1 answer
164 views

Finding unimplemented tasks on Rosetta Code using Requests

This is a Find unimplemented tasks task from Rosetta Code: Given the name of a language on Rosetta Code, find all tasks which are not implemented in that language. There is already a solution ...
Georgy's user avatar
  • 1,939
6 votes
1 answer
2k views

Web scraping Google Trends in Python

Question Recently I have tried to create a web scraping program to get data from Google Trends. It uses an RSS feed to do so. My question is as follows: How can I improve my code such that it is more ...
Ranch Mayonaise's user avatar
1 vote
1 answer
4k views

Python requests downloading big files in slower speed than wget

This is a simple toy downloader using python's requests library. I’ve monitored the download process is slower on an ethernet connected box. The realtime speed is ...
faquir's user avatar
  • 11
4 votes
1 answer
199 views

Python program that gets basic information on a Github user

I've created a basic Python CLI script, which takes a Github username, and returns some basic information associated with said username. I use the requests for HTTP requests (I'm new to this area) and ...
Arnav Borborah's user avatar
5 votes
1 answer
5k views

Python scraping tables

I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
brain_dead_cow's user avatar
5 votes
1 answer
412 views

Scraping web data with Python 3

Below is a portion of the code i have written to scrape the bikesales.com.au website for details of bikes for sales (The full code is here). This finds all the 'href' attributes on each search page ...
theotheraussie's user avatar
7 votes
2 answers
2k views

Repeatable HTTP requests in Python

I have a simple python script that's repeating a HTTP response/request. I would like to know how I could have constructed this better, particularly if there's a way I could have incorporated the slice ...
Michael A's user avatar
  • 1,641
3 votes
1 answer
981 views

PyCryptoScraper to scrape currency rates

I created a PyCryptoScraper script to scrape crypto-currencies from Coingecko. Now I want to know if you could give me some hints or improvements for my script, which works fine. The whole code also ...
Patrick85's user avatar
  • 491
3 votes
1 answer
3k views

Downloading files using Python-requests

I wrote a Python script to download files using multiple (source) IP addresses -- kindly suggest any improvements. ...
avamsi's user avatar
  • 155
2 votes
1 answer
1k views

Object-oriented URL loading and HTTP client with requests in Python

I have an API which I want to consume in Python 3.x using the requests module. Right now, my solution is to create Requests objects for each URL like the ...
Sylvain's user avatar
  • 121
8 votes
1 answer
4k views

Web Scraping with Python + asyncio

I've been working at speeding up my web scraping with the asyncio library. I have a working solution, but am unsure as to how pythonic it is or if I am properly ...
Adam Hammes's user avatar