Questions tagged [python-requests]

USE ONLY FOR THE PYTHON REQUESTS LIBRARY. Requests is a full-featured Python HTTP library with an easy-to-use, logical API.

Filter by
Sorted by
Tagged with
2 votes
1 answer
48 views

Create URL's, then download and print each RSS feed text

Looking for a more efficient way of achieving this. In particular the combine_all_rss_lists so I don't need all those params. Feedparser module doesn't accept lists which is why I've had to initially ...
  • 55
3 votes
1 answer
32 views

request data and print results

On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
  • 55
1 vote
1 answer
55 views

Single API call with authentication in Python

I've written a small script for Zabbix to check the status of our ESXi Hosts. The request needs to be authenticated, so I went for this process: Read auth token from file If file does not exist, ...
2 votes
1 answer
78 views

python: requests large.zip -> unzip -> fix -> filter ->gunzip

I wrote a function to download a large zipfile 5-7gb from Iowa State MRMS data archive. The zip files appear to be malformed and results in a BadZipFileError hence ...
2 votes
1 answer
77 views

Monitoring webpages to be notified

I have been writing a monitoring script where I check for whenever there is changes on a webpage. When a change has happend I want to be notified by printing out that there is a difference. I have ...
2 votes
1 answer
90 views

Compare between two dictionaries

Hello beautiful people! I have currently worked on a small script that I would of course continue to work with whenever I get a good feedback from the best code reviewer in here <3 - I have worked ...
1 vote
1 answer
75 views

A python script to download video from a website

I wrote this script for gaining experience, this script takes in a website link (NSFW) and download the video on that site. It uses regular expressions to parse the website source code to extract urls ...
  • 201
9 votes
2 answers
2k views

Python script to scrape and parse the Stanford Encyclopedia of Philosophy

I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
's user avatar
1 vote
0 answers
44 views

How to make memory and cpu efficient a code

I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
  • 111
3 votes
1 answer
226 views

Python script to POST requests to a networking device

Request to review my code please and see if I am following the best practices. This is sending a POST operation via a Python script using the requests library to a networking cloud orchestration ...
1 vote
1 answer
99 views

Printing a JSON/HTTP response from a Cisco endpoint

Please check over my Python code for a HTTP GET operation using the requests library, and provide any potential pointers for improvement. ...
1 vote
1 answer
49 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
  • 321
3 votes
1 answer
56 views

Acupuncture database builder

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
  • 321
0 votes
0 answers
34 views

Python scraper that gets product information from e-commerce website

I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
4 votes
2 answers
61 views

Search for providers of TV Shows

All my code is below. It uses an API to search for a show and then outputs the providers (like breaking bad is available on Netflix). The main aim of this application was to practice using APIs and ...
  • 41
0 votes
2 answers
121 views

Python script for Web scraping

I wrote a Python script for Web scraping of a Website. Please review my code and suggest me any changes or make me aware of my blunders/mistakes?. I wrote the almost same script for other websites ...
0 votes
1 answer
100 views

Config Variables Initialization | Python

I found interesting the way Python's requests library does the status_code data structure initialization (see code here). I am reusing it in a hobby project, but ...
  • 301
3 votes
1 answer
57 views

Refactor Web Scraper

I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
3 votes
1 answer
93 views

benchmarking requests to localhost

I am writing a benchmarking tool from scratch in Python. However I can't get the performance of other benchmarking tools like wrk or ...
2 votes
2 answers
162 views

Scrape webelements

I have been working with requests where I as easy as it is. Scraping the webpage and add it into a dict and print the payload if we find a new value or not. ...
2 votes
1 answer
1k views

Run python File With get request flask API

Code is Working Review and Recommend What Are Best Practice What I am trying to achieve. API with Flask that runs any python file from the current directory. Run the file and get output in JSON ...
2 votes
2 answers
124 views

Web scraping and design pattern lifting

I would like to get some feed back on my code; the goal is to get all agencies address of banks. I wrote a pretty simple brute force algorithm. I was wondering if you would have any advice to improve ...
  • 273
2 votes
1 answer
61 views

Scraping webpage elements using Python

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
3 votes
0 answers
101 views

Python REST API Data Extraction with requests library

Is there a better way to write this piece of code to extract data from a REST API with the requests library? It currently takes about 10 minutes to finish extracting the data from the REST API. I ...
  • 31
2 votes
1 answer
85 views

Beautifulsoup scrape posts that has the word python in it with no python tag

This is my first time web scraping, and here is the code I whipped up: ...
  • 962
2 votes
1 answer
198 views

Do I need to use inheritance and classes for my OOP webscraper? [closed]

I am currently writing python code that scrapes information from the web. I have to scrape several sites, but there a two types of procedures: Directly scrape from the website Download pdf and scrape ...
  • 41
5 votes
2 answers
288 views

Python Web scraping

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any. What do I want to achieve? I want to scrape ...
1 vote
0 answers
27 views

Trouble returning web scraping output as python dictionary [closed]

So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
  • 11
3 votes
0 answers
592 views

Python Async Request using Tor and User-Agent Rotation

The code below will scrape data from three APIs. The APIs are for property listings. So, for each listing, I will have one request. There will be around 20000 requests. That's why I use ayncio to make ...
  • 131
5 votes
0 answers
81 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
  • 151
1 vote
1 answer
289 views

Python program that gets my hackerrank code and scores using requests [closed]

I'm trying to extend my coding skills to other languages by doing projects. (learned Java in school) and since python is very popular right now thats what I ended up choosing for this project, though ...
  • 11
8 votes
1 answer
144 views

Finding unimplemented tasks on Rosetta Code using Requests

This is a Find unimplemented tasks task from Rosetta Code: Given the name of a language on Rosetta Code, find all tasks which are not implemented in that language. There is already a solution ...
  • 1,913
6 votes
1 answer
2k views

Web scraping Google Trends in Python

Question Recently I have tried to create a web scraping program to get data from Google Trends. It uses an RSS feed to do so. My question is as follows: How can I improve my code such that it is more ...
1 vote
1 answer
3k views

Python requests downloading big files in slower speed than wget

This is a simple toy downloader using python's requests library. I’ve monitored the download process is slower on an ethernet connected box. The realtime speed is ...
  • 11
4 votes
1 answer
137 views

Python program that gets basic information on a Github user

I've created a basic Python CLI script, which takes a Github username, and returns some basic information associated with said username. I use the requests for HTTP requests (I'm new to this area) and ...
5 votes
1 answer
5k views

Python scraping tables

I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
4 votes
1 answer
377 views

Scraping web data with Python 3

Below is a portion of the code i have written to scrape the bikesales.com.au website for details of bikes for sales (The full code is here). This finds all the 'href' attributes on each search page ...
7 votes
2 answers
1k views

Repeatable HTTP requests in Python

I have a simple python script that's repeating a HTTP response/request. I would like to know how I could have constructed this better, particularly if there's a way I could have incorporated the slice ...
  • 1,631
3 votes
1 answer
962 views

PyCryptoScraper to scrape currency rates

I created a PyCryptoScraper script to scrape crypto-currencies from Coingecko. Now I want to know if you could give me some hints or improvements for my script, which works fine. The whole code also ...
  • 491
3 votes
1 answer
3k views

Downloading files using Python-requests

I wrote a Python script to download files using multiple (source) IP addresses -- kindly suggest any improvements. ...
  • 155
2 votes
1 answer
1k views

Object-oriented URL loading and HTTP client with requests in Python

I have an API which I want to consume in Python 3.x using the requests module. Right now, my solution is to create Requests objects for each URL like the ...
  • 121
8 votes
1 answer
4k views

Web Scraping with Python + asyncio

I've been working at speeding up my web scraping with the asyncio library. I have a working solution, but am unsure as to how pythonic it is or if I am properly ...
7 votes
1 answer
10k views

Python auth using requests

I am trying to implement a custom ApiClient in Python, using requests. The way it does authentication is by: login(username, password) -> get back a token if valid, http error code if not set the ...
  • 215
5 votes
0 answers
357 views

Choose the Transport Layer Security protocol - requests python

I am using using the following code to choose the transport layer security protocol while issuing a request to server. Could someone please review it. This is in reference to the following question: ...
5 votes
1 answer
4k views

Python API client

I'm creating a Python API client, after thinking and checking some open source Python clients I created this class, and I'm looking for any feedback (design, best practices...), this is an example URL ...
  • 153
8 votes
1 answer
210 views

Timed requests and parsing JSON in Python

I wrote this tiny script to pull the JSON feed from the CitiBike website: ...
  • 337