Questions tagged [python-requests]

USE ONLY FOR THE PYTHON REQUESTS LIBRARY. Requests is a full-featured Python HTTP library with an easy-to-use, logical API.

Filter by
Sorted by
Tagged with
2 votes
1 answer
129 views

Python script to POST requests to a networking device

Request to review my code please and see if I am following the best practices. This is sending a POST operation via a Python script using the requests library to a networking cloud orchestration ...
user avatar
1 vote
1 answer
81 views

Printing a JSON/HTTP response from a Cisco endpoint

Please check over my Python code for a HTTP GET operation using the requests library, and provide any potential pointers for improvement. ...
user avatar
1 vote
1 answer
34 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
user avatar
  • 321
3 votes
1 answer
51 views

Acupuncture database builder

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
user avatar
  • 321
0 votes
0 answers
26 views

Python scraper that gets product information from e-commerce website

I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
user avatar
4 votes
2 answers
56 views

Search for providers of TV Shows

All my code is below. It uses an API to search for a show and then outputs the providers (like breaking bad is available on Netflix). The main aim of this application was to practice using APIs and ...
user avatar
  • 41
0 votes
2 answers
101 views

Python script for Web scraping

I wrote a Python script for Web scraping of a Website. Please review my code and suggest me any changes or make me aware of my blunders/mistakes?. I wrote the almost same script for other websites ...
user avatar
0 votes
1 answer
63 views

Config Variables Initialization | Python

I found interesting the way Python's requests library does the status_code data structure initialization (see code here). I am reusing it in a hobby project, but ...
user avatar
  • 301
3 votes
1 answer
51 views

Refactor Web Scraper

I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
user avatar
2 votes
2 answers
154 views

Scrape webelements

I have been working with requests where I as easy as it is. Scraping the webpage and add it into a dict and print the payload if we find a new value or not. ...
user avatar
2 votes
1 answer
754 views

Run python File With get request flask API

Code is Working Review and Recommend What Are Best Practice What I am trying to achieve. API with Flask that runs any python file from the current directory. Run the file and get output in JSON ...
user avatar
2 votes
2 answers
87 views

Web scraping and design pattern lifting

I would like to get some feed back on my code; the goal is to get all agencies address of banks. I wrote a pretty simple brute force algorithm. I was wondering if you would have any advice to improve ...
user avatar
  • 273
2 votes
1 answer
58 views

Scraping webpage elements using Python

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
user avatar
3 votes
0 answers
90 views

Python REST API Data Extraction with requests library

Is there a better way to write this piece of code to extract data from a REST API with the requests library? It currently takes about 10 minutes to finish extracting the data from the REST API. I ...
user avatar
  • 31
2 votes
1 answer
74 views

Beautifulsoup scrape posts that has the word python in it with no python tag

This is my first time web scraping, and here is the code I whipped up: ...
user avatar
  • 964
2 votes
1 answer
159 views

Do I need to use inheritance and classes for my OOP webscraper? [closed]

I am currently writing python code that scrapes information from the web. I have to scrape several sites, but there a two types of procedures: Directly scrape from the website Download pdf and scrape ...
user avatar
  • 41
5 votes
2 answers
282 views

Python Web scraping

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any. What do I want to achieve? I want to scrape ...
user avatar
1 vote
0 answers
26 views

Trouble returning web scraping output as python dictionary [closed]

So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
user avatar
  • 11
3 votes
0 answers
498 views

Python Async Request using Tor and User-Agent Rotation

The code below will scrape data from three APIs. The APIs are for property listings. So, for each listing, I will have one request. There will be around 20000 requests. That's why I use ayncio to make ...
user avatar
  • 131
5 votes
0 answers
70 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
user avatar
  • 151
1 vote
1 answer
261 views

Python program that gets my hackerrank code and scores using requests [closed]

I'm trying to extend my coding skills to other languages by doing projects. (learned Java in school) and since python is very popular right now thats what I ended up choosing for this project, though ...
user avatar
  • 11
8 votes
1 answer
134 views

Finding unimplemented tasks on Rosetta Code using Requests

This is a Find unimplemented tasks task from Rosetta Code: Given the name of a language on Rosetta Code, find all tasks which are not implemented in that language. There is already a solution ...
user avatar
  • 1,913
6 votes
1 answer
2k views

Web scraping Google Trends in Python

Question Recently I have tried to create a web scraping program to get data from Google Trends. It uses an RSS feed to do so. My question is as follows: How can I improve my code such that it is more ...
user avatar
1 vote
1 answer
3k views

Python requests downloading big files in slower speed than wget

This is a simple toy downloader using python's requests library. I’ve monitored the download process is slower on an ethernet connected box. The realtime speed is ...
user avatar
  • 11
4 votes
1 answer
113 views

Python program that gets basic information on a Github user

I've created a basic Python CLI script, which takes a Github username, and returns some basic information associated with said username. I use the requests for HTTP requests (I'm new to this area) and ...
user avatar
5 votes
1 answer
5k views

Python scraping tables

I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
user avatar
4 votes
1 answer
366 views

Scraping web data with Python 3

Below is a portion of the code i have written to scrape the bikesales.com.au website for details of bikes for sales (The full code is here). This finds all the 'href' attributes on each search page ...
user avatar
7 votes
2 answers
1k views

Repeatable HTTP requests in Python

I have a simple python script that's repeating a HTTP response/request. I would like to know how I could have constructed this better, particularly if there's a way I could have incorporated the slice ...
user avatar
  • 1,631
3 votes
1 answer
948 views

PyCryptoScraper to scrape currency rates

I created a PyCryptoScraper script to scrape crypto-currencies from Coingecko. Now I want to know if you could give me some hints or improvements for my script, which works fine. The whole code also ...
user avatar
  • 491
3 votes
1 answer
3k views

Downloading files using Python-requests

I wrote a Python script to download files using multiple (source) IP addresses -- kindly suggest any improvements. ...
user avatar
  • 155
2 votes
1 answer
1k views

Object-oriented URL loading and HTTP client with requests in Python

I have an API which I want to consume in Python 3.x using the requests module. Right now, my solution is to create Requests objects for each URL like the ...
user avatar
  • 121
8 votes
1 answer
4k views

Web Scraping with Python + asyncio

I've been working at speeding up my web scraping with the asyncio library. I have a working solution, but am unsure as to how pythonic it is or if I am properly ...
user avatar
7 votes
1 answer
10k views

Python auth using requests

I am trying to implement a custom ApiClient in Python, using requests. The way it does authentication is by: login(username, password) -> get back a token if valid, http error code if not set the ...
user avatar
  • 215
5 votes
0 answers
349 views

Choose the Transport Layer Security protocol - requests python

I am using using the following code to choose the transport layer security protocol while issuing a request to server. Could someone please review it. This is in reference to the following question: ...
user avatar
5 votes
1 answer
4k views

Python API client

I'm creating a Python API client, after thinking and checking some open source Python clients I created this class, and I'm looking for any feedback (design, best practices...), this is an example URL ...
user avatar
  • 153
8 votes
1 answer
203 views

Timed requests and parsing JSON in Python

I wrote this tiny script to pull the JSON feed from the CitiBike website: ...
user avatar
  • 337