Questions tagged [selenium]

Selenium is a tools suite to automate web browsers, and web application testing

Filter by
Sorted by
Tagged with
3 votes
1 answer
68 views

Scraping website with Python and Selenium to collect data from dynamic website

Summary: The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
sangharsh's user avatar
  • 279
2 votes
1 answer
44 views

A selenium web scraper to package NBA data

I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
BluffShove's user avatar
0 votes
0 answers
83 views

Why is my scraping code so slow?

I'm a beginner with scraping with Selenium. And I wrote a couple of scrips. And when the script is used for a website with 10-15 pages, it works as it should. However, when I use it on a larger ...
hemoglobin's user avatar
0 votes
2 answers
57 views

Application install code- Need to remove Nested Loops and make code more compact and readable. Code installs application and its version as per req

This code installs the application, according to the type of application selected like alpha, beta, and released, also it checks for latest version in each type. ...
Rohan's user avatar
  • 29
2 votes
0 answers
85 views

Python script that refreshes network connection at random intervals

This is a followup to Python script that reboots the router every 600 seconds. Recently censorship in China has increased because of the meeting of heads of states, and my previous script has been ...
Ξένη Γήινος's user avatar
12 votes
4 answers
1k views

Python script that reboots the router every 600 seconds

This is a Python script I wrote to reboot the router every 600 seconds. Why did I do it? Well, in case you don't know, I live in China, and I use VPNs to browse the international internet. My ISP ...
Ξένη Γήινος's user avatar
0 votes
1 answer
99 views

Poetry Web Scraping in Python [closed]

I have a script that obtains urls that lead to a specific poem. This code current works and uses multiprocessing pools. I currently am getting restricted or blocked by some way from the website that I ...
watrgoat's user avatar
5 votes
1 answer
67 views

Waits for one of two xpaths to be present, one inside of an iframe, one outside of it

I'm using selenium for testing, and I've written code that waits for one of two locations before it continues. This is really easy when both locations are within the same frame, because then I can ...
Josh Heaps's user avatar
1 vote
1 answer
55 views

Python script for get minimum product price having maximum reviews

Technical task. user visits amazon.com website user fills out a search field with the product name and activates search ==> a page with search results is displayed. user looks for the product ...
lr_lennok's user avatar
0 votes
1 answer
181 views

Selenium/Beautifulsoup web scrape of Grainger

This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
cahilltyler's user avatar
0 votes
3 answers
2k views

To automate scheduling at the Italian Consulate

I'm a beginner developer in Python and I need some tips to improve my code, in performance (speed) and hardware usage. Is it possible to improve? And I have a question: is it possible to keep the ...
user avatar
2 votes
1 answer
51 views

scraping pollution data tables from government website

i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved: ...
Omar Morales Rivera's user avatar
3 votes
1 answer
188 views

Scrape PokeDex and display in tkinter

Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
Salah Zahran's user avatar
3 votes
1 answer
180 views

Crawler/scraper for soccer match results

I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
Olgierd Wiśniewski's user avatar
1 vote
1 answer
68 views

Volunteer registration website Selenium testing

I have a volunteer registration website that I wrote some end-to-end tests for in Selenium. Selenium is of course software that lets you automate clicking around a browser, which makes it great for ...
RedDragonWebDesign's user avatar
1 vote
0 answers
228 views

Web scraper for pokemon prices using Selenium

I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
Moorgort's user avatar
1 vote
1 answer
370 views

Selenium web scraper to scrape Apple tweets of the last 3 years

I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
VaKt's user avatar
  • 11
3 votes
0 answers
86 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
berkersenol's user avatar
4 votes
2 answers
321 views

Scraping campsite availability from a webpage using vba with selenium

I wrote some code to extract the information from a table, but it takes an extremely long time. The table is in the format of a calendar. I need the information on an Excel sheet with column 1 as the ...
Nelj's user avatar
  • 41
1 vote
1 answer
353 views

Web-scrape bonds information from nasdaqomxnordic

I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated! ...
yogi06's user avatar
  • 13
4 votes
2 answers
586 views

Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python

I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths. The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
Nini's user avatar
  • 55
1 vote
1 answer
95 views

Web Scraping DOL with Selenium

This is my code to extract data from the DOL website for a project I'm doing for my portfolio. The code does extract all of the data that I need, however it runs really slowly. I think it took like 15 ...
Nini's user avatar
  • 55
2 votes
0 answers
575 views

My first vba-selenium code

I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company. The asset value is not available other than on ...
Ron Rosenfeld's user avatar
1 vote
2 answers
107 views

WaitFor method for website automation testing with Selenium WebDriver

I am trying to use Selenium for website automation testing tasks and I am new to Selenium testing framework. The the situation I faced is to wait the website components loading and then do the related ...
JimmyHu's user avatar
  • 4,708
4 votes
1 answer
1k views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
Sati's user avatar
  • 417
2 votes
1 answer
957 views

Extract flight arrival data from web

I'm trying to scrape arrival data from Flight Radar 24 My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process? Here's my script: ...
lockey's user avatar
  • 51
1 vote
1 answer
44 views

Simple login test, translated from java to python

I'm reading on how to write proper testing suites here. So I'm trying to follow the selenium example in the docs which is in Java; I'm trying to translate it to Python since my app is written in ...
Tom's user avatar
  • 241
1 vote
0 answers
20 views

Organizing things together to form a minimum viable Scraper App (part 2)

This is a follow-up of my question over here. Response to @Reinderien's answer: I have corrected the more trivial issues highlighted in @Reinderien's answer below as follows. ...
Sati's user avatar
  • 417
1 vote
0 answers
81 views

Optimising this web-scraping code

A member of SO who I immensely respect just told me that the code below makes him uncomfortable. ...
PyNoob's user avatar
  • 179
2 votes
0 answers
2k views

Python script to scrape google maps without API

I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance. It uses selenium ...
WhiteWood's user avatar
2 votes
1 answer
518 views

Looping through search results in Selenium

This is a follow-up of my question over here. I am extending the functionality of @Reinderien's suggested code to enable the automatic looping through of pages in the search result. I've mainly ...
Sati's user avatar
  • 417
0 votes
1 answer
190 views

Unix batch job execution from BDD automation framework

As part of my testing, I have to connect to Unix server using putty and execute few unix jobs. Feature file Steps for test scenario: ...
Sh87's user avatar
  • 89
3 votes
1 answer
285 views

Looping through keywords and multiple pages in Selenium

This is a follow-up of my question over here. I adopted @Reinderien's suggested script to a second website below: fudan.py ...
Sati's user avatar
  • 417
1 vote
1 answer
95 views

How to make use of a class of descriptors to save code and other issues

This is a follow-up to my question over here. Having worked through most of the issues highlighted by @Reinderien in his detailed answer, I somehow got the code to work... Some quick observations: <...
Sati's user avatar
  • 417
3 votes
1 answer
161 views

Getting xpath indices to count forward to preserve table structure

Scraping tables from the web get complicated when there are 2 or more values in a cell. In order to preserve the table structure, I have devised a way to count the row-number index of its xpath, ...
Sati's user avatar
  • 417
-2 votes
1 answer
102 views

Can I refactor my code anymore? [closed]

I have written one method; however I feel I can refactor it even more but not sure what Can someone help me here? ...
Chandler's user avatar
2 votes
0 answers
331 views

A thread-safe WebDriver pool to submit browsing jobs to

For a project, I need to make use of Selenium WebDrivers, and since they're so expensive to start, I decided to write a pool to manage them. ...
Carcigenicate's user avatar
2 votes
2 answers
302 views

Scrape and parse dictionary website and get meaning of each word in a text file

In this program, I have done automation using Selenium, to find english meaning of each sanskrit word from a dictionary website. The function eng_meaning accepts ...
coder's user avatar
  • 2,449
1 vote
0 answers
384 views

Selenium Webscraper for Odd data analyses

I have written the following code to achieve the following: Regularly scrape all live matches on the betting site oddsportal.com Pull odds data into data frame Evaluate data frame for two odd ...
BlackElefant's user avatar
3 votes
0 answers
71 views

Web Scraping data from website using Selenium in python

ApplicationDetail.txt: URL: https://www.abc.com applicationNo : 123456 Application Class: ...
Mr.A I's user avatar
  • 31
2 votes
0 answers
584 views

Discord Bot Python. Selenium screenshots

I wrote my own Discord Bot which is taking screenshot of a specific website. This scrip is very simple and thought it should work fast, but it isn't. I read a lot, and I think I'm not able to improve ...
Lukas's user avatar
  • 21
2 votes
1 answer
3k views

How to make selenium in python faster

I'm making a bot for PlayStation 5. And restock is coming soon so I want to make my bot faster. The approximate duration is 4.90 seconds. Is there any way to make this faster? I found that I should ...
Matija Milicevic's user avatar
4 votes
0 answers
90 views

Python Selenium First Project - loop through pages and check/uncheck tick boxes

New to Python/Selenium and learning on the fly with my first basic selenium script. Request user input - activate or deactivate. Open's website, loops through shop URL as well as activate/deactivate ...
Richard Lee's user avatar
0 votes
1 answer
88 views

Reading configuration variables in a Python project

I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper) I have created .env file at the root of the project ...
Hooman Bahreini's user avatar
5 votes
1 answer
281 views

Automating TypeRacer using selenium

This is a personal project which I created solely to torture my friends ;) I'm pretty sure you've heard about typeracer. And if you have friends like mine who are faster than you, you'd surely be ...
Sriv's user avatar
  • 2,730
0 votes
2 answers
172 views

open multiple tabs using selenium

I am using the following script to read search terms from a file and then search those terms in google (each search result in it's own tab). ...
Ahmad Ismail's user avatar
2 votes
0 answers
87 views

C# Scraper - EventHandler

It might help to check my previous question out, its part of the same application. The only minor changes are with the entity class names. They now each have their own entities, and I use interfaces ...
Josh Hallow's user avatar
  • 1,325
7 votes
1 answer
245 views

Booking an East London Tennis Court

Description I'm not sure if it Covid-19 but lately it is impossible to book a tennis court in my area on time. It's always full or maybe just don't check enough :) To beat the queue and get notified ...
Ludisposed's user avatar
  • 11.5k
6 votes
1 answer
485 views

Web Scraping using Selenium and Python

This is my first program code ever, and it actually works. My goal is to scrape information from the website and store it in a database. It is a site that has historical data on sporting events and ...
user avatar
5 votes
1 answer
1k views

Game scraper for Steam

I made a scraper for Steam that get different info about a Steam game, such as the price, the specs and the supported platforms. The reason I made this was because I have a super slow laptop, so ...
F.M's user avatar
  • 153