Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [selenium]

Selenium is a tools suite to automate web browsers, and web application testing

Filter by
Sorted by
Tagged with
-4 votes
0 answers
6 views

Getting "Cannot invoke "String.equalsIgnoreCase(String)" because "this.a" is null" error [closed]

enter code here package practice; import java.util.Scanner; import org.openqa.selenium.WebDriver; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa....
user avatar
1 vote
0 answers
31 views

Web scraper for pokemon prices using Selenium

I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
user avatar
1 vote
1 answer
32 views

Selenium web scraper to scrape Apple tweets of the last 3 years

I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
user avatar
  • 11
3 votes
0 answers
65 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
user avatar
4 votes
2 answers
94 views

Scraping campsite availability from a webpage using vba with selenium

I wrote some code to extract the information from a table, but it takes an extremely long time. The table is in the format of a calendar. I need the information on an Excel sheet with column 1 as the ...
user avatar
  • 41
0 votes
0 answers
19 views

Python selenium online Stanford-Binet Test solver

This is a Python program that solves a version of Stanford-Binet Test found at https://stanfordbinettest.com, basically it is a selenium bot that goes to the test url and submits what I have ...
user avatar
0 votes
0 answers
45 views

A Python program that scrapes questions from an online Stanford-Binet Test website

This is a simple Python program that scrapes questions from https://stanfordbinettest.com/, it has a version of Stanford-Binet IQ Test which randomly selects 100 questions from a pool of ...
user avatar
1 vote
1 answer
70 views

Web-scrape bonds information from nasdaqomxnordic

I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated! ...
user avatar
  • 13
4 votes
2 answers
393 views

Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python

I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths. The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
user avatar
  • 55
1 vote
1 answer
74 views

Web Scraping DOL with Selenium

This is my code to extract data from the DOL website for a project I'm doing for my portfolio. The code does extract all of the data that I need, however it runs really slowly. I think it took like 15 ...
user avatar
  • 55
2 votes
0 answers
115 views

My first vba-selenium code

I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company. The asset value is not available other than on ...
user avatar
1 vote
2 answers
76 views

WaitFor method for website automation testing with Selenium WebDriver

I am trying to use Selenium for website automation testing tasks and I am new to Selenium testing framework. The the situation I faced is to wait the website components loading and then do the related ...
user avatar
  • 3,302
3 votes
1 answer
440 views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
user avatar
  • 321
2 votes
1 answer
190 views

Extract flight arrival data from web

I'm trying to scrape arrival data from Flight Radar 24 My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process? Here's my script: ...
user avatar
  • 51
1 vote
1 answer
38 views

Simple login test, translated from java to python

I'm reading on how to write proper testing suites here. So I'm trying to follow the selenium example in the docs which is in Java; I'm trying to translate it to Python since my app is written in ...
user avatar
  • 241
1 vote
0 answers
20 views

Organizing things together to form a minimum viable Scraper App (part 2)

This is a follow-up of my question over here. Response to @Reinderien's answer: I have corrected the more trivial issues highlighted in @Reinderien's answer below as follows. ...
user avatar
  • 321
1 vote
0 answers
58 views

Optimising this web-scraping code

A member of SO who I immensely respect just told me that the code below makes him uncomfortable. ...
user avatar
  • 169
2 votes
0 answers
835 views

Python script to scrape google maps without API

I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance. It uses selenium ...
user avatar
2 votes
1 answer
77 views

Looping through search results in Selenium

This is a follow-up of my question over here. I am extending the functionality of @Reinderien's suggested code to enable the automatic looping through of pages in the search result. I've mainly ...
user avatar
  • 321
0 votes
1 answer
104 views

Unix batch job execution from BDD automation framework

As part of my testing, I have to connect to Unix server using putty and execute few unix jobs. Feature file Steps for test scenario: ...
user avatar
  • 89
3 votes
1 answer
167 views

Looping through keywords and multiple pages in Selenium

This is a follow-up of my question over here. I adopted @Reinderien's suggested script to a second website below: fudan.py ...
user avatar
  • 321
1 vote
1 answer
73 views

How to make use of a class of descriptors to save code and other issues

This is a follow-up to my question over here. Having worked through most of the issues highlighted by @Reinderien in his detailed answer, I somehow got the code to work... Some quick observations: <...
user avatar
  • 321
3 votes
1 answer
83 views

Getting xpath indices to count forward to preserve table structure

Scraping tables from the web get complicated when there are 2 or more values in a cell. In order to preserve the table structure, I have devised a way to count the row-number index of its xpath, ...
user avatar
  • 321
-2 votes
1 answer
99 views

Can I refactor my code anymore? [closed]

I have written one method; however I feel I can refactor it even more but not sure what Can someone help me here? ...
user avatar
1 vote
0 answers
80 views

A thread-safe WebDriver pool to submit browsing jobs to

For a project, I need to make use of Selenium WebDrivers, and since they're so expensive to start, I decided to write a pool to manage them. ...
user avatar
2 votes
2 answers
72 views

Scrape and parse dictionary website and get meaning of each word in a text file

In this program, I have done automation using Selenium, to find english meaning of each sanskrit word from a dictionary website. The function eng_meaning accepts ...
user avatar
  • 2,439
1 vote
0 answers
56 views

Selenium Webscraper for Odd data analyses

I have written the following code to achieve the following: Regularly scrape all live matches on the betting site oddsportal.com Pull odds data into data frame Evaluate data frame for two odd ...
user avatar
2 votes
0 answers
36 views

Web Scraping data from website using Selenium in python

ApplicationDetail.txt: URL: https://www.abc.com applicationNo : 123456 Application Class: ...
user avatar
  • 21
2 votes
0 answers
425 views

Discord Bot Python. Selenium screenshots

I wrote my own Discord Bot which is taking screenshot of a specific website. This scrip is very simple and thought it should work fast, but it isn't. I read a lot, and I think I'm not able to improve ...
user avatar
  • 21
2 votes
1 answer
805 views

How to make selenium in python faster

I'm making a bot for PlayStation 5. And restock is coming soon so I want to make my bot faster. The approximate duration is 4.90 seconds. Is there any way to make this faster? I found that I should ...
user avatar
4 votes
0 answers
61 views

Python Selenium First Project - loop through pages and check/uncheck tick boxes

New to Python/Selenium and learning on the fly with my first basic selenium script. Request user input - activate or deactivate. Open's website, loops through shop URL as well as activate/deactivate ...
user avatar
0 votes
1 answer
60 views

Reading configuration variables in a Python project

I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper) I have created .env file at the root of the project ...
user avatar
5 votes
1 answer
188 views

Automating TypeRacer using selenium

This is a personal project which I created solely to torture my friends ;) I'm pretty sure you've heard about typeracer. And if you have friends like mine who are faster than you, you'd surely be ...
user avatar
  • 2,710
0 votes
2 answers
103 views

open multiple tabs using selenium

I am using the following script to read search terms from a file and then search those terms in google (each search result in it's own tab). ...
user avatar
2 votes
0 answers
77 views

C# Scraper - EventHandler

It might help to check my previous question out, its part of the same application. The only minor changes are with the entity class names. They now each have their own entities, and I use interfaces ...
user avatar
6 votes
0 answers
92 views

Booking an East London Tennis Court

Description I'm not sure if it Covid-19 but lately it is impossible to book a tennis court in my area on time. It's always full or maybe just don't check enough :) To beat the queue and get notified ...
user avatar
  • 11.4k
4 votes
1 answer
271 views

Web Scraping using Selenium and Python

This is my first program code ever, and it actually works. My goal is to scrape information from the website and store it in a database. It is a site that has historical data on sporting events and ...
user avatar
5 votes
1 answer
554 views

Game scraper for Steam

I made a scraper for Steam that get different info about a Steam game, such as the price, the specs and the supported platforms. The reason I made this was because I have a super slow laptop, so ...
user avatar
  • 153
3 votes
1 answer
100 views

Project : My Anime Finder

This is a project that finds a specific anime (show) and checks to see if there is a new episode of it . this is also the first time that i used def like this and ...
user avatar
  • 359
6 votes
2 answers
237 views

Web scraper for driver's license test times

I have created a small selenium script that checks for available times to write a test for a drivers license. The program runs every minute and takes approx 50 seconds to run. I have noticed that it's ...
user avatar
6 votes
2 answers
418 views

Finding solutions on GitHub and Stack Overflow

The first part of the code opens Chrome and navigates to GitHub. If I don't find my solution, I open a new tab with ctrlt. Then my code goes to Stack Overflow and searches there for solutions. ...
user avatar
  • 359
5 votes
2 answers
106 views

Crawling a court website and downloading records

I am new to Python and I am using Python 3.8 and Selenium. This is a follow up of my previous question. Since I received several useful suggestions I have re-written the following code. The code works ...
user avatar
  • 231
3 votes
1 answer
128 views

Web Scraping Dynamically Generated Content Python

This is a small web scraping project I made in 2 hours that targets the website remote.co . I am looking forward for improvements in my code. I know about the inconsistency with the WebDriverWait and ...
user avatar
4 votes
3 answers
358 views

Crawl a website and download records

I am new to Python. Amateur, enthusiast and learner. I have developed code which visits one particular website. Crawls through it to reach a certain repository, downloads the records from there to the ...
user avatar
  • 231
4 votes
1 answer
219 views

Webscraping With Selenium - a Course Downloader and Sorter

I've been working on a program that automates the painful task of downloading each file from a website for courses one by one and sorting them using python. This is my first python webscraping project ...
user avatar
  • 1,180
3 votes
2 answers
1k views

Instagram Scraping Using Selenium - Download Posts - Photos - Videos

Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium ...
user avatar
  • 387
5 votes
1 answer
2k views

Scraping Instagram - Download posts, photos - videos

Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium ...
user avatar
  • 387
2 votes
1 answer
237 views

Download pictures (or videos) from Instagram using Selenium

Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium ...
user avatar
  • 387
2 votes
1 answer
939 views

Instagram Scraping Posts Using Selenium

Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium and navigate to ...
user avatar
  • 387
2 votes
1 answer
566 views

Instagram Scraping Using Selenium

Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium and navigate to ...
user avatar
  • 387