I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well.
I tried to follow this best practices:
- Give meaningful names to variables
- Only explain what seems not obvious
- Split code into functions to make it more readable.
Currently the code takes a while, I am not sure how to speed it up just yet.
Code
import puppeteer, {type Page, type Browser} from "puppeteer";
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
const baseURL = "https://www.arnoldclark.com/used-cars";
//based off conditions that I want, done manually in the website
const city = "cityName"
const search =
"?payment_type=cash&min_price=4000&max_price=8000&location=${city}%2C UK&photos_only=true&unreserved_only=true&roadtax_cost=30&mileage=72000&age=6&seats[]=4&seats[]=5&show_click_and_collect_options=true&sort_order=mileage";
await page.goto(baseURL + search);
// car URL from link
const carUrls = await page.$$eval("h2.ac-vehicle__title a", (carLinks) =>
carLinks.map((link) => link.href)
);
const candidates = await getCandidates(browser, carUrls);
console.table(candidates);
await browser.close();
})();
interface Car {
favs: number;
url: string;
}
async function getCandidates(browser: Browser, carUrls: string[]) {
const candidates:Car[] = [];
await Promise.all(
carUrls.map(async (url) => {
const newPage = await browser.newPage();
await newPage.goto(url);
try {
// may be loaded dynamically
const favs = await newPage.$eval(
".ac-productheader__content-wrapper span",
(el) => el.textContent
);
if(favs) candidates.push({ favs: Number.parseInt(favs.split(" ")[0]), url });
} catch (e) {
console.log("no favs");
}
newPage.close()
})
);
return candidates.sort((a, b) => b.favs - a.favs).slice(0, 4);
}
Sample output
┌─────────┬──────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ (index) │ favs │ url │
├─────────┼──────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 0 │ 87 │ 'https://www.arnoldclark.com/used-cars/kia/picanto/1-0-1-5dr/2018/ref/arnai-u-92166' │
│ 1 │ 31 │ 'https://www.arnoldclark.com/used-cars/suzuki/celerio/1-0-sz3-5dr/2018/ref/arnai-u-92266' │
│ 2 │ 31 │ 'https://www.arnoldclark.com/used-cars/ford/ka/1-2-studio-5dr/2017/ref/arnaj-u-41340' │
│ 3 │ 29 │ 'https://www.arnoldclark.com/used-cars/dacia/sandero/1-2-16v-ambiance-5dr/2017/ref/arndv-u-75527' │
└─────────┴──────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
({headless:false})
99% of the time is occupied by the process sending 20 http requests for pages (and each page requests extra assets) at once and waiting the content to arrive. @Elikill58 \$\endgroup\$