3
\$\begingroup\$

I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well.

I tried to follow this best practices:

  • Give meaningful names to variables
  • Only explain what seems not obvious
  • Split code into functions to make it more readable.

Currently the code takes a while, I am not sure how to speed it up just yet.

Code


import puppeteer, {type Page, type Browser} from "puppeteer";

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  const baseURL = "https://www.arnoldclark.com/used-cars";
  //based off conditions that I want, done manually in the website
  const city = "cityName"
  const search =
    "?payment_type=cash&min_price=4000&max_price=8000&location=${city}%2C UK&photos_only=true&unreserved_only=true&roadtax_cost=30&mileage=72000&age=6&seats[]=4&seats[]=5&show_click_and_collect_options=true&sort_order=mileage";

  await page.goto(baseURL + search);

  // car URL from link
  const carUrls = await page.$$eval("h2.ac-vehicle__title a", (carLinks) =>
    carLinks.map((link) => link.href)
  );

  const candidates = await getCandidates(browser, carUrls);

  console.table(candidates);

  await browser.close();
})();

interface Car { 
  favs: number;
  url: string;
}

async function getCandidates(browser: Browser, carUrls: string[]) {
  const candidates:Car[] = [];
  await Promise.all(
    carUrls.map(async (url) => {
      const newPage = await browser.newPage();
      await newPage.goto(url);
      try {
        // may be loaded dynamically
        const favs = await newPage.$eval(
          ".ac-productheader__content-wrapper span",
          (el) => el.textContent
        );
        if(favs) candidates.push({ favs: Number.parseInt(favs.split(" ")[0]), url });
      } catch (e) {
        console.log("no favs");
      }
      newPage.close()
    })
  );
  return candidates.sort((a, b) => b.favs - a.favs).slice(0, 4);
}

Sample output

┌─────────┬──────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ (index) │ favs │                                                url                                                │
├─────────┼──────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│    0    │  87  │       'https://www.arnoldclark.com/used-cars/kia/picanto/1-0-1-5dr/2018/ref/arnai-u-92166'        │
│    1    │  31  │     'https://www.arnoldclark.com/used-cars/suzuki/celerio/1-0-sz3-5dr/2018/ref/arnai-u-92266'     │
│    2    │  31  │       'https://www.arnoldclark.com/used-cars/ford/ka/1-2-studio-5dr/2017/ref/arnaj-u-41340'       │
│    3    │  29  │ 'https://www.arnoldclark.com/used-cars/dacia/sandero/1-2-16v-ambiance-5dr/2017/ref/arndv-u-75527' │
└─────────┴──────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
\$\endgroup\$
4
  • 1
    \$\begingroup\$ Did you checked times to run parts? Like if the request was long, if it's because of sorting etc? \$\endgroup\$
    – Elikill58
    Apr 20 at 7:15
  • \$\begingroup\$ wdym ? I think it just takes time because it is http-ing to each URL @Elikill58 \$\endgroup\$
    – Mah Neh
    Apr 20 at 10:21
  • 1
    \$\begingroup\$ Yes but how much? Does one URL is more longer than others? Are they representing 90% of total time or 10% of it? It can be a good start to know where time is used \$\endgroup\$
    – Elikill58
    Apr 20 at 11:01
  • \$\begingroup\$ Using ({headless:false}) 99% of the time is occupied by the process sending 20 http requests for pages (and each page requests extra assets) at once and waiting the content to arrive. @Elikill58 \$\endgroup\$
    – Mah Neh
    Apr 20 at 12:13

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.