My algorithm scrapes an infinite-scroll page but it takes too long. It scrolls three times but I'm wondering if there is a way to do a ScrollBottom()
so no need of repeated code.
Regarding the site from the example: Scroll is done by jQuery ScrollExtend goo.gl/Sq4vVx triggered when the users scroll beyond a particular tag. When that happens a particular class is added into the tag and removed after the pagination is done.
I think there's room for improvement code and performance wise.
"use strict";
var Xray = require('x-ray');
var phantom = require('x-ray-phantom');
var phantom_opts = {
webSecurity: false,
images: false,
weak: false
};
var x = Xray().driver(phantom(phantom_opts, function (nightmare, done) {
done
.useragent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36")
.goto(nightmare.request.req.url)
.scrollTo(4000, 0)
.wait()
.scrollTo(8000, 0)
.wait()
.scrollTo(12000, 0)
.wait()
}));
x('https://www.compraonline.grupoeroski.com/es/supermercado/2059698-Alimentos-Frescos/2059746-Carnes-y-aves/2059753-Pollo/', '.product_list li',
[{
name: '.description_1',
unitPrice: '.description_2',
image: '.image_line img@src',
price: '.product_price_cont p',
url: '.image_line a@href',
volumen: '.description_1',
medida: '.description_1'
}])(function (err, products) {
if (err) console.log(err);
console.log(products.length);
process.exit(0);
});
scrollTo(4000,0)
will only go as far as the bottom, then so will 8000 and 12000 all three will have been called and the page might reload content once in this time. Maybe twice if you are lucky. This approach will never work. \$\endgroup\$