# Web scraper for Yell

I have created a scraper for yell.com in vba. The scraper is efficient enough to pull data from that site, whatever the search parameter is. If any link from that site is given to my parser, it is able to scrape the whole records irrespective of how many pages it has spread across. There is no "a" tag for the first page in pagination option for this reason it was previously scraping all the records except for the first page. However, I've fixed that issue and now it is working flawlessly pulling all the records available there. I tried to make it accurate yet there are always rooms for improvement.

Sub Yell_parser()
Dim http As New XMLHTTP60
Dim html As New HTMLDocument, html2 As New HTMLDocument
Dim page As Object, newlink As String
Dim I As Long, x As Long

With http
.Open "GET", "https://www.yell.com/ucs/UcsSearchAction.do?keywords=coffee&location=United+Kingdom&scrambleSeed=1370600159", False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
.send
html.body.innerHTML = .responseText
End With

Set page = html.getElementsByClassName("row pagination")(0).getElementsByTagName("a")

' First page first, selected already, 'row pagination' doesn't have 'a' for it
GetPageData x, html

For I = 0 To page.Length - 2
With http
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
.send
html2.body.innerHTML = .responseText
End With

' Next pages start from here

GetPageData x, html2
Next I
End Sub

Sub GetPageData(ByRef x, ByRef html As HTMLDocument)
Dim post As HTMLHtmlElement
x = x + 1
If .Length Then Cells(x + 1, 1) = .item(0).innerText
End With
If .Length > 1 Then Cells(x + 1, 2) = .item(1).innerText
End With
If .Length > 2 Then Cells(x + 1, 3) = .item(2).innerText
End With
If .Length > 3 Then Cells(x + 1, 4) = .item(3).innerText
End With
If .Length > 1 Then Cells(x + 1, 5) = .item(1).innerText
End With
Next post
End Sub


• avoid code duplication - for instance, you have the User-Agent string specified twice - extract it as a constant and re-use. GetPageData also has duplicated code
• some of your locators are layout-oriented which makes them less reliable and less readable - Bootstrap classes like col-lg-12 or col-md-11 have a layout/design meaning and have a high probability of change. row businessCapsule--title can become businessCapsule--title; col-sm-10 col-md-11 col-lg-12 businessCapsule--address would become businessCapsule--address.
• @SMth80 right, leaving a single class name businessCapsule--address would do the trick. Thanks. – alecxe Jul 26 '17 at 16:08