Get a stock quote from a web page using F#

Question

I want to extract a price for a single index fund, the price of which is available on dynamic web pages.

Being new to this, my original idea was to download the single page of static HTML and get the price from that. How hard could it be? Inspired by an article on F# for fun and profit I imagined it would be easy.

Of course, the data isn't in the static page, as anyone who is familiar with web site development knows. Just look at the huge amount of content on w3schools that describes how a modern page is built. Much of it is dynamic and runs client code in the browser, like javascript.

Given that a simple HTTP request doesn't work, I opted to use a web browser control.

I found inspiration from these stackoverflow questions: How to use WebBrowser control DocumentCompleted event in C#?, View Generated Source (After AJAX/JavaScript) in C# and C# how to wait for a webpage to finish loading before continuing.

open System
open System.Windows.Forms
open System.Threading
open System.Text.RegularExpressions

let wb = new WebBrowser()
wb.ScrollBarsEnabled <- false
wb.ScriptErrorsSuppressed <- true

let rec GetPrice (wb:WebBrowser) = 
    Application.DoEvents()  // Get the browser to do work
//    The following doesn't work
//    while (wb.ReadyState <> WebBrowserReadyState.Complete) do Application.DoEvents()

    let m = Regex.Match(wb.Document.Body.InnerText, """£(?<VIGSCA>\d{3}\.\d{4})[^\d]""")
    if m.Success 
    then 
        // by definition, a sucessful match is a float32 (Single) so this parse will work
        Single.Parse(m.Groups.["VIGSCA"].Value) 
    else
        // it takes several seconds to get a result, let's give the CPU a rest before we try again
        Thread.Sleep 100 
        GetPrice wb

let stockURL = Uri "https://www.vanguard.co.uk/uk/portal/detail/mf/overview?portId=9158&assetCode=EQUITY##overview" 
wb.Navigate(stockURL)
while (wb.ReadyState <> WebBrowserReadyState.Complete) do Application.DoEvents()

let price = GetPrice wb
printfn "Price is %A" price

wb.Dispose()

Can you suggest improvements? For example:

1. More idiomatic f#

2. Use an event instead of a loop. This code doesn't work

   let downloadComplete = Async.AwaitEvent (wb.DocumentCompleted) |> Async.Ignore
   Async.RunSynchronously downloadComplete
   

3. An alternative way to download the price that doesn't use a web browser control

These related questions, Use Webbrowser control to get and post data?, How to use System.Windows.Forms.WebBrowser in a web app?, are interesting but don't help.

Answer 1

I have been doing my own research and here is my answer to point 3.

An alternative way to download the price that doesn't use a web browser control

namespace ExcelVIGSCA

module getPrice = 

    open ExcelDna.Integration
    open FSharp.Data

    let saveVIGSCAjson () = 
        // Get a copy of the data shape, and save it in a file so that the compiler can type check it.
        // I used http://www.telerik.com/fiddler to find the correct data.
        let url = @"https://www.vanguard.co.uk/uk/portal/GetMFJsonForPortId.js?portId=9158" 
        let response = Http.RequestString(url)
        let json = (response.Split('=')).[1]
        System.IO.File.WriteAllText(__SOURCE_DIRECTORY__ + "\VIGSCA.json",json) |> ignore

    // Clearly, VIGSCA.json has to exist at compile time, so saveVIGSCAjson() will have to be have been run during development.
    // This makes the program very fagile.
    type VIGSCAData = JsonProvider<"VIGSCA.json">

    // information about attributes & data types: http://exceldna.codeplex.com/wikipage?title=Reference&referringTitle=Documentation 
    [<ExcelFunction(Description = "Price for VIGSCA")>]
    // Excel-DNA needs concrete types for input and output to work: it can't be generic.
    let VIGSCAprice () = 
        saveVIGSCAjson() // save the current data.
        let VIGSCA = VIGSCAData.GetSample()
        // The compiler will use the file it has at **compile time** to build up the type data.  This WILL lead to a runtime 
        // exception if the type has changed.
        VIGSCA.FundData.Fund.[0].NavPrice.Price.[0].Price

After some research, including this illuminating article, I realised just how complicated web scraping is. Extracting some data from a static HTML page would have been easy. But we are dealing with a dynamic web page built on ssl/https connection.

I know that this code is incredibly brittle. I certainly wouldn't want to give it to anyone else to use. Consider it a learning exercise, or a hint at what can be done. To get the web page at all, you have to have previously been to Vanguard's web site and selected the "Individual Investor" option, which presumably saves a cookie to that effect.

My research suggests that there are 2 broad categories of getting data from a dynamic web page.

1. Use a web browser programmatically (as in the question) and then make sure that the whole page, or at least the bit you are interested in, has downloaded before you extract the data. The use of a regular expressions seems OK for this. This is "simple" in the sense that you know what the final page looks like so you know what data to extract.

2. Use a protocol analyser (like fiddler) and inspect the web page source to work out exactly what request has the data you want. Then use a tool, like the wonderful FSharp.Data to extract it. This is definitely more work but you have a better understanding of how the web site is structured so that, hopefully, it is easier to maintain. It is definitely more focused.

Whatever approach is chosen, knowledge of how these dynamic pages are built (including some familiarity with javascript) is necessary. Thank goodness for sites like w3schools