I want to extract a price for a single index fund, the price of which is available on dynamic web pages.
Being new to this, my original idea was to download the single page of static HTML and get the price from that. How hard could it be? Inspired by an article on F# for fun and profit I imagined it would be easy.
Of course, the data isn't in the static page, as anyone who is familiar with web site development knows. Just look at the huge amount of content on w3schools that describes how a modern page is built. Much of it is dynamic and runs client code in the browser, like javascript.
Given that a simple HTTP request doesn't work, I opted to use a web browser control.
I found inspiration from these stackoverflow questions: How to use WebBrowser control DocumentCompleted event in C#?, View Generated Source (After AJAX/JavaScript) in C# and C# how to wait for a webpage to finish loading before continuing.
open System
open System.Windows.Forms
open System.Threading
open System.Text.RegularExpressions
let wb = new WebBrowser()
wb.ScrollBarsEnabled <- false
wb.ScriptErrorsSuppressed <- true
let rec GetPrice (wb:WebBrowser) =
Application.DoEvents() // Get the browser to do work
// The following doesn't work
// while (wb.ReadyState <> WebBrowserReadyState.Complete) do Application.DoEvents()
let m = Regex.Match(wb.Document.Body.InnerText, """£(?<VIGSCA>\d{3}\.\d{4})[^\d]""")
if m.Success
then
// by definition, a sucessful match is a float32 (Single) so this parse will work
Single.Parse(m.Groups.["VIGSCA"].Value)
else
// it takes several seconds to get a result, let's give the CPU a rest before we try again
Thread.Sleep 100
GetPrice wb
let stockURL = Uri "https://www.vanguard.co.uk/uk/portal/detail/mf/overview?portId=9158&assetCode=EQUITY##overview"
wb.Navigate(stockURL)
while (wb.ReadyState <> WebBrowserReadyState.Complete) do Application.DoEvents()
let price = GetPrice wb
printfn "Price is %A" price
wb.Dispose()
Can you suggest improvements? For example:
- More idiomatic f#
Use an event instead of a loop. This code doesn't work
let downloadComplete = Async.AwaitEvent (wb.DocumentCompleted) |> Async.Ignore Async.RunSynchronously downloadComplete
An alternative way to download the price that doesn't use a web browser control
These related questions, Use Webbrowser control to get and post data?, How to use System.Windows.Forms.WebBrowser in a web app?, are interesting but don't help.