3
\$\begingroup\$

I created this script to obtain data from graphs online, from monavisaxmldata. I were wondering if there was a way to obtain the data faster. The program which uses this data gets presented in a presentation which takes roughly 5 minutes. But the data already takes 1.5 minutes to load. Which is quiet a bite in such a short time period.

Is the WebClient() optimized in speed or should I approach the handling of the information differently?

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using System.Net;
using System.Web;

public class PVPanels_Actueel : MonoBehaviour 
{
    private static WebClient myWebClient = new WebClient();
    private List<KeyValuePair<string, List<MonavisaXmlData>>> DataList = new List<KeyValuePair<string, List<MonavisaXmlData>>>();
    private List<MonavisaGraphData> graphReadyData = new List<MonavisaGraphData>();
    public event System.Action<List<MonavisaGraphData>> PVPanelsActDataParsed = delegate { };

private void Start()
{
    myWebClient.DownloadStringCompleted += onDownloadComplete;
}
public static void StartDownload(int day, int month, int year)
{
    string dateAndTime = string.Format("{0}-{1}-{2}+23:00", day, month, year);
    MonavisaRequestForm myRequest = new MonavisaRequestForm
        (
            "foo",
            "bar",
            string.Format("http://www.monavisa.info/CreateGraphData?graphs=1&graph[0]=1226&todate={0}&period=1&step=0&b_id=194&inter=1&other_graph=true", dateAndTime),
            ref myWebClient
            );
    MonavisaFetch.instance.getData(ref myRequest);
}

private void onDownloadComplete(object sender, DownloadStringCompletedEventArgs e)
{
    DataList.Clear();
    DataList = MonavisaParse.Parse(e.Result);
    graphReadyData.Clear();

    float totalKwh = 0;
    float totalMw2 = 0;
    string latestDate = "";
    for (int i = 0; i < DataList[1].Value.Count; i++)
    {
        float f0 = float.Parse(DataList[0].Value[i].value);
        float f1 = float.Parse(DataList[1].Value[i].value);
        float f2 = float.Parse(DataList[2].Value[i].value);
        float f3 = float.Parse(DataList[3].Value[i].value);

        if (f0 > 0)
        {
            totalMw2 += f0;
            latestDate = DataList[0].Value[i].date;
        }

        if (f1 > 0 || f2 > 0 || f3 > 0)
        {
            totalKwh += f1 + f2 + f3;
        }
    }

    graphReadyData.Add(new MonavisaGraphData { date = latestDate, value = totalMw2 });
    graphReadyData.Add(new MonavisaGraphData { date = latestDate, value = totalKwh });
    PVPanelsActDataParsed.Invoke(graphReadyData);
}
}

The data is stored on a dutch site. you can go to this link use the log in info DWA-demo and password = welkom (this is a sample login given by the site itself) to see how the data structure is stored. What I currently request is a bit of a different graph, which is stored in a XML file. This file contains the values

Period where:

HOUR = 0;  
DAY = 1;  
WEEK = 2;  
MONTH = 3;  
TWO_MONTHS = 4;  
THREE_MONTHS = 5;  
SIX_MONTHS = 6;  
YEAR = 7;  
FIVE_YEAR = 8;  

and a step where

HOUR = 0;  
DAY = 1;  
WEEK = 2;  
MONTH = 3;  
TWO_MONTHS = 4;  
THREE_MONTHS = 5;  
SIX_MONTHS = 6;  
YEAR = 7;  
FIVE_YEAR = 8;

And currently I am listing graph1&graph[0] which is 1 day per hour of dateAndTime, which today and the last round hour.

string.Format("http://www.monavisa.info/CreateGraphData?graphs=1&graph[0]=1226&todate={0}&period=1&step=0&b_id=194&inter=1&other_graph=true", dateAndTime),

This situation should be repeatable by putting a XML on a server/site where you first have to login, and then grab the data from the XML through requests.

\$\endgroup\$
  • \$\begingroup\$ Is there an English version of monavisa.info? I'd like to poke around a bit and see if they have an API, raw data interface, etc. \$\endgroup\$ – Barry Carter Oct 29 '14 at 17:28
  • \$\begingroup\$ Could you make multiple parallel requests to the site to get all the data at once? \$\endgroup\$ – Barry Carter Oct 29 '14 at 18:44
  • \$\begingroup\$ I meant making multiple connections. Not sure how to do it in Java (you'd need a client that starts the download process in the background, but returns immediately), but GNU parallel and curl could do it from the command-line. However, monavisa.info may consider this unfriendly and/or disallow parallel requests from the same IP address. \$\endgroup\$ – Barry Carter Oct 29 '14 at 18:59
  • \$\begingroup\$ Well, you should ask them first, but obtaining data in parallel doesn't make it any less secure. I also notice you're using http, not https, which is unusual for confidential data. \$\endgroup\$ – Barry Carter Oct 29 '14 at 20:11
  • \$\begingroup\$ @BarryCarter Nice catch, thats a mistake I did not see. \$\endgroup\$ – MX D Oct 29 '14 at 22:22
2
\$\begingroup\$

It looks fast enough to me, nothing screams "major slowdown" in your code, so I would recommend putting a profiler on your code to spot any remaining hotpoints. After that, it's a case of damage control to try to reduce the impact of the data transfer time. As such, consider starting the download as early as possible.

On other matters:

Use var when the right hand side of your declaration makes the type obvious:

string dateAndTime = string.Format("{0}-{1}-{2}+23:00", day, month, year);

should be:

var dateAndTime = string.Format("{0}-{1}-{2}+23:00", day, month, year);

The reason for doing this is that should you change the type of the variable, you only have to change it in one place.

I'm also seeing a few magic strings and numbers that it might be wise to extract as constants or readonly instance variables.

I also don't like your short variable names for your float datapoints. Firstly, if there's nothing that distinguishes them other than the order in which you read them, store them as an array, although I suspect this is not the case, because f0 is given special treatment in the following code.

Secondly, to a maintenance programmer f1 means nothing, unless you're referring to a high-speed motor race. Call them dataPoints if you must, but f1 is extremely unhelpful.

I also note a couple of worrisome points about your implementation:

You don't seem to handle any potential errors at all. What if you can't make a connection, what if the connection drops half way through? Looking at your code literally nothing would happen.

You also only clear your graphReadyData variable when the download completes. If the download fails, this variable will contain the previous correctly-downloaded data. This has the potential to cause errors if not for yourself then for a maintenance programmer, so I would recommend clearing it when you begin the download.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.