I have a C# application and a CakePHP website. These communicate between them using GET and (WebRequest) POST method.

My problem is that it's happening too fast. I have made a diagram that shows some statistics. I retrieve the data from the website using GET method (XML file), such as:


In this XML file we find some information such as a website URL, a random amount of seconds, and some other stuff. The C# application will open the URL in the built-in browser (we used geckofx for that) for, let's say, 30 seconds. After 30 seconds, the C# application will send some info back to the website using (WebRequest) POST method, confirming that the user has viewed the website. Now this is happening over and over again. (Currently we have +160 IP connected.)

The other companies that are in the same business, they do it differently. They download from the website 10 websites, the user views them and then only send back the data.

My problem is I have no idea how that can be done.

My server config : E3-1230 Xeon with 14 GB RAM and 440 GB RAID 1 disk space. CPU and RAM memory are running out.

Here is the C# application code which downloads and uploads the data:

private void LoadUrl() 
            XmlDocument doc1 = new XmlDocument();
            XmlElement root = doc1.DocumentElement;
            XmlNodeList nodes = root.SelectNodes("/response/sites/Site");
            foreach (XmlNode node in nodes)
                string site_id = node["id"].InnerText;
                string address = node["address"].InnerText;
                string surfsec = node["surfsec"].InnerText;
                string Referral = node["referral"].InnerText;

                var result = Convert.ToInt32(surfsec);

                var sec_to_mil = result * 1000;

                label6.Text = "Please wait : ";

                button2.Visible = false;
                button3.Visible = false;

                timeLeft = result;
                HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://www.my-website.com/post_data_comes_here");
                request.Method = "POST";
                request.ContentType = "application/x-www-form-urlencoded";
                string postData = "post_data_here";
                byte[] bytes = Encoding.UTF8.GetBytes(postData);
                request.ContentLength = bytes.Length;

                Stream requestStream = request.GetRequestStream();
                requestStream.Write(bytes, 0, bytes.Length);

                WebResponse response = request.GetResponse();
                Stream stream = response.GetResponseStream();
                StreamReader reader = new StreamReader(stream);

                var results = reader.ReadToEnd();

                timer1.Interval = sec_to_mil;

                geckoWebBrowser1.Navigate(address, GeckoLoadFlags.BypassCache, Referral, null, null);
                 geckoWebBrowser1.JavascriptError += (sender, error) =>
                     GeckoWebBrowser browser = geckoWebBrowser1;
                     string text = "window.alert = function(){};";
                     using (AutoJSContext context = new AutoJSContext(browser.Window.JSContext))
                         string resultjs;

                         context.EvaluateScript(text, (nsISupports)browser.Window.DomWindow, out resultjs);

                textBox1.Text = address;

I will not post the PHP because mainly my problem is with the C# application that need to be rewritten.

I was thinking on a local database, download 10 website in that database, view them and then send that data back to my MySQL server. Is that possible?

  • 1
    \$\begingroup\$ You've already received answers, so you cannot redact the embedded code already posted. \$\endgroup\$
    – Jamal
    Commented Sep 24, 2015 at 14:24
  • \$\begingroup\$ I decided to move the mysql queries into the application, work with the data in there. I spied a little on the competitors applications and I think this is the only reasonable solution for now. Thanks for everyone who join into conversation. I highly appreciate it \$\endgroup\$
    – rgerculy
    Commented Sep 24, 2015 at 14:27

2 Answers 2


Can't help you with your problem (which I don't really understand), but I did notice these issues with your code:

  • result doesn't tell me what it is. Instead of assigning the value of surfsec to a string and then converting that value, do this directly.
  • Why assign sec_to_mil when you only use it once? Just do timer1.Interval = surfsec * 1000;
  • WebResponse is IDisposable, so it should be used within a using block. Same for each of those Streams. Same for StreamReader.
  • You don't do anything with results?

The whole block where you do the request and get the response should move to a separate method. The whole of LoadUrl() should be re-thought and the method renamed, since it's doing a lot of different things, certainly more than just "loading a URL".

  • \$\begingroup\$ result - is a number, eg : 30 - I convert it to milliseconds and then use it. WebResponse - I will look into . results- I don't since I use that code only to send data back. \$\endgroup\$
    – rgerculy
    Commented Sep 24, 2015 at 11:42
  • \$\begingroup\$ Current code loads one website urls, process it, send data back. What I need is Load 10 website urls, process them ( open them in the browser ), send data back ( for all 10 websites ) \$\endgroup\$
    – rgerculy
    Commented Sep 24, 2015 at 11:49
  • \$\begingroup\$ @rgerculy I know what result is, but its name doesn't tell me what it is. WRT "send data back": your code isn't doing this. It does a POST and then gets the response, but nowhere does it send anything back. WRT connecting to ten sites at once: Multithreading a large number of web requests. \$\endgroup\$
    – BCdotWEB
    Commented Sep 24, 2015 at 12:26
  • \$\begingroup\$ In the POST method I send the website and user id to PHP to process it. ( that's my data ). I can add 10 websites to that XML, my problem is how to process them in the application ( view the sites one by one ), and send back all the informations at once. ( 10 website ID's in one POST method for example. ) \$\endgroup\$
    – rgerculy
    Commented Sep 24, 2015 at 12:49

To answer your question "yes it's possible" because you've said your competitors do it. I don't really understand what "it" is so I can't give specific advice. Besides which, code not yet written is off-topic for code review.

On to current code.

  • Your naming is awful. doc1, button1, button2, timer1. You really need to work on that
  • Try to keep to the single responsibility principle - one method should do one thing
  • using statements are your friend - use them for IDisposables

You've also hit on one of my least favourite thing ever:

 sec_to_mil = result * 1000;

The name is bad for several reasons

  1. It sounds more like a method
  2. It shouldn't contain underscores
  3. You've used abbreviations

TimeSpan is all about representing a period of time:

var surfTimeInSeconds = int.Parse(node["surfsec"].InnerText);
var surfTime = TimeSpan.FromSeconds(surfTimeInSeconds);

// surfTime.Ticks
// surfTime.TotalMilliseconds

Isn't that better? They type system is helping you and you won't get confused about whether that int is seconds, minutes, days, milliseconds or number of red socks in your drawer.

Your code is cut short so it's hard to know what else you're doing. Try to concentrate on making your code easy to read. Once you do that, you'll find it's easier to change. Once it's easy to change, you'll find implementing features easier.

  • \$\begingroup\$ Thanks for your help. I will start working on that as soon I get an idea how to move on ... \$\endgroup\$
    – rgerculy
    Commented Sep 24, 2015 at 12:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.