I'm writing my own HTTP client, kinda like cURL. (I already know I'm reinventing the wheel, this is more or less getting an inside look of HTTP 1.x before 2 becomes a thing.)

So far pages download perfectly fine using OS provided socket libs. (Linux/Mac) So what the problem/question is, how can I better my client? One of the main problems with it, is that it's slow. Like 1.00 - 1.30 seconds slow. I know this may not sounds like much, but the 1 second is mainly because of the timeout of the recv. Or maybe I cant do much more except do what other HTTP clients do and cache data.

char   buf[CHUNK_SIZE+1];
    //now it is time to receive the page
    memset(buf, 0, sizeof(buf));
    std::string htmlcontent;

    struct timeval tv;

    tv.tv_sec = 1;
    setsockopt(socket, SOL_SOCKET, SO_RCVTIMEO,(struct timeval *)&tv,sizeof(struct timeval));

        tmpres = recv(socket, buf, CHUNK_SIZE, 0);

        if(tmpres < 1)

        htmlcontent += buf;
        memset(buf, 0, tmpres);

So as you can see, the recv timeout is 1 second. I'm hoping to maybe manipulate this in someway so, it's instead 500 milliseconds for timeout or perhaps look at another method in general when dealing with chunk data from a HTTP server. And of course I googled some topics, but a lot of them had pretty basic examples. Mostly what I have already written.


1 Answer 1


Since it looks like you don't wait for a specific length of data I assume that you have no idea how to get the length of the data up-front. Since you are willing to learn how HTTP/1.x works I recommend you to read the specifications and watch out for something like "Content-length" and "Transfer-Encoding: chunked". Once you've understood why these are important and how to correctly deal with them you can write your code so that it does not timeout.

  • \$\begingroup\$ Yeah I know how those work, I guest I should say how can I can correctly deal with chunk data that the timeout of recv isn't affected. \$\endgroup\$
    – ajm113
    Commented Jan 20, 2015 at 16:31
  • \$\begingroup\$ If you know how these work then why you don't read only a much bytes as required by content-length or chunk size? Of course you will get a timeout if you try to read more since no more available. \$\endgroup\$ Commented Jan 20, 2015 at 17:03
  • \$\begingroup\$ Yes, I made made the adjustments now that you mention that. But is their anyway to shorten the timeout less then a second? \$\endgroup\$
    – ajm113
    Commented Jan 20, 2015 at 17:29
  • \$\begingroup\$ You can use non-blocking sockets so you have no timeout and you can use select or poll to detect when data are available. Basic socket programming and has nothing to do with HTTP. You still have to know how much data you want to read because you cannot be sure that all data arrive immediately. \$\endgroup\$ Commented Jan 20, 2015 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.