HTTP client similar to cURL

Question

I'm writing my own HTTP client, kinda like cURL. (I already know I'm reinventing the wheel, this is more or less getting an inside look of HTTP 1.x before 2 becomes a thing.)

So far pages download perfectly fine using OS provided socket libs. (Linux/Mac) So what the problem/question is, how can I better my client? One of the main problems with it, is that it's slow. Like 1.00 - 1.30 seconds slow. I know this may not sounds like much, but the 1 second is mainly because of the timeout of the recv. Or maybe I cant do much more except do what other HTTP clients do and cache data.

char   buf[CHUNK_SIZE+1];
    //now it is time to receive the page
    memset(buf, 0, sizeof(buf));
    std::string htmlcontent;

    struct timeval tv;

    tv.tv_sec = 1;
    setsockopt(socket, SOL_SOCKET, SO_RCVTIMEO,(struct timeval *)&tv,sizeof(struct timeval));

    while(true){
        tmpres = recv(socket, buf, CHUNK_SIZE, 0);

        if(tmpres < 1)
        {
            break;
        }

        htmlcontent += buf;
        memset(buf, 0, tmpres);
    }

So as you can see, the recv timeout is 1 second. I'm hoping to maybe manipulate this in someway so, it's instead 500 milliseconds for timeout or perhaps look at another method in general when dealing with chunk data from a HTTP server. And of course I googled some topics, but a lot of them had pretty basic examples. Mostly what I have already written.

Answer 1

Since it looks like you don't wait for a specific length of data I assume that you have no idea how to get the length of the data up-front. Since you are willing to learn how HTTP/1.x works I recommend you to read the specifications and watch out for something like "Content-length" and "Transfer-Encoding: chunked". Once you've understood why these are important and how to correctly deal with them you can write your code so that it does not timeout.