VC++ Fast Http Download

Question

I am a C# developer, dont know much about vc++ or c++, never used it, for some reasons i have decided to use a c++ dll in my app for downloading content from the web.

I dont want to use WebClient.

I want it to download html content by providing Url of the resource, the dll will return the string response.

this is the code i have till now, source

#include <iostream>
#include <string>

#include <stdlib.h>
#include <winsock.h>//dont forget to add wsock32.lib to linker dependencies

using namespace std;

#define BUFFERSIZE 1024
void die_with_error(char *errorMessage);
void die_with_wserror(char *errorMessage);

int main(int argc, char *argv[])
{
    string request;
    string response;
    int resp_leng;

    char buffer[BUFFERSIZE];
    struct sockaddr_in serveraddr;
    int sock;

    WSADATA wsaData;
    char *ipaddress = "208.109.181.178";
    int port = 80;

    request+="GET /test.html HTTP/1.0\r\n";
    request+="Host: www.zedwood.com\r\n";
    request+="\r\n";

    //init winsock
    if (WSAStartup(MAKEWORD(2, 0), &wsaData) != 0)
        die_with_wserror("WSAStartup() failed");

    //open socket
    if ((sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)
        die_with_wserror("socket() failed");

    //connect
    memset(&serveraddr, 0, sizeof(serveraddr));
    serveraddr.sin_family      = AF_INET;
    serveraddr.sin_addr.s_addr = inet_addr(ipaddress);
    serveraddr.sin_port        = htons((unsigned short) port);
    if (connect(sock, (struct sockaddr *) &serveraddr, sizeof(serveraddr)) < 0)
        die_with_wserror("connect() failed");

    //send request
    if (send(sock, request.c_str(), request.length(), 0) != request.length())
        die_with_wserror("send() sent a different number of bytes than expected");

    //get response
    response = "";
    resp_leng= BUFFERSIZE;
    while (resp_leng == BUFFERSIZE)
    {
        resp_leng= recv(sock, (char*)&buffer, BUFFERSIZE, 0);
        if (resp_leng>0)
            response+= string(buffer).substr(0,resp_leng);
        //note: download lag is not handled in this code
    }

    //display response
    cout << response << endl;

    //disconnect
    closesocket(sock);

    //cleanup
    WSACleanup();
    return 0;
}

void die_with_error(char *errorMessage)
{
    cerr << errorMessage << endl;
    exit(1);
}

void die_with_wserror(char *errorMessage)
{
    cerr << errorMessage << ": " << WSAGetLastError() << endl;
    exit(1);
}

The instructions with the code says, code will end an http download 'early' if the download is anything slower than instant.

Please suggest better code, or modifications, which can be perfect for use in a Fast Crawler.

Answer 1

The above implementation has little to no chance of working. The HTTP protocol is quite complex, and none of that complexity is contemplated.

My advice is, don't waste time and energy in this. Use a high-level library.

Also, using low-level code will NOT improve the performance of fetching a document via HTTP -- even if your implementation was blazing-fast 100% polished assembly code, you'd still have to wait centuries (from the CPU point of view) for the data to arrive from the network.