I am facing performance issues while reading a large text file from the network. The same file is read locally without any performance problems:

int file_no_lines(char *file_path)
{
    FILE *fp = NULL;
    char line_read[200];
    int linecount = 0;
    fp = fopen(file_path, "r");
    while(1) 
    {
     if (fgets(line_read, 160 ,fp) == NULL)
     {
        if(feof(fp))
        {
           fclose(fp);
           return linecount;
        }
        else
        {
            printf("Bad File Termination");
            fclose(fp);
            return -1 ;
        }
     }
     else
     {
        linecount++;
     }
    }
    fclose(fp);
    return linecount;
}
int file_read_lines(char *file_path)
{
    FILE *fp = NULL;
    char line_read[200];
    fp = fopen(file_path, "r");
    while(1) 
    {
     if (fgets(line_read, 160 ,fp) == NULL)
     {
        if(feof(fp))
        {
           fclose(fp);
           return -1;
        }
        else
        {
            printf("Bad File Termination");
            fclose(fp);
            return -1 ;
        }
     }
     else
     {
        //update structure
     }
    }
    fclose(fp);
    return 1;
}
int _tmain(int argc, _TCHAR* argv[])
{

    int no_lines = file_no_lines("C:/sample.txt"); //count no. of lines
    //allocate structure for no. of lines
    file_read_lines("C:/sample.txt"); // fill allocated array
    return 0;
}

Steps:

  1. Count the no. of valid lines using file_no_lines
  2. Allocate for no. of lines
  3. Update structure, file_read_lines
  • 2
    When you say "from network", what does that mean? From a URL (like HTTP/FTP, etc.), or from a network file system like SMB or NFS? – rolfl Mar 18 '15 at 15:11
  • It is SMB file system. – K P Mar 19 '15 at 5:45

Reading just 160 bytes at a time could be bad for performance, especially if you are using the older SMB 1 protocol. SMB 2.0 introduced pipelining, which allows the client to issue another request before the previous one has completed. Therefore, with SMB 1, you would have to wait for each request to be finished — including network and disk latency — before making the next read. If the read requests are too small, then performance could suffer badly.

What is a good buffer size? Microsoft's KB 223140 suggests that somewhere between 4 kB and 60 kB could be ideal — in any case, much more than 160 bytes.

  • Thanks but my text doesn't have more that 160 bytes per line. – K P Apr 2 '15 at 9:58
  • 1
    Performance would still benefit from reading larger blocks — especially if the lines are short. You could eliminate entire round trips if the next line is entirely contained in the buffer. – 200_success Apr 2 '15 at 10:03

The loop using while (1) feels cumbersome. The code would be more sensible if the while loop had a meaningful termination condition.

int file_no_lines(char *file_path)
{
    char line_read[BLOCK_SIZE];
    int linecount = 0;
    FILE *fp = fopen(file_path, "r");
    while (fgets(line_read, BLOCK_SIZE, fp)) 
    {
       linecount++;
    }

    if (!feof(fp))
    {
        printf("Bad File Termination");
        fclose(fp);
        return -1 ;
    } else {
        fclose(fp);
        return linecount;
    }
}

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.