2
\$\begingroup\$

I have written program which following purpose

  1. Traverse directory in multi threading environment First determine all top level directory and store into vector which have following result:
  • /home/dir1
  • /home/dir2
  • ... more than 1000
  1. create number of database
  2. create number of threads
  3. assign each thread to one directory
  4. Traverse and determine size of directory
  5. There is some issue regarding perforce but I feel code also need to review any better suggestion
#include<boost/tokenizer.hpp>
#include<boost/asio.hpp>
#include <boost/bind/bind.hpp>
#include "scan_dir.h"  //local file
using namespace std::chrono;
/*
 * Process directory fucntion
 * Input : Project path, maxdepth
 */
void process_dir(const std::string &proj, uint64_t &count, std::vector<std::string> &dirs) {
    std::cout<<"Creating Directory"<<std::endl;
    //dirs  = Util::get_top_dir_depth(proj, 0);
    dirs  = Util::traverse_dir(proj, 1);
    count = dirs.size();
}

int main(int argc, char *argv[]) {
    po::options_description desc("DiskAnalyzer Tool");
    po::variables_map vm;
    std::string user, proj;
    uint64_t f_size, maxdepth=0, dir_size=0;
    bool show_dir;
Dirs d;
    
    desc.add_options()
        ("help,h", "DiskAnalyzer option")
        ("proj,p", po::value<string>(),"provide directory path which you would like to search data")
        ("user,u", po::value<string>(), "display file which is associated/Owner with user")
        ("dirsize,ds", po::value<uint64_t>()->default_value(1000000), "display dir which dir_size>=size by default 1000000 Byte:1MB")
        ("showdir,sh", po::value<bool>()->default_value(false), "show only dir which is associated with user")
        ("maxdepth", po::value<uint64_t>()->default_value(5), "show only dir which is associated with user")
        ("filesize,fs", po::value<uint64_t>()->default_value(10000), "display file which file_size>=size by default 10000 Byte:10KB");
    
    try {
        po::store(po::parse_command_line(argc, argv, desc), vm);
        po::notify(vm);
    }catch(const std::exception &err) {
        std::cerr<<err.what()<<std::endl;
        std::cout<<desc<<std::endl;
    }
    catch(...) {
        std::cout<<"Unkown exception"<<std::endl;
    }
    if(vm.count("help")) {
        std::cout<<"scan -p <proj_name> -u <user_name> -maxdepth <maxdepth> -fs <file_size> -d <debug>\n\n";
        std::cout<<desc<<std::endl;
        return 1;
    }
    if(vm.count("user")){
        user = vm["user"].as<string>();
    }
    if(vm.count("proj")){
        proj = vm["proj"].as<string>();
    }
    if(vm.count("filesize")){
        f_size = vm["filesize"].as<uint64_t>();
    }
    if(vm.count("showdir")) {
        show_dir = vm["showdir"].as<bool>();
    }
    if(vm.count("dirsize")) {
        dir_size = vm["dirsize"].as<uint64_t>();
    }
    if(vm.count("maxdepth")){
        maxdepth = vm["maxdepth"].as<uint64_t>();
    }
    
    if(show_dir) {
        d.scan_dir_name(proj, user, dir_size, maxdepth);
        return 0;
    } else {
        uint64_t count = 0;
        std::vector<std::string> dir;
        process_dir(proj, count, dir);
        std::cout<<"createing database["<<proj<<"   "<<count<<"  ]"<<std::endl;
        std::string db_name = Command::basename(proj);
        DataBase db[count];
        for (uint64_t i = 0; i<count; i++){
            db[i].set_db_name("DiskAnalyzer_"+ std::to_string(i)+"_" +db_name);
            if(!db[i].prepare_db()){
                std::cerr<<"[Error] DataBase operation failed"<<std::endl;
                return 1;
            }
        }
        std::size_t max_thread = dir.size() > 1000 ? 1000 : dir.size();
        //max_thread = 10;
        std::cout<<dir.size()<<std::endl;
        //contain directory information
        while(dir.size()){
            std::size_t dir_traverse = 0, db_count = 0;
            boost::asio::io_service io_service;
            boost::asio::io_service::work work(io_service);
            boost::thread_group threads;
            for (std::size_t i = 0; i < max_thread; ++i)
                threads.create_thread(boost::bind(&boost::asio::io_service::run, &io_service));
            
            for(auto it = dir.begin(); it != dir.end() && dir_traverse <max_thread; ++it){
                if(db_count>=count)
                    db_count = 0;
                try {
                    //this function determine determine size of directory. I had expectation
                    // each directory will go each thread
                    io_service.post(boost::bind(&Dirs::scan_dir, boost::ref(d), *it, db[db_count], user));
                } catch(...) {
                    std::cerr<<"got error"<<std::endl;
                    continue;
                }
                dir_traverse++;
                //dir_traverse = dir_traverse + max_thread;
                db_count++;
                //boost::this_thread::sleep(boost::posix_time::seconds(1));
            }
            io_service.stop();
            threads.join_all();
            dir.erase(dir.begin(), dir.begin()+dir_traverse);
            std::cout<<" [Remaining Processing dir cout  "<<dir.size()<<std::endl;
        }
        return 0;
    }
    std::cout<<desc<<std::endl;
    return 0;
}

\$\endgroup\$
2
  • 2
    \$\begingroup\$ Does multi-threading help. Intuition seems to suggest the bottleneck here is disk IO. \$\endgroup\$ Mar 18, 2021 at 9:11
  • 1
    \$\begingroup\$ Console I/O is likely to be a pretty serious bottleneck as well, especially with using std::endl everywhere. :-( \$\endgroup\$ Mar 18, 2021 at 17:17

1 Answer 1

2
\$\begingroup\$

Use of multithreading

Multithreading is normally used when you have a lot of work to do on the CPU, and spreading it out over multiple cores speeds up your program. However, here you don't do much work at all, just traversing the directories. This means that most of the time, your program will be I/O-bound, and throwing more threads at the problem doesn't make it go any faster; you are just waiting for the disk to return data from the directory structures in the filesystem.

A few threads might still be useful, because if the operating system gets multiple I/O requests, it can do some more intelligent scheduling of I/O operations. This is mostly helpful for hard disks, where a lot of the time is spent moving the read heads. For solid state disks this is less of an issue, although depending on the model it might still have some benefits.

Starting threads itself has overhead, so for sure you don't want to start a thousand threads. Try varying max_threads and benchmark the results. I expect you get the maximum performance with just a few threads at most.

Keep threads alive

Again, starting threads is not free. You are starting max_threads threads, each processing a directory, and then you wait for all threads to finish, before you start a new group of max_threads threads. Apart from the repeated overhead of starting threads, you also limit the processing speed to that of the slowest thread in each group.

The normal approach is to start a small number of threads, and have each thread continuously pick some work until there is no more work left to do.

Use of Boost

Boost can be useful, but using Boost::ASIO for this problem is overkill in my opinion. You can easily start a few threads yourself using C++11's standard thread functions.

Avoid std::endl

As mentioned in the comments, avoid using std::endl, and use '\n' instead. The former is equivalent to the latter, but also forces a flush of the output stream, which can negatively impact performance.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.