4
\$\begingroup\$

I'm currently modifying a research project (massive test and benchmark of diverse conjectures to try and eventually solve the SSW conjecture) so that it can run on a big HPC architecture, which does not handle most of the 'classic' multi-process/multi-thread methods (like fork-based in C, python multiprocessing module, or multi-threading libraries).

So I have to go for an OpenMPI based implementation.

I need advice on the manager/worker pattern, and on the 'good and legit' way to use open MPI, good practices included. I'm also taking advices on coding style and the likes, but it's not the main issue here (academic lab coding style :-) ).

The rest of the code (the actual jobs to perform) are working fine, benchmarked, tested, and ... confidential. So here is a toy case, where a job will consist in printing an integer, sleeping for a short time span, and returning no piece of information to the manager. Error managing for workers is also handled at worker level, so we don't need to really care about it here.

The principle is the following: I have a 'manager' which sends jobs to perform to a number of workers (a few thousands in my case). Each worker is totally independent, and does not need to return any information to its manager. Workers errors are handled in the workers code.

Here is my toy program:

/* file: test_mpi.c */

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdbool.h>

#define KEEP_WORKING 0
#define NO_MORE_JOBS 1
#define READY_TO_WORK 2
#define WORKER_NOT_AVAILABLE -1
#define MANAGER_RANK 0
#define NB_JOBS 8


int get_rank_from_ready_worker() {
    MPI_Status status;
    int worker_useless_msg;
    MPI_Recv(&worker_useless_msg,
             1,
             MPI_INT,
             MPI_ANY_SOURCE,
             MPI_ANY_TAG,
             MPI_COMM_WORLD,
             &status);

    if (status.MPI_TAG == READY_TO_WORK) {
        return status.MPI_SOURCE;
    } else {
        return WORKER_NOT_AVAILABLE;
    }
}


void send_job_to_worker(int job, int worker_rank) {
    MPI_Send(&job,
             1,
             MPI_INT,
             worker_rank,
             KEEP_WORKING,
             MPI_COMM_WORLD);
}


void signal_to_manager_that_ready() {
    int empty_msg = 0;
    MPI_Send(&empty_msg,
             1,
             MPI_INT,
             MANAGER_RANK,
             READY_TO_WORK,
             MPI_COMM_WORLD);
}


void do_job(int job, int my_rank) {
    unsigned int sleep_time = 1 + ((rand() + job) % 4);
    sleep(sleep_time);
    printf("[%d] performed job %d\n",
           my_rank, job);
}


bool receive_job_from_manager(int my_rank) {
    int job;
    MPI_Status status;
    MPI_Recv(&job,
             1,
             MPI_INT,
             MANAGER_RANK,
             MPI_ANY_TAG,
             MPI_COMM_WORLD,
             &status);

    if (status.MPI_TAG == KEEP_WORKING) { /* received a job to perform */
        do_job(job, my_rank);
        /* tell manager that it's ready to perform a new job */
        signal_to_manager_that_ready();
        return true;

    } else if (status.MPI_TAG == NO_MORE_JOBS) { /* work is over ! */
        return false;
    } else { /* this should not happen */
        return false;
    }
}

void terminate_worker(int worker_rank) {
    int empty_job = 0;
    MPI_Send(&empty_job,
             1,
             MPI_INT,
             worker_rank,
             NO_MORE_JOBS,
             MPI_COMM_WORLD);
}


int main(int argc, char** argv) {
    /* init MPI environment */
    MPI_Init(&argc, &argv);

    /* Find out rank, size */
    int world_rank, world_size;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    /* Check that we have at least one manager & one worker */
    if (world_size < 2) {
        fprintf(stderr, "World size must be greater than 1 for %s\n", argv[0]);
        MPI_Abort(MPI_COMM_WORLD, 1);
    }

    if (world_rank == MANAGER_RANK) { /* manager part */
        /* create 'jobs'*/
        int jobs[NB_JOBS];
        for (int i = 0; i < NB_JOBS; i++) {
            jobs[i] = i;
        }

        int job_to_send = 0, worker_rank;
        do { /* manager main loop: sending jobs to ready workers */
            worker_rank = get_rank_from_ready_worker();
            if (worker_rank != WORKER_NOT_AVAILABLE) {
                send_job_to_worker(jobs[job_to_send], worker_rank);
                job_to_send ++;
            }
        } while (job_to_send < NB_JOBS);

        /* All jobs done -> terminating workers */
        for (int nb_terminated = 0; nb_terminated < world_size-1; nb_terminated++) {
            terminate_worker(get_rank_from_ready_worker());
        }

    } else { /* worker part */
        /* signal to manager that it's ready to work */
        signal_to_manager_that_ready();

        /* start working loop */
        bool keep_working;
        do {
            keep_working = receive_job_from_manager(world_rank);
        } while (keep_working);
    }

    MPI_Barrier(MPI_COMM_WORLD);
    MPI_Finalize();
    return 0;
}

To compile and run (on unix):

mpicc test_mpi.c -o test_mpi -Wall -Wextra
mpirun -n 4 ./test_mpi

Note that to compile and run this code, Open MPI needs to be installed on the machine.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Please do not update the code in your question to incorporate feedback from answers, doing so goes against the Question + Answer style of Code Review. This is not a forum where you should keep the most updated version in your question. Please see what you may and may not do after receiving answers. \$\endgroup\$
    – Mast
    Nov 26 '19 at 12:32
2
\$\begingroup\$

Small review

Potential bug

rand() + job can overflow leading to undefined behavior. Possible then that 1 + ((rand() + job) % 4) is a negative number and code sleeps for a long time.

e.g. rand() + job --> INT_MAX + 2 --> -INT_MAX and (-INT_MAX) % 4 --> -3. -3 + 1 --> unsigned int sleep_time is UINT_MAX -1. Sleep maybe 136 year, longer than old Rip.

Avoid signed integer overflow. Consider unsigned math.

// unsigned int sleep_time = 1 + ((rand() + job) % 4);
unsigned int sleep_time = 1 + ((0u + rand() + job) % 4);
sleep(sleep_time);
\$\endgroup\$
1
  • 2
    \$\begingroup\$ Thanks for spotting it :-) Even if it is not going to be part of the code that will run on the big machines, I always like to see how easy it is to create potentially catastrophic bugs in C (and this one would go undetected for a long long time, with such a low probability of being triggered). I'd give a second +1 for the reference to rip but unfortunately I can't. \$\endgroup\$
    – m.raynal
    Nov 26 '19 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.