4
\$\begingroup\$

The following code is taken from an archived course. Although my answer to the assignment passes the tests, I'm not sure if it is actually the correct way to write golang code. Primarily because in my solution, there is one extra channel left at the end when the master node is done distributing map/reduce tasks. If it helps, the link to my code is https://github.com/tushar00jain/distribute/blob/master/mapreduce/master.go#L29

Although the code should be self explanatory, other definitions are in the same folder.

I have 2 solutions and I've provided one of them that uses channels below.

The submitJob function is responsible for making the map/reduce RPC call on the workers. If I interchange the lines marked as replace, the code does not work anymore, because there is no channel waiting for mr.registerChannel after the worker from the last job becomes available.

mr.registerChannel basically registers a worker (by sending the ip address) and indicates that it is available, so that a task waiting for this channel can execute. Once the task is done, it makes this channel available again. One thing to be noted is that, I other code will be sending on the mr.registerChannel as well (because there needs to be a worker to begin with).

The code after that just launches goroutines for all the map tasks, which wait for a worker to become available. Then I place blocking code with the done channel so that reduce only starts after all the maps are done. The code for reduce is also similar.

package mapreduce

import "container/list"
import "fmt"

func (mr *MapReduce) RunMaster() *list.List {
    // Your code here
    mr.Workers = make(map[string]*WorkerInfo)
    done := make(chan bool)

    var submitJob func(args *DoJobArgs)
    submitJob = func(args *DoJobArgs) {
        worker := <-mr.registerChannel
        if _, ok := mr.Workers[worker]; !ok {
            mr.Workers[worker] = &WorkerInfo{worker}
        }

        var res DoJobReply
        ok := call(worker, "Worker.DoJob", args, &res)
        if ok {
            done <- true                     // replace
            mr.registerChannel <- worker     // replace
        } else {
            fmt.Printf("DoJob: RPC %s Job error\n", worker)
            delete(mr.Workers, worker)
            // recursively keep trying with a different worker
            submitJob(args)
        }
    }

    go func() {
        for i := 0; i < mr.nMap; i++ {
            args := &DoJobArgs{mr.file, Map, i, mr.nReduce}
            go func(args *DoJobArgs) {
                submitJob(args)
            }(args)
        }
    }()

    for i := 0; i < mr.nMap; i++ {
        <-done
    }

    go func() {
        for i := 0; i < mr.nReduce; i++ {
            args := &DoJobArgs{mr.file, Reduce, i, mr.nMap}
            go func(args *DoJobArgs) {
                submitJob(args)
            }(args)         
        }
    }()

    for i := 0; i < mr.nReduce; i++ {
        <-done
    }

    close(done)
    return mr.KillWorkers()
}
\$\endgroup\$

closed as off-topic by hoffmale, Dannnno, yuri, t3chb0t, hjpotter92 Aug 17 '18 at 11:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Code not implemented or not working as intended: Code Review is a community where programmers peer-review your working code to address issues such as security, maintainability, performance, and scalability. We require that the code be working correctly, to the best of the author's knowledge, before proceeding with a review." – hoffmale, Dannnno, yuri, hjpotter92
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    \$\begingroup\$ The code in the question cannot be meaningfully reviewed without additional clarification, as submitJob doesn't seem to be working correctly (it tries to read from mr.registerChannel before anything gets written to it, resulting in a deadlock). \$\endgroup\$ – hoffmale Aug 14 '18 at 12:12

Browse other questions tagged or ask your own question.