Skip to main content
fixed bug
Source Link
ChrisWue
  • 20.4k
  • 4
  • 42
  • 107

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (magn,(i-1)*slice_size, slice_size), (), ()) for i in ncp]xrange(ncp)]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

You will have to change your rxx_func to accept a start index and a count which defines for how many items it is responsible:

def rxx_func(amp, start_index, count):
    N = len(amp)
    rxx = [0]*count

    for m in xrange(start_index, start_index + count - 1):
        for n in xrange(N-m):
            rxx[m]+=amp[n]*amp[n+m]
    return rxx

I'm sure there is plenty which can be optimized in the above.

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (magn,(i-1)*slice_size, slice_size), (), ()) for i in ncp]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

You will have to change your rxx_func to accept a start index and a count which defines for how many items it is responsible:

def rxx_func(amp, start_index, count):
    N = len(amp)
    rxx = [0]*count

    for m in xrange(start_index, start_index + count - 1):
        for n in xrange(N-m):
            rxx[m]+=amp[n]*amp[n+m]
    return rxx

I'm sure there is plenty which can be optimized in the above.

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (magn,(i-1)*slice_size, slice_size), (), ()) for i in xrange(ncp)]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

You will have to change your rxx_func to accept a start index and a count which defines for how many items it is responsible:

def rxx_func(amp, start_index, count):
    N = len(amp)
    rxx = [0]*count

    for m in xrange(start_index, start_index + count - 1):
        for n in xrange(N-m):
            rxx[m]+=amp[n]*amp[n+m]
    return rxx

I'm sure there is plenty which can be optimized in the above.

fixed code
Source Link
ChrisWue
  • 20.4k
  • 4
  • 42
  • 107

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# split the input into chunks
magn_slices = [magn[i:i+slize_size] for i in xrange(0, len(magn), slice_size)]

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (slicemagn,(i-1)*slice_size, slice_size), (), ()) for slicei in magn_slices]ncp]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

You will have to change your rxx_func to accept a start index and a count which defines for how many items it is responsible:

def rxx_func(amp, start_index, count):
    N = len(amp)
    rxx = [0]*count

    for m in xrange(start_index, start_index + count - 1):
        for n in xrange(N-m):
            rxx[m]+=amp[n]*amp[n+m]
    return rxx

I'm sure there is plenty which can be optimized in the above.

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# split the input into chunks
magn_slices = [magn[i:i+slize_size] for i in xrange(0, len(magn), slice_size)]

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (slice,), (), ()) for slice in magn_slices]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

I'm sure there is plenty which can be optimized in the above.

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (magn,(i-1)*slice_size, slice_size), (), ()) for i in ncp]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

You will have to change your rxx_func to accept a start index and a count which defines for how many items it is responsible:

def rxx_func(amp, start_index, count):
    N = len(amp)
    rxx = [0]*count

    for m in xrange(start_index, start_index + count - 1):
        for n in xrange(N-m):
            rxx[m]+=amp[n]*amp[n+m]
    return rxx

I'm sure there is plenty which can be optimized in the above.

Source Link
ChrisWue
  • 20.4k
  • 4
  • 42
  • 107

I don't do much python so I might be wrong but to my understanding, this:

job = job_server.submit(rxx_func, (arg_n,), (), ())

just submits a single job but it doesn't automatically parallelize it. If you want to process the input in parallel you need to submit n jobs each working on 1-nth of the input and then combine the results. I think your parallel execution code should look something like this:

slice_size = len(magn) / ncp;

# split the input into chunks
magn_slices = [magn[i:i+slize_size] for i in xrange(0, len(magn), slice_size)]

# submit a job for each chunk
jobs = [job_server.submit(rxx_func, (slice,), (), ()) for slice in magn_slices]

# combine the results into one list, requires #import itertools
rxx = list(itertools.chain.from_iterable([job() for job in jobs]))

I'm sure there is plenty which can be optimized in the above.