I am using Support Vector Machine (SVM) algorithm to perform a classification. The satellite image I am using is really big (5GB) that's why I am trying to take advantage of multiprocessing tool to speed up the process.

My problem is that my PC does not use all the available cores. I run the code to my laptop (4 cores) but it takes for ever for the process to finish. It uses all the 4 CPU cores though. When I try to run the same code on the desktop PC which has 12 CPU cores, only 5 of them reaches 100%. below, we see the image to be classified and the training data that are used on the right enter image description here

import os
import numpy as np
from osgeo import gdal, gdal_array, gdalconst
from osgeo import ogr
import pandas as pd
import image_slicer

from sklearn.svm import SVC
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import accuracy_score
from sklearn.model_selection import cross_val_score, StratifiedKFold, StratifiedShuffleSplit
from sklearn.model_selection import validation_curve, GridSearchCV
from sklearn.pipeline import Pipeline

from matplotlib.pylab import *    
from multiprocessing import Pool
import time

img = 'sea_ice.tif' #image to be used for classification, 3000 x 3000 pixels
roi = 'training_data.shp' #training data

X = img[roi_int > 0, 2:] #X is the matrix containing our features
y = roi[roi>0] #y contains the values of our training data

#Split our dataset into training and testing. Test data will be used to make predictions
split_test_data = 0.30
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_test_data, random_state=42)
stratified = StratifiedShuffleSplit(len(y_train), 10, split_test_data, random_state=0)

#use pipeline method to do all the steps automatically
pip = Pipeline([ ('scale', MinMaxScaler()), 
             ('svm', SVC())])
pip.fit(X_train, y_train)

def predict(input_data):
    img_predict = pip.predict(input_data)
    return img_predict

start = time.time()

tfs_shape = (img.shape[0] * img.shape[1], img.shape[2]-2 )
tfs_2D = img[:, :, 2:].reshape(tfs_shape)

# split good data into chunks for parallel processing
tfsChunks = np.copy(tfs_2D)
split = np.array_split(tfsChunks, 9)

# run parallel processing of all data with SVM
pool = Pool(9)
svmLablesGood = pool.map(predict, split)

# join results back from the queue and insert into full matrix
svmLabelsGood = np.hstack(svmLablesGood)

# reshape labels from vector into 2D raster map
svm_reshape = svmLabelsGood.reshape(img.shape[0], img.shape[1])

end = time.time()
print 'the processing time is:', end - start, '\n'

#Evaluate the model using K-fold cross-validation
scores_svm = cross_val_score(pip, X_test, y_test, scoring='accuracy', cv=stratified)
print ('Accuracy: %0.2f (+/- %0.2f)' % (scores_svm.mean(), scores_svm.std())), '\n'
print ('10 parts cross-validation:' , scores_svm), '\n'

score_train = pip.score(X_train,y_train)
score_test = pip.score(X_test,y_test)
print 'training score:', score_train, '\n'
print 'testing score:', score_test, '\n'

enter image description here

below, the results from the classification are shown (after 3 days of processing). I need to classify areas with ice and ice free areas. Yellow areas show the ice.

enter image description here

I cannot wait for 3 days for my laptop to finish the process. I want to take advantage of the 12 CPU cores of the desktop PC to speed up the process, but I do not understand why only 5 cores are used.

  • 1
    \$\begingroup\$ Please provide the full code to make it easier on the reviewers. At least your imports are missing and if it's at all possible, a smaller example image would be helpful. \$\endgroup\$ – Mast Jul 12 '17 at 8:57
  • 1
    \$\begingroup\$ What happens if you call it with pool = Pool(12) instead of pool = Pool(9)? \$\endgroup\$ – Graipher Jul 12 '17 at 10:23
  • \$\begingroup\$ The really obvious question is have you tried pypy? It might not be faster, but it should be your first attempt when trying to gain speed. \$\endgroup\$ – Oscar Smith Dec 9 '17 at 19:40
  • \$\begingroup\$ I know this question is 2 years old, but I'm not sure it's on topic. OP is looking for an answer as to why the multithreading doesn't use all the cores and doesn't seem to want the code reviewed. \$\endgroup\$ – IEatBagels Jul 26 '19 at 19:05

When you want to parallelize a program you have to actually do it. When you're using C, you can hope that the compiler will spread the CPU "tension" to other CPU, which it will do with the right compiler command. But you're using Python so it cannot do the optimization since there is no (separate) compiler.

But even with C it doesn't work that well. Normally you have to do it yourself with threads. If you can make your calculation on threads you should do it. I didn't read your code in details, but when it comes to SVN I guess that the calculation doesn't depend on previous calculation. If that's the case you might be able to parallelize your program with as many threads as you have cores.

But I think that you have to do that step of parallelizing by yourself, you can't count on the machine to guess where it can or cannot parallelize.

And that will possibly reduce the execution time by the number of threads. (even if that not exactly true, that's a max).

  • 4
    \$\begingroup\$ Regarding your threads suggestion, Python has a Global Interpreter Lock (GIL), which can prevent threads of interpreted code from being processed concurrently. However, what OP can do is write a multi-threaded program using the threading module and run it in the IronPython or Jython runtime or use the subprocess module to run multiple Python interpreters and communicate between them or even use twisted.From all of that, I'd choose the multiprocessing module, which can take advantage of multiple cores (it gets around GIL by starting multiple processes transparently) \$\endgroup\$ – Grajdeanu Alex. Jul 12 '17 at 10:47
  • \$\begingroup\$ Late to the party - this just got bumped. Numpy already uses multiprocessing behind the screens for some operations, typically rendering your own 'threading' efforts slower. \$\endgroup\$ – Gerard Feb 7 '18 at 22:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.