I am using Support Vector Machine (SVM) algorithm to perform a classification. The satellite image I am using is really big (5GB) that's why I am trying to take advantage of
multiprocessing tool to speed up the process.
My problem is that my PC does not use all the available cores. I run the code to my laptop (4 cores) but it takes for ever for the process to finish. It uses all the 4 CPU cores though. When I try to run the same code on the desktop PC which has 12 CPU cores, only 5 of them reaches 100%. below, we see the image to be classified and the training data that are used on the right
import os import numpy as np from osgeo import gdal, gdal_array, gdalconst from osgeo import ogr import pandas as pd import image_slicer from sklearn.svm import SVC from sklearn.preprocessing import MinMaxScaler from sklearn.model_selection import train_test_split from sklearn.metrics import confusion_matrix from sklearn.metrics import classification_report from sklearn.metrics import accuracy_score from sklearn.model_selection import cross_val_score, StratifiedKFold, StratifiedShuffleSplit from sklearn.model_selection import validation_curve, GridSearchCV from sklearn.pipeline import Pipeline from matplotlib.pylab import * from multiprocessing import Pool import time img = 'sea_ice.tif' #image to be used for classification, 3000 x 3000 pixels roi = 'training_data.shp' #training data X = img[roi_int > 0, 2:] #X is the matrix containing our features y = roi[roi>0] #y contains the values of our training data #Split our dataset into training and testing. Test data will be used to make predictions split_test_data = 0.30 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_test_data, random_state=42) stratified = StratifiedShuffleSplit(len(y_train), 10, split_test_data, random_state=0) #use pipeline method to do all the steps automatically pip = Pipeline([ ('scale', MinMaxScaler()), ('svm', SVC())]) pip.fit(X_train, y_train) def predict(input_data): img_predict = pip.predict(input_data) return img_predict start = time.time() tfs_shape = (img.shape * img.shape, img.shape-2 ) tfs_2D = img[:, :, 2:].reshape(tfs_shape) # split good data into chunks for parallel processing tfsChunks = np.copy(tfs_2D) split = np.array_split(tfsChunks, 9) # run parallel processing of all data with SVM pool = Pool(9) svmLablesGood = pool.map(predict, split) # join results back from the queue and insert into full matrix svmLabelsGood = np.hstack(svmLablesGood) # reshape labels from vector into 2D raster map svm_reshape = svmLabelsGood.reshape(img.shape, img.shape) plt.imshow(svm_reshape) plt.show() end = time.time() print 'the processing time is:', end - start, '\n' #Evaluate the model using K-fold cross-validation scores_svm = cross_val_score(pip, X_test, y_test, scoring='accuracy', cv=stratified) print ('Accuracy: %0.2f (+/- %0.2f)' % (scores_svm.mean(), scores_svm.std())), '\n' print ('10 parts cross-validation:' , scores_svm), '\n' score_train = pip.score(X_train,y_train) score_test = pip.score(X_test,y_test) print 'training score:', score_train, '\n' print 'testing score:', score_test, '\n'
below, the results from the classification are shown (after 3 days of processing). I need to classify areas with ice and ice free areas. Yellow areas show the ice.
I cannot wait for 3 days for my laptop to finish the process. I want to take advantage of the 12 CPU cores of the desktop PC to speed up the process, but I do not understand why only 5 cores are used.