2
\$\begingroup\$

I am using Matlab to train a convolutional neural network to do a two class image classification problem. I have an imbalanced data set (~1800 images minority class, ~5000 images majority class). As I understand it, the splitEachLabel function will split the data into a train set and a test set. In my case, it will put 1024 images (selected from both classes) into the train set and the rest into the test set.

I am using new random seeds on each iteration to select different sets of images, just to check if the neural network architecture works well when given different subsets of the same data. Is that a good way to do it?

My only issue is my training accuracy is about 20% Lower than my testing accuracy, which I find extremely perplexing.

% delete accuracy between runs if changing number of iterations
for i = 1:6
    rng(i); % for reproducible results
    datastore = imageDatastore(fullfile('.'), 'IncludeSubfolders', true, 'LabelSource', 'foldernames');
    [trainingSet, testingSet] = splitEachLabel(datastore, 1024, 'randomize');
    % convolutional output area = 1+(inputWidth - filterSize + 2*Padding) /
    % Stride
    layers = [imageInputLayer([128 128 3]);
          convolution2dLayer(7,48,'Stride',1,'Padding',3);
          reluLayer();
          maxPooling2dLayer(2,'Stride',2);
          convolution2dLayer(5,128,'Stride',1,'Padding',2);
          reluLayer();
          maxPooling2dLayer(2,'Stride',2);
          convolution2dLayer(3,192,'Stride',1,'Padding',1);
          reluLayer();
          convolution2dLayer(3,192,'Stride',1,'Padding',1);
          reluLayer();
          convolution2dLayer(3,128,'Stride',1,'Padding',1);
          reluLayer();
          maxPooling2dLayer(2,'Stride',2);
          fullyConnectedLayer(256);
          fullyConnectedLayer(2);
          softmaxLayer();
          classificationLayer()];
    options = trainingOptions('sgdm','MaxEpochs',30,'InitialLearnRate',0.0001,'MiniBatchSize',128, ...
    'LearnRateDropPeriod', 6, 'LearnRateDropFactor', 0.2);
    convnet = trainNetwork(trainingSet, layers, options);
    trainPredictions = classify(convnet, trainingSet);
    trainLabels = trainingSet.Labels;
    train_accuracy{i} = sum(trainPredictions == trainLabels)/numel(trainLabels);

    testPredictions = classify(convnet, testingSet);
    testLabels = testingSet.Labels;
    test_accuracy{i} = sum(testPredictions == testLabels)/numel(testLabels);

    disp(train_accuracy{i})
    disp(test_accuracy{i})
end
disp("Mean:")
disp(mean(cell2mat(train_accuracy)))
disp(mean(cell2mat(test_accuracy)))

Edit: I discovered I needed to add a parameter to the training options in order for the learn rate to drop every so many epochs. Specifically, I needed to add 'LearnRateSchedule' with the option 'piecewise'.

Also, increasing the number of training images a little bit and removing one of the 192 filter convolutional layers seems to have made the training and testing accuracy more closely align with each other. So I believe that issue was just a fluke regarding my architecture and limited number of training images versus the size of the testing set.

\$\endgroup\$
  • 1
    \$\begingroup\$ "my training accuracy is about 20% Lower than my testing accuracy" - Stupid question, but did you triple check that it is not the other way around? Then it would be a classical case of overfitting. \$\endgroup\$ – Leander Moesinger Nov 18 '17 at 11:17
  • \$\begingroup\$ Yeah, I know it would be. That's half the reason I put this code here, because I was concerned I was doing something wrong. I have since removed the first convolution2dLyaer(3, 192, 'Stride', 1, 'Padding', 1) and its relu layer and the training/testing accuracies are closer together, with the mean testing just being slightly lower than the mean training. There are some cases where testing is much higher than training, but now I think those cases are just flukes. I still found it odd, though. \$\endgroup\$ – Alpha Bravo Nov 19 '17 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.