1
\$\begingroup\$

I've created an application to predict emotions. But I think the application is being over fitted. And I cannot figure out how to solve the overfitting. I'm using a small data set of 107 images spread across 6 emotions.

Train.py

#Adding Seed so that random initialization is consistent
from numpy.random import seed
seed(1)
tf.set_random_seed(2)


batch_size = 3

#Prepare input data
classes = ['Anger','Disgust','Fear','Happy','Neutral','Sad','Surprise']
num_classes = len(classes)

# 20% of the data will automatically be used for validation
validation_size = 0.2
img_size = 128
num_channels = 3
train_path='C:\\Users\\Jack\\source\\repos\\Emotion Prediction\\Emotion Prediction\\training_images'

# We shall load all the training and validation images and labels into memory using openCV and use that during training
data = dataset.read_train_sets(train_path, img_size, classes, validation_size=validation_size)


print("Complete reading input data. Will Now print a snippet of it")
print("Number of files in Training-set:\t\t{}".format(len(data.train.labels)))
print("Number of files in Validation-set:\t{}".format(len(data.valid.labels)))



session = tf.Session()
x = tf.placeholder(tf.float32, shape=[None, img_size,img_size,num_channels], name='x')

## labels
y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')
y_true_cls = tf.argmax(y_true, dimension=1)



##Network graph params
filter_size_conv1 = 3 
num_filters_conv1 = 32  

filter_size_conv2 = 3
num_filters_conv2 = 32

filter_size_conv3 = 3
num_filters_conv3 = 64

fc_layer_size = 128

def create_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

def create_biases(size):
    return tf.Variable(tf.constant(0.05, shape=[size]))



def create_convolutional_layer(input,
               num_input_channels, 
               conv_filter_size,        
               num_filters):  

    ## We shall define the weights that will be trained using create_weights function.
    weights = create_weights(shape=[conv_filter_size, conv_filter_size, num_input_channels, num_filters])
    ## We create biases using the create_biases function. These are also trained.
    biases = create_biases(num_filters)

    ## Creating the convolutional layer
    layer = tf.nn.conv2d(input=input,
                     filter=weights,
                     strides=[1, 1, 1, 1],
                     padding='SAME')

    layer += biases                             #gets input and weights. strides = filter moves 1 step in both x and y directions (width and height)

    ## We shall be using max-pooling.  
    layer = tf.nn.max_pool(value=layer,
                            ksize=[1, 2, 2, 1],
                            strides=[1, 2, 2, 1],   #middle values define shape of max pooling in x and y directions, same with strides as this while down-sample and halve the input size
                            padding='SAME')
    ## Output of pooling is fed to Relu which is the activation function for us.
    layer = tf.nn.relu(layer)

    return layer



def create_flatten_layer(layer):
    #We know that the shape of the layer will be [batch_size img_size img_size num_channels] 
    # But let's get it from the previous layer.
    layer_shape = layer.get_shape()

    ## Number of features will be img_height * img_width* num_channels. But we shall calculate it in place of hard-coding it.
    num_features = layer_shape[1:4].num_elements()

    ## Now, we Flatten the layer so we shall have to reshape to num_features
    layer = tf.reshape(layer, [-1, num_features])

    return layer


def create_fc_layer(input,          
             num_inputs,    
             num_outputs,
             use_relu=True):

    #Let's define trainable weights and biases.
    weights = create_weights(shape=[num_inputs, num_outputs])
    biases = create_biases(num_outputs)

    # Fully connected layer takes input x and produces wx+b.Since, these are matrices, we use matmul function in Tensorflow
    layer = tf.matmul(input, weights) + biases
    #all_weights=[]
    #w=layer.get_weights()
    #all_weights.append(w)
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer


layer_conv1 = create_convolutional_layer(input=x,
               num_input_channels=num_channels,
               conv_filter_size=filter_size_conv1,
               num_filters=num_filters_conv1)
layer_conv2 = create_convolutional_layer(input=layer_conv1,
               num_input_channels=num_filters_conv1,
               conv_filter_size=filter_size_conv2,
               num_filters=num_filters_conv2)

layer_conv3= create_convolutional_layer(input=layer_conv2,
               num_input_channels=num_filters_conv2,
               conv_filter_size=filter_size_conv3,
               num_filters=num_filters_conv3)

layer_flat = create_flatten_layer(layer_conv3)

layer_fc1 = create_fc_layer(input=layer_flat,
                     num_inputs=layer_flat.get_shape()[1:4].num_elements(),
                     num_outputs=fc_layer_size,
                     use_relu=True)

layer_fc2 = create_fc_layer(input=layer_fc1,
                     num_inputs=fc_layer_size,
                     num_outputs=num_classes,
                     use_relu=False) 

y_pred = tf.nn.softmax(layer_fc2,name='y_pred')

y_pred_cls = tf.argmax(y_pred, dimension=1)
session.run(tf.global_variables_initializer())
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
                                                    labels=y_true)
cost = tf.reduce_mean(cross_entropy)            #takes in softmax of matrix multiplication and compares it to the training target via cross-entropy
optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cost)
correct_prediction = tf.equal(y_pred_cls, y_true_cls)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


session.run(tf.global_variables_initializer()) 


def show_progress(epoch, feed_dict_train, feed_dict_validate, val_loss):
    acc = session.run(accuracy, feed_dict=feed_dict_train)
    val_acc = session.run(accuracy, feed_dict=feed_dict_validate)
    msg = "Training Epoch {0} --- Training Accuracy: {1:>6.1%}, Validation Accuracy: {2:>6.1%},  Validation Loss: {3:.3f}"
    print(msg.format(epoch + 1, acc, val_acc, val_loss))

total_iterations = 0

saver = tf.train.Saver()
def train(num_iteration):
    global total_iterations

    for i in range(total_iterations,
                   total_iterations + num_iteration):

        x_batch, y_true_batch, _, cls_batch = data.train.next_batch(batch_size)
        x_valid_batch, y_valid_batch, _, valid_cls_batch = data.valid.next_batch(batch_size)


        feed_dict_tr = {x: x_batch,
                           y_true: y_true_batch}
        feed_dict_val = {x: x_valid_batch,
                              y_true: y_valid_batch}

        session.run(optimizer, feed_dict=feed_dict_tr)

        if i % int(data.train.num_examples/batch_size) == 0: 
            val_loss = session.run(cost, feed_dict=feed_dict_val)
            epoch = int(i / int(data.train.num_examples/batch_size))    

            show_progress(epoch, feed_dict_tr, feed_dict_val, val_loss)
            saver.save(session, './emotions-model') 


    total_iterations += num_iteration

train(num_iteration=3000)

Dataset.py

def load_train(train_path, image_size, classes):
    images = []
    labels = []
    img_names = []
    cls = []

    print('Going to read training images')
    for fields in classes:   
        index = classes.index(fields)
        print('Now going to read {} files (Index: {})'.format(fields, index))
        path = os.path.join(train_path, fields, '*g')
        files = glob.glob(path)
        for fl in files:
            image = cv2.imread(fl)
            imag

e = cv2.resize(image, (image_size, image_size),0,0, cv2.INTER_LINEAR)
                image = image.astype(np.float32)
                image = np.multiply(image, 1.0 / 255.0)
                images.append(image)
                label = np.zeros(len(classes))
                label[index] = 1.0
                labels.append(label)
                flbase = os.path.basename(fl)
                img_names.append(flbase)
                cls.append(fields)
        images = np.array(images)
        labels = np.array(labels)
        img_names = np.array(img_names)
        cls = np.array(cls)

        return images, labels, img_names, cls


    class DataSet(object):

      def __init__(self, images, labels, img_names, cls):
        self._num_examples = images.shape[0]

        self._images = images
        self._labels = labels
        self._img_names = img_names
        self._cls = cls
        self._epochs_done = 0
        self._index_in_epoch = 0

      @property
      def images(self):
        return self._images

      @property
      def labels(self):
        return self._labels

      @property
      def img_names(self):
        return self._img_names

      @property
      def cls(self):
        return self._cls

      @property
      def num_examples(self):
        return self._num_examples

      @property
      def epochs_done(self):
        return self._epochs_done

      def next_batch(self, batch_size):
        """Return the next `batch_size` examples from this data set."""
        start = self._index_in_epoch
        self._index_in_epoch += batch_size

        if self._index_in_epoch > self._num_examples:
          # After each epoch we update this
          self._epochs_done += 1
          start = 0
          self._index_in_epoch = batch_size
          assert batch_size <= self._num_examples
        end = self._index_in_epoch

        return self._images[start:end], self._labels[start:end], self._img_names[start:end], self._cls[start:end]


    def read_train_sets(train_path, image_size, classes, validation_size):
      class DataSets(object):
        pass
      data_sets = DataSets()

      images, labels, img_names, cls = load_train(train_path, image_size, classes)
      images, labels, img_names, cls = shuffle(images, labels, img_names, cls)  

      if isinstance(validation_size, float):
        validation_size = int(validation_size * images.shape[0])

      validation_images = images[:validation_size]
      validation_labels = labels[:validation_size]
      validation_img_names = img_names[:validation_size]
      validation_cls = cls[:validation_size]

      train_images = images[validation_size:]
      train_labels = labels[validation_size:]
      train_img_names = img_names[validation_size:]
      train_cls = cls[validation_size:]

      data_sets.train = DataSet(train_images, train_labels, train_img_names, train_cls)
      data_sets.valid = DataSet(validation_images, validation_labels, validation_img_names, validation_cls)

      return data_sets

Log

Training Epoch 1 --- Training Accuracy:  66.7%, Validation Accuracy:  33.3%,  Validation Loss: 1.748
Training Epoch 2 --- Training Accuracy:  33.3%, Validation Accuracy:   0.0%,  Validation Loss: 1.646
Training Epoch 3 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.817
Training Epoch 4 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.625
Training Epoch 5 --- Training Accuracy:  33.3%, Validation Accuracy:   0.0%,  Validation Loss: 1.912
Training Epoch 6 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.750
Training Epoch 7 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.307
Training Epoch 8 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.583
Training Epoch 9 --- Training Accuracy:  33.3%, Validation Accuracy:   0.0%,  Validation Loss: 1.462
Training Epoch 10 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.722
Training Epoch 11 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.644
Training Epoch 12 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.390
Training Epoch 13 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.851
Training Epoch 14 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.318
Training Epoch 15 --- Training Accuracy:  33.3%, Validation Accuracy:  33.3%,  Validation Loss: 1.612
Training Epoch 16 --- Training Accuracy:  66.7%, Validation Accuracy:  33.3%,  Validation Loss: 1.054
Training Epoch 17 --- Training Accuracy:  66.7%, Validation Accuracy:  66.7%,  Validation Loss: 1.386
Training Epoch 18 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 2.472
Training Epoch 19 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 1.062
Training Epoch 20 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.774
Training Epoch 21 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 1.122
Training Epoch 22 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 2.418
Training Epoch 23 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 1.389
Training Epoch 24 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 2.487
Training Epoch 25 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 4.330
Training Epoch 26 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 0.823
Training Epoch 27 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 2.173
Training Epoch 28 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 1.558
Training Epoch 29 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 3.549
Training Epoch 30 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 2.107
Training Epoch 31 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 4.092
Training Epoch 32 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.280
Training Epoch 33 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 2.862
Training Epoch 34 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 3.318
Training Epoch 35 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 0.827
Training Epoch 36 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 3.734
Training Epoch 37 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 2.328
Training Epoch 38 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 5.210
Training Epoch 39 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.896
Training Epoch 40 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 4.436
Training Epoch 41 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 4.195
Training Epoch 42 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.137
Training Epoch 43 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 4.350
Training Epoch 44 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 2.736
Training Epoch 45 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 6.058
Training Epoch 46 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 7.786
Training Epoch 47 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 5.402
Training Epoch 48 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 4.846
Training Epoch 49 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.314
Training Epoch 50 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 4.814
Training Epoch 51 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 3.047
Training Epoch 52 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 6.672
Training Epoch 53 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 8.441
Training Epoch 54 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 6.159
Training Epoch 55 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 5.220
Training Epoch 56 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.452
Training Epoch 57 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 5.171
Training Epoch 58 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 3.304
Training Epoch 59 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 7.131
Training Epoch 60 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 8.937
Training Epoch 61 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 6.786
Training Epoch 62 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 5.584
Training Epoch 63 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.554
Training Epoch 64 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 5.472
Training Epoch 65 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 3.526
Training Epoch 66 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 7.537
Training Epoch 67 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 9.353
Training Epoch 68 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 7.322
Training Epoch 69 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 5.857
Training Epoch 70 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.651
Training Epoch 71 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 5.719
Training Epoch 72 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 3.716
Training Epoch 73 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 7.873
Training Epoch 74 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 9.711
Training Epoch 75 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 7.728
Training Epoch 76 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.105
Training Epoch 77 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.723
Training Epoch 78 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 5.940
Training Epoch 79 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 3.877
Training Epoch 80 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.170
Training Epoch 81 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 10.021
Training Epoch 82 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.164
Training Epoch 83 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.326
Training Epoch 84 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.792
Training Epoch 85 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 6.139
Training Epoch 86 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 4.036
Training Epoch 87 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.436
Training Epoch 88 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 10.302
Training Epoch 89 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.543
Training Epoch 90 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.520
Training Epoch 91 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.853
Training Epoch 92 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 6.312
Training Epoch 93 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 4.167
Training Epoch 94 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.675
Training Epoch 95 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 10.554
Training Epoch 96 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.883
Training Epoch 97 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.700
Training Epoch 98 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 1.906
Training Epoch 99 --- Training Accuracy: 100.0%, Validation Accuracy:  66.7%,  Validation Loss: 6.469
Training Epoch 100 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 4.293
Training Epoch 101 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 8.899
Training Epoch 102 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 10.787
Training Epoch 103 --- Training Accuracy: 100.0%, Validation Accuracy:   0.0%,  Validation Loss: 9.221
Training Epoch 104 --- Training Accuracy: 100.0%, Validation Accuracy:  33.3%,  Validation Loss: 6.876
\$\endgroup\$
2
  • \$\begingroup\$ Welcome to code review where we review code that is working as expected so that we can provide suggestions on how that code can be improved. We are not able to debug code or add features to code. Unfortunately this question indicates that the code is not working as expected, and therefore is off-topic. Please check our guidelines at codereview.stackexchange.com/help/asking. I don't know if there is an AI related stackexchange Q&A site, but that would be the place to ask this question if there is. \$\endgroup\$
    – pacmaninbw
    Apr 15 '20 at 17:10
  • 1
    \$\begingroup\$ Considering the low validation accuracy, I do not consider this code to be working the way it should. This basically means it's too early for a review of it. Please take a look at the help center. \$\endgroup\$
    – Mast
    Apr 15 '20 at 19:33
0
\$\begingroup\$

Would you please check read_train_set function and make sure 20% of validation data consist of all seven classes? It's preferred to have all classes balanced in the training and testing data set. One way to do it is to split the data per class. Adding a dropout layer in CNN might help reduce over-fitting. The model might be able to detect 'happy' from 'sad' better than detect 'surprise' from 'fear'. A more detailed validation accuracy analysis will show some insight.

\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.