Keras stacked LSTM model for multiclass classification

Question

I am working on a multiple classification problem and after dabbling with multiple neural network architectures, I settled for a stacked LSTM structure as it yields the best accuracy for my use-case. Unfortunately the network takes a long time (almost 48 hours) to reach a good accuracy (~1000 epochs) even when I use GPU acceleration.

At this point, I would like some feedback about my code and my results, did I miss something? and hopefully some suggestions on how to further speed-up and improve the training process/code.

My code is the following:

# -*- coding: utf-8 -*-
import keras
import numpy as np
from time import time
from utils import dmanip, vis
from keras.models import Sequential
from keras.layers import LSTM, Dense
from keras.utils import to_categorical
from keras.callbacks import TensorBoard
from sklearn.preprocessing import LabelEncoder
from tensorflow.python.client import device_lib
from sklearn.model_selection import train_test_split

###############################################################################
####################### Extract the data from .csv file #######################
###############################################################################
# get data
data, column_names = dmanip.get_data(file_path='../data_one_outcome.csv')

# split data
X = data.iloc[:, :-1]
y = data.iloc[:, -1:].astype('category')

###############################################################################
########################## init global config vars ############################
###############################################################################
# check if GPU is used
print(device_lib.list_local_devices())

# init
n_epochs = 1500
n_comps = X.shape[1]

###############################################################################
################################## Keras RNN ##################################
###############################################################################
# encode the classification labels
le = LabelEncoder()
yy = to_categorical(le.fit_transform(y))

# split the dataset
x_train, x_test, y_train, y_test = train_test_split(X, yy, test_size=0.35,
                                                    random_state=True,
                                                    shuffle=True)

# exapand dimensions
x_train = np.expand_dims(x_train, axis=2)
x_test = np.expand_dims(x_test, axis=2)

# define model
model = Sequential()
model.add(LSTM(units=n_comps, return_sequences=True,
               input_shape=(x_train.shape[1], 1),
               dropout=0.2, recurrent_dropout=0.2))
model.add(LSTM(64, return_sequences=True, dropout=0.2, recurrent_dropout=0.2))
model.add(LSTM(32, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(4 ,activation='softmax'))

# print model architecture summary
print(model.summary())

# compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Create a TensorBoard instance with the path to the logs directory
tensorboard = TensorBoard(log_dir='./logs/rnn/{}'.format(time()))

# fit the model
history = model.fit(x_train, y_train, epochs=n_epochs, batch_size=100,
                    validation_data=(x_test, y_test), callbacks=[tensorboard])

# plot results
vis.plot_nn_stats(history=history, stat_type="accuracy", fname="RNN-accuracy")
vis.plot_nn_stats(history=history, stat_type="loss", fname="RNN-loss")

The resulting accuracy and loss functions are:

My data is essentially a big matrix (38607, 150), where 150 is the number of features and 38607 is the number of samples, with a target vector including 4 classes.

Answer 1

If I understand your problem, your samples are 150-dimensional vectors which is to be classified into one of four categories. If so, an LSTM architecture is a very poor fit because there are no relations between your samples. I.e what is in the n:th sample doesn't impact what is in the n + 1:th sample.

Try something like this instead:

from keras.layers import Dense
from keras.models import Sequential
from keras.utils import to_categorical
import numpy as np

n = 38607
d = 150
k = 4

# Generate data
X = np.random.randn(n, d)
Y = to_categorical(np.random.randint(k, size = n), k)

model = Sequential()
model.add(Dense(128, input_dim = d, activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(4, activation = 'softmax'))
model.compile(
    loss = 'binary_crossentropy',
    optimizer='adam', metrics=['accuracy'])
model.summary()

model.fit(X, Y, epochs = 100, batch_size = 128, verbose = 1)

I haven't used any test data so the model quickly overfits (accuracy > 25% implies overfitting). Which brings me to my next point. On your graphs, you get better performance on the test data than on the training data which is very suspect. I suggest you try and train your network without any dropout first to see if it behaves as expected before adding it back.