I'm running the following code using numpy arrays, I get a MemoryError in Ubuntu, while the same code runs on Mac OSX. (Pagination is automatically setup in Mac) This process consumes around 30 GB. Original post here.
I'm trying to replace
numpy array with another structure which doesn't consume so much memory and improve the
Original code here.
The following function:
maximum sequence length, and the dictionary of all
photos, will transform the data into input-output pairs of data for a training model. There are two input arrays to the model: one for photo features and one for the encoded text. There is one output for the model which is the encoded next word in the text sequence.
The input text is encoded as integers, which will be fed to a word embedding layer. The photo features will be fed directly to another part of the model. The model will output a prediction, which will be a probability distribution over all words in the vocabulary.
The output data will therefore be a one-hot encoded version of each word, representing an idealized probability distribution with 0 values at all word positions except the actual word position, which has a value of 1.
def create_sequences(tokenizer, max_length, descriptions, photos): """Creates sequences of images, input sequences and output words for an image. X1, X2 (text sequence), y (word) photo startseq, little photo startseq, little, girl photo startseq, little, girl, running photo startseq, little, girl, running, in photo startseq, little, girl, running, in, field photo startseq, little, girl, running, in, field, endseq :param tokenizer: :param max_length: :param descriptions: :param photos: :return: """ X1, X2, y = , ,  # Walk through each image identifier. for desc_key, desc_list in descriptions.iteritems(): # Walk through each description for the image. for desc in desc_list: # Encode the sequence. seq = tokenizer.texts_to_sequences([desc]) # Split one sequence into multiple X,Y pairs. for i in range(1, len(seq)): # Split into input and output pair. in_seq, out_seq = seq[:i], seq[i] # Pad input sequence. in_seq = pad_sequences([in_seq], maxlen=max_length) # Encode output sequence out_seq = to_categorical([out_seq], num_classes=vocab_size) # Store. X1.append(photos[desc_key]) X2.append(in_seq) y.append(out_seq) print len(X1), len(X2), len(y) print type(X1) return array(X1), array(X2), array(y)
Dataset: 6000 train images. Descriptions: train=6000 Vocabulary Size: 7579 Photos: train=6000 Description Length: 34 Preparing text sequences for training. 306404 306404 306404 <type 'numpy.ndarray'>
sys.getsizeof() for the arrays I get:
print sys.getsizeof(X1), sys.getsizeof(X2), sys.getsizeof(y) 2678104 2678104 2678104