I'm running the following code using numpy arrays, I get a MemoryError in Ubuntu, while the same code runs on Mac OSX. (Pagination is automatically setup in Mac) This process consumes around 30 GB. Original post here.
I'm trying to replace numpy
array with another structure which doesn't consume so much memory and improve the for loops
.
Original code here.
The following function: create_sequences
Given the tokenizer
, a maximum sequence length
, and the dictionary of all descriptions
and photos
, will transform the data into input-output pairs of data for a training model. There are two input arrays to the model: one for photo features and one for the encoded text. There is one output for the model which is the encoded next word in the text sequence.
The input text is encoded as integers, which will be fed to a word embedding layer. The photo features will be fed directly to another part of the model. The model will output a prediction, which will be a probability distribution over all words in the vocabulary.
The output data will therefore be a one-hot encoded version of each word, representing an idealized probability distribution with 0 values at all word positions except the actual word position, which has a value of 1.
def create_sequences(tokenizer, max_length, descriptions, photos):
"""Creates sequences of images, input sequences and output words for an image.
X1, X2 (text sequence), y (word)
photo startseq, little
photo startseq, little, girl
photo startseq, little, girl, running
photo startseq, little, girl, running, in
photo startseq, little, girl, running, in, field
photo startseq, little, girl, running, in, field, endseq
:param tokenizer:
:param max_length:
:param descriptions:
:param photos:
:return:
"""
X1, X2, y = [], [], []
# Walk through each image identifier.
for desc_key, desc_list in descriptions.iteritems():
# Walk through each description for the image.
for desc in desc_list:
# Encode the sequence.
seq = tokenizer.texts_to_sequences([desc])[0]
# Split one sequence into multiple X,Y pairs.
for i in range(1, len(seq)):
# Split into input and output pair.
in_seq, out_seq = seq[:i], seq[i]
# Pad input sequence.
in_seq = pad_sequences([in_seq], maxlen=max_length)[0]
# Encode output sequence
out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]
# Store.
X1.append(photos[desc_key][0])
X2.append(in_seq)
y.append(out_seq)
print len(X1), len(X2), len(y)
print type(X1[0])
return array(X1), array(X2), array(y)
Output:
Dataset: 6000 train images.
Descriptions: train=6000
Vocabulary Size: 7579
Photos: train=6000
Description Length: 34
Preparing text sequences for training.
306404 306404 306404
<type 'numpy.ndarray'>
Memory utilization
Using sys.getsizeof()
for the arrays I get:
print sys.getsizeof(X1), sys.getsizeof(X2), sys.getsizeof(y)
2678104 2678104 2678104
CProfile:
tokenizer
,max_length
,descriptions
, orphotos
are \$\endgroup\$goal clearly is to reduce the amount of memory being used
this may be your goal in posting here - the title should reflect the purpose of the code presented. It would probably help to know a) the purpose of the overall processing c) how splitting sequences fromtokenizer.texts_to_sequences()
at each and every index givesinput
andoutput
. \$\endgroup\$