I'm running the following code using numpy arrays, I get a MemoryError in Ubuntu, while the same code runs on Mac OSX. (Pagination is automatically setup in Mac) This process consumes around 30 GB. Original post here.

I'm trying to replace numpy array with another structure which doesn't consume so much memory and improve the for loops. Original code here.

The following function: create_sequences

Given the tokenizer, a maximum sequence length, and the dictionary of all descriptions and photos, will transform the data into input-output pairs of data for a training model. There are two input arrays to the model: one for photo features and one for the encoded text. There is one output for the model which is the encoded next word in the text sequence.

The input text is encoded as integers, which will be fed to a word embedding layer. The photo features will be fed directly to another part of the model. The model will output a prediction, which will be a probability distribution over all words in the vocabulary.

The output data will therefore be a one-hot encoded version of each word, representing an idealized probability distribution with 0 values at all word positions except the actual word position, which has a value of 1.

def create_sequences(tokenizer, max_length, descriptions, photos):
    """Creates sequences of images, input sequences and output words for an image.

    X1,     X2 (text sequence),                         y (word)
    photo   startseq,                                   little
    photo   startseq, little,                           girl
    photo   startseq, little, girl,                     running
    photo   startseq, little, girl, running,            in
    photo   startseq, little, girl, running, in,        field
    photo   startseq, little, girl, running, in, field, endseq

    :param tokenizer:
    :param max_length:
    :param descriptions:
    :param photos:
    :return:
    """
    X1, X2, y = [], [], []
    # Walk through each image identifier.
    for desc_key, desc_list in descriptions.iteritems():
        # Walk through each description for the image.
        for desc in desc_list:
            # Encode the sequence.
            seq = tokenizer.texts_to_sequences([desc])[0]
            # Split one sequence into multiple X,Y pairs.
            for i in range(1, len(seq)):
                # Split into input and output pair.
                in_seq, out_seq = seq[:i], seq[i]
                # Pad input sequence.
                in_seq = pad_sequences([in_seq], maxlen=max_length)[0]
                # Encode output sequence
                out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]
                # Store.
                X1.append(photos[desc_key][0])
                X2.append(in_seq)
                y.append(out_seq)
    print len(X1), len(X2), len(y)
    print type(X1[0])
    return array(X1), array(X2), array(y)

Output:

Dataset: 6000 train images.
Descriptions: train=6000
Vocabulary Size: 7579
Photos: train=6000
Description Length: 34
Preparing text sequences for training.
306404 306404 306404
<type 'numpy.ndarray'>

Memory utilization

Memory utilization

Using sys.getsizeof() for the arrays I get:

print sys.getsizeof(X1), sys.getsizeof(X2), sys.getsizeof(y)
2678104 2678104 2678104

CProfile:

https://pastebin.com/raw/UkAEaAjq

  • what in your code is using an nd.array? Can you attach the code you use to call this? – Oscar Smith Dec 11 '17 at 15:36
  • 3
    There still is not enough info in the question to give any useful feedback. I have no idea what the code is trying to do, or what tokenizer, max_length, descriptions, or photos are – Oscar Smith Dec 11 '17 at 17:48
  • 3
    The title of the post should say what the code is supposed to do -- what kind of a program it is . "Numpy" is already a tag. "Array" is implied by numpy. "Enhancements" applies to every question on this site. – Snowbody Dec 11 '17 at 18:27
  • 4
    goal clearly is to reduce the amount of memory being used this may be your goal in posting here - the title should reflect the purpose of the code presented. It would probably help to know a) the purpose of the overall processing c) how splitting sequences from tokenizer.texts_to_sequences() at each and every index gives input and output. – greybeard Dec 13 '17 at 7:55
  • 2
    When I say "goal" I mean the code's reason for being. You didn't write the code to "save memory", you wrote the code to solve some real-world problem. The title of the post is supposed to communicate the purpose of your code. ("A portfolio optimizer"/"A database of melodies"/"A guessing game"). In the body of the question, state what you'd like reviewers to focus on. – Snowbody Dec 13 '17 at 7:57

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.