2
\$\begingroup\$

I have an array of dim 1000 x 150 where I have 1000 strings each of length 150. And the strings only contain ATCGN (these are DNA sequences).

I am converting the letters to one-hot vectors such that A is [1,0,0,0,0] and N is [0,0,0,0,1].

I wrote the following code but it takes way too long when there are about 20 million strings.

def seqs2onehot(seqs, num_val):
    def one_hot_encode(seq):
        mapping = dict(zip("ACGTN", range(num_val)))
        seq = [mapping[i] for i in seq]
        return np.eye(num_val)[seq]
    onehotvecs = []
    for i in seqs:
        onehotvecs.append(np.array(one_hot_encode(i), dtype='bool'))
    return onehotvecs

To generate input:

num_inputs = 1000
inputs = []
for num in range(num_inputs):
    inputs.append(''.join(random.choice('ATCGN') for i in range(150)))

Outputs are of dimension 1000 x 150 x 5 x 1. You can create the final outputs by doing this:

inputs = np.array(inputs)
inputs = np.expand_dims(np.array(seqs2onehot(inputs, 5)), -1)
inputs.shape

Is there a way to do this without iterating through the list and just using matrix operations?

\$\endgroup\$
4
  • \$\begingroup\$ Can you provide an example of the expected formats for the input and output? I think that would help make this a bit clearer regarding your intentions. Otherwise, I'd say one of your main hangups is in using append 20 million times on a list. If you know your final size, pre-allocate your memory. Similarly, you do a lot of "regeneration on the fly" for things that could be defined statically and reused rather than created, used, and discarded, only to be re-created every single iteration. \$\endgroup\$
    – zephyr
    Feb 8 at 16:49
  • \$\begingroup\$ @zephyr good idea, I've added in input output code. \$\endgroup\$
    – Jonathan
    Feb 8 at 20:07
  • 1
    \$\begingroup\$ What is the rationale for using “one hot vectors”? What is the rationale for encoding them in the way you have? \$\endgroup\$ Feb 9 at 4:36
  • \$\begingroup\$ This is for a neural net that I'm training. But even otherwise, I come across similar coding problems and by learning through this example, I can extend it elsewhere... \$\endgroup\$
    – Jonathan
    Feb 9 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.