Training a Reinforcement Learning algorithm

Question

Problem: I'm trying to reduce the computation time and optimize specific functions of a Reinforcement Learning algorithm in the training phase and observed that one block of code takes too much computation overhead (the for loop described below).

for loop description

The function takes in range self.n = 3 (n is the number of agents in an RL algorithm) and the index is a list which contains a list of random numbers (1024 - fixed list) to retrieve a set of observations, actions, next observations etc. But the problem is when self.n (number of agents) increases from 3 to 20 or 48 or even bigger numbers, this loop is taking to much time as each agent has to collect a set of 1024 data points (obs, next_obs, actions etc.) from their buffer (self._storage) and append all the agent's data to a single list (in the for loop). As each agent does not have any data dependency to collect the data. My doubt is, can we try to execute the for loop on multiple threads or use MPI to make them run in parallel and collect the data faster to achieve some speedup?

For better understanding, I run the whole algorithm with self.n =3 and self.n=6, for 60,000 episodes (the more episodes the loop is triggered more often) and the computation times are 1758.40 sec and 4616.31 sec, respectively. I'm glad to answer any questions.

Code:

for i in range(self.n):
    obs, act, rew, obs_next, done = agents[i].replay_buffer.sample_index(index)
    obs_n.append(obs)
    obs_next_n.append(obs_next)
    act_n.append(act)

The connector functions are:

def sample_index(self, idxes):
    return self._encode_sample(idxes)

def _encode_sample(self, idxes):
    obses_t, actions, rewards, obses_tp1, dones = [], [], [], [], []
    for i in idxes:
        data = self._storage[i]
        obs_t, action, reward, obs_tp1, done = data
        obses_t.append(np.array(obs_t, copy=False))
        actions.append(np.array(action, copy=False))
        rewards.append(reward)
        obses_tp1.append(np.array(obs_tp1, copy=False))
        dones.append(done)
    return np.array(obses_t), np.array(actions), np.array(rewards), np.array(obses_tp1), np.array(dones)

Also,

self.replay_buffer = ReplayBuffer(1e6)

Here, ReplayBuffer is a class

Better details about the loop can be found here: https://github.com/openai/maddpg/blob/master/maddpg/trainer/maddpg.py#L173

Screenshot of index is: (list of random numbers)

Answer 1

How do you imagine that the sequence of append operations in your loop can be sped up by parallelization? It can not, because appending is strictly sequential.

Dynamic data structures and parallelism don't go together. Instead of a dynamic append, pre-allocate your list of results, and have the iterations insert their values in the appropriate location.