Problem: I'm trying to reduce the computation time and optimize specific functions of a Reinforcement Learning algorithm in the training phase and observed that one block of code takes too much computation overhead (the for loop described below).
for loop description
The function takes in range self.n = 3 (n is the number of agents in an RL algorithm) and the index is a list which contains a list of random numbers (1024 - fixed list) to retrieve a set of observations, actions, next observations etc. But the problem is when self.n (number of agents) increases from 3 to 20 or 48 or even bigger numbers, this loop is taking to much time as each agent has to collect a set of 1024 data points (obs, next_obs, actions etc.) from their buffer (self._storage) and append all the agent's data to a single list (in the for loop). As each agent does not have any data dependency to collect the data. My doubt is, can we try to execute the for loop on multiple threads or use MPI to make them run in parallel and collect the data faster to achieve some speedup?
For better understanding, I run the whole algorithm with self.n =3 and self.n=6, for 60,000 episodes (the more episodes the loop is triggered more often) and the computation times are 1758.40 sec and 4616.31 sec, respectively. I'm glad to answer any questions.
for i in range(self.n): obs, act, rew, obs_next, done = agents[i].replay_buffer.sample_index(index) obs_n.append(obs) obs_next_n.append(obs_next) act_n.append(act)
The connector functions are:
def sample_index(self, idxes): return self._encode_sample(idxes) def _encode_sample(self, idxes): obses_t, actions, rewards, obses_tp1, dones = , , , ,  for i in idxes: data = self._storage[i] obs_t, action, reward, obs_tp1, done = data obses_t.append(np.array(obs_t, copy=False)) actions.append(np.array(action, copy=False)) rewards.append(reward) obses_tp1.append(np.array(obs_tp1, copy=False)) dones.append(done) return np.array(obses_t), np.array(actions), np.array(rewards), np.array(obses_tp1), np.array(dones)
self.replay_buffer = ReplayBuffer(1e6)
ReplayBuffer is a class
Better details about the loop can be found here: https://github.com/openai/maddpg/blob/master/maddpg/trainer/maddpg.py#L173