# Experience replay memory

I have just begun learning Julia. Here is an implementation of experience replay memory for use in a reinforcement learning algorithm. It is pretty simple, essentially a ring buffer with the following requirements:

• Used to store 1D arrays of numbers, typically Float32 or Float64. All the stored arrays are the same size.
• Has a maximum capacity, after which new entries overwrite old ones
• Has a sample function for retrieving a given number of entries

import Base.length

struct Memory{T <: Real}
max_size::UInt32
experiences::Vector{Vector{T}}
end

Memory{T}(max_size) where {T <: Real} = Memory{T}(max_size, Vector{Vector{T}}())

length(memory::Memory) = length(memory.experiences)

function remember!(memory::Memory, experience)
size = length(memory)
if size == memory.max_size
memory.experiences[1 + size % memory.max_size] = experience
else
push!(memory.experiences, experience)
end
end

function sample(memory::Memory{T}, count::Integer) where {T <: Real}
size = length(memory)
@assert count <= size
return [memory.experiences[1 + rand(UInt32) % size] for i in 1:count]
end


• There's a CircularBuffer type in DataStructures, which does pretty much the same. Have a look at its code for some inspiration. Or reuse it, if it fits your needs and you don't hesitate to add that dependency.
• Regarding the point before, I'd especially recommend to implement the common interfaces, and probably iterate as well. remember! makes sense semantically, but push! is the standard name for this functionality. You could implement remember! in terms of push! and export both, or just use push! and mention it in the docs.
• There isn't really a need to be so specific about types. I'd just use

struct Memory{T}
max_size::UInt32
experiences::Vector{T}
end


Nothing is lost by generalizing in this way, since you never use information about the content. But who knows, maybe you later want switch to different types for experiences, eg. StaticVectors.

• I would doubt whether using UInt32 saves more than it complicates. Int is pretty much standard for everything of this kind: lengths, indices, offsets, etc.

• Depending on the sizes of your use case, it might be better to preallocate experiences as Vector{T}(undef, max_size) and use just indexing (and keeping track of the actual length), instead of push!ing until full -- see the DataStructures implementation. If not, calling sizehint!(experiences, max_size) might be good, if you expect to always exhaust the full capacity.
• Instead of providing sample, I'd recommend hooking into rand, thereby getting more methods for free. In the simplest case, this just consists of defining

Random.rand(rng::Random.AbstractRNG, s::Random.SamplerTrivial{Memory{T}}) where {T} = rand(rng, s[].experiences)


which gives you all variations: rand(m), rand(m, N) (corresponging to your sample), rand(rng, m), etc. Especially the form with a provided RNG is relevant if you want to write reproducible experiments. The rand! methods seem not to work by default, but can be added easily if you need them. (If you switch to a preallocation + indexing solution, you need to make sure to sample only from @view experiences[1:length].)

And note that you can directly sample from an array a using rand(a).