I need to implement a Ring Buffer/FIFO for data coming from a TCP socket.
It must support the following operations:
- Append a the recv()'ed chunk of bytes.
- Allow me to peek at the beginning of the buffer, since i get differently-sized packets, and I must decode a small fixed-size header to know how many bytes to process.
- Remove a chunk of bytes from the beginning of the buffer for processing.
I guess my needs are pretty standard for a TCP-based streaming protocol, but surprisingly I have never found a "best practice" method for doing this.
There are similar questions on SO already, most suggest to use collections.deque
, which is fine but has some shortcomings that must be worked around for my needs:
- It doesn't easily allow peeking.
- It doesn't allow removal of chunks of bytes.
Mixing various suggestions I came up with the following implementation, which seems working but I wonder: can I do any better, performance-wise? Removing a single byte at a time in get()
doesn't look optimal at all.
import collections
import itertools
class RingBuffer (object):
"""Ring buffer"""
def __init__ (self, size = 4096):
self._buf = collections.deque (maxlen = size)
def put (self, data):
"""Adds data to the end of the buffer"""
self._buf.extend (data)
def get (self, size):
"""Retrieves data from the beginning of the buffer"""
data = str ()
for i in xrange (size):
data += self._buf.popleft ()
return data
def peek (self, size):
"""\"Peeks\" at the beginning of the buffer (i.e.: retrieves data without removing them from the buffer)"""
return str (bytearray (itertools.islice (self._buf, size)))
def len (self):
"""Returns the length of the buffer"""
return len (self._buf)
If you are wondering why I am returning a string from peek()
, that is because I need to process its return value with struct.unpack_from()
.
StringIO
instance as you buffer would enable easier peeking and getting chunks of bytes. \$\endgroup\$ – Justin Fay Mar 27 '15 at 10:01