UDP image livestream from Android device to C# desktop application

Question

After searching a lot, on how to do it and not finding any good solutions, I implemented my own UDP livestream from an Android device to a C#/WPF desktop application. It works, however, since I get complete JPEG images from the camera2 API each frame in android, I need to wait until I have data for the complete image before I can display it (instead of updating image areas). All in all, slicing up the images and putting them together on the other end is a pain. I ended up buffering up to 4 images, always discarding the oldest one as new ones come along. I still have to implement connection losses and so on...

Anyway, here is the Android side (kotlin):

// gets called as soon as a new image is available
//livestreamSocket is a DatagramSocket, connection is basically set up at this point.
fun sendLivestreamImage(img: Image) { 
    if(livestreamSocket.isClosed) return

    /** Protocol: [8bytes timestamp][4byte imagesize][4bytes startindex][4bytes payloadlength][x bytes payload] */

    var buffer = img.planes[0].buffer
    var imgSize = buffer.remaining()

    var payloadSize = liveStreamPacketSize - livestreamHeaderSize

    var numbPackets = (imgSize / payloadSize) + 1

    //Log.d(TAG, "size: ${imgSize}, numbPackets: ${numbPackets}")

    for(i in 1..numbPackets){

        if(i < numbPackets){
            var bytes = ByteArray(liveStreamPacketSize)
            var startIndex = buffer.position()
            var headbuffer = ByteBuffer.allocate(livestreamHeaderSize).putLong(img.timestamp).putInt(8, imgSize).putInt(12, startIndex).putInt(16, payloadSize)
            headbuffer.rewind()
            headbuffer.get(bytes, 0, livestreamHeaderSize)
            buffer.get(bytes, livestreamHeaderSize, payloadSize)
            var packet = DatagramPacket(bytes, bytes.size, udpLivestreamAddress, udpLivestreamPort)
            livestreamSocket.send(packet)
        }
        else {
            payloadSize = buffer.remaining()
            var startIndex = buffer.position()
            var bytes = ByteArray(payloadSize + livestreamHeaderSize)
            var headbuffer = ByteBuffer.allocate(livestreamHeaderSize).putLong(img.timestamp).putInt(8, imgSize).putInt(12, startIndex).putInt(16, payloadSize)
            headbuffer.rewind()
            headbuffer.get(bytes, 0, livestreamHeaderSize)
            buffer.get(bytes, livestreamHeaderSize, payloadSize)
            var packet = DatagramPacket(bytes, bytes.size, udpLivestreamAddress, udpLivestreamPort)
            livestreamSocket.send(packet)
        }
    }
}

On the C# side I have the following function listening to the UDP connection:

/// <summary>
/// waits and receives livestream images
/// </summary>
public async Task ReceiveLivestream(IProgress<BitmapImage> imageReceivedProgress)
{

    var udpEndpoint = new IPEndPoint(IPAddress.Any, GenDefInt.UdpLiveStreamPort);
    liveStreamReceiver = new UdpClient(udpEndpoint);
    var imgBuilder = new LivestreamImageBuilder();

    while (true)
    {
        var recv = await liveStreamReceiver.ReceiveAsync();

        long timestamp = IPAddress.NetworkToHostOrder(BitConverter.ToInt64(recv.Buffer, 0));
        int imageSize = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(recv.Buffer, 8));
        int startIndex = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(recv.Buffer, 12));
        int payloadLength = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(recv.Buffer, 16));

            var img = imgBuilder.CreateImageFromUdpPackets(timestamp, imageSize, startIndex, payloadLength, recv.Buffer);
            if (img == null) continue;
            else imageReceivedProgress.Report(img); //reports the updated image back to the viewmodel of the application, where it will update the GUI
    }
}

The important parts are happening in the LivestreamImageBuilder class:

public class LivestreamImageBuilder
{
    #region Private Fields
    private byte[][] imageBuffers = new byte[4][];
    private int[] copiedBytes = new int[4];
    private long[] timeStamps = new long[4];
    #endregion

    #region Public Properties
    #endregion

    #region Constructors
    #endregion

    #region Private Methods
    /// <summary>
    /// returns a BitmapImage from a full imagebuffer bytearray
    /// </summary>
    /// <param name="imgBuffer"></param>
    /// <returns></returns>
    private BitmapImage GetImageFromBuffer(byte[] imgBuffer)
    {

        var img = new BitmapImage();
        using (MemoryStream ms = new MemoryStream(imgBuffer))
        {
            img.BeginInit();
            img.CacheOption = BitmapCacheOption.OnLoad;
            img.StreamSource = ms;
            img.EndInit();
        }
        return img;
    }
    #endregion

    #region Public Methods
    /// <summary>
    /// Manages and fills up to four image buffers with data from the latest udp packet
    /// if the respective image buffer is full, returns a BitmapImage, else returns null
    /// </summary>
    /// <param name="timeStamp"></param>
    /// <param name="imageSize"></param>
    /// <param name="startIndex"></param>
    /// <param name="payloadLength"></param>
    /// <param name="buffer"></param>
    /// <returns></returns>
    public BitmapImage CreateImageFromUdpPackets(long timeStamp, int imageSize, int startIndex, int payloadLength, byte[] buffer)
    {
        for (uint i = 0; i < 4; i++)
        {
            if (timeStamp == timeStamps[i])
            {
                Buffer.BlockCopy(buffer, 20, imageBuffers[i], startIndex, payloadLength);
                copiedBytes[i] += payloadLength;
                if (copiedBytes[i] >= imageSize)
                {
                    return GetImageFromBuffer(imageBuffers[i]);
                }
                else return null;
            }
        }

        //find oldest buffer
        int oldest = 0;
        for (int i = 0; i < timeStamps.Length; i++)
        {
            if (timeStamps[i] < timeStamps[oldest])
            {
                oldest = i;
            }
        }

        timeStamps[oldest] = timeStamp;
        imageBuffers[oldest] = new byte[imageSize];
        copiedBytes[oldest] = 0;
        Buffer.BlockCopy(buffer, 20, imageBuffers[oldest], startIndex, payloadLength);
        copiedBytes[oldest] += payloadLength;
        if (copiedBytes[oldest] >= imageSize)
        {
            return GetImageFromBuffer(imageBuffers[oldest]);
       }
        else return null;

    }
    #endregion
}

Again, it works. I just found it weird that I couldn't find solutions to this (I'm sure, somebody has already solved this in a better, more efficient way), so I would appreciate all kinds of suggestions for improvements, errors I overlooked or any other kind of input :)

Obviously, If there are questions concerning the code, please just ask away -> I'm a bit lazy with comments^^

Answer 1

I have no experience with Kotlin, so I'll purely be focusing on the C# part

---

Empty #regions:
Delete them, they just take up unnecessary space. In the same vein, I personally add an empty line after the #region and just before endregion, makes it more readable (IMO).

Naming:
Microsoft's own naming conventions aren't complete, so I follow what everyone else, and even Microsoft themselves does. Namely: Prefix private members in a class with an underscore. This turns your LivestreamImageBuilder member declarations into this:

private byte[][] _imageBuffers = new byte[4][];
private int[] _copiedBytes = new int[4];
private long[] _timeStamps = new long[4];

This then allows you to use the proper name as the parameter in GetImageFromBuffer, i.e

private BitmapImage GetImageFromBuffer(byte[] imageBuffer)

Even though MS's naming conventions are sparce, they tell us:

DO NOT use abbreviations or contractions as part of identifier names.

So it depends on how pedantic you want to be. Is img an abbreviation? Yes, does basically everyone understand that it means image? Also yes, but you should still technically not call it img.

Documentation:
You are using XMLDocs, but their not quite complete. For example, you have zero documentation concerning parameters which I think is quite important. Especially when they could be confusing, like long timeStamp. Why is it a long and not a DateTime? What are acceptable values? or is anything not acceptable? Also there is no documentation for what a method can return. In your CreateImageFromUdpPackets method I think it's especially important, because it can return null, a simple

Returns the newest Image received, if none are completely received returns null

would suffice.

Comments:
I know, I know, they're a PITA, but important. Some code blocks are quite difficult to understand, for example:

Buffer.BlockCopy(buffer, 20, imageBuffers[i], startIndex, payloadLength);
copiedBytes[i] += payloadLength;
if (copiedBytes[i] >= imageSize)
{
    return GetImageFromBuffer(imageBuffers[i]);
}
// ...

Could use a comment, something like

Copies newly received image into the local buffer, and returns the image if it's complete

For loop:
Currently you're iterating through your timeStamps in a for loop with a fixed max, this should be replaced by this:

for (int i = 0; i < timeStamps.Length; i++)
{
    if (timeStamp == timeStamps[i])
   // ...
}

Because what if you decide you want to keep 5 images around? Or 3? Or however many.

If else statements:
You have multiple unnecessary else statements, they should be removed. So turn

if (copiedBytes[i] >= imageSize)
{
    return GetImageFromBuffer(imageBuffers[i]);
}
else return null;

Into

if (copiedBytes[i] >= imageSize)
    return GetImageFromBuffer(imageBuffers[i]) // The curly braces can optionally be left out

return null;

Custom network data class:
Why not use a custom network data class instead of passing 5 parameters to your CreateImageFromUdpPackets? You could even implement methods on it construct it directly from a byte array. Like so:

public class LivestreamImagePacket
{
    public long Timestamp {get; set;}

    public int ImageSize {get; set;}

    public int StartIndex {get; set;}

    public int PayloadLength {get; set;}

    public LiveStreamImagePacket(long timestamp, int imageSize, int startIndex, int payloadLength)
    {
        Timestamp = timestamp;
        ImageSize = imageSize;
        StartIndex = startIndex;
        PayloadLength = payloadLength;
    }

    public static LivestreamImagePacket FromBytes(byte[] bytes)
    {
        long timestamp = IPAddress.NetworkToHostOrder(BitConverter.ToInt64(recv.Buffer, 0));
        int imageSize = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(recv.Buffer, 8));
        int startIndex = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(recv.Buffer, 12));
        int payloadLength = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(recv.Buffer, 16));

        return new LivestreamImagePacket(timestamp, imageSize, startIndex, payloadLength);
    }
}

Then you can use it like so:

while (true)
{
    var recv = await liveStreamReceiver.ReceiveAsync();
    var packet = LivestreamImagePacket.FromBytes(recv.Buffer);
    var img = imgBuilder.CreateImageFromUdpPackets(packet, recv.Buffer);
}

Of course you'd need to change the CreateImageFromUdpPackets signature to this CreateImageFromUdpPackets(LivestreamImagePacket packet, byte[] buffer) and replace the usages of timestamp with packet.Timestamp