I've been developing a small screen sharing project for the last months, it's a desktop application, pct protocol based. I'm going to show the logic and the code and I hope to get reviews. I would like to hear every suggestion.
First of all, I'm using a service called Desktop Duplication API. Here's the C# implementation.
Using this, allows me a rapid access to the screen buffer and access the dirty regions of the screen(Rectangle which gets updated and changed) so I don't have to handle 2 shots (previous and current bitmaps) and scan them for differences.
After using the desktop API for a while, I noticed that the DirtyRegions aren't exactly pure DirtyRegions. These rectangles contained way more data than they supposed to have. (that's probably related to the desktop refreshing.. I'm not sure why). For example, if I just opened a small video, and few pixels moved in the video sometimes it would even give me the entire video player bounds and even more!
I noticed that it had a massive effect on the network traffic size so I decided to write a small algorithm to find the exact changed bounds for each region itself. (Actually I rely on the DirtyRegions property, I don't have to scan the whole bitmap area, but each region scan is necessary to prevent sending extra unnecessary data.)
After I found a change, I just crop(using the changed rectangle bounds) from the current desktop bitmap (the full one), compress it as jpeg, and send.
This is the Differences finding code:
private unsafe Rectangle GetBoundingBoxForChanges(Bitmap _prevBitmap, Bitmap _newBitmap)
{
BitmapData bmNewData = _newBitmap.LockBits(
new Rectangle(0, 0, _newBitmap.Width, _newBitmap.Height),
ImageLockMode.ReadOnly, _newBitmap.PixelFormat);
BitmapData bmPrevData = _prevBitmap.LockBits(
new Rectangle(0, 0, _prevBitmap.Width, _prevBitmap.Height),
ImageLockMode.ReadOnly, _prevBitmap.PixelFormat);
var strideNew = bmNewData.Width;
var scanNew0 = bmNewData.Scan0;
var scanPrev0 = bmPrevData.Scan0;
foreach (var region in frame.UpdatedRegions)
{
var width = region.Width;
var height = region.Height;
var left = width;
var right = 0;
var top = height;
var bottom = 0;
var pNew = (int*)scanNew0.ToPointer();
var pPrev = (int*)scanPrev0.ToPointer();
for (var y = 0; y < height; ++y)
{
// For pixels up to the current bound (left to right)
//
for (var x = 0; x < left; ++x)
{
// Use pointer arithmetic to index the
// next pixel in this row.
//
if ((pNew + x)[0] != (pPrev + x)[0])
{
// Found a change.
//
if (x < left)
{
left = x;
}
if (x > right)
{
right = x;
}
if (y < top)
{
top = y;
}
if (y > bottom)
{
bottom = y;
}
}
}
// Move the pointers to the next row.
//
pNew += strideNew;
pPrev += strideNew;
}
// Second Pass - The first pass found at
// least one different pixel and has set
// the left & top bounds. In addition, the
// right & bottom bounds have been initialized.
// Adapt the number of pixels scanned from right
// to left so we only scan up to the current bound.
// In addition, there is no need to scan past
// the top bound.
//
// Set the pointers to the first element of the
// bottom row.
//
pNew = (int*)(void*)scanNew0;
pPrev = (int*)(void*)scanPrev0;
pNew += (_newBitmap.Height - 1) * strideNew;
pPrev += (_prevBitmap.Height - 1) * strideNew;
// For each row (bottom to top)
//
for (int y = _newBitmap.Height - 1; y > top; y--)
{
// For each column (right to left)
//
for (int x = _newBitmap.Width - 1; x > right; x--)
{
// Use pointer arithmetic to index the
// next pixel in this row.
//
if ((pNew + x)[0] != (pPrev + x)[0])
{
// Found a change.
//
if (x > right)
{
right = x;
}
if (y > bottom)
{
bottom = y;
}
}
pNew -= strideNew;
pPrev -= strideNew;
}
}
var diffImgWidth = right - left + 1;
var diffImgHeight = bottom - top + 1;
if (diffImgHeight < 0 || diffImgWidth < 0)
{
// Nothing changed
continue;
}
// Return the bounding box.
//
_newBitmap.UnlockBits(bmNewData);
_prevBitmap.UnlockBits(bmPrevData);
Rectangle rect= new Rectangle(left, top, diffImgWidth, diffImgHeight);
return rect;
}
In the second side, the client receives these blocks and merges them on the initial big bitmap -- and this is how the updated screen displayed.
I did many benchamrks, for the differences processing part, the .jpeg compressing, the processing of the bitmap in the client side(and the drawing to the screen part), all these operation were extremely quick, running this project in my LAN network was very very fast.
This all lead me to one simple conclusion, the main bottleneck here is the network part. When there's no massive change on the screen there won't even be a network traffic, but when there are big changes on the screen, each frame size (after the jpeg compression) could get to 150Kb!
I'm looking for more ways or techniques to improve the efficiency of the project since it's kind of a real time project, it has to work quite fast.
Things I already do in order to achieve this:
- Reusing buffer and bitmap object in purpose of prevent their allocation.
- Disabling Windows Aero effects in the server screen (which gets captured).
- Scanning for differences and sending them only.
I'm looking for ways to reduce the network traffic (mostly).
I've been suggested to move to UDP and didn't actually understand why and how (is it promised faster transmission of the data? and doesn't it have a buffer size limit? last time I tried I was unable to send larger buffer than 65k).
I'm not about to reinvent the wheel or something. This is only a small project I develop for myself.
