# Updating WPF image from camera frame grabber thread

I have created a WPF application in .NET 4.6.1. to display video from the frame grabbing event for a machine vision camera.

In XAML there is an image control bound to a WriteableBitmap

<Image x:Name="CameraImage" Grid.Column="1" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Source="{Binding CameraImage}"/>


In the View Model:

public event PropertyChangedEventHandler PropertyChanged;
protected void NotifyPropertyChanged([CallerMemberName] string propertyName = null)
{
//Console.WriteLine(propertyName);
PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
}

private WriteableBitmap cameraImage;
private IntPtr cameraImagePtr;
public WriteableBitmap CameraImage
{
get { return cameraImage; }
set
{
cameraImage = value;
cameraImagePtr = cameraImage.BackBuffer;
NotifyPropertyChanged();
}
}


The WriteableBitmap is initialised as so:

CameraImage = new WriteableBitmap(2448, 2048, 0, 0, PixelFormats.Bgr24, null);


In the grab event generated by the camera we have:

// Occurs when an image has been acquired and is ready to be processed.
private void OnImageGrabbed(Object sender, ImageGrabbedEventArgs e)
{
try
{
// Get the grab result.
IGrabResult grabResult = e.GrabResult;

// Check if the image can be displayed.
if (grabResult.IsValid)
{
byte[] buffer = grabResult.PixelData as byte[];

converter.OutputPixelFormat = PixelType.BGR8packed;
lock(cameraImage)
{
// Converts camera Bayer array image to BGR image and saves at cameraImagePtr
converter.Convert(cameraImagePtr, 2448 * 2048 * 3, grabResult);
}

// Make WriteableBitmap dirty (on UI thread) so that the WPF Image control redraws (parent is UI object)
parent.Dispatcher.Invoke(new Action(() =>
{
cameraImage.Lock();
cameraImage.Unlock();
}), DispatcherPriority.Render);
}
}
catch (Exception exception)
{
Console.WriteLine(exception.Message);
throw exception;
}
finally
{
// Dispose the grab result if needed for returning it to the grab loop.
e.DisposeGrabResultIfClone();
}
}


converter is Basler.Pylon.PixelDataConverter. It converts the grabbed frame which is in Bayer array format into a normal BRG image.

The lock is in place because there is another routine to save the current image, and I don't want to do this while it is being updated:

    public void SaveImage()
{
if (cameraImage == null) return;
lock(cameraImage)
{
String filename = folder + "\\" + stem + "-" + set.ToString("D2") + "-" + subset.ToString("D2") + ".png";
if (filename != string.Empty)
{
using (FileStream stream = new FileStream(filename, FileMode.Create))
{
PngBitmapEncoder encoder = new PngBitmapEncoder();
encoder.Save(stream);
}
}
}
}


Questions

• I can't just pass cameraImage.BackBuffer to the writing (Converter.convert()) function, I have to do cameraImagePtr = cameraImage.BackBuffer; and then use cameraImagePtr. Why is that?

Answer: Because another thread owns the cameraImage object

• The fans on my laptop spin up when this is running - it seems like it is using a lot of CPU. Is there a more efficient way? From what I can tell I am not making extra copies of the frame.

Answer: On another PC the CPU is only 18%, so not sure why the laptop fan is going so hard.

Update

The camera images are quite large - 2448 x 2048 x 3 bytes.

As is, I get 20 fps (camera true fps is 30) with CPU usage of 21%. There is frame tearing occuring.

I did some changes to the code with the following results:

1. Moved converter.Convert(...) into the Invoke block. FPS dropped to 14 and the UI was a tiny bit more unresponsive. Frame tearing stopped.

2. Removed the lock. No difference.

3. Create a temporary byte array and pass it to converter.Convert(...) instead of cameraImagePtr, then inside the Invoke block use 'cameraImage.WritePixels(...)' to copy to the WriteableBitmap on the UI thread. No tearing, but fps dropped to 14.

4. Same as above, but using BeginInvoke instead. Fps back up to 20 but no tearing, and now the lock can be removed!

So it seems option 4 is the best, the code is now:

if (grabResult.IsValid)
{
converter.OutputPixelFormat = PixelType.BGR8packed;
byte[] bgrBuffer = new byte[2448 * 2048 * 3];
converter.Convert(bgrBuffer, grabResult);

parent.Dispatcher.BeginInvoke(new Action(() =>
{
cameraImage.Lock();
cameraImage.WritePixels(new Int32Rect(0, 0, 2448, 2048), bgrBuffer, 2448 * 3, 0);
cameraImage.Unlock();
}), DispatcherPriority.Render);
frames++;
}

• What you mean by saying "I can't just pass"? What error is shown? Also, can you show what is that converter? – Aram Kocharyan Dec 26 '17 at 9:46
• @AramKocharyan The error is "The calling thread cannot access this object because a different thread owns it." Converter writes an image to an IntPtr or a byte array. – geometrikal Dec 26 '17 at 13:26

1) Are you sure that it is safe to write to cameraImagePtr from background thread the way you do it? Because documentation clearly states that it is not:

Update the back buffer only between calls to the Lock and Unlock methods. If you do not follow the Lock/Unlock workflow described in the WriteableBitmap class remarks, undefined behaviors, such as tearing, can occur.

I suggest you convert your image to separate byte[] buffer and then use WriteableBitmap.WritePixels in-between lock/unlock to block-copy the pixel data.

2) The way you use lock statement looks fishy. What exactly are you locking and why? If events come on different threads and you can't handle them fast enough, then you have to come up with a throttling mechanism, that would drop some of the camera frames. Otherwise events will just keep piling up on this lock in no particular order and eat up your CPU (which is what you observe).

3) Depending on what else is going on in your app, Dispatcher.Invoke call can be pretty expensive performance-wise. Dispatcher.BeginInvoke on the other hand is much cheaper, but you will have to be really careful with your buffers, so that background threads do not overwrite the buffer, that is currently being rendered on UI thread.

• Thanks, I'm on my way into work at the moment and will respond shortly. – geometrikal Dec 26 '17 at 10:24
• 1) Yes, there is tearing. I did some experiments with different methods based on your comments, and 'converting' the image into a temporary buffer then using WriteableBitmap.WritePixels inside a BeginInvoke block instead of Invoke was just as fast but with no tearing. (See question edits) – geometrikal Dec 26 '17 at 13:58
• 2) I was locking because there is another routine that saves the WriteableBitmap to disk if a button is clicked. With the changes you proposed, it is no longer needed as the BackBuffer is written on the same thread. – geometrikal Dec 26 '17 at 13:59
• 3) BeginInvoke was indeed faster with the temporary buffer method. If I just put converter.Convert in there I get an error (from the camera library) that the image is invalid, presumable because grabResult is being accessed asynchronously from the UI thread. – geometrikal Dec 26 '17 at 14:02
• Now the next question, posted above, is that conversion is CPU heavy and so fps is less than it should be. (I think the camera doesn't send frame events if the previous one is not finished). What is a good method to offload the conversion? ConcurrentQueue and Task? – geometrikal Dec 26 '17 at 14:04