Streaming H264 video from PiCamera to a JavaFX ImageView

Question

I'm currently working on a robotics application where a video feed is being displayed from a Raspberry Pi 3.

I've been working on a way to stream the video directly into JavaFX (the rest of the UI is created in this), however, my knowledge of video streaming is very limited. The goal for the video system is to maintain decent video quality and FPS while reducing latency as much as possible (looking for sub 100 ms). H264 video was chosen as the format for it's speed, but I hear that sending raw video could be faster as there is no compression (could not get raw video to work well at all).

Running my code I am capable of streaming a Pi camera at about 120-130ms of latency and ~48 frames per second. I would like to continue to reduce the latency of this application, and would like to make sure that I'm making decisions for the correct reasons.

The largest issue I have so far is start-up time; it takes about 15-20 seconds for the video to initially launch and catch up to the latest frame.

The following code is an MCVE of the video system. If anyone is interested in reproducing this, you can get it running on a Raspberry Pi (mine is a Raspberry Pi 3) with python-picamera installed. You'll also need a Java Client with JavaCV installed. My version info is org.bytedeco:javacv-platform:1.3.2.

Python side:

We decided to use a Python library to control the video stream because it provides a nice wrapper around the picamera command-line tool. The output from the video is being sent over a TCP connection and will be received by a Java client. (The way we remotely launch this application has been left out of the review because I just wanted this post to focus on the video aspects)

import picamera
import socket
import signal
import sys

with picamera.PiCamera() as camera:
    camera.resolution = (1296, 720)
    camera.framerate = 48

    soc = socket.socket()
    soc.connect((sys.argv[1], int(sys.argv[2])))
    file = soc.makefile('wb')

    try:
        camera.start_recording(
            file,
            format='h264',
            intra_period=0,
            quality=0,
            bitrate=25000000)
        while True:
            signal.pause()
    finally:
        file.close()
        soc.close()

Why did I choose these values?:

• camera.resolution = (1296, 720), camera.framerate = 48 were the largest images I could output at a frame-rate fast enough to reduce latency.
• intra_period=0 Wanted the images to remain small, and by setting the intra_period to zero, no I frames/full frames (apart from the first frame) will be sent; reducing the time between frames
• quality=0 from the docstring: Quality 0 is special and seems to be a "reasonable quality" default
• bitrate=25000000 Wanted to set the bitrate as high as possible to not slow down video transfer when lots of changes in the frames (when P frames/partial frames become large)

Java side:

The Java decoder was written using JavaCV and sends the TCP H264 stream into an FFmpegFrameGrabber. The decoder then converts the Frame into a BufferedImage, and then into a WritableImage for JavaFX.

public class FFmpegFXImageDecoder {
    private FFmpegFXImageDecoder() { }

    public static void streamToImageView(
        final ImageView view,
        final int port,
        final int socketBacklog,
        final String format,
        final double frameRate,
        final int bitrate,
        final String preset,
        final int numBuffers
    ) {
        try (final ServerSocket server = new ServerSocket(port, socketBacklog);
             final Socket clientSocket = server.accept();
             final FrameGrabber grabber = new FFmpegFrameGrabber(
                 clientSocket.getInputStream());
        ) {
            final Java2DFrameConverter converter = new Java2DFrameConverter();
            grabber.setFrameRate(frameRate);
            grabber.setFormat(format);
            grabber.setVideoBitrate(bitrate);
            grabber.setVideoOption("preset", preset);
            grabber.setNumBuffers(numBuffers);
            grabber.start();
            while (!Thread.interrupted()) {
                final Frame frame = grabber.grab();
                if (frame != null) {
                    final BufferedImage bufferedImage = converter.convert(frame);
                    if (bufferedImage != null) {
                        Platform.runLater(() ->
                            view.setImage(SwingFXUtils.toFXImage(bufferedImage, null)));
                    }
                }
            }
        }
        catch (final IOException e) {
            e.printStackTrace();
        }
    }
}

This can then be placed into a JavaFX view like below:

public class TestApplication extends Application {

    static final int WIDTH = 1296;

    static final int HEIGHT = 720;

    @Override
    public void start(final Stage primaryStage) throws Exception {
        final ImageView imageView = new ImageView();
        final BorderPane borderPane = new BorderPane();

        imageView.fitHeightProperty().bind(borderPane.widthProperty()
            .divide(WIDTH).multiply(HEIGHT));
        imageView.fitWidthProperty().bind(borderPane.widthProperty());

        borderPane.setPrefSize(WIDTH, HEIGHT);
        borderPane.setCenter(imageView);

        final Scene scene = new Scene(borderPane);
        primaryStage.setScene(scene);
        primaryStage.show();

        new Thread(() -> FFmpegFXImageDecoder.streamToImageView(
            imageView, 12345, 100, "h264", 96, 25000000, "ultrafast", 0)
        ).start();
    }
}

Why did I choose these values?:

• frameRate=96 Wanted the framerate of the Client to be twice the speed of the stream such that I'm not waiting on frames
• bitrate=25000000 to match the stream
• VideoOption preset="ultrafast" To try and reduce the startup time for the stream.

Final Questions:

What are some ways I improve the latency of this system?

How can I reduce the start-up time of this stream? It currently takes about 15 seconds to launch and catch up.

Are the parameters chosen for JavaCV and PiCamera logical? Is my understanding of them correct?

Answer 1

Architectural Ideas

Let's start at the Architecture of your Application and the data transfer. There's basically two places where we can optimize the performance of your application. I'm ignoring latency for now, since that is mostly determined by the network and the performance of the image processing.

This means if we can improve the speed of image processing, latency will also go down. That's the first component.

The second component is to make the network transfer less vulnerable to overhead and stalling. A great recap of TCP versus UDP is this joelonsoftware article.
Taking the information there into account, I'd think you're better off sending your video over UDP.

Performance review

It's problematic to require a full rendering of an image to update the image-view. Most realtime rendering applications use the idea of a frame-/backbuffer. What happens there is the following:

1. An image is rendered to the backbuffer.
2. When it's finished, the framebuffer and the backbuffer are swapped
3. The next image is rendered to the backbuffer while the framebuffer is displayed.

As far as I can tell, you're missing out on a lot of performance by ineffectively handling how data is passed between network and image-view. It'd help to see what Java2DFrameConverter does exactly.

Code style review

There's a few things that struck me as odd in your code. The following is a review without taking the performance into account directly:

streamToImageView takes a lot of arguments. You can drastically reduce their number by partially applying them outside of the method. Additionally the converter can be a static field, though I can understand if you want to have an instance for every invocation of the method.

This might also be the place for the backbuffer idea, since you can reuse Image instances when rendering. I'm not sure, but you might be able to just set the Image to the imageView once and then reuse the already set instance.

The method then looks like this:

public static void streamImageToView(ImageView view, int port, int socketBacklog, Consumer<Grabber> grabberSettings) {
    try (final ServerSocket server = new ServerSocket(port, socketBacklog);
         final Socket clientSocket = server.accept();
         final FrameGrabber grabber = new FFmpegFrameGrabber(clientSocket.getInputStream());
    ) {
        grabberSettings.accept(grabber);
        grabber.start();
        while (!Thread.interrupted()) {
            final Frame frame = grabber.grab();
            if (frame == null) {
                continue;
            }
            final BufferedImage bufferedImage = converter.convert(frame);
            if (bufferedImage != null) {
                Platform.runLater(() -> {
                    SwingFXUtils.toFXImage(bufferedImage, view.getImage());
                    view.setImage(view.getImage()); 
                    // might not be required. Forces repaint
                });
            }
        } catch (IOException ex) {
            // same as before
        }
    } 
}

I like very much that you're making all the variables final wherever possible. FWIW I leave the partial application of the streamToImageView arguments to you as an exercise ;)