16
\$\begingroup\$

I'm currently working on a robotics application where a video feed is being displayed from a Raspberry Pi 3.

I've been working on a way to stream the video directly into JavaFX (the rest of the UI is created in this), however, my knowledge of video streaming is very limited. The goal for the video system is to maintain decent video quality and FPS while reducing latency as much as possible (looking for sub 100 ms). H264 video was chosen as the format for it's speed, but I hear that sending raw video could be faster as there is no compression (could not get raw video to work well at all).

Running my code I am capable of streaming a Pi camera at about 120-130ms of latency and ~48 frames per second. I would like to continue to reduce the latency of this application, and would like to make sure that I'm making decisions for the correct reasons.

The largest issue I have so far is start-up time; it takes about 15-20 seconds for the video to initially launch and catch up to the latest frame.

JavaFX python video

The following code is an MCVE of the video system. If anyone is interested in reproducing this, you can get it running on a Raspberry Pi (mine is a Raspberry Pi 3) with python-picamera installed. You'll also need a Java Client with JavaCV installed. My version info is org.bytedeco:javacv-platform:1.3.2.

Python side:

We decided to use a Python library to control the video stream because it provides a nice wrapper around the picamera command-line tool. The output from the video is being sent over a TCP connection and will be received by a Java client. (The way we remotely launch this application has been left out of the review because I just wanted this post to focus on the video aspects)

import picamera
import socket
import signal
import sys

with picamera.PiCamera() as camera:
    camera.resolution = (1296, 720)
    camera.framerate = 48

    soc = socket.socket()
    soc.connect((sys.argv[1], int(sys.argv[2])))
    file = soc.makefile('wb')

    try:
        camera.start_recording(
            file,
            format='h264',
            intra_period=0,
            quality=0,
            bitrate=25000000)
        while True:
            signal.pause()
    finally:
        file.close()
        soc.close()

Why did I choose these values?:

  • camera.resolution = (1296, 720), camera.framerate = 48 were the largest images I could output at a frame-rate fast enough to reduce latency.
  • intra_period=0 Wanted the images to remain small, and by setting the intra_period to zero, no I frames/full frames (apart from the first frame) will be sent; reducing the time between frames
  • quality=0 from the docstring: Quality 0 is special and seems to be a "reasonable quality" default
  • bitrate=25000000 Wanted to set the bitrate as high as possible to not slow down video transfer when lots of changes in the frames (when P frames/partial frames become large)

Java side:

The Java decoder was written using JavaCV and sends the TCP H264 stream into an FFmpegFrameGrabber. The decoder then converts the Frame into a BufferedImage, and then into a WritableImage for JavaFX.

public class FFmpegFXImageDecoder {
    private FFmpegFXImageDecoder() { }

    public static void streamToImageView(
        final ImageView view,
        final int port,
        final int socketBacklog,
        final String format,
        final double frameRate,
        final int bitrate,
        final String preset,
        final int numBuffers
    ) {
        try (final ServerSocket server = new ServerSocket(port, socketBacklog);
             final Socket clientSocket = server.accept();
             final FrameGrabber grabber = new FFmpegFrameGrabber(
                 clientSocket.getInputStream());
        ) {
            final Java2DFrameConverter converter = new Java2DFrameConverter();
            grabber.setFrameRate(frameRate);
            grabber.setFormat(format);
            grabber.setVideoBitrate(bitrate);
            grabber.setVideoOption("preset", preset);
            grabber.setNumBuffers(numBuffers);
            grabber.start();
            while (!Thread.interrupted()) {
                final Frame frame = grabber.grab();
                if (frame != null) {
                    final BufferedImage bufferedImage = converter.convert(frame);
                    if (bufferedImage != null) {
                        Platform.runLater(() ->
                            view.setImage(SwingFXUtils.toFXImage(bufferedImage, null)));
                    }
                }
            }
        }
        catch (final IOException e) {
            e.printStackTrace();
        }
    }
}

This can then be placed into a JavaFX view like below:

public class TestApplication extends Application {

    static final int WIDTH = 1296;

    static final int HEIGHT = 720;

    @Override
    public void start(final Stage primaryStage) throws Exception {
        final ImageView imageView = new ImageView();
        final BorderPane borderPane = new BorderPane();

        imageView.fitHeightProperty().bind(borderPane.widthProperty()
            .divide(WIDTH).multiply(HEIGHT));
        imageView.fitWidthProperty().bind(borderPane.widthProperty());

        borderPane.setPrefSize(WIDTH, HEIGHT);
        borderPane.setCenter(imageView);

        final Scene scene = new Scene(borderPane);
        primaryStage.setScene(scene);
        primaryStage.show();

        new Thread(() -> FFmpegFXImageDecoder.streamToImageView(
            imageView, 12345, 100, "h264", 96, 25000000, "ultrafast", 0)
        ).start();
    }
}

Why did I choose these values?:

  • frameRate=96 Wanted the framerate of the Client to be twice the speed of the stream such that I'm not waiting on frames
  • bitrate=25000000 to match the stream
  • VideoOption preset="ultrafast" To try and reduce the startup time for the stream.

Final Questions:

What are some ways I improve the latency of this system?

How can I reduce the start-up time of this stream? It currently takes about 15 seconds to launch and catch up.

Are the parameters chosen for JavaCV and PiCamera logical? Is my understanding of them correct?

\$\endgroup\$
7
  • 1
    \$\begingroup\$ Not a review, but your Python code is good. You might want to use argparse, but the rest of it is great. I also agree with using try except, rather than with, in that use case. \$\endgroup\$
    – Peilonrayz
    May 15, 2017 at 14:08
  • \$\begingroup\$ Did you end up implementing a more finalized version of this? I'm looking to do basically the same thing + some OpenCV toppings, and I would really love to use this as a starting point. \$\endgroup\$
    – JMY1000
    Jan 26, 2019 at 5:28
  • \$\begingroup\$ @JMY1000 I ended up using mostly this configuration, with some small tweaks. Viewing the cameras on a stronger client computer helps drastically, and there are some tradeoffs that can be made for better latency. If you use x264 encoding you will be able to stream in UDP as it can tolerate high packet losses, but you increase CPU usage on the pi, limiting max resolution. This article is really helpful and most of these settings can be configured in the raspi video python wrapper google.com/amp/s/blog.maxofs2d.net/post/171294188718/… \$\endgroup\$
    – flakes
    Jan 26, 2019 at 8:41
  • \$\begingroup\$ Awesome, thanks. Do you have your code posted somewhere I could look? \$\endgroup\$
    – JMY1000
    Jan 26, 2019 at 20:57
  • \$\begingroup\$ @JMY1000, unfortunately no. That code is now part of a closed source project. However, you should be able to get started using the code listed here. Once you have a handle on how the video is getting transmitted from the client to an opencv handler, it wont be difficult to start implementing changes. I will also point out that java doesn't have the nicest api for working with opencv. Re-writing the client in python will make your life far easier. Good luck! \$\endgroup\$
    – flakes
    Jan 26, 2019 at 21:33

1 Answer 1

5
\$\begingroup\$

Architectural Ideas

Let's start at the Architecture of your Application and the data transfer. There's basically two places where we can optimize the performance of your application. I'm ignoring latency for now, since that is mostly determined by the network and the performance of the image processing.

This means if we can improve the speed of image processing, latency will also go down. That's the first component.

The second component is to make the network transfer less vulnerable to overhead and stalling. A great recap of TCP versus UDP is this joelonsoftware article.
Taking the information there into account, I'd think you're better off sending your video over UDP.

Performance review

It's problematic to require a full rendering of an image to update the image-view. Most realtime rendering applications use the idea of a frame-/backbuffer. What happens there is the following:

  1. An image is rendered to the backbuffer.
  2. When it's finished, the framebuffer and the backbuffer are swapped
  3. The next image is rendered to the backbuffer while the framebuffer is displayed.

As far as I can tell, you're missing out on a lot of performance by ineffectively handling how data is passed between network and image-view. It'd help to see what Java2DFrameConverter does exactly.

Code style review

There's a few things that struck me as odd in your code. The following is a review without taking the performance into account directly:

streamToImageView takes a lot of arguments. You can drastically reduce their number by partially applying them outside of the method. Additionally the converter can be a static field, though I can understand if you want to have an instance for every invocation of the method.

This might also be the place for the backbuffer idea, since you can reuse Image instances when rendering. I'm not sure, but you might be able to just set the Image to the imageView once and then reuse the already set instance.

The method then looks like this:

public static void streamImageToView(ImageView view, int port, int socketBacklog, Consumer<Grabber> grabberSettings) {
    try (final ServerSocket server = new ServerSocket(port, socketBacklog);
         final Socket clientSocket = server.accept();
         final FrameGrabber grabber = new FFmpegFrameGrabber(clientSocket.getInputStream());
    ) {
        grabberSettings.accept(grabber);
        grabber.start();
        while (!Thread.interrupted()) {
            final Frame frame = grabber.grab();
            if (frame == null) {
                continue;
            }
            final BufferedImage bufferedImage = converter.convert(frame);
            if (bufferedImage != null) {
                Platform.runLater(() -> {
                    SwingFXUtils.toFXImage(bufferedImage, view.getImage());
                    view.setImage(view.getImage()); 
                    // might not be required. Forces repaint
                });
            }
        } catch (IOException ex) {
            // same as before
        }
    } 
}

I like very much that you're making all the variables final wherever possible. FWIW I leave the partial application of the streamToImageView arguments to you as an exercise ;)

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.