No Reference Image Quality Assessment for Video in Matlab

Question

I am trying to perform the No-Reference Image Quality Assessment (NR-IQA) calculation on video. Several NR-IQA methods including Naturalness Image Quality Evaluator (NIQE), Perception based Image Quality Evaluator (PIQE) and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) are used and the calculation results are both show on video frame and export to an Excel file. The file BigBuckBunny is used as the input experimental video.

The experimental implementation

%% Read and process a video into MATLAB
%   Reference: https://www.mathworks.com/solutions/image-video-processing/video-processing.html

tic
% Setup: create Video Reader and Writer
videoFilename = ['BigBuckBunny_320x180'];
videoPath = ['.'];

videoFullFilename = fullfile(videoPath, [videoFilename '.mp4']);
videoFileReader = VideoReader(videoFullFilename);
% Reference: https://www.mathworks.com/matlabcentral/answers/123189-get-the-number-of-frame-of-a-video-using-vision-videofilereader-and-videoreader
TotalFrameCount = floor(videoFileReader.Duration * videoFileReader.FrameRate);

% Setup: create output folder
OutputFolderRoot = 'NIQE_PIQE_BRISQUE';
if ~exist(OutputFolderRoot, 'dir')
    mkdir(OutputFolderRoot);
end
outputFilename = fullfile(OutputFolderRoot, [videoFilename '.avi']);
outputVideo = VideoWriter(outputFilename);

% Setup: create data record variables
% Reference: https://www.mathworks.com/help/matlab/ref/round.html
NIQE_results = zeros(1, round(TotalFrameCount, 0));
PIQE_results = zeros(1, round(TotalFrameCount, 0));
BRISQUE_results = zeros(1, round(TotalFrameCount, 0));

FrameIndex = 1;

% Setup: create deployable video player
depVideoPlayer = vision.DeployableVideoPlayer;

open(outputVideo);

%% Detect faces in each frame
while hasFrame(videoFileReader)
    % read video frame
    videoFrame = readFrame(videoFileReader);

    [videoFrame, NIQE_results, PIQE_results, BRISQUE_results] = processEachFrame(videoFrame, FrameIndex, NIQE_results, PIQE_results, BRISQUE_results);

    % Display video frame to screen
    depVideoPlayer(videoFrame);

    % Write frame to final video file
    writeVideo(outputVideo, videoFrame);
    
    % Show progress
    fprintf('The calculation of frame %d is finished!\n', FrameIndex);
    
    FrameIndex = FrameIndex + 1;
end
close(outputVideo);
depVideoPlayer.delete();
ElapsedTime = toc;

% Write calculated results
% Reference: https://www.mathworks.com/help/matlab/ref/fprintf.html
fileID = fopen(fullfile(OutputFolderRoot, [videoFilename '_ElapsedTime.txt']),'w');
nbytes = fprintf(fileID,'Elapsed time is %f seconds.\n', ElapsedTime);

output_filename = fullfile(OutputFolderRoot, [videoFilename '_Results.xlsx']);
% Write title
output_title = [{'Frame Index'} {'NIQE'} {'PIQE'} {'BRISQUE'}];
writecell(output_title, output_filename, 'Sheet', 1, 'Range', 'A1');
index_column = 1:TotalFrameCount;
output_data = [index_column' NIQE_results' PIQE_results' BRISQUE_results'];
writematrix(output_data, output_filename, 'Sheet', 1, 'Range', 'A2');

function [output, NIQE_results, PIQE_results, BRISQUE_results] = processEachFrame(input, FrameIndex, NIQE_results, PIQE_results, BRISQUE_results)
    output = input;
    NIQE_score = niqe(input);
    PIQE_score = piqe(input);
    BRISQUE_score = brisque(input);
    NIQE_position = [10 10];
    PIQE_position = [10 40];
    BRISQUE_position = [10 70];
    output = insertText(output, NIQE_position, ['NIQE_score: ' num2str(NIQE_score)], 'BoxOpacity', 0.6);
    output = insertText(output, PIQE_position, ['PIQE_score: ' num2str(PIQE_score)], 'BoxOpacity', 0.6);
    output = insertText(output, BRISQUE_position, ['BRISQUE_score: ' num2str(BRISQUE_score)], 'BoxOpacity', 0.6);
    NIQE_results(FrameIndex) = NIQE_score;
    PIQE_results(FrameIndex) = PIQE_score;
    BRISQUE_results(FrameIndex) = BRISQUE_score;
end

All suggestions are welcome.

Reference:

1. Big Buck Bunny: https://peach.blender.org/

Answer 1

MATLAB Editor warnings

On these two lines, the MATLAB Editor gives a warning "Use of brackets [] is unnecessary". This is because they're not concatenating anything, there's only one thing in each pair.

videoFilename = ['BigBuckBunny_320x180'];
videoPath = ['.'];

Instead write

videoFilename = 'BigBuckBunny_320x180';
videoPath = '.';

Redundant rounding

TotalFrameCount is computed as floor(...), then is passed through round before use, which obviously does nothing:

TotalFrameCount = floor(videoFileReader.Duration * videoFileReader.FrameRate);
% ...
NIQE_results = zeros(1, round(TotalFrameCount, 0));
PIQE_results = zeros(1, round(TotalFrameCount, 0));
BRISQUE_results = zeros(1, round(TotalFrameCount, 0));

Consider using modern string arrays

If you don't need your code to run on Octave, you should consider using strings instead of char vectors. Some things become easier with them. For example, instead of [videoFilename '_ElapsedTime.txt'] you would write videoFilename + "_ElapsedTime.txt", and instead of ['NIQE_score: ' num2str(NIQE_score)] you would write "NIQE_score: " + NIQE_score.

Cell array construction

Instead of [{'Frame Index'} {'NIQE'} {'PIQE'} {'BRISQUE'}], write {'Frame Index', 'NIQE', 'PIQE', 'BRISQUE'} (the commas are not necessary, but I always prefer making them explicit). Or, if using string arrays, write ["Frame Index", "NIQE", "PIQE", "BRISQUE"].

Transpose vs complex-conjugate transpose

The ' operator applies a complex-conjugate transpose. If the array it is applied to is real-valued, then it's just a transpose, but it's best to explicitly use the transpose operator, .'. Getting used to picking the right one will eventually prevent some very difficult to discover bugs (there are some of these you can find on Stack Overflow).

Hard-coded score names

You have variable names containing the names of the three scores you compute. I would consider these names data, rather than code. To change this, you could simplify code a bit. For example, instead of

NIQE_results = zeros(1, round(TotalFrameCount, 0));
PIQE_results = zeros(1, round(TotalFrameCount, 0));
BRISQUE_results = zeros(1, round(TotalFrameCount, 0));

you could do

results = zeros(TotalFrameCount, ScoreCount);

This simplifies processEachFrame() as well, you wouldn't need to pass these three arrays as both input and output, it's just one array.

Later you concatenate them anyway:

output_data = [index_column' NIQE_results' PIQE_results' BRISQUE_results'];

This would become

output_data = [index_column' results];

(note I created the results array in the right orientation to avoid the transpose).

You also pass FrameIndex into processEachFrame(), just so you can write the results to the right location of the three results arrays. Instead, just have the function return the values, and have the calling code write those values to the right location:

[videoFrame, results(FrameIndex,:)] = processEachFrame(videoFrame);

Avoid repetitive code, use a loop instead

If you think of the scores being computed as data rather than code, it becomes easier to avoid repetitive code and use a loop instead. Your function processEachFrame() could be something like this:

function [frame, results] = processEachFrame(frame)
    scores = struct('name', {'NIQE', 'PIQE', 'BRISQUE'}, ...
                    'function', {@niqe, @piqe, @brisque}, ...
                    'position', {[10 10], [10 40], [10 70]});
    n_scores = numel(scores);
    results = zeros(1, n_scores);
    for ii = 1:n_scores
        results(ii) = scores(ii).function(frame);
        frame = insertText(frame, scores(ii).position, [scores(ii).name, '_score: ', num2str(results(ii))], 'BoxOpacity', 0.6);
    end
end

Not only is this shorter and simpler, but it makes it really easy to add a score later on, there won't be a need to change the code at all. scores should probably be defined at the top of the script, outside the function.

Note that if we name the input and output variable the same (here frame), and when calling the function we use the same variable as both input and output (videoFrame in your code), then MATLAB will optimize and not duplicate the array. See Loren's blog post.

Display of progress

I can imagine that printing a line of text for every frame processed will quickly drown the command window in nearly identical lines of text. You could try one of the many progress bars posted on the File Exchange. For example, I just found this one that works within the Command Window (haven't tried it myself). But even just printing the frame number (without a newline) would be an easy and convenient approach.

Comments

You have helpful comments, for the most part, indicating what each section of code does. This makes it easy to find relevant code without having to read the whole script. However, one comment seems copy-pasted from different code, and describes something you don't do:

%% Detect faces in each frame

Wrong comments are more harmful than no comments.

Variable names

Your variable names are clear and descriptive, and you're not afraid of long names. This is good.

I would suggest you pick one style (snake_case, CamelCase or lowerCamelCase). Mixing all styles makes it look like code was composed by copy-pasting from multiple sources.

Answer 2

The Issue of Frame Rate

In this experimental implementation, the part of frame rate of outputVideo may different from the input video. After checking the document of VideoWriter, there is a parameter FrameRate and its default value is 30. It might be better to set this parameter to make the frame rate of outputVideo match the input video with the following way.

outputVideo.FrameRate = videoFileReader.FrameRate;

Note: This setting should be put before open operation.