# C++ multiple object recognition

I am working on a multi-object recognition program. I have succeeded recognising two objects. However, the speed of my program is really slow and laggy. Can somebody please tell a way to speed the program up?

So far, I have seen that ORB is quite fast at matching features. However, my program is still too slow. I have also noticed that there are three for loops, possibly making the program slow down. Interestingly on YouTube, I have seen videos where the output is really smooth.

Is there a way to fix this? Thanks.

This is my code:

#include <stdio.h>
#include <iostream>
#include "opencv2/core/core.hpp"
#include "opencv2/features2d/features2d.hpp"
#include "opencv2/nonfree/features2d.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/calib3d/calib3d.hpp"

using namespace std;
using namespace cv;

FlannBasedMatcher matcher(new flann::LshIndexParams(20, 10, 2));

void ttea(vector<KeyPoint> bwImgKP, Mat objectDS, Mat bwImgDS, vector<KeyPoint> objectKP, vector<Point2f> objectCRN, Mat colorImg, string text)
{
vector<vector<DMatch>> matches;
vector<DMatch> matchesGD;
vector<Point2f> obj;
vector<Point2f> scene;
vector<Point2f> sceneCRN(4);
Mat H;

if (bwImgKP.empty() || objectDS.empty() || bwImgDS.empty())
{
return;
}

matcher.knnMatch(objectDS, bwImgDS, matches, 2);

for (int i = 0; i < min(bwImgDS.rows - 1, (int)matches.size()); i++)
{
if ((matches[i][0].distance < 0.6*(matches[i][1].distance)) && ((int)matches[i].size() <= 2 && (int)matches[i].size() > 0))
{
matchesGD.push_back(matches[i][0]);
}
}

if (matchesGD.size() >= 4)
{
for (int i = 0; i < matchesGD.size(); i++)
{
obj.push_back(objectKP[matchesGD[i].queryIdx].pt);
scene.push_back(bwImgKP[matchesGD[i].trainIdx].pt);
}

H = findHomography(obj, scene, CV_RANSAC);

perspectiveTransform(objectCRN, sceneCRN, H);

line(colorImg, sceneCRN[0], sceneCRN[1], Scalar(255, 0, 0), 4);
line(colorImg, sceneCRN[1], sceneCRN[2], Scalar(255, 0, 0), 4);
line(colorImg, sceneCRN[2], sceneCRN[3], Scalar(255, 0, 0), 4);
line(colorImg, sceneCRN[3], sceneCRN[0], Scalar(255, 0, 0), 4);
putText(colorImg, text, sceneCRN[1], FONT_HERSHEY_DUPLEX, 1, Scalar(0, 0, 255), 1, 8);
}
}

int main()
{
OrbFeatureDetector detector;
OrbDescriptorExtractor extractor;

VideoCapture capture(0);

vector<KeyPoint> object0KP;
detector.detect(object0, object0KP);

Mat object0DS;
extractor.compute(object0, object0KP, object0DS);

vector<Point2f> object0CRN(4);
object0CRN[0] = (cvPoint(0, 0));
object0CRN[1] = (cvPoint(object0.cols, 0));
object0CRN[2] = (cvPoint(object0.cols, object0.rows));
object0CRN[3] = (cvPoint(0, object0.rows));

vector<KeyPoint> object1KP;
detector.detect(object1, object1KP);

Mat object1DS;
extractor.compute(object1, object1KP, object1DS);

vector<Point2f> object1CRN(4);
object1CRN[0] = (cvPoint(0, 0));
object1CRN[1] = (cvPoint(object1.cols, 0));
object1CRN[2] = (cvPoint(object1.cols, object1.rows));
object1CRN[3] = (cvPoint(0, object1.rows));

while (true)
{
Mat bwImg;
Mat bwImgDS;
vector<KeyPoint> bwImgKP;

Mat colorImg;

cvtColor(colorImg, bwImg, CV_BGR2GRAY);

detector.detect(bwImg, bwImgKP);
extractor.compute(bwImg, bwImgKP, bwImgDS);

ttea(bwImgKP, object0DS, bwImgDS, object0KP, object0CRN, colorImg, "Play");
ttea(bwImgKP, object1DS, bwImgDS, object1KP, object1CRN, colorImg, "Magazine");

imshow("Fish Smart", colorImg);

if (waitKey(1) == 27)
{
return 0;
}
}
}


## Intro

I'm not an expert "well-versed in the art", so this is mostly a style review of the C++ code rather than providing the performance-improving hints that you're really looking for; I hope someone else will step up with those insights for you!

We don't seem to be using <stdio.h> or <iostream>, but do require <vector> and <string>.

My system doesn't have "opencv2/nonfree/features2d.hpp", but it doesn't seem to be required.

Unless your OpenCV headers are installed into a non-standard location, it's probably better to include them with <> rather than "".

The only thing we're using from the std namespace is std::vector and a solitary std::string, so we can be a lot more specific about what we import.

More arguably, we might be more specific about cv at file scope and only bring in the entire namespace within the functions. For this small program, it's reasonably to import all of namespace cv at file scope, so I'll leave that as is.

We now have:

#include <opencv2/core/core.hpp>
#include <opencv2/features2d/features2d.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/calib3d/calib3d.hpp>

#include <string>
#include <vector>

using namespace cv;

using std::vector;


## Globals

Does matcher need to be a global variable? Perhaps it could be a static variable within ttea()? Perhaps it would be better as a non-static local variable - that would allow ttea() to be called concurrently from multiple threads, which would significantly help with your performance problems. Perhaps a thread-local variable is the right compromise? I've no idea what the construction cost of a cv::FlannBasedMatcher is, so it's hard for me to give a solid recommendation here, but you could ask the questions above and work out the best choice from that.

## ttea() function

I've no idea what the name ttea means - is it obvious to someone likely to be solving the same problem? If not, consider a more descriptive name, or at least a comment. The same goes for its formal parameter names.

I think it's better to declare variables where they are needed - we're writing C++, not (old) C.

This loop looks a bit strange:

for (int i = 0;  i < min(bwImgDS.rows - 1, (int)matches.size());  i++)


It's not clear where bwImgDS.rows comes into this. Is it just an optimisation? It might be possible to write this as a standard range-based loop:

for (const auto& match: matches)


perhaps with a break if there's a need to terminate early. If it can't be re-written as a range-based for, I'd still advise writing i++ as a postincrement rather than as a preincrement: although it should be equivalent here, defaulting to this form is a good habit that can avert surprises when incrementing non-primitive types.

The condition within the loop confuses me:

    if ((matches[i][0].distance < 0.6*(matches[i][1].distance)) && ((int)matches[i].size() <= 2 && (int)matches[i].size() > 0))


I don't know where that constant 0.6 comes from - is it a tunable, or is it a fundamental constant in the algorithm? Either way, it deserves to be named. In the other expressions, it doesn't make sense to cast size_t to int and then compare against zero - we should be able to safely write matches[i].size() <= 2. Given that we're accessing elements 0 and 1, it probably makes sense to test matches[i].size() exactly equal to 2 as the first test:

    if (matches[i].size() == 2  &&  matches[i][0].distance < 0.6 * matches[i][1].distance)


Later, when we iterate over matchesGD, we can avoid a warning by making i a size_t rather than int:

    for (std::size_t i = 0;  i < matchesGD.size();  ++i)


But this is easily converted to range-based form.

The most helpful thing we can change in ttea() is to reduce the amount of copying of its parameters. Everything except colorImg can be passed as reference to const object:

void ttea(const vector<KeyPoint>& bwImgKP,
const Mat& objectDS,
const Mat& bwImgDS,
const vector<KeyPoint>& objectKP,
const vector<Point2f>& objectCRN,
Mat colorImg,
const std::string& text)
{
if (bwImgKP.empty() || objectDS.empty() || bwImgDS.empty())
return;

FlannBasedMatcher matcher(new flann::LshIndexParams(20, 10, 2));
vector<vector<DMatch>> matches;
matcher.knnMatch(objectDS, bwImgDS, matches, 2);

vector<DMatch> matchesGD;
for (int i = 0;  i < bwImgDS.rows - 1 && i < (int)matches.size();  ++i)
{
const auto& match = matches[i];
if (match.size() == 2  &&  match[0].distance < 0.6 * match[1].distance)
matchesGD.push_back(match[0]);
}

if (matchesGD.size() >= 4)
{
vector<Point2f> obj;
vector<Point2f> scene;
for (const auto& match: matchesGD)
{
obj.push_back(objectKP[match.queryIdx].pt);
scene.push_back(bwImgKP[match.trainIdx].pt);
}

auto H = findHomography(obj, scene, CV_RANSAC);

vector<Point2f> sceneCRN(4);
perspectiveTransform(objectCRN, sceneCRN, H);

line(colorImg, sceneCRN[0], sceneCRN[1], Scalar(255, 0, 0), 4);
line(colorImg, sceneCRN[1], sceneCRN[2], Scalar(255, 0, 0), 4);
line(colorImg, sceneCRN[2], sceneCRN[3], Scalar(255, 0, 0), 4);
line(colorImg, sceneCRN[3], sceneCRN[0], Scalar(255, 0, 0), 4);
putText(colorImg, text, sceneCRN[1],
FONT_HERSHEY_DUPLEX, 1, Scalar(0, 0, 255), 1, 8);
}
}


## main() function

As I don't have the required input images, I have less to say about main(), other than noting that

while (true) {
...
if (condition)
break;
}


is equivalent to

do {
...
} while (!condition);


So the final loop could be

do {
Mat bwImg;
Mat bwImgDS;
vector<KeyPoint> bwImgKP;

Mat colorImg;