Aligning the playing time of a video with the observed time within the video

I have a video that displays a soccer match. Video file match.mp4 consist of some opening commercials, the actual match and the half-time end/break. As a result, the video file is always lengthier than the match itself. The properties of match.mp4 looks as follows:

So according to my Windows computer, the file has a length of 00:48:36. I want to check this using OpenCV:

import cv2   # version 3.4.2

video = cv2.VideoCapture('match.mp4')
while (True):
fps = video.get(cv2.CAP_PROP_FPS)                      # 30
video.set(cv2.CAP_PROP_POS_AVI_RATIO, 1)               # set video to the end
duration = video.get(cv2.CAP_PROP_POS_MSEC)            # 2,916,633 ms
total_frames = video.get(cv2.CAP_PROP_FRAME_COUNT)     # 87,499 frames


2,916,633 / (1,000 * 60) = 48 minutes and ~ 37 seconds. I'm happy to work this and calculate the secondsPerFrame:

secondsPerFrame =  87,499 / 2,916,633 = 0,030000003428611


So, now I know for each frame that passes how many seconds of videoTime passes. Within the video, I am also observing the gameTime:

I try to find the gameTime using TesseractOCR and contour-detecting. Sometimes, I get the correct result, sometimes I don't get any result and sometimes I get incorrect results. Edit: these values are not giving an error or anything. They are simply not usable for my gameTime-calculation. As a result, I need a solution to estimate the gameTime for each frame, independent of usable/non-usable gameTime-data

Correct/Usable:

Frame: 211, gameTime: 00:17
Frame: 212, gameTime: 00:18
Frame: 213, gameTime: 00:18
Frame: 214, gameTime: 00:18
Frame: 215, gameTime: 00:18
Frame: 216, gameTime: 00:18
....


None/Unusable:

Frame: 228, gameTime: N/A
Frame: 229, gameTime: N/A
Frame: 230, gameTime: N/A
Frame: 231, gameTime: N/A
Frame: 232, gameTime: N/A
Frame: 233, gameTime: N/A
...


Outliers/Unusable:

Frame: 1742, gameTime: N/A
Frame: 1743, gameTime: 61:09
Frame: 1744, gameTime: N/A
Frame: 1745, gameTime: 61:09
Frame: 1746, gameTime: 61:09
Frame: 1747, gameTime: 01:09     # should be 61:09
...


These values are the input for my final estimation of gameTime. The goal to get an estimation of matchTime, indepedent of that particular frame gives a usable/non-usable OCR-interpretation of gameTime. Important to note is the relationship between videoTime and gameTime. Obviously, the two are different as videoTime starts with the commercials, whereas gameTime starts only when the game begins and the time on the scorebox is observed for the first time. Still, both variables are closely related as there is a simple linear relationship (with slope 1) between the video and the game time. 1 sec in videoTime is also 1 sec in gameTime.

We can write the linear relationship between the two as:

y = mx + b where y is gameTime, m = 1, x = videoTime, b = (-)delay between gameTime and videoTime


So, my question is how can we determine b as quickly and accurate as possible? Below are my considerations:

• I want to use a limited number of frames. There are events happening in the first couple of seconds of the match that need to be linked as soon as possible to a timestamp
• It needs to be able to handle outliers. So, my idea was to use RANCSAC to cope with eventual misreadings of the time and filter these outliers.
• My overall script runs in real-time so the method needs to be fast. Running regressions (normal/RANCSAC) take a lot of time. More and more frames are included into the dataset as the match progresses.

My script:

import cv2
from skimage.measure import LineModelND, ransac

video = cv2.VideoCapture('match.mp4')
previous_frames = []
frame_count = []
match_time = []
counter = 0

def ransac_regression(data, counter):
model = ransac(data, LineModelND, min_samples=5, residual_threshold=1, max_trials=5)[0]
matchTime = model.predict_y(counter)
print(matchTime)

while True:
if ret:
if len(previous_frames) == 200:
time = get_matchtime()
boolean = checktime()
if boolean:
print("Frame: {0}, time: {1}".format(counter, time))
frame_count.append(counter)
match_time.append(timestamp)
else:
print("Frame: {0}, time: N/A".format(counter))
previous_frames = previous_frames[1:]
previous_frames.append(frame)
else:
previous_frames.append(frame)
if len(match_time) >= 25:
data = np.column_stack([frame_count, match_time])
ransac = ransac_regression(data, counter)
counter += 1
key = cv2.waitKey(5)
if key == ord('q'):
break

video.release()
cv2.destroyAllWindows()


My script works, but I get the feeling that I am over-complicating things and there are easier/faster ways to determine the delay between videoTime and gameTime. Thanks in advance for your help!

• If the code works only some of the time, then this question is off-topic for the code review site. We can't help you debug the code. – pacmaninbw Dec 5 '19 at 13:44
• The code works all the time. I always get an estimated gameTime. I am posting this because I want to check if there is an easier/faster way – HJA24 Dec 5 '19 at 13:49
• You might want to remove the section about missing results and wrong results in that case. – pacmaninbw Dec 5 '19 at 13:53
• If the output is incorrect, it doesn't work. – Mast Dec 5 '19 at 14:20
• The output is not incorrect. You and @pacmaninbw are talking about the intermediate output which is usable or not usable/"incorrect". However, this is the input for the final output, namely estimated gameTime – HJA24 Dec 5 '19 at 14:37