dup = 1;
you should break. There's no need to execute the rest of the loop.
Consider using <stdbool.h>, making bool dup = false, later assigning it true, and writing if (!dup).
In practical terms, an array of five values poses no computational cost. However, if your prof cares about complexity ...
Use better indentation. Always the same, and especially not just one space. Four spaces are the standard.
You're inconsistent, I'd replace fin.readline() with another next(fin).
open has mode 'r' by default, no need to specify. And you don't really want + when writing.
For the file variable, personally I prefer just f, and use fin and fout ...
Nice solution and implementation. Few suggestions:
Naming: the name main is generally used for the entry point of the program. In this case, a better name could be steps_to_sort_cows.
Sublist bounds: since n is fixed, instead of sublist = nums[i:n] you can write sublist = nums[i:]
The time complexity is \$O(n*n*log(n))\$. Basically, for each cow ...
Allocate the temporary array once, and pass it down to the recursive invocations.
Do not fall back to the insertion sort immediately. Postpone it until all the recursions complete, and insertion sort the entire array once.
It feels scary to insertion sort the entire array. Wouldn't it degrade performance to \$O(n^2)\$? The answer is no....
Your indentation is off. if should be at the same level as the corresponding else or else if, their bodies being indented once. Don't wander all the way to the right.
You are missing the include for std::string, <string>.
using namespace std; seems convenient, right? Unfortunately, throwing everything and the kitchen sink into the global namespace ...
If the input really does have commas between the numbers, then we'll want to allow for that here:
In any case, it's important to check the return value from scanf(), otherwise we could be using values that haven't been properly initialised.
It's probably worth writing separate functions for the input, output and processing, ...
On your array that has 5 elements
the 1st iteration of the outerloop (CX=4) has to do 4 compares in the innerloop
the 2nd iteration of the outerloop (CX=3) has to do 3 compares in the innerloop
the 3rd iteration of the outerloop (CX=2) has to do 2 compares in the innerloop
the 4th iteration of the outerloop (CX=1) has to do 1 compare in the innerloop
In your ...