Return to Answer

By the way, I also noticed another bug here. When you exit the loop you currently are skipping the add ebx, 4 line. But you need to advance ebx or else the remainder part below won't have the right value for ebx. This works correctly in my modified version because I always add to ebx inside the loop.

.loopP: movaps  xmm1, [esi+ebx*4]
        movaps  xmm2, [edi+ebx*4]
        subps   xmm1, xmm2
        add     ebx, 4
        mulps   xmm1, xmm1
        dec     ecx
        haddps  xmm1, xmm1
        haddps  xmm1, xmm1
        addps   xmm0, xmm1
        jnz      .loopP

        cmp     edx, 0
        je      .endD
        sub     ebx, 4

.loopP: movaps  xmm1, [esi+ebx*4]
        movaps  xmm2, [edi+ebx*4]
        subps   xmm1, xmm2
        add     ebx, 4
        mulps   xmm1, xmm1
        dec     ecx
        haddps  xmm1, xmm1
        haddps  xmm1, xmm1
        addps   xmm0, xmm1
        jnz      .loopP

        cmp     edx, 0
        je      .endD

##Remove one jump from main loop##

.dU1:   shl     edx, 2
        movaps  xmm7, [mask1 + edx*4 - 16]

Explanation: shr is the shift right instruction. When you shift right by 2 it is the same thing as dividing by 4. To get the remainder of something divided by 4, you just need to and that number with 3. C equivalent code:

    remainder = length & 3;
    length >>= 2;

##Remove one jump from main loop##

.dU1:   shl     edx, 4
        movaps  xmm7, [mask1 + edx - 16]

Explanation: Your masks are located in memory in consecutive order like an array of 128 bit values. So you can load them by index. Note that edx is 1..3 here. The shl instruction is shift left, and shifting left by 4 multiplies edx by 16. Then we load from mask1 + edx - 16. The minus 16 is because edx started at one instead of zero. Equivalent C code is something like this:

        uint8_t mask1[48] = { /* values */ };
        mask128 = *(__m128 *) &mask1[(remainder<<4)-16];