##Wrong increment##

First, there is a bug in your main loop.  This line:

        add     ebx, 1

should be:

        add     ebx, 4

because you are operating on 4 floats at a time.  Right now, you are doing array elements [0..3] following by [1..4].  On my computer the program crashed because it had a problem doing an unaligned load.

##Simplify divide by 4##


This part where you divide ecx by 4:

        push    eax                     
        mov     edx, 0                  
        mov     eax, ecx                
        mov     ebx, 4                  
        div     ebx                     
        mov     ecx, eax                ; ecx becomes the counter for the packed iterations
        pop     eax          

can be simplified to this:

        mov     edx, ecx
        and     edx, 3
        shr     ecx, 2

##Remove one jump from main loop##

Your main loop:

    .loopP: movaps  xmm1, [esi+ebx*4]
            movaps  xmm2, [edi+ebx*4]
            subps   xmm1, xmm2
            mulps   xmm1, xmm1
            dec     ecx
            haddps  xmm1, xmm1
            haddps  xmm1, xmm1
            addps   xmm0, xmm1
            jz      .loopC
            add     ebx, 4
            jmp     .loopP
    
    .loopC  cmp     edx, 0
            je      .endD

Can be rewritten to remove one of the jumps at the end:

    .loopP: movaps  xmm1, [esi+ebx*4]
            movaps  xmm2, [edi+ebx*4]
            subps   xmm1, xmm2
            add     ebx, 4
            mulps   xmm1, xmm1
            dec     ecx
            haddps  xmm1, xmm1
            haddps  xmm1, xmm1
            addps   xmm0, xmm1
            jnz      .loopP
    
            cmp     edx, 0
            je      .endD
            sub     ebx, 4

##Loading the mask##

The part where you do a chain of compares to load one of three masks:

    .dU1:   cmp     edx, 1          
            jne     .dU2
            movaps  xmm7, [mask1]           
            jmp     .dU
    
    .dU2:   cmp     edx, 2
            jne     .dU3
            movaps  xmm7, [mask2]
            jmp     .dU
    
    .dU3:   movaps  xmm7, [mask3]

can be simplified to this:

    .dU1:   shl     edx, 2
            movaps  xmm7, [mask1 + edx*4 - 16]

##Stack push/pop##

I'm not sure if this is part of your calling convention, but your prologue and epilogue have extraneous instructions:

        push    ebp
        mov     ebp, esp
        sub     esp, 4
        push    ebx
        push    esi
        push    edi

        pop     edi 
        pop     esi
        pop     ebx
        mov     esp, ebp
        pop     ebp
        ret

Could be changed to:

        push    ebp
        mov     ebp, esp
        push    ebx
        push    esi
        push    edi

        pop     edi
        pop     esi
        pop     ebx
        pop     ebp
        ret

Furthermore, you are not utilizing `eax`.  If you used `eax` instead of `ebx` everywhere in your function, you wouldn't have to save `ebx` because you wouldn't ever use it.  You can load the result pointer into `eax` at the very end instead of at the beginning.

##Comments##

In general I liked all your comments.  The one thing I thought you could add a comment for was that this double instruction:

            haddps  xmm1, xmm1
            haddps  xmm1, xmm1

adds all 4 floats in `xmm1` into the first float in `xmm1`.  I wasn't familiar with the `haddps` instruction and I initially thought you made a typo (I thought a single `haddps` would do the whole job).