Converting Array of `Float32` (`float`) to Array of `UINT8` (`unsigned char`) Using AVX2 [closed]

Question

Given input array of Float32 (float) with numElements how could one efficiently convert it into array of UINT8 (unsigned char)?
The tricky part here is to apply Unsigned Saturation in the conversion

For instance, here is a vanilla code for it (Pay attention there is a scaling operation):

void ConvertToUint8(unsigned char* mO, float* mI, int numElements, float scalingFctr)
{
    int ii;
    for (ii = 0; ii < numElements; ii++) {
        mO[ii] = (unsigned char)(fmin(fmax(mI[ii] * scalingFctr, 0.0), 255.0));
    }
}

Where mO is the output array.

I'm looking for a way to optimize (Performance wise) this code on AVX2 enabled CPU's. Any idea, Intrinsics included, is welcome.

Pay attention the above code apply unsigned saturation manually (Is there a function for unsigned saturation based casting in C?). I think in practice SSE and AVX have it built in (See _mm_packus_epi16() for SSE).

The objective is to yield faster code than the vanilla example as in Compiler Explorer - ConvertToUint8.

For simplicity one could assume the arrays are aligned.

Answer 1

Harold's comment is correct.

Consider what happens for float inputs like 5000000000 * 1.0. Conversion to int32_t with cvtps2dq will give you -2147483648 from that out-of-range positive float. (2's complement integer bit-pattern 0x80000000 is the "indefinite integer value" described by Intel's documentation for this case.)

In that case, your vectorized version that clamps via integer saturation will start with a negative (and ultimately do unsigned saturation to 0), not matching your fmin which clamps before even converting to integer, resulting in 255.

So you have to be able to rule out such inputs if you want to vectorize without clamping in the FP domain before conversion to integer.

Remember that IEEE754 binary32 float can represent values outside the range of int32_t or int64_t, and what x86 FP->int conversions do in that case.

Answer 2

This is a tryout I cam up with - ConvertToUint8():

#include <immintrin.h> // AVX & AVX2

#define AVX_STRIDE          8
#define AVX_STRIDE_DOUBLE   16
#define AVX_STRIDE_TRIPLE   24
#define AVX_STRIDE_QUAD     32

void ConvertToUint8(unsigned char* mO, float* mI, int numElements, float scalingFctr)
{
    int ii;
    float *ptrInputImage;
    int *ptrOutputImage;

    __m256  floatPx1, floatPx2, floatPx3, floatPx4;
    __m256  scalingFactor;
    __m256i int32Px1, int32Px2, int32Px3, int32Px4;
    __m256i uint8Px1, uint8Px2;
    __m256i *ptrOutputImageAvx;

    for (ii = 0; ii < numElements; ii += AVX_STRIDE_QUAD) {
        ptrInputImage       = mI;
        ptrOutputImageAvx   = (__m256i*)(mO);
        // AVX Pack is 8 Floats (8 * 32 Bit) -> 32 UINT8 (32 * 8 Bit)
        // Hence loading 4 * 8 Floats which will be converted into 32 UINT8

        floatPx1 = _mm256_loadu_ps(ptrInputImage);
        floatPx2 = _mm256_loadu_ps(ptrInputImage + AVX_STRIDE);
        floatPx3 = _mm256_loadu_ps(ptrInputImage + AVX_STRIDE_DOUBLE);
        floatPx4 = _mm256_loadu_ps(ptrInputImage + AVX_STRIDE_TRIPLE);

        ptrInputImage += AVX_STRIDE_QUAD;

        // See https://stackoverflow.com/questions/51778721
        int32Px1 = _mm256_cvtps_epi32(_mm256_mul_ps(floatPx1, scalingFactor)); // Converts the 8 SP FP values of a to 8 Signed Integers (32 Bit).
        int32Px2 = _mm256_cvtps_epi32(_mm256_mul_ps(floatPx2, scalingFactor));
        int32Px3 = _mm256_cvtps_epi32(_mm256_mul_ps(floatPx3, scalingFactor));
        int32Px4 = _mm256_cvtps_epi32(_mm256_mul_ps(floatPx4, scalingFactor));
        uint8Px1 = _mm256_packs_epi32(uint16Px1, uint16Px2); // Saturating and packing 2 of 8 Integers into 16 of INT16
        uint8Px2 = _mm256_packs_epi32(uint16Px3, uint16Px4); // Saturating and packing 2 of 8 Integers into 16 of INT16
        uint8Px1 = _mm256_packus_epi16(uint8Px1, uint8Px2); // Saturating and packing 2 of 16 INT16 into 32 of UINT8
        uint8Px1 = _mm256_permutevar8x32_epi32(uint8Px1, _mm256_setr_epi32(0, 4, 1, 5, 2, 6, 3, 7)); // Permitation for Linear Orderinmg
        _mm256_storeu_si256(ptrOutputImageAvx++, uint8Px1); // Storing 32 UINT8, Promoting the pointer

    }
}

The code is based on answer of Peter Cordes - How to Convert 32 [Bit] Float to 8 [Bit] Signed char?
I'd love to hear thoughts about it.