Efficient Strip Special Characters

Question

I have a private function in PHP that takes a given URL string and just leave the Alpha or numeric and certain punctuation left. I need the stripping of string due to having it match against ElasticSearch and ElasticSearch doesn't take it too kindly for special characters when it comes to exact match.

The following function so far is working fine.

protected function stripSpecialCharater($string)
{
    $string = iconv("Windows-1252","UTF-8",urldecode($string));
    $string = preg_replace(array('/\s{2,}/', '/[\t\n]/'), ' ', $string);
    return preg_replace('/[^a-zA-Z0-9ç&-©‚ƒ‘\s]/', '', $string);
}

Can it be better? is there a lib that does this better? I've used Laravel Validation service where you can specify alpha_dash but this is not for validation but rather I need the left over to see if there is a match to the Database Storage for a search.

Answer 1

Regex101.com seems to think you can simply combine \s{2,} and [\t\n] into \s{2,}[\t\n] and it should have the same effect:

$string = preg_replace(array('/\s{2,}/', '/[\t\n]/'), ' ', $string);

into:

$string = preg_replace('/\s{2,}[\t\n]/', ' ', $string);

---

You should add whitespace between your binary operators in this line:

$string = iconv("Windows-1252","UTF-8",urldecode($string));

into:

$string = iconv("Windows-1252", "UTF-8", urldecode($string));

(I'm not personally familiar with iconv, so I can't give too much of a review on your usage of the function.)

---

Your third regex could be slimmed down from a-zA-Z into a-z with an case insensitive search.

return preg_replace('/[^a-zA-Z0-9ç&-©‚ƒ‘\s]/', '', $string);

return preg_replace('/[^a-z0-9ç&-©‚ƒ‘\s]/i', '', $string);

---

In the same way you use a preg_replace with an array, you combine the first pair, with the secondary statement into one line if you like.

Answer 2

You are handling multibyte strings so additional care must be taken. If you do not include the u pattern modifier, your multibyte return values will be damaged. (Demo)

Let's dissect your posted patterns:

Patterns to be replaced by a single space:

/\s{2,}/  // match two or more whitespace characters
/[\t\n]/  // then re-iterate the potentially modified string and match any tab or line-feed (newline) character

Rather than making two separate passes, you can safely merge the logic into one pattern. To provide a more comprehensive "mop up" of newline characters, I'll recommend \R.

/\s{2,}|[\R\t]/u

As for the pattern to remove whitelisted characters, you have some overlapping logic which I believe is due to an ill-positioned hyphen.
Reference: https://www.ascii.cl/htmlcodes.htm

 pattern | ASCII decimal place
 -----------------------------
   [a-z] | range 97 to 122
   [A-Z] | range 65 to 90
   [0-9] | range 48 to 57
     Ã   | 195
     §   | 167
   [&-©] | range 38 to 169
     ‚   | 44
     Â   | 194
     ƒ   | 402
     â   | 226
     €   | 8364
     ˜   | 126
    \s   | *(equal to [\p{Z}\h\v])  <-- I don't feel like writing it all out

The position of your hyphen between two non-ranged entities in your negated character class tells the regex engine that you which to declare a range of characters. I interpret this to be a mistake. Rather than escaping the hyphen, I always recommend positioning the hyphen as the final character in the character class.

My suggested pattern will condense the alphabetic ranges by writing the case-insensitive pattern modifier i, using \d to signify digits, moving character ranges to the front, moving the hyphen to the back, and then (to be OCD) I've ordered single characters by their decimal value.

/[^a-z\d\s&‚~§©ÂÃ⃀-]/ui

You could implement this without retyping $string to create a oneliner, but if you prefer shorter lines of code, you can break it up over multiple lines or declare variables.

function stripSpecialCharater($string)
{
    return preg_replace(['/\s{2,}|[\t\R]/u', '/[^a-z\d\s&‚~§©ÂÃ⃀-]/ui'], [' ', ''], iconv("Windows-1252", "UTF-8", urldecode($string)));
}

Now we don't have any sample input to work with, but this may be a valuable read. It might be more stable to perform the replacement on the raw string, but that is for you to determine with your project data. There are plenty of ways to mince multibyte characters that are not always obvious; I recommend testing against a large battery of cases.

Raw replacement:

return preg_replace(['/\s{2,}|[\t\R]/u', '/[^a-z\d\s&‚~§©ÂÃ⃀-]/ui'], [' ', ''], $string);

As a final consideration, you may want to perform the whitespace reduction after the whitelist replacements so that you can handle newly consecutive whitespace characters. A final trim() might be advisable too. Add some realistic test strings to this demo and see what processes ultimately provide the cleanest output.