EDIT: Here's my totally-revised PHP...
$text = preg_replace("~[^ a-z0-9'-]~"," ",strtolower($INPUT));
for($i=1;$i<strlen($text)-1;$i++) {
if(preg_match("~['-]~",$text[$i]) && ( !preg_match("~[a-z0-9]~",$text[$i-1]) || !preg_match("~[a-z0-9]~",$text[$i+1])) ){
$text[$i] = " ";
}
}
while( preg_match ("~ ~",$text) ) $text = preg_replace( "~ ~", " ", $text);
if(preg_match("~[' -]~",$text[0])) $text = substr($text,1,strlen($text)-1);
if(preg_match("~[' -]~",$text[strlen($text)-1])) $text = substr($text,0,strlen($text)-2);
Now, what I'd said still applies..
This regex seems to work for me, but I'm curious if anyone can think of a breaking case, or tell me anything I did wrong.
(If it seems the code does what I say I want it do to, say so, then for fun you can help me become more neurotic about what I actually want it to do, which hinges on the question of defining an English "word".)
Desired final form of $text: replace, with spaces, all characters in $INPUT except for letters, digits, and any hyphens or apostrophes that are directly between two letters/digits (and hence "part of a word"). Then collapse all whitespace into single spaces, and if necessary, drop the leading and closing space.
End result should be a lowercase series of words separated by spaces, of which some words may contain (entirely "inside" the word) one or more non-consecutive apostraphes, and/or one or more non-consecutive hyphens.
Does this do that, with no exceptions?
After this step, the next part will be to split (or explode, whichever is better) the string by spaces, then generate a list of words and frequencies. I'm pretty sure how to do that on my own (especially because my teacher actually told us two algorithms). In fact I'll probably hand the assignment in before choosing an answer to this; I'm mostly asking out of curiosity.
My thinking is...
- English "words" tend to be case insensitive for distinction (though there is the proper noun issue, which I'll just ignore).
- Some "words" are digits (The word 88), or include digits, like 2Pac or se7en.
- An apostrophe can also be a single quote, so it shouldn't be considered part of the word in case that's the intention, even though it sometimes might be, as with What is goin' on? (Fortunately, "proper" contractions don't begin or end with an apostrophe.)
- Hyphens can occur at the beginning or end of a word, but that doesn't make for a distinct word (An eater asked if I was a fire-juggler or -eater has two *eater*s, not one with the hyphen and one without).
- Two or more consecutive hyphens are meant as a dash of some kind, separating words
- Two consecutive apostrophes are likewise assumed to separate words, despite typos like don''t
- In general, digits, hyphens, and apostrophes are the only relevant "non-letter letters" for word-frequency counting.
- Despite all my overthinking here, I don't actually want to be maximally thorough, just "good enough" (and "good enough" is the question I'm ovethinking!).
Old code:
$text = preg_replace( "~((?<![a-z0-9])[-'](?![a-z0-9]))|([^ a-z0-9'-])~", " ", strtolower($INPUT));
while(preg_match("~ ~",$text)) $text = str_replace(" ", " ", $text);
if($text[0] == " ") $text = substr($text,1,strlen($text)-2);
if($text[strlen($text)-1] == " ") $text = substr($text,0,strlen($text)-2);
