# Text tokenizer - extract words and positions from text

I have Set of character delimiters (DELIMITERS), eg . , etc. Using this I want to split text and get words with their position in text. String.split() works fine if you want only words. The same with StringTokenizer. Wrote some simple method to deal with this, but maybe there is a better way to achieve this result?

public List<String> extractWords(String text){
List<String> words = new ArrayList<>();
List<WordPos> positions = new ArrayList<>();
int wordStart = -1;
for(int i=0; i < text.length(); i++){
if(DELIMITERS.contains(text.charAt(i))){
if(wordStart >=0){ //word just ended
String word = text.substring(wordStart, i);
}
wordStart = -1;
}else{ //not delimiter == valid word
if(wordStart < 0){ //word just started
wordStart = i;
}
}
}
return words;
}

// inner static class for words positions
public static class WordPos{
int start;
int end;
public WordPos(int start, int end){
this.start = start;
this.end = end;
}
}

• In general, tokenization is a nontrivial problem - right now you want a better solution than just breaking on whitespace and you'll include separators, but then you'll get issues with things such as abbreviations, parts of people names, urls, emoticons, etc. If this list of separators is enough for your needs, then great, but if at some point you decide to go further, then an appropriate solution would be to reuse an existing solution for natural language tokenization (and further processing). For java, nlp.stanford.edu/software/corenlp.shtml is one option, but there are others. – Peteris Feb 9 '15 at 13:11
• I know this is a nontrivial problem, but for this purpose it's enough, it's just has to be fast :) – bartektartanus Feb 9 '15 at 21:37

You shouldn't shorten class names. So WordPos should become WordPosition.

Right now you are creating WordPos items which you add to a List<WordPos> which is lost after returning from the method.

Instead of using substring() you could use a StringBuilder object, where you append if the current char is contained in DELIMITERS.

You would need to take care (which you don't do right now) about the case that the text ends without any of your delimiting chars.

public List<String> extractWords(String text) {
List<String> words = new ArrayList<>();
StringBuilder sb = new StringBuilder(text.length());

for (int i = 0; i < text.length(); i++) {
char current = text.charAt(i);

if (DELIMITERS.contains(current)) {
if (sb.length() > 0) {
sb.setLength(0);
}
} else {
sb.append(current);
}
}
if (sb.length() > 0) {
}
return words;
}

• This was just prototype, I was more curious if this could be achieved using some other functions. But thanks anyway :) – bartektartanus Feb 9 '15 at 11:11

String.split is a special case of using regular expressions. Just because it does not return the position of the words, does not mean that regular expressions themselves are not the right solution.

Consider the following:

private static final Pattern DELIMETERS = Pattern.compile("[,. ]+");


Then, using that pattern, you can match the words.

Matcher matcher = DELIMETERS.match(input);
int start = 0;
List<WordPos> words = new ArrayList<>();
while (matcher.find()) {