Text tokenizer - extract words and positions from text

Question

I have Set of character delimiters (DELIMITERS), eg . , etc. Using this I want to split text and get words with their position in text. String.split() works fine if you want only words. The same with StringTokenizer. Wrote some simple method to deal with this, but maybe there is a better way to achieve this result?

public List<String> extractWords(String text){
    List<String> words = new ArrayList<>();
    List<WordPos> positions = new ArrayList<>();
    int wordStart = -1;
    for(int i=0; i < text.length(); i++){
        if(DELIMITERS.contains(text.charAt(i))){
            if(wordStart >=0){ //word just ended
                String word = text.substring(wordStart, i);
                positions.add(new WordPos(wordStart, i));
                words.add(word);
            }
            wordStart = -1;
        }else{ //not delimiter == valid word
            if(wordStart < 0){ //word just started
                wordStart = i;
            }
        }
    }
    return words;
}

// inner static class for words positions
public static class WordPos{
    int start;
    int end;
    public WordPos(int start, int end){
        this.start = start;
        this.end = end;
    }
}

Answer 1

You shouldn't shorten class names. So WordPos should become WordPosition.

---

Right now you are creating WordPos items which you add to a List<WordPos> which is lost after returning from the method.

---

Instead of using substring() you could use a StringBuilder object, where you append if the current char is contained in DELIMITERS.

You would need to take care (which you don't do right now) about the case that the text ends without any of your delimiting chars.

public List<String> extractWords(String text) {
    List<String> words = new ArrayList<>();
    StringBuilder sb = new StringBuilder(text.length());

    for (int i = 0; i < text.length(); i++) {
        char current = text.charAt(i);

        if (DELIMITERS.contains(current)) {
            if (sb.length() > 0) {
                words.add(sb.toString());
                sb.setLength(0);
            }
        } else {
            sb.append(current);
        }
    }
    if (sb.length() > 0) {
        words.add(sb.toString());
    }
    return words;
}

Answer 2

String.split is a special case of using regular expressions. Just because it does not return the position of the words, does not mean that regular expressions themselves are not the right solution.

Consider the following:

private static final Pattern DELIMETERS = Pattern.compile("[,. ]+");

Then, using that pattern, you can match the words.

Matcher matcher = DELIMETERS.match(input);
int start = 0;
List<WordPos> words = new ArrayList<>();
while (matcher.find()) {
    words.add(new WordPos(start, matcher.start());
    start = matcher.end();
}
if (input.length() > start) {
    words.add(new WordPos(start, input.length());
}
return words;

Note that if you want to include the actual word part, then you can use substring on the input.