2
\$\begingroup\$

I have Set of character delimiters (DELIMITERS), eg . , etc. Using this I want to split text and get words with their position in text. String.split() works fine if you want only words. The same with StringTokenizer. Wrote some simple method to deal with this, but maybe there is a better way to achieve this result?

public List<String> extractWords(String text){
    List<String> words = new ArrayList<>();
    List<WordPos> positions = new ArrayList<>();
    int wordStart = -1;
    for(int i=0; i < text.length(); i++){
        if(DELIMITERS.contains(text.charAt(i))){
            if(wordStart >=0){ //word just ended
                String word = text.substring(wordStart, i);
                positions.add(new WordPos(wordStart, i));
                words.add(word);
            }
            wordStart = -1;
        }else{ //not delimiter == valid word
            if(wordStart < 0){ //word just started
                wordStart = i;
            }
        }
    }
    return words;
}

// inner static class for words positions
public static class WordPos{
    int start;
    int end;
    public WordPos(int start, int end){
        this.start = start;
        this.end = end;
    }
}
\$\endgroup\$
  • 2
    \$\begingroup\$ In general, tokenization is a nontrivial problem - right now you want a better solution than just breaking on whitespace and you'll include separators, but then you'll get issues with things such as abbreviations, parts of people names, urls, emoticons, etc. If this list of separators is enough for your needs, then great, but if at some point you decide to go further, then an appropriate solution would be to reuse an existing solution for natural language tokenization (and further processing). For java, nlp.stanford.edu/software/corenlp.shtml is one option, but there are others. \$\endgroup\$ – Peteris Feb 9 '15 at 13:11
  • \$\begingroup\$ I know this is a nontrivial problem, but for this purpose it's enough, it's just has to be fast :) \$\endgroup\$ – bartektartanus Feb 9 '15 at 21:37
4
\$\begingroup\$

You shouldn't shorten class names. So WordPos should become WordPosition.


Right now you are creating WordPos items which you add to a List<WordPos> which is lost after returning from the method.


Instead of using substring() you could use a StringBuilder object, where you append if the current char is contained in DELIMITERS.

You would need to take care (which you don't do right now) about the case that the text ends without any of your delimiting chars.

public List<String> extractWords(String text) {
    List<String> words = new ArrayList<>();
    StringBuilder sb = new StringBuilder(text.length());

    for (int i = 0; i < text.length(); i++) {
        char current = text.charAt(i);

        if (DELIMITERS.contains(current)) {
            if (sb.length() > 0) {
                words.add(sb.toString());
                sb.setLength(0);
            }
        } else {
            sb.append(current);
        }
    }
    if (sb.length() > 0) {
        words.add(sb.toString());
    }
    return words;
}
\$\endgroup\$
  • \$\begingroup\$ This was just prototype, I was more curious if this could be achieved using some other functions. But thanks anyway :) \$\endgroup\$ – bartektartanus Feb 9 '15 at 11:11
4
\$\begingroup\$

String.split is a special case of using regular expressions. Just because it does not return the position of the words, does not mean that regular expressions themselves are not the right solution.

Consider the following:

private static final Pattern DELIMETERS = Pattern.compile("[,. ]+");

Then, using that pattern, you can match the words.

Matcher matcher = DELIMETERS.match(input);
int start = 0;
List<WordPos> words = new ArrayList<>();
while (matcher.find()) {
    words.add(new WordPos(start, matcher.start());
    start = matcher.end();
}
if (input.length() > start) {
    words.add(new WordPos(start, input.length());
}
return words;

Note that if you want to include the actual word part, then you can use substring on the input.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.