Extracting small word parts (quadgrams) from text using R

Question

I would like to extracts small word parts (quadgrams) from text. Example:

hello world

results in:

_hel, hell, ello, llo_, lo_w ...

This my basic attempt thus far:

text <- "hello world"

number_of_characters <- nchar(text)
quad_gram_list <- ""[-1]

for (i in 1:number_of_characters) {

    end <- i + 3
    if (end > number_of_characters) {
        end <- number_of_characters
    }

    temp <- substring(text, i, end)

    if (nchar(temp) == 4) {
        quad_gram_list <- append(quad_gram_list, temp)
    }
}

Any improvement suggestions would be very much appreciated.

Answer 1

Your loop approach works but seems a bit convoluted. For instance you could rewrite the for loop this way:

for (i in 1:number_of_characters) {

  end <- i + 3

  if (end <= number_of_characters) {
    temp <- substring(text, i, end)
    quad_gram_list <- append(quad_gram_list, temp)
  }
}

But anyway it's generally better to avoid loops in R.

Also note that:

• you should use character(0) instead of ""[-1]
• : is to be avoided because of undesired behaviour on edge cases. It's better to use seq_len().

Here are two alternative solutions:

1) Using base R:

unlist(lapply(seq_len(nchar(text) - 3), function(i) substring(text, i, i + 3)))
# [1] "hell" "ello" "llo " "lo w" "o wo" " wor" "worl" "orld"

2) Using tokenizers:

library(tokenizers)

tokenize_character_shingles(
  text,
  n = 4,
  strip_non_alphanum = FALSE,
  simplify = TRUE
)
# [1] "hell" "ello" "llo " "lo w" "o wo" " wor" "worl" "orld"