3
\$\begingroup\$

I would like to extracts small word parts (quadgrams) from text. Example:

hello world

results in:

_hel, hell, ello, llo_, lo_w ...

This my basic attempt thus far:

text <- "hello world"

number_of_characters <- nchar(text)
quad_gram_list <- ""[-1]

for (i in 1:number_of_characters) {

    end <- i + 3
    if (end > number_of_characters) {
        end <- number_of_characters
    }

    temp <- substring(text, i, end)

    if (nchar(temp) == 4) {
        quad_gram_list <- append(quad_gram_list, temp)
    }
}

Any improvement suggestions would be very much appreciated.

\$\endgroup\$
5
\$\begingroup\$

Your loop approach works but seems a bit convoluted. For instance you could rewrite the for loop this way:

for (i in 1:number_of_characters) {

  end <- i + 3

  if (end <= number_of_characters) {
    temp <- substring(text, i, end)
    quad_gram_list <- append(quad_gram_list, temp)
  }
}

But anyway it's generally better to avoid loops in R.

Also note that:

  • you should use character(0) instead of ""[-1]
  • : is to be avoided because of undesired behaviour on edge cases. It's better to use seq_len().

Here are two alternative solutions:

1) Using base R:

unlist(lapply(seq_len(nchar(text) - 3), function(i) substring(text, i, i + 3)))
# [1] "hell" "ello" "llo " "lo w" "o wo" " wor" "worl" "orld"

2) Using tokenizers:

library(tokenizers)

tokenize_character_shingles(
  text,
  n = 4,
  strip_non_alphanum = FALSE,
  simplify = TRUE
)
# [1] "hell" "ello" "llo " "lo w" "o wo" " wor" "worl" "orld"
\$\endgroup\$
  • 2
    \$\begingroup\$ substring is vectorized so you can do: start <- head(seq(nchar(text)), -3); stop <- tail(seq(nchar(text)), -3); substring(text, start, stop) \$\endgroup\$ – flodel May 14 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.