# Using Redis to store word occurences

I am doing a project where I am analyzing a group of Tweets.

What I need to do is find the occurrence of each individual word. I am fairly new to Redis, so I am not quite sure that my solution is optimal. I am storing a key value pair for each word. The key is a word, and the value is the occurrence count. I then use a list to keep track of every word that goes into the cache.

$cache = new Redis;$words = array('foo', 'bar', 'baz');

foreach ($words as$word) {
if ($cache->exists($word) {
$cache->incr($word);
} else {
$cache->put($word, 1);
$cache->lpush('words',$word);
}
}

$data = array();$list = $cache->lrange('words', 0, -1); foreach ($list as $word) {$data[$word] =$cache->get($word); } asort($data);


Your code is alright, really. However, by your using redis, I suspect you're after a fast way to store tiny bits of data. The varname $cache somewhat confirms my suspicion. If this is the case, your using lrange might not be the best choice, nor is exists really meaningful: INCR creates a new key, if required, and it returns the new value of that key in every case, so you can ditch the use of put, and simplify your first loop to: foreach ($words as $word) { if ($cache->incr($word) == 1)$cache->lpush('words', $word); }  Both these functions are atomic, so I don't think you can much improve on that. lrange has a time-complexity of O(S+N) according to the documentation. Given that you're just fetching the entire words collection, I'd opt for the O(n) alternative SMEMBERS. If you are going to use lrange, if you want to use pagination, for exampe, I'd consider using SCARD first, to determine the total number of members of your words collection. Anyway, here's how I'd write the second loop: $list = $cache->smembers('words'); foreach ($list as $word) {$data[$word] =$cache->get($word); }  I'll look into using redis itself a bit more, because it's quite easy to do away with that words collection, too: foreach ($words as $word)$cache->incr($word);$data = array();
$list =$cache->keys('*');//get all keys in collection
foreach ($list as$word)
$data[$word] = $cache->get($word);


But this would imply the entire database contains nothing but words, which isn't what you want, probably. If I find the time, I'll attempt to put together a function or script that simply iterates over that words collection, and performs something like this

EVAL "return {KEYS[1], redis.call('get', KEYS[1])}" 1 <a word>


for each word, returning a series of redis objects that look like this:

1) <a word>
2) 123


where the 2) actually is the occurance count

Unrelated
I notice you're constructing an object of the Redis class. This, to me, tells me you're using a library other than the official one. The PRedis class was, last time I checked, the preferred one. Perhaps consider changing to that lib, and see if it works for you. Anyway, look into the libs with a star next to their name :)

• Awesome answer. Thanks a lot! I wasn't aware of the ->keys('') way of fetching the words. Wouldn't i be able to use a prefix for each word and do something like ->keys('word-')? Btw. i am using the PRedis package, it just works under an alias in my application. – Stromgren May 16 '14 at 16:48
• @Stromgren: Prefixing the collections would do the trick, too, I believe. Just try \$ redis-cli and then SELECT 1 (where 1 is the db, of course). Try the commands there. – Elias Van Ootegem May 17 '14 at 10:09