5
\$\begingroup\$

I am writing a cron job to manipulate and transfer remote data. I need to cache data from a table, process it, merge with previous table, iterate the process, and eventually send the result data to remote database.

Here is a basic in-memory cache storage with thin simple CRUD type API. It has to store hashes, enforce equal keys and unique ids. The data come from tables and immediately transformed into hashes (key-value pairs). That way hashes are portable and no reference to tables is needed. Also they can be easily ported to NoSQLs.

The design is inspired by Parse.com and StackMob.

Somehow I could not find any similar light weight library. I need it portable to easily transfer to any server. I would rather avoid any heavy installation or configuration.

An older version was posted here.

Any critique is welcome.

// I am using 'i' as primary key for all hashes
define('ID_KEY', 'i');

/** Local In-Memory Cache: Tables referred by their string names
 * Storing hashes:
 *    all hashes must have ID_KEY as key
 *    all hashes must have keys other than ID_KEY
 *    and hashes in the same table must have the same key structure (enforced)
 * Basic CRUD interface + 'readOne' for the first entry + 'realDelete'
 * arrays packed into json strings, which allows for array values,
 * e.g. array('key1'=>'val1', 'key2'=> array(1,2,3))
 */
class LocStore {
   // assoc. array('table' => array of keys)
   private $_keys = array();
   // assoc. array of strings indexed by table and id (must be nonempty)
   private $_values = array(); 

   /** create entry, enforce the same keys and new id
    * @param string $table
    * @param ass. array $hash, must have ID_KEY not null, must have other keys
    */
   public function create ($table, array $hash) {
      $id = $hash[ID_KEY];
      if (! $id) throw new Exception('Id is empty');
      unset($hash[ID_KEY]);
      if (! $hash) throw new Exception('Empty hash beside id');
      // sort by keys alphabetically to ensure the same order for all entries
      ksort($hash);
      if ( empty($this->_keys[$table]) ) {
         // first hash for $table - cache keys
         $this->_keys[$table] = array_keys($hash);
      } elseif ($this->_keys[$table] != array_keys($hash)) {
         throw new Exception('Array keys mismatch: '. json_encode($hash) );
      }
      if ( isset($this->_values[$table]) && ! empty($this->_values[$table][$id]) ) {
         throw new Exception("Id '$id' already exists");
      }
      $this->_values[$table][$id] = json_encode(array_values($hash));
      // for chaining
      return $this;
   }

   // read one entry, empty array if nothing is there
   // reset pointer for iteration
   public function reset ($table) {
      if (! empty($this->_values[$table]) ) {
         reset($this->_values[$table]);         
      }
   }

   // read next hash and advance pointer
   public function readNext ($table) {
      if (empty($this->_values[$table])) return array();
      $id = key($this->_values[$table]);
      next($this->_values[$table]);
      if (! $id) return array ();
      return $this->read($table, $id);    
   }

   // read one entry if there are some or else return empty array
   // in contrast to readNext, the result is always non-empty as long as table has hashes
   public function readOne ($table) {
      if ( empty($this->_values[$table]) ) return array();
      foreach ($this->_values[$table] as $id => $val) {
         return $this->read($table, $id);         
      }
      return array();
   }

   // read by id, empty array if not found
   public function read ($table, $id) {
      if (! $id) throw new Exception('Id is empty');
      if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      //$keys = $this->_toArray($this->_keys[$table]);
      $keys = $this->_keys[$table];
      //$values = $this->_toArray($this->_values[$table][$id]);
      $values = json_decode($this->_values[$table][$id]);
      $result = array_combine($keys, $values);
      $result[ID_KEY] = (string) $id;
      return $result;
   }

   // read and delete one nonempty array or return empty if nothing is left
   public function readDeleteOne ($table) {
      if ( empty($this->_values[$table]) ) return array();
      foreach ($this->_values[$table] as $id => $val) {
         return $this->readDelete($table, $id);         
      }
      return array();
   }

   // read and delete by id, empty array if not found
   public function readDelete ($table, $id) {
      if (! $id) throw new Exception('Id is empty');
      if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      $keys = $this->_keys[$table];
      $values = json_decode($this->_values[$table][$id]);
      $result = array_combine($keys, $values);
      $result[ID_KEY] = (string) $id;
      // deletion is here
      unset($this->_values[$table][$id]);
      return $result;
   }

   // delete by id, exception if not found
   public function delete ($table, $id) {
      if (! $id) throw new Exception('Id is empty');
      if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) {
         throw new Exception('Deleting non-existant entry');
      }
      unset($this->_values[$table][$id]);
      return $this;
   }

   public function deleteAll ($table) {
      $this->_values[$table] = array();
      return $this;
   }
}

EDIT:

Here is the (only) method from another class using reset and readNext (where $this->_locStore points to the class locStore above):

public function loadChains ($table, $mergeTable, $relKey) {
  $this->_locStore->reset($mergeTable);
  while ( $hash = $this->_locStore->readNext($mergeTable) ) {
     $chain = $this->getChain($mergeTable, $hash[ID_KEY], $relKey);
     $hash[$relKey] = $chain;
     $this->_locStore->create($table, $hash);
  }
  $this->_locStore->deleteAll($mergeTable);
}
\$\endgroup\$
  • \$\begingroup\$ Actually I can not understand why do you call it cache? \$\endgroup\$ – zavg Dec 14 '13 at 20:56
  • \$\begingroup\$ @zavg Cache is where you store your data temporarily, isn't it? \$\endgroup\$ – Dmitri Zaitsev Dec 15 '13 at 1:23
  • 1
    \$\begingroup\$ Cache is used to store data so that future requests for that data can be served faster. \$\endgroup\$ – zavg Dec 15 '13 at 6:59
  • \$\begingroup\$ @zavg I've seen Laravel calling Cache its abstract storage API that can be e.g. database, memcached, redis ... Is this a bad terminology? \$\endgroup\$ – Dmitri Zaitsev Dec 15 '13 at 10:51
  • \$\begingroup\$ 1) Cache, to me, is something that outlives the request being processed (data stored in files, which can be loaded come the next request). 2) $hash[ID_KEY]: what is ID_KEY? it seems to me to be a constant, so whenever you call this method, you're using the same key of an array, why, then, are you passing an array? \$\endgroup\$ – Elias Van Ootegem Dec 16 '13 at 8:34
4
\$\begingroup\$

Ok, since there is no answer, I'll write it myself what would have been really a great help and what I have discovered:

The main problem with this code is the usage of many expensive operations responsible for slow execution. It seems the split into keys and values before storing as well as sorting and then re-combining into arrays is really slow.

Why it matters? Because I am running my code using this storage on Iron.io as IronWorker. Iron is great but has limitations: each worker gets 320 MB memory and 1 hour maximum to run. In the past I hit the memory limit because I stored my data in as hash of hashes, i.e. in the form

$mainHash = array(
    id1 => array('key1' => 'value1', ... ),
       ...
);

with many rows! Bad idea!

Then after hitting the memory limit I turned to obsessive memory saving strategy, replacing row hashes by their JSONs. However, since all keys are repeated, I considered it a waste of memory to store them on each hash. So I split off keys in a separate array and only stored values! This is exactly what the above code does! Memory usage went nicely under 100 MB! But ... the execution time didn't fit into the 1 hour frame. Which cost me some headache for the few past days until today I had this great idea to try with actual JSONs:

$mainHash = array(
    id1 => json_encode(array('key1' => 'value1', ... )),
       ...
);

And the result is amazing! Memory is under 110 MB and I am inside 1 hour!!! So only marginal memory loss with huge win to get everything done in one worker!

So here is the winning cleaner solution:

class LocStoreFast extends LocStore {
   /** just store json hashes, no enforcement
    * @param string $table
    * @param associative array $hash, must have ID_KEY not null, must have other keys
    */
   public function create ($table, array $hash) {
      $id = $hash[ID_KEY];
      if ( isset($this->_values[$table]) && ! empty($this->_values[$table][$id]) ) {
         throw new Exception("Id '$id' already exists");
      }
      $this->_values[$table][$id] = json_encode($hash);
   }

    // read by id, empty array if not found
   public function read ($table, $id) {
      if ( ! $id || empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      return json_decode($this->_values[$table][$id], true);
   }

     // read and delete by id, empty array if not found
   public function readDelete ($table, $id) {
      if ( ! $id || empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      $result = json_decode($this->_values[$table][$id], true);
      unset($this->_values[$table][$id]);
      return $result;
   }

}
\$\endgroup\$
  • 4
    \$\begingroup\$ +1 awesome selfie! Sorry your question hasn't had more attention. Feel free to accept your own answer! \$\endgroup\$ – Mathieu Guindon Jan 31 '14 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.