In-memory data cache architecture for data transfer

Question

I am writing a cron job to manipulate and transfer remote data. I need to cache data from a table, process it, merge with previous table, iterate the process, and eventually send the result data to remote database.

Here is a basic in-memory cache storage with thin simple CRUD type API. It has to store hashes, enforce equal keys and unique ids. The data come from tables and immediately transformed into hashes (key-value pairs). That way hashes are portable and no reference to tables is needed. Also they can be easily ported to NoSQLs.

The design is inspired by Parse.com and StackMob.

Somehow I could not find any similar light weight library. I need it portable to easily transfer to any server. I would rather avoid any heavy installation or configuration.

An older version was posted here.

Any critique is welcome.

// I am using 'i' as primary key for all hashes
define('ID_KEY', 'i');

/** Local In-Memory Cache: Tables referred by their string names
 * Storing hashes:
 *    all hashes must have ID_KEY as key
 *    all hashes must have keys other than ID_KEY
 *    and hashes in the same table must have the same key structure (enforced)
 * Basic CRUD interface + 'readOne' for the first entry + 'realDelete'
 * arrays packed into json strings, which allows for array values,
 * e.g. array('key1'=>'val1', 'key2'=> array(1,2,3))
 */
class LocStore {
   // assoc. array('table' => array of keys)
   private $_keys = array();
   // assoc. array of strings indexed by table and id (must be nonempty)
   private $_values = array(); 

   /** create entry, enforce the same keys and new id
    * @param string $table
    * @param ass. array $hash, must have ID_KEY not null, must have other keys
    */
   public function create ($table, array $hash) {
      $id = $hash[ID_KEY];
      if (! $id) throw new Exception('Id is empty');
      unset($hash[ID_KEY]);
      if (! $hash) throw new Exception('Empty hash beside id');
      // sort by keys alphabetically to ensure the same order for all entries
      ksort($hash);
      if ( empty($this->_keys[$table]) ) {
         // first hash for $table - cache keys
         $this->_keys[$table] = array_keys($hash);
      } elseif ($this->_keys[$table] != array_keys($hash)) {
         throw new Exception('Array keys mismatch: '. json_encode($hash) );
      }
      if ( isset($this->_values[$table]) && ! empty($this->_values[$table][$id]) ) {
         throw new Exception("Id '$id' already exists");
      }
      $this->_values[$table][$id] = json_encode(array_values($hash));
      // for chaining
      return $this;
   }

   // read one entry, empty array if nothing is there
   // reset pointer for iteration
   public function reset ($table) {
      if (! empty($this->_values[$table]) ) {
         reset($this->_values[$table]);         
      }
   }

   // read next hash and advance pointer
   public function readNext ($table) {
      if (empty($this->_values[$table])) return array();
      $id = key($this->_values[$table]);
      next($this->_values[$table]);
      if (! $id) return array ();
      return $this->read($table, $id);    
   }

   // read one entry if there are some or else return empty array
   // in contrast to readNext, the result is always non-empty as long as table has hashes
   public function readOne ($table) {
      if ( empty($this->_values[$table]) ) return array();
      foreach ($this->_values[$table] as $id => $val) {
         return $this->read($table, $id);         
      }
      return array();
   }

   // read by id, empty array if not found
   public function read ($table, $id) {
      if (! $id) throw new Exception('Id is empty');
      if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      //$keys = $this->_toArray($this->_keys[$table]);
      $keys = $this->_keys[$table];
      //$values = $this->_toArray($this->_values[$table][$id]);
      $values = json_decode($this->_values[$table][$id]);
      $result = array_combine($keys, $values);
      $result[ID_KEY] = (string) $id;
      return $result;
   }

   // read and delete one nonempty array or return empty if nothing is left
   public function readDeleteOne ($table) {
      if ( empty($this->_values[$table]) ) return array();
      foreach ($this->_values[$table] as $id => $val) {
         return $this->readDelete($table, $id);         
      }
      return array();
   }

   // read and delete by id, empty array if not found
   public function readDelete ($table, $id) {
      if (! $id) throw new Exception('Id is empty');
      if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      $keys = $this->_keys[$table];
      $values = json_decode($this->_values[$table][$id]);
      $result = array_combine($keys, $values);
      $result[ID_KEY] = (string) $id;
      // deletion is here
      unset($this->_values[$table][$id]);
      return $result;
   }

   // delete by id, exception if not found
   public function delete ($table, $id) {
      if (! $id) throw new Exception('Id is empty');
      if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) {
         throw new Exception('Deleting non-existant entry');
      }
      unset($this->_values[$table][$id]);
      return $this;
   }

   public function deleteAll ($table) {
      $this->_values[$table] = array();
      return $this;
   }
}

EDIT:

Here is the (only) method from another class using reset and readNext (where $this->_locStore points to the class locStore above):

public function loadChains ($table, $mergeTable, $relKey) {
  $this->_locStore->reset($mergeTable);
  while ( $hash = $this->_locStore->readNext($mergeTable) ) {
     $chain = $this->getChain($mergeTable, $hash[ID_KEY], $relKey);
     $hash[$relKey] = $chain;
     $this->_locStore->create($table, $hash);
  }
  $this->_locStore->deleteAll($mergeTable);
}

Answer 1

Ok, since there is no answer, I'll write it myself what would have been really a great help and what I have discovered:

The main problem with this code is the usage of many expensive operations responsible for slow execution. It seems the split into keys and values before storing as well as sorting and then re-combining into arrays is really slow.

Why it matters? Because I am running my code using this storage on Iron.io as IronWorker. Iron is great but has limitations: each worker gets 320 MB memory and 1 hour maximum to run. In the past I hit the memory limit because I stored my data in as hash of hashes, i.e. in the form

$mainHash = array(
    id1 => array('key1' => 'value1', ... ),
       ...
);

with many rows! Bad idea!

Then after hitting the memory limit I turned to obsessive memory saving strategy, replacing row hashes by their JSONs. However, since all keys are repeated, I considered it a waste of memory to store them on each hash. So I split off keys in a separate array and only stored values! This is exactly what the above code does! Memory usage went nicely under 100 MB! But ... the execution time didn't fit into the 1 hour frame. Which cost me some headache for the few past days until today I had this great idea to try with actual JSONs:

$mainHash = array(
    id1 => json_encode(array('key1' => 'value1', ... )),
       ...
);

And the result is amazing! Memory is under 110 MB and I am inside 1 hour!!! So only marginal memory loss with huge win to get everything done in one worker!

So here is the winning cleaner solution:

class LocStoreFast extends LocStore {
   /** just store json hashes, no enforcement
    * @param string $table
    * @param associative array $hash, must have ID_KEY not null, must have other keys
    */
   public function create ($table, array $hash) {
      $id = $hash[ID_KEY];
      if ( isset($this->_values[$table]) && ! empty($this->_values[$table][$id]) ) {
         throw new Exception("Id '$id' already exists");
      }
      $this->_values[$table][$id] = json_encode($hash);
   }

    // read by id, empty array if not found
   public function read ($table, $id) {
      if ( ! $id || empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      return json_decode($this->_values[$table][$id], true);
   }

     // read and delete by id, empty array if not found
   public function readDelete ($table, $id) {
      if ( ! $id || empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array();
      $result = json_decode($this->_values[$table][$id], true);
      unset($this->_values[$table][$id]);
      return $result;
   }

}