# Speeding up storage of about 220000 values in Core Data

I use this code on the app's first run to save about 220000 values into core data. The file Navaids.txt has about 200000 and Waypoints.txt has about 20000 lines, where I get data from each line and put it in an entity in core data.

I am still kind of new to CD but would like to see how I can improve this code on app's first launch to make it as fast as possible. Right now it takes about 10 seconds to run on an iPad but would like to see how I can improve.

Let me know if I should provide samples of my .txt files or if you need to see any of my delegate class.

if (self.firstRun == TRUE) {
NSString* navaidPath = [[NSBundle mainBundle] pathForResource:@"Navaids" ofType:@"txt"];
NSString* navContent = [NSString stringWithContentsOfFile:navaidPath encoding:NSUTF8StringEncoding error:NULL];
NSArray *navContentLines = [navContent componentsSeparatedByString:@"\r\n"];

NSString* wayptPath = [[NSBundle mainBundle] pathForResource:@"Waypoints" ofType:@"txt"];
NSString* wayptContent = [NSString stringWithContentsOfFile:wayptPath encoding:NSUTF8StringEncoding     error:NULL];
NSArray *wayptContentLines = [wayptContent componentsSeparatedByString:@"\r\n"];

@autoreleasepool {
for (int i = 0; i < [wayptContentLines count]; i++) {
@autoreleasepool {

NSArray *brokenFix = [[wayptContentLines objectAtIndex:i] componentsSeparatedByString:@"|"];

newContact = [NSEntityDescription
insertNewObjectForEntityForName:@"Waypoints"
inManagedObjectContext:context];

[newContact setValue: [brokenFix objectAtIndex:0] forKey:@"name"];
[newContact setValue: [brokenFix objectAtIndex:1] forKey:@"latitude"];
[newContact setValue: [brokenFix objectAtIndex:2] forKey:@"longitude"];

}
}
}

@autoreleasepool {
for (int i = 0; i < [navContentLines count]; i++) {
@autoreleasepool {

NSArray *brokenNav = [[navContentLines objectAtIndex:i] componentsSeparatedByString:@"|"];

newContact = [NSEntityDescription
insertNewObjectForEntityForName:@"Navaids"
inManagedObjectContext:context];

[newContact setValue: [brokenNav objectAtIndex:0] forKey:@"icao"];
[newContact setValue: [brokenNav objectAtIndex:1] forKey:@"name"];
[newContact setValue: [brokenNav objectAtIndex:6] forKey:@"latitude"];
[newContact setValue: [brokenNav objectAtIndex:7] forKey:@"longitude"];
}
}
}

NSError *error;
[context save:&error];


Ten seconds for reading/writing nearly half a million lines from/to disk is actually relatively decent.

The autorelease pool doesn't actually slow us down. Autorelease pool is handled by the preprocessor. So is ARC. Instead of us manually writing our release/retains, ARC, or autorelease pool, will do this for us. However, I actually would make a change to how you're handling the autorelease pools so that you're minimizing the amount of information you're keeping in RAM at any given time. For example:

@autoreleasepool {
// read waypoints from file into memory
// loop through waypoints array putting them into core data
}

@autoreleasepool {
// read navaids from file into memory
// loop through navaids array putting them into core data
}


Make no reference to waypoints outside the first autorelease pool and no reference to navaids outside the second autorelease pool and you'll be using approximately half as much RAM throughout this method.

Though... a cleaner way might be to just put each in their own method and then the same is accomplished without autorelease pools (which again, will not slow down your code).

We can however speed up your code. If we use a forin loop rather than a traditional for loop the code will run faster as we'll overall be passing significantly fewer messages. For example:

for (NSString *waypoint in wayptContentLines) {
NSArray *brokenFix = [waypoint componentsSeparatedByString:@"|"];

newContact = [NSEntityDescription insertNewObjectForEntityForName:@"Waypoints"
inManagedObjectContext:context];

[newContact setValue: [brokenFix objectAtIndex:0] forKey:@"name"];
[newContact setValue: [brokenFix objectAtIndex:1] forKey:@"latitude"];
[newContact setValue: [brokenFix objectAtIndex:2] forKey:@"longitude"];
}


Now, instead of calling to wayptContentLines twice per iteration (nearly a million times) (once to check length, once to grab the object at index), we're instead calling to the array less than once per iteration (the exact amount varies).

A forin loop in Objective-C reads several objects into a buffer then deals with the objects in the buffer before reading the objects in. I think on average it reads in about 8 objects, so you go from 2/i to 1/8i messages to wayptContentLines. In English, a for loops probably sends about 16 times as many messages to the array as a forin loop would... and over 400,000+ iterations, thats a LOT of messages and it will take up a lot of time.

• Thanks for this, the code is a bit faster now which is nice. I guess you're right 10sec for over 200000 objects is decent. I've implemented the changed you suggested such as the new forin loops. :) – m179 Oct 2 '14 at 12:38

### Common

Don't allocate additional memory for temporary objects like NSArray *brokenFix. Go through the array's element by hand, instead. You have array with a lot of sting formated objects in it, but you need to allocate only few of them. It will decrease number of allocations.

If it's still not enough, try to write clean c++ or c code.

• Hello, I removed the autoreleasepools, thanks, but I'm not sure what you mean by going through the array's element by hand... – m179 Sep 30 '14 at 21:50
• @MichaelShamash, I'm not shure that it will be much more efficient, but you can try do not allocate not used objects at all, you can extract substring by range and pass it as param to enother method. IMHO: high perfomance is not objective-c purpose, it is more c/c++. – outoftime Sep 30 '14 at 22:07
• ok I understand. Thought that's what you meant just wanted to make sure. This wouldn't be good for me though since my array of brokenFix is created from strings of varying lengths which is not possible to do substringFromIndex then. Thanks! – m179 Sep 30 '14 at 22:24
• @MichaelShamash, find indexes by looping through line – outoftime Sep 30 '14 at 22:39

You can improve performance by reducing the batch size. In my testing, if I saved these in batches of 1000-2000, I got the best performance (and it's also more memory efficient than doing all 220,000 objects at once). You might want to try different batch sizes and see if it differs given your object model.

I also, personally, wrap each of these batches in an autorelease pool, which ensures that memory consumed by autorelease objects is recovered. I would not, though, generally have an autorelease pool for each object saved. Just one autorelease pool around each group. (The objects are held in memory until it encounters save, anyway, so there is modest benefit in having @autoreleasepool more frequently than that.)

NSInteger count = ...;
NSInteger i = 0;

while (i < count) {
NSInteger batchEnd = MIN(i + kBatchSize, count);
@autoreleasepool {
while (i < batchEnd) {
NSManagedObject *object = [NSEntityDescription insertNewObjectForEntityForName:@"..."
inManagedObjectContext:context];
[object setValue:... forKey:@"..."];
[object setValue:... forKey:@"..."];

i++;
}

NSError *error;
if (![context save:&error]) {
NSLog(@"error saving %@", error);
}
}
}