Improving the speed of creation for three Perlin Noise Maps in Python?

Question

I am interested in learning how I can improve the speed of the code in this pygame file. I iterate over 6400 * 1800 * 3 or 34,560,000 elements of various numpy arrays here to apply noise values to them. The noise library I'm using can be found on GitHub here.

I am calling static variables from a class called ST here. ST.MAP_WIDTH = 6400 and ST.MAP_HEIGHT = 1800. All other ST attributes called here are assigned in the code. They are the noise-maps I'm making.

from __future__ import division
from singleton import ST
import numpy as np
import noise
import timeit
import random
import math


def __noise(noise_x, noise_y, octaves=1, persistence=0.5, lacunarity=2.0):
    """
    Generates and returns a noise value.

    :param noise_x: The noise value of x
    :param noise_y: The noise value of y
    :return: numpy.float32
    """

    value = noise.pnoise2(noise_x, noise_y,
                          octaves, persistence, lacunarity,
                          random.randint(1, 9999))

    return np.float32(value)


def __elevation_mapper(noise_x, noise_y):
    """
    Finds and returns the elevation noise for the given noise_x and
    noise_y parameters.

    :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer
    :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer
    :return: float
    """
    return __noise(noise_x, noise_y,  8, 0.9)


def __climate_mapper(y, noise_x, noise_y):
    """
    Finds and returns the climate noise for the given noise_x and
    noise_y parameters.

    :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer
    :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer
    :return: float
    """
    # find distance from bottom of map and normalize to range [0, 1]
    distance = math.sqrt((y - (ST.MAP_HEIGHT >> 1))**2) / ST.MAP_HEIGHT

    value = __noise(noise_x, noise_y,  8, 0.7)

    return (1 + value - distance) / 2


def __rainfall_mapper(noise_x, noise_y):
    """
    Finds and returns the rainfall noise for the given noise_x and
    noise_y parameters.

    :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer
    :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer
    :return: float
    """
    return __noise(noise_x, noise_y,  4, 0.65, 2.5)


def create_map_arr():
    """
    This function creates the elevation, climate, and rainfall noise maps,
    normalizes them to the range [0, 1], and then assigns them to their
    appropriate attributes in the singleton ST.
    """

    start = timeit.default_timer()

    elevation_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32)
    climate_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32)
    rainfall_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32)

    randomizer = random.uniform(0.0001, 0.9999)

    # assign noise map values
    for y in range(ST.MAP_HEIGHT):
        for x in range(ST.MAP_WIDTH):
            noise_x = x / ST.MAP_WIDTH - randomizer
            noise_y = y / ST.MAP_HEIGHT - randomizer

            elevation_arr[y][x] = __elevation_mapper(noise_x, noise_y)
            climate_arr[y][x] = __climate_mapper(y, noise_x, noise_y)
            rainfall_arr[y][x] = __rainfall_mapper(noise_x, noise_y)

    # normalize to range [0, 1] and assign to relevant ST attributes
    ST.ELEVATIONS = (elevation_arr - elevation_arr.min()) / \
                    (elevation_arr.max() - elevation_arr.min())

    ST.CLIMATES = (climate_arr - climate_arr.min()) / \
                  (climate_arr.max() - climate_arr.min())

    ST.RAINFALLS = (rainfall_arr - rainfall_arr.min()) / \
                   (rainfall_arr.max() - rainfall_arr.min())

    stop = timeit.default_timer()
    print("GENERATION TIME: " + str(stop - start))

Answer 1

Losing your Loops

Austin Hastings' comment gives you a good hint where to look at. The main takeaway for you should be:

(Most) loops are damn slow in Python. Especially multiple nested loops.

NumPy can help to vectorize your code, i.e. in this case that more of the looping is done in the C backend instead of in the Python interpreter. I would highly recommend to have a listen to the talk Losing your Loops: Fast Numerical Computing with NumPy by Jake VanderPlas. Although primarily tailored towards data science, it gives a good overview on the topic.

I did some slight modifications to your original script to include some of the vectorization ideas while still using your chosen Perlin noise library. (Sidenote: I changed the __ prefix to a single _, because that is the convention most Python programmers use for internal functions. See PEP8 style guide.)

# -*- coding: utf-8 -*-
from __future__ import division, print_function
import numpy as np
import noise
import timeit


class ST(object):
    MAP_HEIGHT = 1800
    MAP_WIDTH = 6400


def _noise(noise_x, noise_y, octaves=1, persistence=0.5, lacunarity=2.0):
    """
    Generates and returns a noise value.

    :param noise_x: The noise value of x
    :param noise_y: The noise value of y
    :return: numpy.float32
    """
    if isinstance(noise_x, np.ndarray):
        #rand_seed = np.random.randint(1, 9999, noise_x.size)
        rand_seed = np.ones((noise_x.size, ))   # just for comparison
        value = np.array([noise.pnoise2(x, y, octaves, persistence, lacunarity, r)
                          for x, y, r in zip(noise_x.flat, noise_y.flat, rand_seed)])
        return value.reshape(noise_x.shape)
    else:
        value = noise.pnoise2(noise_x, noise_y,
                              octaves, persistence, lacunarity,
                              1.0)   # just for comparison
                              #np.random.randint(1, 9999))
        return np.float32(value)


def _elevation_mapper(noise_x, noise_y):
    """
    Finds and returns the elevation noise for the given noise_x and
    noise_y parameters.

    :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer
    :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer
    :return: float
    """
    return _noise(noise_x, noise_y, 8, 0.9)


def _climate_mapper(y, noise_x, noise_y):
    """
    Finds and returns the climate noise for the given noise_x and
    noise_y parameters.

    :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer
    :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer
    :return: float
    """
    # find distance from bottom of map and normalize to range [0, 1]
    distance = np.sqrt((y - (ST.MAP_HEIGHT >> 1))**2) / ST.MAP_HEIGHT

    value = _noise(noise_x, noise_y, 8, 0.7)

    return (1.0 + value - distance) / 2.0


def _rainfall_mapper(noise_x, noise_y):
    """
    Finds and returns the rainfall noise for the given noise_x and
    noise_y parameters.

    :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer
    :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer
    :return: float
    """
    return _noise(noise_x, noise_y, 4, 0.65, 2.5)


def create_map_arr():
    """
    This function creates the elevation, climate, and rainfall noise maps,
    normalizes them to the range [0, 1], and then assigns them to their
    appropriate attributes in the singleton ST.
    """
    # assign noise map values
    randomizer = np.random.uniform(0.0001, 0.9999)

    start_arr = timeit.default_timer()

    X, Y = np.mgrid[0:ST.MAP_WIDTH, 0:ST.MAP_HEIGHT]
    noise_x = X / ST.MAP_WIDTH - randomizer
    noise_y = Y / ST.MAP_HEIGHT - randomizer
    elevation_arr_np = _elevation_mapper(noise_x, noise_y)
    climate_arr_np = _climate_mapper(Y, noise_x, noise_y)
    rainfall_arr_np = _rainfall_mapper(noise_x, noise_y)

    duration_arr = timeit.default_timer() - start_arr

    start_loop = timeit.default_timer()

    elevation_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32)
    climate_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32)
    rainfall_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32)
    for y in range(ST.MAP_HEIGHT):
        for x in range(ST.MAP_WIDTH):
            noise_x = x / ST.MAP_WIDTH - randomizer
            noise_y = y / ST.MAP_HEIGHT - randomizer

            elevation_arr[y, x] = _elevation_mapper(noise_x, noise_y)
            climate_arr[y, x] = _climate_mapper(y, noise_x, noise_y)
            rainfall_arr[y, x] = _rainfall_mapper(noise_x, noise_y)

    duration_loop = timeit.default_timer() - start_loop

    print(np.allclose(elevation_arr, elevation_arr_np.T))
    print(np.allclose(climate_arr, climate_arr_np.T))
    print(np.allclose(rainfall_arr, rainfall_arr_np.T))

    print("GENERATION TIME: loop: {:.6f}, array: {:.6f}".format(duration_loop, duration_arr))

if __name__ == "__main__":
    create_map_arr()

The bottleneck is still in

value = np.array([noise.pnoise2(x, y, octaves, persistence, lacunarity, r)
                  for x, y, r in zip(noise_x.flat, noise_y.flat, rand_seed)])

and it would be highly favorable to use an implementation which supports 2D input, preferably from NumPy, directly (see further reading below).

Nevertheless, the modifications bring the execution time down to a third of the original time on my machine (which is not that powerful):

True
True
True
GENERATION TIME: loop: 338.094228, array: 101.549388

Those three Trues are from a little test I added to check if the generated maps are the same within reasonable accuracy. For this purpose the additional random value in _noise was disabled.

Further reading

There have also already been similar questions on Code Review (see, e.g. here), where a reviewer created a Perlin noise implementation purely in Numpy. There also seems to be a GitHub project also doing Perlin noise with Numpy. So maybe have a look at them if your not forced to stick with noise.