Since you have numpy
arrays, you should use their vectorized methods wherever possible. This can make your code a lot faster:
In [1]: x = np.arange(10000000)
In [2]: %timeit max(x)
988 ms ± 42.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [3]: %timeit x.max()
9.67 ms ± 114 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
This includes not casting your arrays to list
.
I would also make this a function that normalizes a single array:
import pickle
import numpy as np
from typing import Iterable, Any
def normalize_one(x: Iterable[Any]) -> np.ndarray:
if not isinstance(x, np.ndarray):
x = np.array(list(x))
low, diff = x.min(), x.ptp()
return (x - low) / diff
# -- load data
prop_1 = np.random.rand(10)*20
prop_2 = np.random.rand(10)*10
prop_3 = list(np.random.rand(10)*30
# -- normalize
prop_1 = normalize_one(prop_1)
prop_2 = normalize_one(prop_2)
prop_3 = normalize_one(prop_3)
If you do have many arrays that need to be normalized, you can always do it in a list comprehension:
properties = [prop_1, prop_2, prop_3]
properties = [normalize_one(prop) for prop in properties]
If you have many of them and they all have the same structure, I would use something like this (now limited to numpy
arrays as input):
def normalize(x: np.ndarray, axis: int = 1) -> np.ndarray:
"""Normalize the array to lie between 0 and 1.
By default, normalizes each row of the 2D array separately.
"""
low, diff = x.min(axis=axis), x.ptp(axis=axis)
# Indexing needed to help numpy broadcasting
return (x - low[:,None]) / diff[:,None]
properties = np.random.rand(3, 10)
properties[0] *= 20
properties[1] *= 10
properties[2] *= 30
properties = normalize(properties)
For props = np.random.rand(10000, 10)
I get the following timings:
Author Timed function call Time [s]
Blade* list(normalize_blade(props)) 68.7 ms ± 749 µs
Linny list(normalize_linny(*props)) 127 ms ± 1.42 ms
Graipher [normalize_one(prop) for prop in props] 119 ms ± 7.4 ms
Graipher normalize(props) 2.32 ms ± 113 µs
The code I used for the test with the code in the OP is this one, which is just the generalization to many properties:
def normalize_blade(properties):
l_bound, u_bound = [], []
properties = [list(prop) for prop in properties]
for prop in properties:
l_bound.append(min(prop))
u_bound.append(max(prop))
for i, prop in enumerate(properties):
yield (np.array(prop) - l_bound[i]) / (u_bound[i] - l_bound[i])
prop_*
as loaded bypickle
? \$\endgroup\$ – Reinderien Mar 30 '20 at 1:41f
three times. What am I missing? \$\endgroup\$ – Reinderien Mar 30 '20 at 1:47with open(dataset_name, "wb") as f: pickle.dump(prop_1, f) pickle.dump(prop_2, f) pickle.dump(prop_3, f)
\$\endgroup\$ – Blade Mar 30 '20 at 1:49