Average across direction-specific average speeds and probabilities

Question

I have a wind rose with 72 evenly-spaced directions, spanning 360°, each describing a direction-specific average wind speed and associated probability. We must condense this information to 36 evenly-spaced direction bins. "probability" is the normalized frequency of occurrences. The probabilities should sum to one. If 5° has a 0.1 probability. that means that the wind came from that direction 10% of the time.

I average across the midpoints of each bin, which results in undesirably shifting the wind rose so that the center is at 2.5 degrees. Is there a more elegant way to do this in theory or implementation?

import pandas as pd
import numpy as np
import wget

# Load data!
url = 'https://raw.githubusercontent.com/scls19fr/windrose/master/samples/amalia_directionally_averaged_speeds.txt'
wget.download(url, 'amalia_directionally_averaged_speeds.txt')
dat = pd.read_csv('./amalia_directionally_averaged_speeds.txt',
                  sep=r'\s+', header=None, skiprows=[0])
dat.columns = ['direction', 'average_speed', 'probability']

# two steps
#   1. average frequency across speeds
#   2. compute average speed weighted by frequencies
#     (the scheme is to average across both midpoints. this
#      shift the centers by 2.5 degrees and bin by 10)
directions, speeds, frequencies = [], [], []
for i in range(dat.shape[0] / 2):
    directions.append(dat.direction[i * 2] + 2.5)
    frequencies.append(np.mean(dat.probability[[i * 2, i * 2 + 1]].values))
    speeds.append((dat.direction[i * 2] * dat.average_speed[i*2] +
                  dat.direction[i*2 + 1] * dat.average_speed[i * 2 + 1]) /
                  (dat.direction[i * 2] + dat.direction[i * 2+1]))

# save data
done = pd.DataFrame({'direction': directions,
                    'speed': speeds, 'probability': frequencies})
done.to_csv('wind_rose.csv')

Answer 1

The biggest issue with this code is it iterates over a DataFrame. Iterating over a DataFrame negates any performance improvements gained by using pandas. As much as possible, use operations that apply to the entire Series or an entire DataFrame at once. Allow the C code to do the looping for you so the data doesn't get pulled into slow Python objects.

There is no reason to download the data into a file using wget. This data can be downloaded directly into memory, and pandas supports http urls.

Look at where you place comments before blocks of code. These are perfect places to split up your code into functions.

"""Process unweighted wind data to file."""
import pandas as pd

URL = ('https://raw.githubusercontent.com/scls19fr/windrose/master/samples/'
       'amalia_directionally_averaged_speeds.txt')
OUTPUT_PATH = 'wind_rose.csv'


def load_wind_data(data_url):
    """Load wind data from url into a dataframe.

    Args:
        data_url(str|io):
            A URL, path, or buffer containing wind data.

    Returns:
        A dataframe containing wind data
    """

    dat = pd.read_csv(data_url, sep=r'\s+', header=None, skiprows=1,
                      names=['direction', 'average_speed', 'probability'])
    return dat

You can use the names argument of read_csv instead of setting columns. wget is not used and the url is passed into read_csv since read_csv supports http.

def average_wind_data(dat):
    """Average and weight wind data.

    Args:
        dat(pd.DataFrame):
            A dataframe containing unweighted wind data.
    Returns:
        A dataframe containing weighted, averaged wind data.

    1. average frequency across speeds
    2. compute average speed weighted by frequencies
    (the scheme is to average across both midpoints. this
     shift the centers by 2.5 degrees and bin by 10)
    """

    avg_df = pd.DataFrame()
    # shift(-1) leaves the NaN value at the end of the dataframe
    adj_sum_dat = dat + dat.shift(-1)
    avg_df['direction'] = (dat['direction'][::2] + 2.5).values
    avg_df['probability'] = (adj_sum_dat['probability'][::2]/2).values
    unweighted_speed = dat['direction'] * dat['average_speed']
    adj_speed_sum = unweighted_speed + unweighted_speed.shift(-1)
    avg_df['speed'] = (adj_speed_sum/adj_sum_dat['direction'])[::2].values
    return avg_df

Use shift() on Series or DataFrame objects to do vector operations on adjacent data. Use slice notation to get even data points ([::2] means all the entire index stepped over by 2). Do vector operations on entire series instead of each item on the series. This section of code is about 6x faster than the original iteration code.

def main(input_url, output_path):
    """Load wind data at input_url and save processed data to csv.

    Args:
        input_url(str|io):
            Location of wind data.
        output_path(str):
            File location to save processed wind data.
    """
    dat = load_wind_data(input_url)
    processed_df = average_wind_data(dat)
    processed_df.to_csv(output_path)


if __name__ == "__main__":
    main(URL, OUTPUT_PATH)

IO should be done at the top level of execution.