Description:
This would make a first part of a currency exchange simulator The first phase includes downloading currency historical data from histdata.com api, plotting given period and given time frames using different plot types and adding some forex technical indicators to the data. I will be posting follow ups when I'm done with further phases of the project.
Feedback:
I need feedback regarding the whole program structure, and general suggestions for improvements/optimizations/features to add are welcome. I'm intending to add a simulation functionality with virtual money and automate a bot that runs on the historical data which predicts future values and maximize profit and maybe a GUI. I also need suggestions on predicting future values for a given time period in the future using machine learning I don't know maybe neural networks - regression - nlp ... I need guidance on focus points(what libraries/techniques...) for building a robust machine learning prediction system as I'm not quite experienced in ml.
Links:
Please download the following zip file and extract the contents(2 folders beginning with 'DAT') in the same folder as the code's (to be able to run the simulator on the period(2017-2018) for eurusd pair:
2017-2018 eurusd M1(1 minute data)
Files
forex_data_handler.py: This is used to download forex historical data from histdata.com api (Use pip3 install histdata beforehand if you're going to download extra data) and this is optional (Links are provided above for minimal trial-run) and of course feel free to download whatever data you want using this module for trying extra examples.fx_simulator.py: This is the simulation part of the code and since it's the first part of the project, it contains the following features:A) The
FxSimulatorconstructor accepts parameters such as currency pairs, years and a few other parameters(you'll find everything in the docstrings)B)
get_interval()method which returns a specific time frame pandas DF. Time frames ex: M1 = 1 min, D5 = 5 days ... check the documentation.C)
compare_years()andcompare_year_months()methods to enable comparisons of months and years periods using a selection of parameters includinginterval,comparison_value...D)
plot_initial_data()method to plot the historical data (scatter, line, histogram ...E)
add_indicators()method to add forex technical indicator values to the pandas df. Indicators include(Bollinger Bands, RSI, Moving Averages ...)pairs.txt: Contains a list of 66 different currency pairs to choose from while downloading the data(optional) from histdata.com api.
Example Plots:
- 2010-2018 period eurusd 'Close' 1 Day(D1) year comparisons(1 figure):
- 2010-2018 period eurusd 'Close' 1 Day(D1) year comparisons(Multiple plots):
- 2019 Months comparison with Moving average(14 days) for eurusd 'Close' 1 Day(D1)
- 2019 Seasonal decompose for eurusd 'Close' 1 Day(D1)
forex_data_handler.py:
from histdata import download_hist_data
from concurrent.futures import ThreadPoolExecutor, as_completed
from time import perf_counter
import os
def get_folder_name(year, platform, time_frame, pair, month=None, compressed=True):
"""
Produce folder name in histdata.com download format.
Args:
year: ex: 2010
platform: MT, ASCII, XLSX, NT, MS.
time_frame: M1 (one minute) or T (tick data).
pair: ex: 'eurgbp'
month: int range(1, 12) inclusive
compressed: If compressed, a folder name ending with '.zip' is returned
"""
if not month:
folder_name = '_'.join(['DAT', platform, pair.upper(), time_frame, str(year)])
if compressed:
return folder_name + '.zip'
else:
return folder_name
if month and month <= 9:
month = '0' + str(month)
folder_name = '_'.join(['DAT', platform, pair.upper(), time_frame, str(year) + str(month)])
if compressed:
return folder_name + '.zip'
if not compressed:
return folder_name
def download_fx_data(start_year, end_year, pairs, threads=50, time_frame='M1', platform='MT', output_directory='.',
current_year=2019, current_month=12):
"""
Download Forex data over a given time range from histdata api.
Args:
start_year: int: The starting year.
end_year: int: The ending year.
pairs: A list of pairs ex: ['eurgbp', 'eurusd']
threads: Number of parallel threads.
time_frame: M1 (one minute) or T (tick data).
platform: MT, ASCII, XLSX, NT, MS.
output_directory: Where to dump the data.
current_year: Current year.
current_month: Current month.
"""
month_pairs = []
if end_year >= current_year:
last_year = current_year - 1
else:
last_year = end_year
years_pairs = [(year, pair) for year in range(start_year, last_year + 1) for pair in pairs
if get_folder_name(year, platform, time_frame, pair) not in os.listdir(output_directory)]
if end_year >= current_year:
month_pairs.extend([(month, pair) for pair in pairs for month in range(1, current_month)
if get_folder_name(current_year, platform, time_frame, pair, month)
not in os.listdir(output_directory)])
with ThreadPoolExecutor(max_workers=threads) as executor:
future_years_data = {executor.submit(download_hist_data, str(year), None, pair, time_frame,
platform, output_directory): (year, pair) for year, pair in years_pairs}
future_months_data = {executor.submit(download_hist_data, str(current_year), str(month), pair, time_frame,
platform, output_directory): (month, pair) for month, pair in month_pairs}
for future_file in as_completed(future_years_data):
try:
future_file.result()
except AssertionError:
print(f'Failure to retrieve {future_years_data[future_file]}')
for future_file in as_completed(future_months_data):
try:
future_file.result()
except AssertionError:
print(f'Failure to retrieve {future_months_data[future_file]}')
if __name__ == '__main__':
start_time = perf_counter()
# Change the next line for specific choices, this is set to download all the data available.
# pair_list = [pair.rstrip() for pair in open('pairs.txt').readlines()]
# download_fx_data(2000, 2020, pair_list)
end_time = perf_counter()
print(f'Process finished ... {end_time - start_time} seconds.')
fx_simulator.py:
from statsmodels.tsa.seasonal import seasonal_decompose
from forex_data_handler import get_folder_name
import matplotlib.pyplot as plt
import pandas as pd
import os
class FxSimulator:
"""A tool for conducting forex backtesting and simulation."""
def __init__(self, pair, years, history_data_path='.', platform='MT', time_frame='M1', current_year=2019,
current_month=12):
"""
Initialize year data.
Args:
pair: A string of currency pair ex: 'eurusd'
years: a list of years.
history_data_path: Folder path containing the historical data.
platform: MT, ASCII, XLSX, NT, MS.
time_frame: M1 or T
current_year: Current year ex: 2019
current_month: Current month ex: 8
"""
not_supported_years = [yr for yr in years if yr not in range(2000, current_year + 1) or yr > current_year]
if not_supported_years:
raise ValueError(f'Invalid year {not_supported_years[0]}'
f'\nYears supported are in range 2000-current year.')
self.currency_pair = pair
self.years = years
self.path = history_data_path
self.platform = platform
self.time_frame = time_frame
self.current_year = current_year
self.current_month = current_month
self.column_names = {'M1': ['Date', 'Time', 'Open', 'High', 'Low', 'Close', 'Volume']}
def combine_frames(self, folders, sep=','):
"""
Combine folder data.
Args:
folders: A list of folders.
sep: Separator of the csv file.
Return:
A data frame of all folders combined.
"""
frames = []
for folder_name in folders:
try:
os.chdir(self.path + folder_name)
current_frame = pd.read_csv(folder_name + '.csv', sep=sep, names=self.column_names[self.time_frame],
parse_dates=True)
frames.append(current_frame)
except FileNotFoundError:
print(f'Folder: {folder_name} not found.')
if frames:
return pd.concat(frames)
def load_data(self, sep=',', year=None):
"""
Load data of the given year range from csv.
Args:
sep: Separator of the csv file.
year: To load a particular year data.
Return:
A data frame with the following columns (check self.column_names)
"""
current_year_month_folders = [get_folder_name(
self.current_year, self.platform, self.time_frame, self.currency_pair, month, False)
for month in range(1, self.current_month)]
if year:
if year not in self.years:
raise ValueError(f'Year {year} not included in self.years')
if year != self.current_year:
folder_name = get_folder_name(year, self.platform, self.time_frame, self.currency_pair, None, False)
return self.combine_frames([folder_name], sep)
if year == self.current_year:
return self.combine_frames(current_year_month_folders, sep)
folder_names = [get_folder_name(yr, self.platform, self.time_frame, self.currency_pair, None, False)
for yr in self.years if yr != self.current_year]
if self.current_year in self.years:
folder_names.extend(current_year_month_folders)
return self.combine_frames(folder_names, sep)
def get_interval(self, interval, year=None, day_timing='12:30'):
"""
Set desired interval.
Args:
interval:
'M' + Minute interval(int 1 - 60) ex: M15 --> 15 minute interval.
'H' + Hour interval(int 1 - 24) ex: H4 --> 4 hour interval.
'D' + Day interval(int 1 - 31) ex: D1 --> 1 day interval.
'W' + Week interval(int 1 - 4)
year: If year, interval of the year will be returned.
day_timing: Timing of the day to get intervals.
Return:
Adjusted data frame.
"""
if year:
period_data = self.load_data(year=year)
else:
period_data = self.load_data()
if 'M' in interval:
current_minutes = period_data['Time'].apply(lambda timing: int(timing.split(':')[-1]))
return period_data[current_minutes % int(interval[1:]) == 0]
if 'H' in interval:
hour_data = period_data[period_data['Time'].apply(lambda timing: timing.split(':')[-1] == '00')]
only_hours = hour_data['Time'].apply(lambda timing: int(timing.split(':')[0]))
valid_hours = only_hours % int(interval[1:]) == 0
return hour_data[valid_hours]
if 'D' in interval:
day_data = period_data[period_data['Time'] == day_timing]
day_only = day_data['Date'].apply(lambda timing: int(timing.split('.')[-1]))
return day_data[day_only % int(interval[1:]) == 0]
if 'W' in interval:
week_data = period_data[period_data['Date'].apply(lambda timing: int(timing.split('.')[-1]) % 7) == 0]
valid_weeks = week_data[week_data['Date'].apply(lambda timing:
int(timing.split('.')[-1]) % int(interval[1:])) == 0]
return valid_weeks[valid_weeks['Time'] == day_timing]
raise ValueError(f'Invalid interval: {interval}')
def compare_years(self, interval, comparison_value):
"""
Get A monthly indexed data for years in self.years.
Args:
interval: Period string indication ex: 'M1', 'D5' ...
comparison_value: A string indication of value to compare: ex: 'Open', Close ...
Return:
A data frame containing year data per month.
"""
frames = []
for year in self.years:
data = self.get_interval(interval, year)
data['Month'] = data['Date'].apply(lambda timing: timing.split('.')[1])
data.reset_index(drop=True, inplace=True)
frames.append(data)
all_years = pd.concat(frames, axis=1).dropna()
all_years.set_index('Month', inplace=True)
all_years.reset_index(inplace=True)
all_years['Month'] = all_years['Month'].apply(lambda months: set(months).pop())
years_data = all_years[['Month', comparison_value]]
years_data.columns = ['Month'] + [comparison_value + '(' + str(yr) + ')' for yr in self.years]
return years_data
def compare_year_months(self, year, interval):
"""
Get A certain year months.
Args:
year: A year specified in self.years
interval: Period string indication ex: 'M1', 'D5' ...
Return:
A data frame containing months of a certain year.
"""
if year not in self.years:
raise ValueError(f'Year {year} not included in self.years')
year_data = self.get_interval(interval, year)
year_data['Month'] = year_data['Date'].apply(lambda timing: timing.split('.')[1])
return year_data
def plot_initial_data(self, plot_type, interval, year=None, years=False, comparison_value=None,
moving_average=None, **kwargs):
"""
Plot initial data in several graph forms.
Args:
plot_type: A string indication to graph type.
- li: Line plot.
- h: Histogram.
- bw: Box & whiskers plot.
- lp: Lag plot.
- ac: Auto-correlation plot.
- sda: Additive Seasonal decompose(Observed-Trend-Seasonal-Residual).
- sdm: Multiplicative Seasonal decompose(Observed-Trend-Seasonal-Residual).
interval: Period string indication ex: 'M1', 'D5' ...
year: If True, only months of given year will be plotted.
years: If True, years will be compared.
comparison_value: If years, this would be a string indication of value to compare against
('Open', Close ...)
moving_average: int representing the moving average window to be displayed (works when a comparison
value is specified).
**kwargs: Additional keyword arguments.
Return:
None
"""
if years and not comparison_value:
raise ValueError(f'Comparison value not specified for years=True')
if year and years:
raise ValueError(f'Cannot compare a single year and all years, please specify year or years')
if moving_average and not comparison_value:
raise ValueError(f'Comparison value not specified for moving average {moving_average}')
if plot_type in ['sda', 'sdm'] and not comparison_value:
raise ValueError(f'Must specify a comparison value for seasonal decomposition')
period_data = pd.DataFrame()
if not year and not years:
period_data = self.get_interval(interval)
period_data.set_index('Date', inplace=True)
if comparison_value:
period_data = period_data.filter(like=comparison_value)
if moving_average:
period_data['Moving Average'] = period_data[comparison_value].rolling(moving_average).mean()
if years:
period_data = self.compare_years(interval, comparison_value)
period_data.set_index('Month', inplace=True)
if moving_average:
for column in period_data.columns:
if comparison_value in column:
period_data['MA ' + column] = period_data[column].rolling(moving_average).mean()
if year:
period_data = self.compare_year_months(year, interval)
period_data.set_index('Month', inplace=True)
if comparison_value:
period_data = period_data.filter(like=comparison_value)
if moving_average:
period_data['Moving Average'] = period_data[comparison_value].rolling(moving_average).mean()
if plot_type == 'li':
period_data.plot(**kwargs)
if plot_type == 'h':
period_data.hist(**kwargs)
if plot_type == 'bw':
period_data.boxplot(**kwargs)
if plot_type == 'sda':
result = seasonal_decompose(period_data.filter(like=comparison_value), model='additive', freq=1)
result.plot()
if plot_type == 'sdm':
result = seasonal_decompose(period_data.filter(like=comparison_value), model='multiplicative', freq=1)
result.plot()
plt.show()
def add_indicators(self, interval, indicators, significant_value='Close', year=None, day_timing='12:30',
rsi_period=14, si_period=14, bb_period=20, kn_period=9, kj_period=26, fast_ema=12,
slow_ema=26, signal_line=9, adx_period=14):
"""
Add Forex technical indicators for the specified interval.
Args:
interval:
'M' + Minute interval(int 1 - 60) ex: M15 --> 15 minute interval.
'H' + Hour interval(int 1 - 24) ex: H4 --> 4 hour interval.
'D' + Day interval(int 1 - 31) ex: D1 --> 1 day interval.
'W' + Week interval(int 1 - 4).
indicators: A list of Technical indicators including:
- 'rsi': Relative strength indicator(RSI).
- 'si': Stochastic oscillator indicator(SI).
- 'bb': Bollinger Bands indicator.
- 'ic': Ichimoku Cloud.
- 'macd': Moving Average Convergence Divergence(MACD).
- 'pp': Pivot Point.
- 'adx': Average Directional movement(ADX)
significant_value: Close - Open - High - Low.
year: If year, interval of the year will be returned.
day_timing: Timing of the day to get intervals.
rsi_period: Period of averaging for the relative strength(RSI) indicator.
si_period: Period of averaging for the stochastic indicator(SI).
bb_period: Period of averaging for the Bollinger Bands indicator.
kn_period: Period of averaging for the Ichimoku Cloud indicator(Kenkan-Sen).
kj_period: Period of averaging for the Ichimoku Cloud indicator(Kijun Sen).
fast_ema: Period of exponential moving average(12 days)
slow_ema: Period of exponential moving average(26 days)
signal_line: MACD signal line.
adx_period: Average True Range(ATR) period.
Return:
Data frame with the adjusted technical indicators.
"""
period_data = self.get_interval(interval, year, day_timing).reset_index(drop=True)
if 'rsi' in indicators:
period_data['Change'] = period_data[significant_value] - period_data[significant_value].shift(1)
period_data['Upward Movement'] = period_data['Change'].apply(lambda change: change if change > 0 else 0)
period_data['Downward Movement'] = period_data['Change'].apply(lambda change: abs(change)
if change < 0 else 0)
period_data['Average Upward Movement'] = period_data['Upward Movement'].rolling(rsi_period).mean()
period_data['Average Downward Movement'] = period_data['Downward Movement'].rolling(rsi_period).mean()
period_data['Relative Strength(RS)'] = (period_data['Average Upward Movement']
/ period_data['Average Downward Movement'])
period_data['Relative Strength Index(RSI)'] = 100 - (100 / (1 + period_data['Relative Strength(RS)']))
if 'si' in indicators:
period_data['Lowest Low'] = period_data['Low'].rolling(si_period).min()
period_data['Highest High'] = period_data['High'].rolling(si_period).max()
period_data['Stochastic Oscillator Index'] = 100 * (period_data['Close'] - period_data['Lowest Low']) / (
period_data['Highest High'] - period_data['Lowest Low'])
if 'bb' in indicators:
period_data['Bollinger Middle Band'] = period_data[significant_value].rolling(bb_period).mean()
period_data['Bollinger Standard Deviation'] = period_data[significant_value].rolling(bb_period).std()
period_data['Upper Bollinger Band'] = period_data['Bollinger Middle Band'] + (
2 * period_data['Bollinger Standard Deviation'])
period_data['Lower Bollinger Band'] = period_data['Bollinger Middle Band'] - (
2 * period_data['Bollinger Standard Deviation'])
if 'ic' in indicators:
period_data['Lowest Low(Kenkan-Sen)'] = period_data['Low'].rolling(kn_period).min()
period_data['Highest High(Kenkan-Sen)'] = period_data['High'].rolling(kn_period).max()
period_data['Kenkan-Sen'] = (
period_data['Highest High(Kenkan-Sen)'] + period_data['Lowest Low(Kenkan-Sen)']) / 2
period_data['Lowest Low(Kijun Sen)'] = period_data['Low'].rolling(kj_period).min()
period_data['Highest High(Kijun Sen)'] = period_data['High'].rolling(kj_period).max()
period_data['Kijun-Sen'] = (
period_data['Highest High(Kijun Sen)'] + period_data['Lowest Low(Kijun Sen)']) / 2
period_data['Chikou-Span'] = period_data[significant_value].shift(-kj_period)
period_data['Senkou(Span A)'] = ((period_data['Kenkan-Sen'] + period_data['Kijun-Sen']) / 2).shift(
kj_period)
period_data['Lowest Low(52)'] = period_data['Low'].rolling(2 * kj_period).min()
period_data['Highest High(52)'] = period_data['High'].rolling(2 * kj_period).max()
period_data['Senkou(Span B)'] = ((period_data['Lowest Low(52)'] + period_data['Highest High(52)']) / 2
).shift(kj_period)
if 'macd' in indicators:
period_data['Fast EMA'] = period_data[significant_value].rolling(fast_ema).mean()
period_data['Slow EMA'] = period_data[significant_value].rolling(slow_ema).mean()
period_data['MACD'] = period_data['Fast EMA'] - period_data['Slow EMA']
period_data['MACD Signal'] = period_data['MACD'].rolling(signal_line).mean()
period_data['MACD Histogram'] = period_data['MACD'] - period_data['MACD Signal']
if 'pp' in indicators:
period_data['Pivot point'] = (period_data['High'] + period_data['Low'] + period_data['Close']) / 3
period_data['First Resistance(R1)'] = (2 * period_data['Pivot point']) - period_data['Low']
period_data['First Support(S1)'] = (2 * period_data['Pivot point']) - period_data['High']
period_data['Second Resistance(R2)'] = period_data['Pivot point'] + period_data[
'High'] - period_data['Low']
period_data['Second Support(S2)'] = period_data['Pivot point'] - (period_data['High'] - period_data['Low'])
period_data['Third Resistance(R3)'] = period_data['High'] + 2 * (
period_data['Pivot point'] - period_data['Low'])
period_data['Third Support(S3)'] = period_data['Low'] - 2 * (
period_data['High'] - period_data['Pivot point'])
if 'adx' in indicators:
period_data['High-Low Difference'] = period_data['High'] - period_data['Low']
period_data['High-Previous Close Difference'] = period_data['High'] - period_data['Close'].shift(1)
period_data['Previous Close and Current Low Difference'] = period_data['Close'].shift(1) - period_data[
'Low']
period_data['True Range(TR)'] = period_data[['High-Low Difference', 'High-Previous Close Difference',
'Previous Close and Current Low Difference']].dropna().max(
axis=1)
period_data['Average True Range(ATR)'] = period_data['True Range(TR)'].rolling(adx_period).mean()
period_data['High-Previous High Difference'] = period_data['High'] - period_data['High'].shift(1)
period_data['Previous Low-Low Difference'] = period_data['Low'].shift(1) - period_data['Low']
positive_condition = period_data[
'High-Previous High Difference'] > period_data['Previous Low-Low Difference']
period_data['Positive DX'] = period_data['High-Previous High Difference'][positive_condition]
period_data['Positive DX'] = period_data['Positive DX'].fillna(0)
negative_condition = period_data[
'High-Previous High Difference'] < period_data['Previous Low-Low Difference']
period_data['Negative DX'] = period_data['Previous Low-Low Difference'][negative_condition]
period_data['Negative DX'] = period_data['Negative DX'].fillna(0)
period_data['Smooth Positive DX'] = period_data['Positive DX'].rolling(adx_period).mean()
period_data['Smooth Negative DX'] = period_data['Negative DX'].rolling(adx_period).mean()
period_data['Positive Directional Movement Index(+DMI)'] = (
period_data['Smooth Positive DX'] / period_data['Average True Range(ATR)']) * 100
period_data['Negative Directional Movement Index(-DMI)'] = (
period_data['Smooth Negative DX'] / period_data['Average True Range(ATR)']) * 100
period_data['Directional Index(DX)'] = abs(
period_data['Positive Directional Movement Index(+DMI)'] -
period_data['Negative Directional Movement Index(-DMI)']) / (
period_data[['Positive Directional Movement Index(+DMI)', 'Negative Directional Movement Index(-DMI)']]
).sum(axis=1) * 100
period_data['Average Directional Index(ADX)'] = period_data['Directional Index(DX)'].rolling(
adx_period).mean()
return period_data
if __name__ == '__main__':
test_years = [year for year in range(2017, 2019)]
x = FxSimulator('eurusd', test_years)
x.plot_initial_data('li', 'D1', comparison_value='Close')
plt.show()
pairs.txt:
eurusd
eurchf
eurgbp
eurjpy
euraud
usdcad
usdchf
usdjpy
usdmxn
gbpchf
gbpjpy
gbpusd
audjpy
audusd
chfjpy
nzdjpy
nzdusd
xauusd
eurcad
audcad
cadjpy
eurnzd
grxeur
nzdcad
sgdjpy
usdhkd
usdnok
usdtry
xauaud
audchf
auxaud
eurhuf
eurpln
frxeur
hkxhkd
nzdchf
spxusd
usdhuf
usdpln
usdzar
xauchf
zarjpy
bcousd
etxeur
eurczk
eursek
gbpaud
gbpnzd
jpxjpy
udxusd
usdczk
usdsek
wtiusd
xaueur
audnzd
cadchf
eurdkk
eurnok
eurtry
gbpcad
nsxusd
ukxgbp
usddkk
usdsgd
xagusd
xaugbp
