Backtrader: Alpha Vantage Data - Direct Ingest

As noted back in April last year, many of the good quality, free equity data sources from the likes of Yahoo and Quandl are disappearing. This is why companies such as Alpha Vantage are an absolute godsend for the tinkerer and cash-challenged retail trader!

In that first post last year, we created a simple script to download and save EOD (End Of Day) data to a CSV File. Now, as a follow up to that script, we will take things a step further. Rather than saving the data to a CSV file, the example code in this post will download the data and directly ingest it into backtrader as a Pandas data feed. This will be useful for those who do not wish to store and manage a large library of CSV files or just want to test some random tickers from time to time.

Getting an API Key

Before we dive into the code. In order to download the data from Alpha Vantage, you must register for an API key. It is free and will grant you lifetime access. If you don’t have one already, head over to the following link and sign up before continuing.

Source: https://www.alphavantage.co/support/#api-key

Installation

Once we have an API key, we also need to install a python package to interact with the API. Romel Torres has created an excellent python wrapper for the API. To install it simply type:

pip install alpha_vantage

Remember to replace pipwithpip3if you already have python2 installed.

alpha_vantagerequires a number of dependencies such as pandas. However, Pip should handle it, download and install all required packages for you.

The Code

'''
Author: www.backtest-rookies.com

MIT License

Copyright (c) 2019 backtest-rookies.com

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
'''

from alpha_vantage.timeseries import TimeSeries
import pandas as pd
import numpy as np
import backtrader as bt
from datetime import datetime

# IMPORTANT!
# ----------
# Register for an API at:
# https://www.alphavantage.co/support/#api-key
# Then insert it here.
Apikey = 'INSERT YOUR API KEY'


def alpha_vantage_eod(symbol_list, compact=False, debug=False, *args, **kwargs):
    '''
    Helper function to download Alpha Vantage Data.

    This will return a nested list with each entry containing:
        [0] pandas dataframe
        [1] the name of the feed.
    '''
    data_list = list()

    size = 'compact' if compact else 'full'

    for symbol in symbol_list:

        if debug:
            print('Downloading: {}, Size: {}'.format(symbol, size))

        # Submit our API and create a session
        alpha_ts = TimeSeries(key=Apikey, output_format='pandas')

        data, meta_data = alpha_ts.get_daily(symbol=symbol, outputsize=size)

        #Convert the index to datetime.
        data.index = pd.to_datetime(data.index)
        data.columns = ['Open', 'High', 'Low', 'Close','Volume']

        if debug:
            print(data)

        data_list.append((data, symbol))

    return data_list

class TestStrategy(bt.Strategy):

    def __init__(self):

        pass

    def next(self):

        for i, d in enumerate(self.datas):

            bar = len(d)
            dt = d.datetime.datetime()
            dn = d._name
            o = d.open[0]
            h = d.high[0]
            l = d.low[0]
            c = d.close[0]
            v = d.volume[0]


            print('{} Bar: {} | {} | O: {} H: {} L: {} C: {} V:{}'.format(dt, bar,dn,o,h,l,c,v))


# Create an instance of cerebro
cerebro = bt.Cerebro()

# Add our strategy
cerebro.addstrategy(TestStrategy)

# Download our data from Alpha Vantage.
symbol_list = ['LGEN.L','LLOY.L']
data_list = alpha_vantage_eod(
                symbol_list,
                compact=False,
                debug=False)

for i in range(len(data_list)):

    data = bt.feeds.PandasData(
                dataname=data_list[i][0], # This is the Pandas DataFrame
                name=data_list[i][1], # This is the symbol
                timeframe=bt.TimeFrame.Days,
                compression=1,
                fromdate=datetime(2018,1,1),
                todate=datetime(2019,1,1)
                )

    #Add the data to Cerebro
    cerebro.adddata(data)

print('Starting to run')
# Run the strategy
cerebro.run()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

'''

Author: www.backtest-rookies.com

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy

of this software and associated documentation files (the "Software"), to deal

in the Software without restriction, including without limitation the rights

to use, copy, modify, merge, publish, distribute, sublicense, and/or sell

copies of the Software, and to permit persons to whom the Software is

furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

SOFTWARE.

'''

from alpha_vantage.timeseries import TimeSeries

import pandas as pd

import numpy as np

import backtrader as bt

from datetime import datetime

# IMPORTANT!

# ----------

# Register for an API at:

# https://www.alphavantage.co/support/#api-key

# Then insert it here.

Apikey = 'INSERT YOUR API KEY'

def alpha_vantage_eod(symbol_list, compact=False, debug=False, *args, **kwargs):

'''

Helper function to download Alpha Vantage Data.

This will return a nested list with each entry containing:

[0] pandas dataframe

[1] the name of the feed.

'''

data_list = list()

size = 'compact' if compact else 'full'

for symbol in symbol_list:

if debug:

print('Downloading: {}, Size: {}'.format(symbol, size))

# Submit our API and create a session

alpha_ts = TimeSeries(key=Apikey, output_format='pandas')

data, meta_data = alpha_ts.get_daily(symbol=symbol, outputsize=size)

#Convert the index to datetime.

data.index = pd.to_datetime(data.index)

data.columns = ['Open', 'High', 'Low', 'Close','Volume']

if debug:

print(data)

data_list.append((data, symbol))

return data_list

class TestStrategy(bt.Strategy):

def __init__(self):

pass

def next(self):

for i, d in enumerate(self.datas):

bar = len(d)

dt = d.datetime.datetime()

dn = d._name

o = d.open[0]

h = d.high[0]

l = d.low[0]

c = d.close[0]

v = d.volume[0]

print('{} Bar: {} | {} | O: {} H: {} L: {} C: {} V:{}'.format(dt, bar,dn,o,h,l,c,v))

# Create an instance of cerebro

cerebro = bt.Cerebro()

# Add our strategy

cerebro.addstrategy(TestStrategy)

# Download our data from Alpha Vantage.

symbol_list = ['LGEN.L','LLOY.L']

data_list = alpha_vantage_eod(

symbol_list,

compact=False,

debug=False)

for i in range(len(data_list)):

data = bt.feeds.PandasData(

dataname=data_list[i][0], # This is the Pandas DataFrame

name=data_list[i][1], # This is the symbol

timeframe=bt.TimeFrame.Days,

compression=1,

fromdate=datetime(2018,1,1),

todate=datetime(2019,1,1)

)

#Add the data to Cerebro

cerebro.adddata(data)

print('Starting to run')

# Run the strategy

cerebro.run()

Code Commentary

Before trying to run the code, be sure to update the API key:

# IMPORTANT!
# ----------
# Register for an API at:
# https://www.alphavantage.co/support/#api-key
# Then insert it here.
Apikey = 'INSERT YOUR API KEY'

# IMPORTANT!

# ----------

# Register for an API at:

# https://www.alphavantage.co/support/#api-key

# Then insert it here.

Apikey = 'INSERT YOUR API KEY'

Most of the magic in this script happens within the alpha_vantage_eod()function. It is here that we download the data and massage it into a format that we can easily pass to Backtrader. As such, it makes sense to go over the function parameters:

symbol_list: Is a simple list of symbols/tickers that we will download. If you just want to download a single instrument then simply place a single ticker inside a list like so: symbol_list = ['AAPL']
compact: This is useful for testing. It will download only the last 100 data points and as such data download speeds are much quicker. Set this parameter to Trueif you wish to speed things up.
debug: Will just provide some extra prints to the terminal in case you are getting some weird results. Using this option may allow you to see which symbol/ticker is giving you pain.

The alpha_vantage_eod()function will return a nested list of data feeds. In other words, each entry in the returned list is another list. The nested (second) lists contain two items. The first item (at index position [0]) is the pandas dataframe and the second item (at index position [1]) is the symbol string for that dataframe. Having easy access to the symbol allows us to add it as the name of the datafeed in backtrader.

Regarding the strategy, it was only designed for testing / checking the feeds are added correctly. However, one thing that can be noted is that the next()method (function) has been coded in a way to support multiple data feeds. As such, the strategy can support one or more data feeds and handle them appropriately to print out their OHLCVvalues. For more information on how to create a more complete strategy that supports multiple data feeds, see this tutorial.

For those interested, further API information can be found here:

Reference: https://www.alphavantage.co/documentation/

Output

Running the script you should see some output that looks like this:

Notes and Further Improvements:

Alpha Vantage has API call limits. When using their free API, you are limited to 5 API calls per minute and a total of 500 a day. Therefore, if you have a large symbol list (over 5) then you will need to add a sleep function to wait 12 seconds between downloads. This is obviously slow down the initialization of the backtest too. So if you plan on making a large number of API calls, you may want to consider saving the output to CSV as shown in the previous post.
The example code above downloads unadjusted data. If you require adjusted data, Alpha Vantage does provide this. However, they only provide an adjusted close value. Therefore, some extra work is required to massage the open, high and low values if you still want to work with OHLCdata. A future post may cover this!
Alpha Vantage also provide Intra-day Equity, FX and Crypto data! Readers may want to use this code example as a base to access these other feeds.

2 Responses

michael March 4, 2019 at 4:51 am Reply

For those of you that have a LINUX platform I created a stock-list-end-of-day-updater in GitHub

https://github.com/michael-b4/stock-list-end-of-day-updater/blob/master/README
Patrick Collins January 8, 2020 at 3:02 am Reply

Amazing!! Glad I found this!

Backtrader: Alpha Vantage Data – Direct Ingest

Getting an API Key

Installation

The Code

Code Commentary

Output

Notes and Further Improvements:

Related

2 Responses

Leave a Reply

Getting an API Key

Installation

The Code

Code Commentary

Output

Notes and Further Improvements:

Share this:

Related

2 Responses

Leave a Reply