Free and well-maintained end of day (EOD) stock data seems to be getting harder and harder to come by these days. Quandl was a great resource for free EOD data. However, that seems to be slowly changing. First, we lost the London Stock Exchange with only paid subscription as an alternative. Now the US-focused EOD Wiki is no longer actively supported. If you head over to the Quandl EOD Wiki page, you will see a banner which reads:
As of April 11, 2018 this data feed is no longer actively supported by the Quandl community. We will continue to host this data feed on Quandl, but we do not recommend using it for investment or analysis.
Now I understand we cannot expect everything to be free and handed to us on a plate but in recent years EOD data was not so hard to come by. Yahoo, Google, Quandl all provided quality free data at one point in time. The landscape seems to be changing and in Quandl’s case, even if we are willing to pay, most of the premium data is actually quite expensive for the average Joe to purchase. So where can we go now?
Alpha Vantage
Luckily all is not lost. We still have some other options available to us if we are willing to dig around and work with other API’s. For example, many brokers and/or spreadbet providers have API’s that can be used by their customers. There are also some independent data providers such as Alpha Vantage who provide excellent free services and it is Alpha Vantage that we will focus on for this post. Alpha Vantange claims to be:
Composed of a tight-knit community of researchers, engineers, and business professionals, Alpha Vantage Inc. is a leading provider of free APIs for realtime and historical data on stocks, physical currencies, and digital/crypto currencies. Our success is driven by rigorous research, cutting edge technology, and a disciplined focus on democratizing access to data.
Source: https://www.alphavantage.co/
Great stuff! Not only does it have the EOD data we are looking to replace, they also provide access to real-time data along with a bunch of other assets and technical indicators! So let’s take a look at how to use Alpha Vantage’s API to download and save EOD data.
Getting an API Key
First things first. In order to use the API, you must register for an API key. This API key will grant you lifetime access to Alpha Vantage data. Head over to the following link and sign up for one before continuing.
Source: https://www.alphavantage.co/support/#api-key
The application form is simple, quick and Alpha Vantage appears to be true to their word. Not a single piece of spam email has made it’s way to my inbox since signing up. Thumbs up!
Accessing Data
As always, the official docs are a good place to start. Alpha Vantage is no exception and their API documentation is available here:
Documentation: https://www.alphavantage.co/documentation/
For new programmers and beginners, the API documentation might be a little daunting at first. It describes the API well but there is no hand holding or step by step instructions on how to use it. There are some examples but you are expected to know how to interpret them in your language of choice. Perhaps this is where we hope we can help…
So in order to use the API and access data, you have two options available. Actually, you have many options available but we will focus from a Python perspective:
- The first option is to follow the API documentation and use a python package such as
urllib
orrequests
to make URL requests to the Alpha Vantage servers. For a beginner, this is the harder way. - The second option is easier. All we need to do is install Romel Torres’s python wrapper directly from pip to use convenient Python functions that handle the HTTP request for you.
Take a look at the project’s github page and if you have an account, give Romel a star.
Git Hub Page: https://github.com/RomelTorres/alpha_vantage
To install the package simply type:
pip install alpha_vantage
Remember to replace pip
withpip3
if you already have python2 installed.
alpha_vantage
requires a number of dependencies such as pandas
. However, Pip should handle it, download and install all required packages for you.
Time to download
Now we should have everything we need to start downloading data. So let’s move onto the code.
In the spirit of keeping things simple and not re-inventing the wheel, this post is going to focus on using the python wrapper rather than making HTTP requests ourselves. We shall use the alpha_vantage
package to download and eventually save the data as a CSV file that we can later ingest into Backtrader using the GenericCSVData
feed.
''' Author: www.backtest-rookies.com MIT License Copyright (c) 2018 backtest-rookies.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ''' from alpha_vantage.timeseries import TimeSeries import argparse import pandas as pd def parse_args(): parser = argparse.ArgumentParser(description='CCXT Market Data Downloader') parser.add_argument('-s','--symbol', type=str, required=True, help='The Symbol of the Instrument/Currency Pair To Download') parser.add_argument('-o', '--outfile', type=str, required=True, help='The output directory and file name to save the data') return parser.parse_args() # Get our arguments args = parse_args() print(args) # Submit our API and create a session alpha_ts = TimeSeries(key='INSERT YOUR API', output_format='pandas') # Get the data data, meta_data = alpha_ts.get_daily(symbol=args.symbol, outputsize='compact') # Save the data data.to_csv(args.outfile)
Commentary
As you will see the script is quite simple. In fact, the MIT license is almost longer than the code itself! However, we should be careful calling it simple. Simple to one person may not be simple to another. This is especially true if you are just starting out. As such, let’s take a look at a few parts.
First, don’t forget to insert your API key before you try to run the code. Look for this line and update it accordingly:
alpha_ts = TimeSeries(key='INSERT YOUR API', output_format='pandas)'
Secondly, it is worth noting why we choose to download the data as a Panda’s dataframe first rather than directly as CSV. The reason is due to some API calls not supporting the CSV format. Therefore, if you want to use this code as a base and expand it to work with other api calls, it is better to use pandas so you can access everything. Take a look at this comment from Romel’s GitHub page:
Please note that some API calls do not support the csv format (namely
ForeignExchange, SectorPerformances and TechIndicators
) because the API endpoint does not support the format on their calls either.
Third, for speed and the purposes of this tutorial, we are downloading compact data rather than full data. However, in practice, you might want to download as much data as they have available. The documentation states that this will result in files larger than 1MB and thus may take some time to run the script. So be patient if you want to download a lot of data.
To switch to full data change the line:
data, meta_data = alpha_ts.get_daily(symbol=args.symbol, outputsize='compact')
To:
data, meta_data = alpha_ts.get_daily(symbol=args.symbol, outputsize='full')
Note: The compact output size is actually the default. Therefore you do not actually need to specify it. I have included it in the code to be explicit and bring it to your attention.
Finally, you will notice that the get_daily()
function call returns both data
and meta_data
. In our script the meta_data
is not used. However, it is worth noting that it exists as it could come in useful should you choose to further explore the API. The meta_data
variable contains a dictionary containing, you guessed it, metadata. Items such as the symbol, when it was last updated, the output size and timezone. You can take a look at its contents by adding print(meta_data)
to the end of the script.
Find This Post Useful?
If this post saved you time and effort, please consider support the site! There are many ways to support us and some won’t even cost you a penny.
Backtest Rookies is a registered with Brave publisher!
Brave users can drop us a tip.
Alternatively, support us by switching to Brave using this referral link and we will receive some BAT!
Referral Link
Enjoying the content and thinking of subscribing to Tradingview? Support this site by clicking the referral link before you sign up!
3HxNVyh5729ieTfPnTybrtK7J7QxJD9gjD
0x9a2f88198224d59e5749bacfc23d79507da3d431
M8iUBnb8KamHR18gpgi2sSG7jopddFJi8S
Hello!
I’ve been studying your articles fow around two week now and they were of very much use. Thank you for the wonderful content and keep up the great work!
However I have encountered a problem, that I have no clue how to solve. Therefore I ask for your help and look forward to getting a response from you)
The problem goes as following:
I use .csv Files as the main source of Data for backtesting and the code from your wevsite (modyified very little) just doesn’t seem to work. I man no positions are entered whatsoever even though there are certainly conditions that fit the necessary criteria. What is even more puzzling, is the fact that the output of the code should be both a plotted chart and a printed net profit. And I get the chart. So the Data itself is perfectly viable and backtrader sees it as a bunch of candles but it just doesn’t run the strategy.
P.S. I have checked the code line by line and compared it with the original (FirstScript snippet) and I have all the “addstrategy”, “cerebro.run” parts.
the code itself:
import backtrader as bt
from datetime import datetime
import ccxt
import sys
import datetime
import backtrader.feeds as btfeed
class firstStrategy(bt.Strategy):
def __init__(self):
self.rsi = bt.indicators.RSI_SMA(self.data.close, period=21)
def next(self):
if not self.position:
if self.rsi 70:
self.sell(size=100)
startcash = 10000
cerebro = bt.Cerebro()
cerebro.addstrategy(firstStrategy)
data = btfeed.GenericCSVData(
dataname=’BTCL.csv’,
dtformat = “%Y-%m-%d”,
# %H:%M:%S”,
datetime=0,
high=2,
low=3,
open=1,
close=4,
volume=5,
openinterest=-1
)
cerebro.adddata(data)
cerebro.broker.setcash(startcash)
cerebro.run()
portvalue = cerebro.broker.getvalue()
pnl = portvalue – startcash
print(‘Final Portfolio Value: ${}’.format(portvalue))
print(‘P/L: ${}’.format(pnl))
cerebro.plot(style=’candlestick’)
Hi Anton,
Thanks for the kind words!
What data are you using? It could be that your size is too large and therefore you do not have enough cash to buy and sell.
The size in the example is 100 and the starting cash is only 10,000 so if your data price is over $100, you might have some issues.
Hello,
first I want to thank you for the blog, it is very useful.
But I am not able to incorporate Alpha Vantage as my data feed source.
If you can help me, thanks.
Hi there!
Have you tried to load the CSV file into a
bt.feeds.GenericCSVData
object?https://www.backtrader.com/docu/datafeed-develop-csv.html?highlight=csv
That would be the first thing to do…
In future, I may do a post that will directly download and load the Panda’s dataframe into a backtest.
Hello,
I tried to implement the basic instruction you provided.
I went to my Anaconda prompt and performed the pip install command and the following was the result:
Successfully built alpha-vantage
Installing collected packages: alpha-vantage
Successfully installed alpha-vantage-2.1.0
Then I copied the code into Spyder and ran the script. The following was the result:
Traceback (most recent call last):
ModuleNotFoundError: No module named ‘alpha_vantage’
———————-
Any suggestions about what I need to do to get the script to work?
I’m mostly still a beginner programmer.
My goal is simply to find an alternative to quandl to “get” Stock Price DATA to work with
Thank You.
Respectfully,
Michael
Hi Michael
I am not too familiar with Anaconda – during installation did you see which directory pip was installing it to?
Did you double check the python version you installed it to? If you installed it with python2, then run the script with python3 (or the other way around, you may encounter the error.
Cheers
So I’m about a year late, but I’ve run into this before too.
Basically, your anaconda environment isn’t always the same as your python environment! Crazy right…
There are a few different ways to address this issue.
The simplest way is to find the location of your anaconda python environment, mine was located here:
/Users/patrick/anaconda3/bin/python
And then just run:
/Users/patrick/anaconda3/bin/python -m pip install alpha_vantage
This will install the package to that python directory
For more information, on how to do it a little more, check out this stack overflow.
https://stackoverflow.com/questions/41060382/using-pip-to-install-packages-to-anaconda-environment
Just found this blog post, love it
😀
– PatrickAlphaVantage