Replacing Quandl Wiki Data with Alpha Vantage

Free and well-maintained end of day (EOD) stock data seems to be getting harder and harder to come by these days. Quandl was a great resource for free EOD data. However, that seems to be slowly changing. First, we lost the London Stock Exchange with only paid subscription as an alternative. Now the US-focused EOD Wiki is no longer actively supported. If you head over to the Quandl EOD Wiki page, you will see a banner which reads:

As of April 11, 2018 this data feed is no longer actively supported by the Quandl community. We will continue to host this data feed on Quandl, but we do not recommend using it for investment or analysis.

Now I understand we cannot expect everything to be free and handed to us on a plate but in recent years EOD data was not so hard to come by. Yahoo, Google, Quandl all provided quality free data at one point in time. The landscape seems to be changing and in Quandl’s case, even if we are willing to pay, most of the premium data is actually quite expensive for the average Joe to purchase. So where can we go now?

Alpha Vantage

Luckily all is not lost. We still have some other options available to us if we are willing to dig around and work with other API’s. For example, many brokers and/or spreadbet providers have API’s that can be used by their customers. There are also some independent data providers such as Alpha Vantage who provide excellent free services and it is Alpha Vantage that we will focus on for this post. Alpha Vantange claims to be:

Composed of a tight-knit community of researchers, engineers, and business professionals, Alpha Vantage Inc. is a leading provider of free APIs for realtime and historical data on stocks, physical currencies, and digital/crypto currencies. Our success is driven by rigorous research, cutting edge technology, and a disciplined focus on democratizing access to data.

Source: https://www.alphavantage.co/

Great stuff! Not only does it have the EOD data we are looking to replace, they also provide access to real-time data along with a bunch of other assets and technical indicators! So let’s take a look at how to use Alpha Vantage’s API to download and save EOD data.

Getting an API Key

First things first. In order to use the API, you must register for an API key. This API key will grant you lifetime access to Alpha Vantage data. Head over to the following link and sign up for one before continuing.

Source: https://www.alphavantage.co/support/#api-key

The application form is simple, quick and Alpha Vantage appears to be true to their word. Not a single piece of spam email has made it’s way to my inbox since signing up. Thumbs up!

Accessing Data

As always, the official docs are a good place to start. Alpha Vantage is no exception and their API documentation is available here:

Documentation: https://www.alphavantage.co/documentation/

For new programmers and beginners, the API documentation might be a little daunting at first. It describes the API well but there is no hand holding or step by step instructions on how to use it.  There are some examples but you are expected to know how to interpret them in your language of choice. Perhaps this is where we hope we can help…

So in order to use the API and access data, you have two options available. Actually, you have many options available but we will focus from a Python perspective:

• The first option is to follow the API documentation and use a python package such as urlliborrequeststo make URL requests to the Alpha Vantage servers. For a beginner, this is the harder way.
• The second option is easier. All we need to do is install Romel Torres’s python wrapper directly from pip to use convenient Python functions that handle the HTTP request for you.

Take a look at the project’s github page and if you have an account, give Romel a star.

Git Hub Page: https://github.com/RomelTorres/alpha_vantage

To install the package simply type:

pip install alpha_vantage

Remember to replace pipwithpip3if you already have python2 installed.

alpha_vantagerequires a number of dependencies such as pandas. However, Pip should handle it, download and install all required packages for you.

Time to download

Now we should have everything we need to start downloading data. So let’s move onto the code.

In the spirit of keeping things simple and not re-inventing the wheel, this post is going to focus on using the python wrapper rather than making HTTP requests ourselves. We shall use the alpha_vantagepackage to download and eventually save the data as a CSV file that we can later ingest into Backtrader using the GenericCSVDatafeed.

'''
Author: www.backtest-rookies.com

MIT License

Copyright (c) 2018 backtest-rookies.com

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
'''

from alpha_vantage.timeseries import TimeSeries
import argparse
import pandas as pd


def parse_args():
    parser = argparse.ArgumentParser(description='CCXT Market Data Downloader')


    parser.add_argument('-s','--symbol',
                        type=str,
                        required=True,
                        help='The Symbol of the Instrument/Currency Pair To Download')

    parser.add_argument('-o', '--outfile',
                        type=str,
                        required=True,
                        help='The output directory and file name to save the data')

    return parser.parse_args()

# Get our arguments
args = parse_args()

print(args)

# Submit our API and create a session
alpha_ts = TimeSeries(key='INSERT YOUR API', output_format='pandas')

# Get the data
data, meta_data = alpha_ts.get_daily(symbol=args.symbol, outputsize='compact')

# Save the data
data.to_csv(args.outfile)

Commentary

As you will see the script is quite simple. In fact, the MIT license is almost longer than the code itself! However, we should be careful calling it simple. Simple to one person may not be simple to another. This is especially true if you are just starting out. As such, let’s take a look at a few parts.

First, don’t forget to insert your API key before you try to run the code. Look for this line and update it accordingly:

alpha_ts = TimeSeries(key='INSERT YOUR API', output_format='pandas)'

Secondly, it is worth noting why we choose to download the data as a Panda’s dataframe first rather than directly as CSV. The reason is due to some API calls not supporting the CSV format. Therefore, if you want to use this code as a base and expand it to work with other api calls, it is better to use pandas so you can access everything. Take a look at this comment from Romel’s GitHub page:

Please note that some API calls do not support the csv format (namely ForeignExchange, SectorPerformances and TechIndicators) because the API endpoint does not support the format on their calls either.

Third, for speed and the purposes of this tutorial, we are downloading compact data rather than full data. However, in practice, you might want to download as much data as they have available. The documentation states that this will result in files larger than 1MB and thus may take some time to run the script. So be patient if you want to download a lot of data. 

To switch to full data change the line:

data, meta_data = alpha_ts.get_daily(symbol=args.symbol, outputsize='compact')

To:

data, meta_data = alpha_ts.get_daily(symbol=args.symbol, outputsize='full')

Note: The compact output size is actually the default. Therefore you do not actually need to specify it. I have included it in the code to be explicit and bring it to your attention.

Finally, you will notice that the get_daily()function call returns both data and meta_data. In our script the meta_datais not used. However, it is worth noting that it exists as it could come in useful should you choose to further explore the API. The meta_datavariable contains a dictionary containing, you guessed it, metadata. Items such as the symbol, when it was last updated, the output size and timezone. You can take a look at its contents by adding print(meta_data)to the end of the script.