Download Cryptocurrency Data with CCXT

CCXT is a cross-language library for interacting with Crypto exchange API’s. At the time of writing, the library provides a single interface to over 100 cryptocurrency exchanges. In this post, we are going to use the library to build a tool that will allow us to download cryptocurrency data for backtesting. For more information on the library, visit CCXT’s GitHub page (referenced below). For those of you who prefer a quick overview, here is a snippet straight from the horse’s mouth:

It is intended to be used by coders, developers, technically-skilled traders, data-scientists and financial analystsfor building trading algorithms on top of it. Current feature list:

• support for many exchange markets, even more upcoming soon
• fully implemented public and private APIs for all exchanges
• all currencies, altcoins and symbols, prices, order books, trades, tickers, etc…
• optional normalized data for cross-exchange or cross-currency analytics and arbitrage
• an out-of-the box unified all-in-one API extremely easy to integrate
• works in Node 7.6+, Python 2 and 3, PHP 5.3+, web browsers

You can find out more about the project here: https://github.com/ccxt/ccxt

Installing the Library

As with most modules, installing CCXT is made simple via pip. Just open up the command line and type: pip install ccxt If you happen to havepython2installed alongsidepython3and want to install it for python3, simply replace pipwithpip3.

Other Dependencies

If this is your first time visiting the site, it is assumed you know how to install Python 3 and run some basic scripts. If that is not you, try looking at the Getting Setup: Python and Backtrader post first. Lastly, the script below requires pandasto also be installed. However, the code could be updated to not require it. The download tool only uses pandas to massage and export the data received. To install it use the same pipcommand as above but replace ccxtwithpandas. Reference:  https://pandas.pydata.org/

Download Tool Overview

In the first iteration, the download tool shall be a simple command line tool. It shall allow simple arguments to be specified at runtime using the argeparsemodule for flexibility. This means we can avoid altering the code everytime we want to download different data. For a deeper look at argparseand how it can be used with Backtrader, see here: Using argparse to change strategy parameters The download tool shall allow users to specify the following options:

• Instrument (Currency Pair)
• Timeframe (1m, 5m, 1 hour etc)
• Exchange

Notice how there are no inputs for start and end dates? The tool shall download everything that is available using CCXT’s fetchOHLCVmethod. Once the data is downloaded, it shall be exported to a CSV file in the current folder. You may wish to tweak this and add anotherargparseargument to specify the output directory.

The Code

'''
Author: www.backtest-rookies.com

MIT License

Copyright (c) 2018 backtest-rookies.com

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
'''

import ccxt
from datetime import datetime, timedelta, timezone
import math
import argparse
import pandas as pd


def parse_args():
    parser = argparse.ArgumentParser(description='CCXT Market Data Downloader')


    parser.add_argument('-s','--symbol',
                        type=str,
                        required=True,
                        help='The Symbol of the Instrument/Currency Pair To Download')

    parser.add_argument('-e','--exchange',
                        type=str,
                        required=True,
                        help='The exchange to download from')

    parser.add_argument('-t','--timeframe',
                        type=str,
                        default='1d',
                        choices=['1m', '5m','15m', '30m','1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1y'],
                        help='The timeframe to download')


    parser.add_argument('--debug',
                            action ='store_true',
                            help=('Print Sizer Debugs'))

    return parser.parse_args()

# Get our arguments
args = parse_args()

# Get our Exchange
try:
    exchange = getattr (ccxt, args.exchange) ()
except AttributeError:
    print('-'*36,' ERROR ','-'*35)
    print('Exchange "{}" not found. Please check the exchange is supported.'.format(args.exchange))
    print('-'*80)
    quit()

# Check if fetching of OHLC Data is supported
if exchange.has["fetchOHLCV"] == False:
    print('-'*36,' ERROR ','-'*35)
    print('{} does not support fetching OHLC data. Please use another exchange'.format(args.exchange))
    print('-'*80)
    quit()

# Check requested timeframe is available. If not return a helpful error.
if args.timeframe not in exchange.timeframes:
    print('-'*36,' ERROR ','-'*35)
    print('The requested timeframe ({}) is not available from {}\n'.format(args.timeframe,args.exchange))
    print('Available timeframes are:')
    for key in exchange.timeframes.keys():
        print('  - ' + key)
    print('-'*80)
    quit()

# Check if the symbol is available on the Exchange
exchange.load_markets()
if args.symbol not in exchange.symbols:
    print('-'*36,' ERROR ','-'*35)
    print('The requested symbol ({}) is not available from {}\n'.format(args.symbol,args.exchange))
    print('Available symbols are:')
    for key in exchange.symbols:
        print('  - ' + key)
    print('-'*80)
    quit()


# Get data
data = exchange.fetch_ohlcv(args.symbol, args.timeframe)
header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume']
df = pd.DataFrame(data, columns=header).set_index('Timestamp')
# Save it
symbol_out = args.symbol.replace("/","")
filename = '{}-{}-{}.csv'.format(args.exchange, symbol_out,args.timeframe)
df.to_csv(filename)

The code is quite straightforward. After using argparseto capture our exchange, instrument and timezone requirements, the bulk of the work is spent error checking before and calling CCXT’s fetch_ohlcvmethod. Speaking of error checking, not all exchanges support fetching of OHLCV data. Furthermore, not all exchanges support the same timeframes and currency pairs. Since there are over 100 exchanges supported at the time of writing, it is easy to imagine that there will be a lot of variances between exchanges. To avoid frustration with bumping into error after error trying to guess which timeframes and/or currency pairs are supported, the code performs a series of checks before attempting to download data. It detected something is not supported, it will notify the user and provide a list of alternative parameters where applicable. Other than that, it is worth paying attention to the exchange.load_markets()call if you plan to expand upon the code or work with the library yourself. Admittedly, I spent a little too much some time scratching my head, trying to figure out why every exchange I tested seemed to have empty markets, symbolsand markets_by_iddictionaries. It turns out that snuggled in the documentation is the answer, you need to call load_markets()before any data is available. This was a good reminder to read the documentation carefully! A little time spent reading can save a lot of time. Finally, if you are new to programming, the getattr(ccxt, args.exchange)might be new to you. getattr()is a python built-in function which allows you to “get an attribute” from an object using a string. In this case, we are getting the correct exchange object from the given argument string. Reference: https://docs.python.org/3/library/functions.html#getattr

Usage Examples

Please note that the following usage examples assume the script is saved as ccxt_market_data.py. ETH/NEO Binance Daily python .\ccxt_market_data.py -s NEO/ETH -e binance -t 1d BTC/USD

Unsupported Timeframe

The example demonstrates the expected result when an unsupported timeframe is given as an argument: python .\ccxt_market_data.py -s NEO/ETH -e bittrex -t 12h

------------------------------------  ERROR  -----------------------------------
The requested timeframe (12h) is not available from bittrex

Available timeframes are:
  - 1m
  - 5m
  - 30m
  - 1h
  - 1d
--------------------------------------------------------------------------------

Unsupported Symbol

The second example has a fair bit of output. It demonstrates the expected result when an unsupported symbol is given as an argument: python .\ccxt_market_data.py -s ATP/USD -e bittrex -t 5m

------------------------------------  ERROR  -----------------------------------
The requested symbol (ATP/USD) is not available from bittrex

Available symbols are:
  - 1ST/BTC
  - 1ST/ETH
  - 2GIVE/BTC
  - ABY/BTC
  - ADA/BTC
  - ADA/ETH
  - ADA/USDT
  - ADT/BTC
  - ADT/ETH
  - ADX/BTC
  - ADX/ETH
  - AEON/BTC
  - AGRS/BTC
  - AMP/BTC
  - ANT/BTC
  - ANT/ETH
  - ARDR/BTC
  - ARK/BTC
  - AUR/BTC
  - BAT/BTC
  - BAT/ETH
  - BAY/BTC
  - BCH/BTC
  - BCH/ETH
  - BCH/USDT
  - BCPT/BTC
  - BCPT/ETH
  - BCY/BTC
  - BITB/BTC
  - BLITZ/BTC
  - BLK/BTC
  - BLOCK/BTC
  - BNT/BTC
  - BNT/ETH
  - BRK/BTC
  - BRX/BTC
  - BSD/BTC
  - BTC/USDT
  - BTG/BTC
  - BTG/ETH
  - BTG/USDT
  - BURST/BTC
  - BYC/BTC
  - CANN/BTC
  - CFI/BTC
  - CFI/ETH
  - CLAM/BTC
  - CLOAK/BTC
  - CLUB/BTC
  - COVAL/BTC
  - CPC/BTC
  - CRB/BTC
  - CRB/ETH
  - CRW/BTC
  - CURE/BTC
  - CVC/BTC
  - CVC/ETH
  - DASH/BTC
  - DASH/ETH
  - DASH/USDT
  - DCR/BTC
  - DCT/BTC
  - DGB/BTC
  - DGB/ETH
  - DMD/BTC
  - DNT/BTC
  - DNT/ETH
  - DOGE/BTC
  - DOPE/BTC
  - DTB/BTC
  - DYN/BTC
  - EBST/BTC
  - EDG/BTC
  - EFL/BTC
  - EGC/BTC
  - EMC/BTC
  - EMC2/BTC
  - ENG/BTC
  - ENG/ETH
  - ENRG/BTC
  - ERC/BTC
  - ETC/BTC
  - ETC/ETH
  - ETC/USDT
  - ETH/BTC
  - ETH/USDT
  - EXCL/BTC
  - EXP/BTC
  - FAIR/BTC
  - FCT/BTC
  - FCT/ETH
  - FLDC/BTC
  - FLO/BTC
  - FTC/BTC
  - GAM/BTC
  - GAME/BTC
  - GBG/BTC
  - GBYTE/BTC
  - GCR/BTC
  - GEO/BTC
  - GLD/BTC
  - GNO/BTC
  - GNO/ETH
  - GNT/BTC
  - GNT/ETH
  - GOLOS/BTC
  - GRC/BTC
  - GRS/BTC
  - GUP/BTC
  - GUP/ETH
  - HMQ/BTC
  - HMQ/ETH
  - IGNIS/BTC
  - INCNT/BTC
  - INFX/BTC
  - IOC/BTC
  - ION/BTC
  - IOP/BTC
  - KMD/BTC
  - KORE/BTC
  - LBC/BTC
  - LGD/BTC
  - LGD/ETH
  - LMC/BTC
  - LSK/BTC
  - LTC/BTC
  - LTC/ETH
  - LTC/USDT
  - LUN/BTC
  - LUN/ETH
  - MAID/BTC
  - MANA/BTC
  - MANA/ETH
  - MCO/BTC
  - MCO/ETH
  - MEME/BTC
  - MER/BTC
  - MLN/BTC
  - MONA/BTC
  - MUE/BTC
  - MUSIC/BTC
  - NAV/BTC
  - NBT/BTC
  - NEO/BTC
  - NEO/ETH
  - NEO/USDT
  - NEOS/BTC
  - NLG/BTC
  - NMR/BTC
  - NMR/ETH
  - NXC/BTC
  - NXS/BTC
  - NXT/BTC
  - NXT/USDT
  - OK/BTC
  - OMG/BTC
  - OMG/ETH
  - OMG/USDT
  - OMNI/BTC
  - PART/BTC
  - PAY/BTC
  - PAY/ETH
  - PDC/BTC
  - PINK/BTC
  - PIVX/BTC
  - PKB/BTC
  - POT/BTC
  - POWR/BTC
  - POWR/ETH
  - PPC/BTC
  - PTC/BTC
  - PTOY/BTC
  - PTOY/ETH
  - QRL/BTC
  - QRL/ETH
  - QTUM/BTC
  - QTUM/ETH
  - QWARK/BTC
  - RADS/BTC
  - RBY/BTC
  - RCN/BTC
  - RCN/ETH
  - RDD/BTC
  - REP/BTC
  - REP/ETH
  - RLC/BTC
  - RLC/ETH
  - SALT/BTC
  - SALT/ETH
  - SBD/BTC
  - SC/BTC
  - SC/ETH
  - SEQ/BTC
  - SHIFT/BTC
  - SIB/BTC
  - SLR/BTC
  - SLS/BTC
  - SNRG/BTC
  - SNT/BTC
  - SNT/ETH
  - SPHR/BTC
  - SPR/BTC
  - SRN/BTC
  - SRN/ETH
  - START/BTC
  - STEEM/BTC
  - STORJ/BTC
  - STORJ/ETH
  - STRAT/BTC
  - STRAT/ETH
  - SWIFT/BTC
  - SWT/BTC
  - SYNX/BTC
  - SYS/BTC
  - THC/BTC
  - TIX/BTC
  - TIX/ETH
  - TKS/BTC
  - TRST/BTC
  - TRST/ETH
  - TRUST/BTC
  - TRX/BTC
  - TRX/ETH
  - TUSD/BTC
  - TX/BTC
  - UBQ/BTC
  - UKG/BTC
  - UKG/ETH
  - UNB/BTC
  - VEE/BTC
  - VEE/ETH
  - VIA/BTC
  - VIB/BTC
  - VIB/ETH
  - VOX/BTC
  - VRC/BTC
  - VRM/BTC
  - VTC/BTC
  - VTR/BTC
  - WAVES/BTC
  - WAVES/ETH
  - WAX/BTC
  - WAX/ETH
  - WINGS/BTC
  - WINGS/ETH
  - XCP/BTC
  - XDN/BTC
  - XEL/BTC
  - XEM/BTC
  - XEM/ETH
  - XLM/BTC
  - XLM/ETH
  - XMG/BTC
  - XMR/BTC
  - XMR/ETH
  - XMR/USDT
  - XMY/BTC
  - XRP/BTC
  - XRP/ETH
  - XRP/USDT
  - XST/BTC
  - XVC/BTC
  - XVG/BTC
  - XVG/USDT
  - XWC/BTC
  - XZC/BTC
  - ZCL/BTC
  - ZEC/BTC
  - ZEC/ETH
  - ZEC/USDT
  - ZEN/BTC
  - ZRX/BTC
  - ZRX/ETH
--------------------------------------------------------------------------------

Unsupported Exachange

The final example demostrates the expected result when an exchange does not support fethcing of OHLCV data. python .\ccxt_market_data.py -s BTC/USD -e gdax -t 15m

------------------------------------  ERROR  -----------------------------------
gdax does not support fetching OHLC data. Please use another exchange
--------------------------------------------------------------------------------

Issues and Improvements

The code tries to be user-friendly in that it will tell the user when incorrect data is provided. Having said that, not all edge cases have been considered and there is definitely room for improvement in that regard. In addition to this, the amount of data available on most exchanges for download is limited. CCXT does provide a sinceparameter for the fetchOHLCV()method but I did not see any detailed description for this in the fetchOHLCV()documentation. Having said that, an explanation of a parameter with the same name does appear the “Trades, Executions and Transactions” section as follows (Note: That the parameter appears frequently, so I assume it will have consistent functionality but that is no guarantee!):

The second optional argument since reduces the array by timestamp, the third limit argument reduces by number (count) of returned items.

Reference: https://github.com/ccxt/ccxt/wiki/Manual#trades-executions-transactions So taking that into account, it appears sinceshould reduce the amount of data provided. However, it would be good to know for sure. If anyone pokes through the codebase and has an answer, I would be happy to hear it. Leave a comment below!