Download Cryptocurrency Data with CCXT

CCXT is a cross-language library for interacting with Crypto exchange API’s. At the time of writing, the library provides a single interface to over 100 cryptocurrency exchanges. In this post, we are going to use the library to build a tool that will allow us to download cryptocurrency data for backtesting. For more information on the library, visit CCXT’s GitHub page (referenced below). For those of you who prefer a quick overview, here is a snippet straight from the horse’s mouth:

It is intended to be used by coders, developers, technically-skilled traders, data-scientists and financial analystsfor building trading algorithms on top of it.
Current feature list:

support for many exchange markets, even more upcoming soon

fully implemented public and private APIs for all exchanges

all currencies, altcoins and symbols, prices, order books, trades, tickers, etc…

optional normalized data for cross-exchange or cross-currency analytics and arbitrage

an out-of-the box unified all-in-one API extremely easy to integrate

works in Node 7.6+, Python 2 and 3, PHP 5.3+, web browsers

You can find out more about the project here: https://github.com/ccxt/ccxt

Installing the Library

As with most modules, installing CCXT is made simple via pip. Just open up the command line and type:
pip install ccxt
If you happen to havepython2installed alongsidepython3and want to install it for python3, simply replace pipwithpip3.

Other Dependencies

If this is your first time visiting the site, it is assumed you know how to install Python 3 and run some basic scripts. If that is not you, try looking at the Getting Setup: Python and Backtrader post first.
Lastly, the script below requires pandasto also be installed. However, the code could be updated to not require it. The download tool only uses pandas to massage and export the data received. To install it use the same pipcommand as above but replace ccxtwithpandas.
Reference: https://pandas.pydata.org/

Download Tool Overview

In the first iteration, the download tool shall be a simple command line tool. It shall allow simple arguments to be specified at runtime using the argeparsemodule for flexibility. This means we can avoid altering the code everytime we want to download different data. For a deeper look at argparseand how it can be used with Backtrader, see here: Using argparse to change strategy parameters
The download tool shall allow users to specify the following options:

Instrument (Currency Pair)
Timeframe (1m, 5m, 1 hour etc)
Exchange

Notice how there are no inputs for start and end dates? The tool shall download everything that is available using CCXT’s fetchOHLCVmethod. Once the data is downloaded, it shall be exported to a CSV file in the current folder. You may wish to tweak this and add anotherargparseargument to specify the output directory.

The Code

'''
Author: www.backtest-rookies.com

MIT License

Copyright (c) 2018 backtest-rookies.com

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
'''

import ccxt
from datetime import datetime, timedelta, timezone
import math
import argparse
import pandas as pd


def parse_args():
    parser = argparse.ArgumentParser(description='CCXT Market Data Downloader')


    parser.add_argument('-s','--symbol',
                        type=str,
                        required=True,
                        help='The Symbol of the Instrument/Currency Pair To Download')

    parser.add_argument('-e','--exchange',
                        type=str,
                        required=True,
                        help='The exchange to download from')

    parser.add_argument('-t','--timeframe',
                        type=str,
                        default='1d',
                        choices=['1m', '5m','15m', '30m','1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1y'],
                        help='The timeframe to download')


    parser.add_argument('--debug',
                            action ='store_true',
                            help=('Print Sizer Debugs'))

    return parser.parse_args()

# Get our arguments
args = parse_args()

# Get our Exchange
try:
    exchange = getattr (ccxt, args.exchange) ()
except AttributeError:
    print('-'*36,' ERROR ','-'*35)
    print('Exchange "{}" not found. Please check the exchange is supported.'.format(args.exchange))
    print('-'*80)
    quit()

# Check if fetching of OHLC Data is supported
if exchange.has["fetchOHLCV"] != True:
    print('-'*36,' ERROR ','-'*35)
    print('{} does not support fetching OHLC data. Please use another exchange'.format(args.exchange))
    print('-'*80)
    quit()

# Check requested timeframe is available. If not return a helpful error.
if (not hasattr(exchange, 'timeframes')) or (args.timeframe not in exchange.timeframes):
    print('-'*36,' ERROR ','-'*35)
    print('The requested timeframe ({}) is not available from {}\n'.format(args.timeframe,args.exchange))
    print('Available timeframes are:')
    for key in exchange.timeframes.keys():
        print('  - ' + key)
    print('-'*80)
    quit()

# Check if the symbol is available on the Exchange
exchange.load_markets()
if args.symbol not in exchange.symbols:
    print('-'*36,' ERROR ','-'*35)
    print('The requested symbol ({}) is not available from {}\n'.format(args.symbol,args.exchange))
    print('Available symbols are:')
    for key in exchange.symbols:
        print('  - ' + key)
    print('-'*80)
    quit()


# Get data
data = exchange.fetch_ohlcv(args.symbol, args.timeframe)
header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume']
df = pd.DataFrame(data, columns=header).set_index('Timestamp')
# Save it
symbol_out = args.symbol.replace("/","")
filename = '{}-{}-{}.csv'.format(args.exchange, symbol_out,args.timeframe)
df.to_csv(filename)

100

101

102

103

104

105

106

107

108

109

'''

Author: www.backtest-rookies.com

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy

of this software and associated documentation files (the "Software"), to deal

in the Software without restriction, including without limitation the rights

to use, copy, modify, merge, publish, distribute, sublicense, and/or sell

copies of the Software, and to permit persons to whom the Software is

furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

SOFTWARE.

'''

import ccxt

from datetime import datetime, timedelta, timezone

import math

import argparse

import pandas as pd

def parse_args():

parser = argparse.ArgumentParser(description='CCXT Market Data Downloader')

parser.add_argument('-s','--symbol',

type=str,

required=True,

help='The Symbol of the Instrument/Currency Pair To Download')

parser.add_argument('-e','--exchange',

type=str,

required=True,

help='The exchange to download from')

parser.add_argument('-t','--timeframe',

type=str,

default='1d',

choices=['1m', '5m','15m', '30m','1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1y'],

help='The timeframe to download')

parser.add_argument('--debug',

action ='store_true',

help=('Print Sizer Debugs'))

return parser.parse_args()

# Get our arguments

args = parse_args()

# Get our Exchange

try:

exchange = getattr (ccxt, args.exchange) ()

except AttributeError:

print('-'*36,' ERROR ','-'*35)

print('Exchange "{}" not found. Please check the exchange is supported.'.format(args.exchange))

print('-'*80)

quit()

# Check if fetching of OHLC Data is supported

if exchange.has["fetchOHLCV"] != True:

print('-'*36,' ERROR ','-'*35)

print('{} does not support fetching OHLC data. Please use another exchange'.format(args.exchange))

print('-'*80)

quit()

# Check requested timeframe is available. If not return a helpful error.

if (not hasattr(exchange, 'timeframes')) or (args.timeframe not in exchange.timeframes):

print('-'*36,' ERROR ','-'*35)

print('The requested timeframe ({}) is not available from {}\n'.format(args.timeframe,args.exchange))

print('Available timeframes are:')

for key in exchange.timeframes.keys():

print(' - ' + key)

print('-'*80)

quit()

# Check if the symbol is available on the Exchange

exchange.load_markets()

if args.symbol not in exchange.symbols:

print('-'*36,' ERROR ','-'*35)

print('The requested symbol ({}) is not available from {}\n'.format(args.symbol,args.exchange))

print('Available symbols are:')

for key in exchange.symbols:

print(' - ' + key)

print('-'*80)

quit()

# Get data

data = exchange.fetch_ohlcv(args.symbol, args.timeframe)

header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume']

df = pd.DataFrame(data, columns=header).set_index('Timestamp')

# Save it

symbol_out = args.symbol.replace("/","")

filename = '{}-{}-{}.csv'.format(args.exchange, symbol_out,args.timeframe)

df.to_csv(filename)

The code is quite straightforward. After using argparseto capture our exchange, instrument and timezone requirements, the bulk of the work is spent error checking before and calling CCXT’s fetch_ohlcvmethod.
Speaking of error checking, not all exchanges support fetching of OHLCV data. Furthermore, not all exchanges support the same timeframes and currency pairs. Since there are over 100 exchanges supported at the time of writing, it is easy to imagine that there will be a lot of variances between exchanges. To avoid frustration with bumping into error after error trying to guess which timeframes and/or currency pairs are supported, the code performs a series of checks before attempting to download data. It detected something is not supported, it will notify the user and provide a list of alternative parameters where applicable.
Other than that, it is worth paying attention to the exchange.load_markets()call if you plan to expand upon the code or work with the library yourself. Admittedly, I spent a little too much some time scratching my head, trying to figure out why every exchange I tested seemed to have empty markets, symbolsand markets_by_iddictionaries. It turns out that snuggled in the documentation is the answer, you need to call load_markets()before any data is available. This was a good reminder to read the documentation carefully! A little time spent reading can save a lot of time.
Finally, if you are new to programming, the getattr(ccxt, args.exchange)might be new to you. getattr()is a python built-in function which allows you to “get an attribute” from an object using a string. In this case, we are getting the correct exchange object from the given argument string.
Reference: https://docs.python.org/3/library/functions.html#getattr

Usage Examples

Please note that the following usage examples assume the script is saved as ccxt_market_data.py.
ETH/NEO Binance Daily
python .ccxt_market_data.py -s NEO/ETH -e binance -t 1d
BTC/USD

Unsupported Timeframe

The example demonstrates the expected result when an unsupported timeframe is given as an argument:
python .ccxt_market_data.py -s NEO/ETH -e bittrex -t 12h

------------------------------------  ERROR  -----------------------------------
The requested timeframe (12h) is not available from bittrex
Available timeframes are:
  - 1m
  - 5m
  - 30m
  - 1h
  - 1d
--------------------------------------------------------------------------------

------------------------------------ ERROR -----------------------------------

The requested timeframe (12h) is not available from bittrex

Available timeframes are:

- 1m

- 5m

- 30m

- 1h

- 1d

--------------------------------------------------------------------------------

Unsupported Symbol

The second example has a fair bit of output. It demonstrates the expected result when an unsupported symbol is given as an argument:
python .ccxt_market_data.py -s ATP/USD -e bittrex -t 5m

------------------------------------  ERROR  -----------------------------------
The requested symbol (ATP/USD) is not available from bittrex
Available symbols are:
  - 1ST/BTC
  - 1ST/ETH
  - 2GIVE/BTC
  - ABY/BTC
  - ADA/BTC
  - ADA/ETH
  - ADA/USDT
  - ADT/BTC
  - ADT/ETH
  - ADX/BTC
  - ADX/ETH
  - AEON/BTC
  - AGRS/BTC
  - AMP/BTC
  - ANT/BTC
  - ANT/ETH
  - ARDR/BTC
  - ARK/BTC
  - AUR/BTC
  - BAT/BTC
  - BAT/ETH
  - BAY/BTC
  - BCH/BTC
  - BCH/ETH
  - BCH/USDT
  - BCPT/BTC
  - BCPT/ETH
  - BCY/BTC
  - BITB/BTC
  - BLITZ/BTC
  - BLK/BTC
  - BLOCK/BTC
  - BNT/BTC
  - BNT/ETH
  - BRK/BTC
  - BRX/BTC
  - BSD/BTC
  - BTC/USDT
  - BTG/BTC
  - BTG/ETH
  - BTG/USDT
  - BURST/BTC
  - BYC/BTC
  - CANN/BTC
  - CFI/BTC
  - CFI/ETH
  - CLAM/BTC
  - CLOAK/BTC
  - CLUB/BTC
  - COVAL/BTC
  - CPC/BTC
  - CRB/BTC
  - CRB/ETH
  - CRW/BTC
  - CURE/BTC
  - CVC/BTC
  - CVC/ETH
  - DASH/BTC
  - DASH/ETH
  - DASH/USDT
  - DCR/BTC
  - DCT/BTC
  - DGB/BTC
  - DGB/ETH
  - DMD/BTC
  - DNT/BTC
  - DNT/ETH
  - DOGE/BTC
  - DOPE/BTC
  - DTB/BTC
  - DYN/BTC
  - EBST/BTC
  - EDG/BTC
  - EFL/BTC
  - EGC/BTC
  - EMC/BTC
  - EMC2/BTC
  - ENG/BTC
  - ENG/ETH
  - ENRG/BTC
  - ERC/BTC
  - ETC/BTC
  - ETC/ETH
  - ETC/USDT
  - ETH/BTC
  - ETH/USDT
  - EXCL/BTC
  - EXP/BTC
  - FAIR/BTC
  - FCT/BTC
  - FCT/ETH
  - FLDC/BTC
  - FLO/BTC
  - FTC/BTC
  - GAM/BTC
  - GAME/BTC
  - GBG/BTC
  - GBYTE/BTC
  - GCR/BTC
  - GEO/BTC
  - GLD/BTC
  - GNO/BTC
  - GNO/ETH
  - GNT/BTC
  - GNT/ETH
  - GOLOS/BTC
  - GRC/BTC
  - GRS/BTC
  - GUP/BTC
  - GUP/ETH
  - HMQ/BTC
  - HMQ/ETH
  - IGNIS/BTC
  - INCNT/BTC
  - INFX/BTC
  - IOC/BTC
  - ION/BTC
  - IOP/BTC
  - KMD/BTC
  - KORE/BTC
  - LBC/BTC
  - LGD/BTC
  - LGD/ETH
  - LMC/BTC
  - LSK/BTC
  - LTC/BTC
  - LTC/ETH
  - LTC/USDT
  - LUN/BTC
  - LUN/ETH
  - MAID/BTC
  - MANA/BTC
  - MANA/ETH
  - MCO/BTC
  - MCO/ETH
  - MEME/BTC
  - MER/BTC
  - MLN/BTC
  - MONA/BTC
  - MUE/BTC
  - MUSIC/BTC
  - NAV/BTC
  - NBT/BTC
  - NEO/BTC
  - NEO/ETH
  - NEO/USDT
  - NEOS/BTC
  - NLG/BTC
  - NMR/BTC
  - NMR/ETH
  - NXC/BTC
  - NXS/BTC
  - NXT/BTC
  - NXT/USDT
  - OK/BTC
  - OMG/BTC
  - OMG/ETH
  - OMG/USDT
  - OMNI/BTC
  - PART/BTC
  - PAY/BTC
  - PAY/ETH
  - PDC/BTC
  - PINK/BTC
  - PIVX/BTC
  - PKB/BTC
  - POT/BTC
  - POWR/BTC
  - POWR/ETH
  - PPC/BTC
  - PTC/BTC
  - PTOY/BTC
  - PTOY/ETH
  - QRL/BTC
  - QRL/ETH
  - QTUM/BTC
  - QTUM/ETH
  - QWARK/BTC
  - RADS/BTC
  - RBY/BTC
  - RCN/BTC
  - RCN/ETH
  - RDD/BTC
  - REP/BTC
  - REP/ETH
  - RLC/BTC
  - RLC/ETH
  - SALT/BTC
  - SALT/ETH
  - SBD/BTC
  - SC/BTC
  - SC/ETH
  - SEQ/BTC
  - SHIFT/BTC
  - SIB/BTC
  - SLR/BTC
  - SLS/BTC
  - SNRG/BTC
  - SNT/BTC
  - SNT/ETH
  - SPHR/BTC
  - SPR/BTC
  - SRN/BTC
  - SRN/ETH
  - START/BTC
  - STEEM/BTC
  - STORJ/BTC
  - STORJ/ETH
  - STRAT/BTC
  - STRAT/ETH
  - SWIFT/BTC
  - SWT/BTC
  - SYNX/BTC
  - SYS/BTC
  - THC/BTC
  - TIX/BTC
  - TIX/ETH
  - TKS/BTC
  - TRST/BTC
  - TRST/ETH
  - TRUST/BTC
  - TRX/BTC
  - TRX/ETH
  - TUSD/BTC
  - TX/BTC
  - UBQ/BTC
  - UKG/BTC
  - UKG/ETH
  - UNB/BTC
  - VEE/BTC
  - VEE/ETH
  - VIA/BTC
  - VIB/BTC
  - VIB/ETH
  - VOX/BTC
  - VRC/BTC
  - VRM/BTC
  - VTC/BTC
  - VTR/BTC
  - WAVES/BTC
  - WAVES/ETH
  - WAX/BTC
  - WAX/ETH
  - WINGS/BTC
  - WINGS/ETH
  - XCP/BTC
  - XDN/BTC
  - XEL/BTC
  - XEM/BTC
  - XEM/ETH
  - XLM/BTC
  - XLM/ETH
  - XMG/BTC
  - XMR/BTC
  - XMR/ETH
  - XMR/USDT
  - XMY/BTC
  - XRP/BTC
  - XRP/ETH
  - XRP/USDT
  - XST/BTC
  - XVC/BTC
  - XVG/BTC
  - XVG/USDT
  - XWC/BTC
  - XZC/BTC
  - ZCL/BTC
  - ZEC/BTC
  - ZEC/ETH
  - ZEC/USDT
  - ZEN/BTC
  - ZRX/BTC
  - ZRX/ETH
--------------------------------------------------------------------------------

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

------------------------------------ ERROR -----------------------------------

The requested symbol (ATP/USD) is not available from bittrex

Available symbols are:

- 1ST/BTC

- 1ST/ETH

- 2GIVE/BTC

- ABY/BTC

- ADA/BTC

- ADA/ETH

- ADA/USDT

- ADT/BTC

- ADT/ETH

- ADX/BTC

- ADX/ETH

- AEON/BTC

- AGRS/BTC

- AMP/BTC

- ANT/BTC

- ANT/ETH

- ARDR/BTC

- ARK/BTC

- AUR/BTC

- BAT/BTC

- BAT/ETH

- BAY/BTC

- BCH/BTC

- BCH/ETH

- BCH/USDT

- BCPT/BTC

- BCPT/ETH

- BCY/BTC

- BITB/BTC

- BLITZ/BTC

- BLK/BTC

- BLOCK/BTC

- BNT/BTC

- BNT/ETH

- BRK/BTC

- BRX/BTC

- BSD/BTC

- BTC/USDT

- BTG/BTC

- BTG/ETH

- BTG/USDT

- BURST/BTC

- BYC/BTC

- CANN/BTC

- CFI/BTC

- CFI/ETH

- CLAM/BTC

- CLOAK/BTC

- CLUB/BTC

- COVAL/BTC

- CPC/BTC

- CRB/BTC

- CRB/ETH

- CRW/BTC

- CURE/BTC

- CVC/BTC

- CVC/ETH

- DASH/BTC

- DASH/ETH

- DASH/USDT

- DCR/BTC

- DCT/BTC

- DGB/BTC

- DGB/ETH

- DMD/BTC

- DNT/BTC

- DNT/ETH

- DOGE/BTC

- DOPE/BTC

- DTB/BTC

- DYN/BTC

- EBST/BTC

- EDG/BTC

- EFL/BTC

- EGC/BTC

- EMC/BTC

- EMC2/BTC

- ENG/BTC

- ENG/ETH

- ENRG/BTC

- ERC/BTC

- ETC/BTC

- ETC/ETH

- ETC/USDT

- ETH/BTC

- ETH/USDT

- EXCL/BTC

- EXP/BTC

- FAIR/BTC

- FCT/BTC

- FCT/ETH

- FLDC/BTC

- FLO/BTC

- FTC/BTC

- GAM/BTC

- GAME/BTC

- GBG/BTC

- GBYTE/BTC

- GCR/BTC

- GEO/BTC

- GLD/BTC

- GNO/BTC

- GNO/ETH

- GNT/BTC

- GNT/ETH

- GOLOS/BTC

- GRC/BTC

- GRS/BTC

- GUP/BTC

- GUP/ETH

- HMQ/BTC

- HMQ/ETH

- IGNIS/BTC

- INCNT/BTC

- INFX/BTC

- IOC/BTC

- ION/BTC

- IOP/BTC

- KMD/BTC

- KORE/BTC

- LBC/BTC

- LGD/BTC

- LGD/ETH

- LMC/BTC

- LSK/BTC

- LTC/BTC

- LTC/ETH

- LTC/USDT

- LUN/BTC

- LUN/ETH

- MAID/BTC

- MANA/BTC

- MANA/ETH

- MCO/BTC

- MCO/ETH

- MEME/BTC

- MER/BTC

- MLN/BTC

- MONA/BTC

- MUE/BTC

- MUSIC/BTC

- NAV/BTC

- NBT/BTC

- NEO/BTC

- NEO/ETH

- NEO/USDT

- NEOS/BTC

- NLG/BTC

- NMR/BTC

- NMR/ETH

- NXC/BTC

- NXS/BTC

- NXT/BTC

- NXT/USDT

- OK/BTC

- OMG/BTC

- OMG/ETH

- OMG/USDT

- OMNI/BTC

- PART/BTC

- PAY/BTC

- PAY/ETH

- PDC/BTC

- PINK/BTC

- PIVX/BTC

- PKB/BTC

- POT/BTC

- POWR/BTC

- POWR/ETH

- PPC/BTC

- PTC/BTC

- PTOY/BTC

- PTOY/ETH

- QRL/BTC

- QRL/ETH

- QTUM/BTC

- QTUM/ETH

- QWARK/BTC

- RADS/BTC

- RBY/BTC

- RCN/BTC

- RCN/ETH

- RDD/BTC

- REP/BTC

- REP/ETH

- RLC/BTC

- RLC/ETH

- SALT/BTC

- SALT/ETH

- SBD/BTC

- SC/BTC

- SC/ETH

- SEQ/BTC

- SHIFT/BTC

- SIB/BTC

- SLR/BTC

- SLS/BTC

- SNRG/BTC

- SNT/BTC

- SNT/ETH

- SPHR/BTC

- SPR/BTC

- SRN/BTC

- SRN/ETH

- START/BTC

- STEEM/BTC

- STORJ/BTC

- STORJ/ETH

- STRAT/BTC

- STRAT/ETH

- SWIFT/BTC

- SWT/BTC

- SYNX/BTC

- SYS/BTC

- THC/BTC

- TIX/BTC

- TIX/ETH

- TKS/BTC

- TRST/BTC

- TRST/ETH

- TRUST/BTC

- TRX/BTC

- TRX/ETH

- TUSD/BTC

- TX/BTC

- UBQ/BTC

- UKG/BTC

- UKG/ETH

- UNB/BTC

- VEE/BTC

- VEE/ETH

- VIA/BTC

- VIB/BTC

- VIB/ETH

- VOX/BTC

- VRC/BTC

- VRM/BTC

- VTC/BTC

- VTR/BTC

- WAVES/BTC

- WAVES/ETH

- WAX/BTC

- WAX/ETH

- WINGS/BTC

- WINGS/ETH

- XCP/BTC

- XDN/BTC

- XEL/BTC

- XEM/BTC

- XEM/ETH

- XLM/BTC

- XLM/ETH

- XMG/BTC

- XMR/BTC

- XMR/ETH

- XMR/USDT

- XMY/BTC

- XRP/BTC

- XRP/ETH

- XRP/USDT

- XST/BTC

- XVC/BTC

- XVG/BTC

- XVG/USDT

- XWC/BTC

- XZC/BTC

- ZCL/BTC

- ZEC/BTC

- ZEC/ETH

- ZEC/USDT

- ZEN/BTC

- ZRX/BTC

- ZRX/ETH

--------------------------------------------------------------------------------

Unsupported Exachange

The final example demostrates the expected result when an exchange does not support fethcing of OHLCV data.
python .ccxt_market_data.py -s BTC/USD -e gdax -t 15m

------------------------------------  ERROR  -----------------------------------
gdax does not support fetching OHLC data. Please use another exchange
--------------------------------------------------------------------------------

------------------------------------ ERROR -----------------------------------

gdax does not support fetching OHLC data. Please use another exchange

--------------------------------------------------------------------------------

Issues and Improvements

The code tries to be user-friendly in that it will tell the user when incorrect data is provided. Having said that, not all edge cases have been considered and there is definitely room for improvement in that regard.
In addition to this, the amount of data available on most exchanges for download is limited. CCXT does provide a sinceparameter for the fetchOHLCV()method but I did not see any detailed description for this in the fetchOHLCV()documentation. Having said that, an explanation of a parameter with the same name does appear the “Trades, Executions and Transactions” section as follows (Note: That the parameter appears frequently, so I assume it will have consistent functionality but that is no guarantee!):

The second optional argument since reduces the array by timestamp, the third limit argument reduces by number (count) of returned items.

Reference: https://github.com/ccxt/ccxt/wiki/Manual#trades-executions-transactions
So taking that into account, it appears sinceshould reduce the amount of data provided. However, it would be good to know for sure. If anyone pokes through the codebase and has an answer, I would be happy to hear it. Leave a comment below!

Hi,

As other users tell in their comments, format of timestamp as printed in the CSV file is not that easy to use (at least for a newbie).
More specifically, I would like to import back the data in backtrader.
As you have used panda.Dataframe for export, it would seem that using Panda data feed in backtrader would me seamless.
I used the script given in backtrader doc here:
https://www.backtrader.com/docu/pandas-datafeed/pandas-datafeed.html?highlight=panda

But I have trouble, date format is not recognised.
I don’t know how solving the issue.
Please, do you think you could show an updated version of your script or below script to be able to make such data transfer between CCXT & Backtrader, relying on panda.dataFrame?

from __future__ import (absolute_import, division, print_function, unicode_literals)


import argparse
import backtrader as bt

import backtrader.feeds as btfeeds
import pandas
def runstrat():

    args = parse_args()
    # Create a cerebro entity

    cerebro = bt.Cerebro(stdstats=False)
    # Add a strategy

    cerebro.addstrategy(bt.Strategy)
    # Get a pandas dataframe

    datapath = ('../../datas/2006-day-001.txt')
    # Simulate the header row isn't there if noheaders requested

    skiprows = 1 if args.noheaders else 0

    header = None if args.noheaders else 0
    dataframe = pandas.read_csv(datapath,

                                skiprows=skiprows,

                                header=header,

                                parse_dates=True,

                                index_col=0)
    if not args.noprint:

        print('--------------------------------------------------')

        print(dataframe)

        print('--------------------------------------------------')
    # Pass it to the backtrader datafeed and add it to the cerebro

    data = bt.feeds.PandasData(dataname=dataframe)
    cerebro.adddata(data)
    # Run over everything

    cerebro.run()
    # Plot the result

    cerebro.plot(style='bar')
def parse_args():

    parser = argparse.ArgumentParser(

        description='Pandas test script')
    parser.add_argument('--noheaders', action='store_true', default=False,

                        required=False,

                        help='Do not use header rows')
    parser.add_argument('--noprint', action='store_true', default=False,

                        help='Print the dataframe')
    return parser.parse_args()

if __name__ == '__main__': runstrat()

I hope you will be able to consider positively this demand.
Have a good day and thanks to share all your tutorials.
Bests,
Pierrot

36 Responses

Jaspal March 17, 2018 at 2:26 am Reply

Hi,

many thanks for the tutorial.

I am trying to import the data into your optimise strategy which I have modified.

However it does not seem to work.

How would you modify your optimise example to include ccxt?

Rookie1 March 17, 2018 at 9:06 pm Reply

Hi Jaspal,
It is hard to say without seeing the strategy. However, one thing I can think of is to check the way you are adding the data. You will need to import the data downloaded as CSV and make sure the columns are mapped correctly.

Try taking a look at the GenericCSV docs here:

https://www.backtrader.com/docu/datafeed.html#genericcsvdata

Jaspal March 22, 2018 at 3:45 am Reply

Hi, I managed to figure it out. I had to declare the format of the csv file in my code with:

# CSV INPUT FILE FORMAT CONFIGURATION
class dataFeed(btfeed.GenericCSVData):
params = (
(‘dtformat’, ‘%Y-%m-%d %H:%M:%S’),
(‘datetime’, 0),
(‘open’, 1),
(‘high’, 2),
(‘low’, 3),
(‘close’, 4),
(‘volume’, 5),
(‘openinterest’, -1)
)

Thanks again!

Rookie1 March 22, 2018 at 9:03 am Reply

Awesome!

gito May 9, 2018 at 4:58 pm Reply

@Jaspal : exactly what I was looking for! thank you
Jorge May 12, 2018 at 5:28 am Reply

Amazing! Thanks a lot for sharing this post, it´s really useful

Another thing…watching the CSV file generated with the call..the first column corresponding to Timestamp…it would be better if we change the time format using datetime in order to get something readable.
Some May 29, 2018 at 10:28 pm Reply

Hello,

Thanks for the great code.
I thought I’d share, with all teh other noobs, a problem I had.

… If you name this code ccxt.py, specifically, it will not fire.
It will come back with an “Exchange “bitfinex” not found. Please check the exchange is supported.” error, no matter what exchange is entered.

Renaming the file to any name other than ccxt.py will solve this issue.

Some Guy

STEPHAN November 18, 2018 at 7:43 am Reply

Hi Some Guy, I have the same exchange not supported error no matter what exchange is entered. I tried changing the name of my file to no avail. Any ideas?

gaorufeng September 3, 2018 at 3:17 pm Reply

error begin 20180902

python /root/ccxt_market_data.py -s BTC/USDT -e binance -t 6h

1535911200000,7298.38,7333.33,7237.06,7302.01,7507.444406
1535932800000,7302.0,7325.0,7191.63,7225.0,7222.945765
1535954400000,7225.02,7260.0,7213.25,7250.79,1612.071928

the close is ok, but the open,high,low are error.

https://imgur.com/a/kD6fGL4

Rookie1 September 5, 2018 at 2:01 pm Reply

Hi Garufeng,

The script simply saved the results returned from the exchange. If you think there are data errors, it might be worth checking it with the exchange.

Some exchanges have API explorers where you can make test calls to the API and compare it against the results.

Good luck!

Thom October 9, 2018 at 12:04 pm Reply

Thank you for the help . how do you change the format of date in the csv file?
Lucas Pinsdorf October 26, 2018 at 4:06 am Reply

Hello!
Thanks for the post!
Do you know how can I get more than just one pair from the same exchange? Do you I have to separate the markets with a comma or something like that?
Thanks!

Rookie1 October 26, 2018 at 8:58 am Reply

Hi Lucas!
The code example would need to be expanded to support multiple downloads. We would need to handle a list of pairs with some in argparse and also run the download in a loop.

Alternatively, you could create a bash script to run this script multiple times, each time with a different pair.

Cheers!

Lucas Pinsdorf November 1, 2018 at 12:18 am Reply

Thanks, Rookie! We created a bash script here and run smoothly 🙂

Thank you so much for the code.

One last question, if I may, is there a way to get 1 year timeframe in all exchanges supported or only the timeframes provided by the specific API?

Bittrex, for instance, I couldn’t get this information. What I specifically need is the volume of that specific pair in Bittrex in the last year. Is there a way to do it or I would have to buy some market data, such as the one from CoinMktCap?

Stephan November 18, 2018 at 7:42 am Reply

Hello! Noob here. For some reason, I always return “Exchange “coinbase” not found. Please check the exchange is supported.” with the same error for any exchange. Any idea what I’m doing wrong?

Rookie1 November 23, 2018 at 9:46 am Reply

Hi Stepan – Sorry for the late reply.

Coinbase does not support downloading OHLC data. The error message should tell you this.

Try to use GDAX instead. (CoinbasePro)

Just tested – This line is working:

python .\ccxt_market_data.py -exchange -e gdax -s BTC/USD -t 1m

Replace python with python3 if you also have python2 installed!

Lucas Pinsdorf December 20, 2018 at 1:03 am Reply

Hello! I described an error that I was having with CCXT and BackTest code and got an insightful answer from one of the maintainers, you should check it out! 🙂

https://github.com/ccxt/ccxt/issues/4356

Cheers!

Rookie1 December 20, 2018 at 4:03 pm Reply

Awesome Lucas! It is always good to improve the code.

The suggestions from the maintainer have been added to the code above.

Gabriel December 20, 2018 at 3:33 am Reply

Hello, how are you?

When I run this to get the volume from bitfinex, it only gives me 101 entries, starting and ending in 2017.

Is there some way I can get only the latest 3 months (~101 entries)?
Thanks!

Rookie1 December 20, 2018 at 4:16 pm Reply

Hi Gabriel,
I think Bitfinex provide their data in reverse order and limit the amount of data points returned.

As such, you would need add some extra parameters to the exchange.fetch_ohlcv() call.

The parameters you need are since limit and params

since controls the date we request from, limit controls the download size (for requesting more than 101 data points) and params can be used to request the data in reverse order.

See here for more info:

https://github.com/ccxt/ccxt/wiki/Manual#ohlcv-candlestick-charts

Here is a snippet regarding since

” Some exchanges will return candles from the beginning of time, others will return most recent candles only, the exchanges’ default behaviour is expected. Thus, without specifying since the range of returned candles will be exchange-specific. One should pass the since argument to ensure getting precisely the history range needed.”

Long January 17, 2019 at 6:32 pm Reply

Dear all,

Could you help me how to download ccxt program and install in my computer?
Chris March 5, 2019 at 9:35 pm Reply

Thank you for the syntax!
I am completely new to pyhton and get stuck to the line
# Get our arguments
args = parse_args()

It returns:

File “”, line 31
args = parse_args()

Are there any solution for that?

Thank you so much for your help!

Chris March 6, 2019 at 4:17 pm Reply

I have figured that out and it works now!
Such a silly question I asked.
Thank you for the script!!

henry March 20, 2019 at 5:04 am Reply

Admittedly I’m a complete novice in python, and you could argue I should go way back and learn the basics before attempting this….

But I just cant get from one step to the next. If I copy the above script for the argparse and save it down, it does not allow me to call that script running the above command.

If I type in “python3 argparse-tutorial.py –help” and run it I get the following error:

———————————————————————————————-

C:\Users\xxxx\PycharmProjects\untitled\venv\Scripts\python.exe C:/Users/xxxx/PycharmProjects/untitled/venv/Lib/site-packages/backtrader/e.py
File “C:/Users/xxxx/PycharmProjects/untitled/venv/Lib/site-packages/backtrader/e.py”, line 1
python argparse-tutorial.py –help
^
SyntaxError: invalid syntax

Process finished with exit code 1

——————————————————————————————————-

I can’t understand what’s going wrong, it seems like there is a step missing in this tutorial, and the one for the Cyrpto data, which is what you do with the argparse script, and how you run another script querying it?

Sorry for the dumb question.

H

Henry March 20, 2019 at 5:31 am Reply

Ignore me, I’m obviously too noob to comprehend. I will explain the issue so any other noobs don’t run into the same issue.

When you save the script you need to run the command in command prompt(windows) or bash(Mac), for example:

> open command prompt
> go to the directory the script is saved in
> run command (eg. “python .ccxt_market_data.py -s NEO/ETH -e binance -t 1d”)

For mega noobs like me, I thought i was running a SECOND python script which was dependant on the argparse script, but it was in fact a command line prompt that should be ran.

(I know how stupid I look, but I wanted to detail the issue I had so that I might save another noob the time I just spent trying to work it out)

Rookie1 March 20, 2019 at 10:10 am Reply

Hi Henry,
Don’t worry about it! It is never stupid to ask a question. We are all here to learn.

Actually, the fact you had an issue means you are likely not the only one and so your comment here will help others.

Thanks and good luck with the learning!

Chris April 5, 2019 at 1:01 am Reply

Hello,

Currently I can only get 500 row of data.
Is it possible to get more?
Thank you!

JS July 20, 2019 at 12:37 pm Reply

Hi,

@Rookie1 Thanks very much for the code. This is really helpful.

I am also facing this issue. Can only fetch 500 rows of data in csv.
Could someone please suggest how to control for start and end dates.

Thanks!

Pierrot May 1, 2019 at 6:49 pm Reply

Hi,

As other users tell in their comments, format of timestamp as printed in the CSV file is not that easy to use (at least for a newbie).
More specifically, I would like to import back the data in backtrader.
As you have used panda.Dataframe for export, it would seem that using Panda data feed in backtrader would me seamless.
I used the script given in backtrader doc here:
https://www.backtrader.com/docu/pandas-datafeed/pandas-datafeed.html?highlight=panda

But I have trouble, date format is not recognised.
I don’t know how solving the issue.
Please, do you think you could show an updated version of your script or below script to be able to make such data transfer between CCXT & Backtrader, relying on panda.dataFrame?

from __future__ import (absolute_import, division, print_function, unicode_literals)
import argparse import backtrader as bt import backtrader.feeds as btfeeds import pandas def runstrat(): args = parse_args() # Create a cerebro entity cerebro = bt.Cerebro(stdstats=False) # Add a strategy cerebro.addstrategy(bt.Strategy) # Get a pandas dataframe datapath = ('../../datas/2006-day-001.txt') # Simulate the header row isn't there if noheaders requested skiprows = 1 if args.noheaders else 0 header = None if args.noheaders else 0 dataframe = pandas.read_csv(datapath, skiprows=skiprows, header=header, parse_dates=True, index_col=0) if not args.noprint: print('--------------------------------------------------') print(dataframe) print('--------------------------------------------------') # Pass it to the backtrader datafeed and add it to the cerebro data = bt.feeds.PandasData(dataname=dataframe) cerebro.adddata(data) # Run over everything cerebro.run() # Plot the result cerebro.plot(style='bar') def parse_args(): parser = argparse.ArgumentParser( description='Pandas test script') parser.add_argument('--noheaders', action='store_true', default=False, required=False, help='Do not use header rows') parser.add_argument('--noprint', action='store_true', default=False, help='Print the dataframe') return parser.parse_args()
if __name__ == '__main__': runstrat()

I hope you will be able to consider positively this demand.
Have a good day and thanks to share all your tutorials.
Bests,
Pierrot
Pierrot May 1, 2019 at 11:03 pm Reply

Hi,
to the interested readers that would like to write timestamp in iso format, I found the solution given in CCXT bug tracker here:
https://github.com/ccxt/ccxt/issues/4478
So a tiniy single line does the trick (maybe to be included in your script David?)
# convert timestamps to iso8601 data = [[exchange.iso8601(candle[0])] + candle[1:] for candle in data]

To be added in the “Get data” section:
# Get data data = exchange.fetch_ohlcv(args.symbol, args.timeframe) # convert timestamps to iso8601 data = [[exchange.iso8601(candle[0])] + candle[1:] for candle in data] header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume'] df = pd.DataFrame(data, columns=header).set_index('Timestamp')

Hope this will be of some help.
Have a good afternoon.
Best regards,
Pierrot
Jakub June 14, 2019 at 6:11 pm Reply

Hello.

How I can add a ‘symbol’ to CSV?

and more specific question. What I can do to have exact format in CSV like this one:

Date,Symbol,Open,High,Low,Close,Volume BTC,Volume USD
2019-05-17 01-AM,BTCUSD,7881.86,7897.92,7881.86,7897.92,0.0,0.0
Dim Norin July 15, 2019 at 9:54 pm Reply

Correct read data from saved csv to backtrader:

filename = ‘{}-{}-{}.csv’.format(‘binance’, symbol.replace(“/”,””), timeframe)
df = pd.read_csv(filename,
header=0,
index_col=’Timestamp’,
parse_dates=[‘Timestamp’],
date_parser=lambda x: pd.to_datetime(int(x)/1000,unit=’s’)
)
data = bt.feeds.PandasData(dataname=df)
mauro October 14, 2019 at 9:50 am Reply

only fetch 500 rows of data in csv.
Could someone please suggest how to control for start and end dates. thanks!
Sandro Böhme January 10, 2020 at 1:18 am Reply

Regarding the since parameter https://github.com/ccxt/ccxt/wiki/Manual#ohlcv-candlestick-charts states:
> The since argument is an integer UTC timestamp in milliseconds
> (everywhere throughout the library with all unified methods).

With Python 3.3 and above one could use something along the lines of this for the since parameter:
int(datetime.strptime('1.1.2017 1:00', '%d.%m.%Y %H:%M').timestamp() * 1000)

As a sidenote: E.g. with Binance you can download a maximum limit of 1000 lines of ohlc data with one go. See:
https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#klinecandlestick-data
Sandro January 10, 2020 at 1:49 am Reply

I have to correct myself.
The above time conversion would assume the local timezone. To get around that UTC should be specified explicitly like this:

int(datetime.strptime(args.since, '%d.%m.%Y %H:%M').replace(tzinfo=timezone.utc).timestamp() * 1000)

You could regenerate the resulting timestamp with a website like https://www.epochconverter.com/timezones?q=1578445200000&tz=UTC and crosscheck the ohlc data from the file for this candle with the one in tradingview.com
Wyllyam January 15, 2020 at 11:45 pm Reply

Hello!! Thanks very much for the code 😉
I cant run the code. The error message should tell you this:

Error 2
Looking for stack:
> File “C:\Users\Shinobi\AppData\Local\temporary projects\pythonapplication1\pythonapplication1.py”, line 33, in parse_args
> return parser.parse_args()
> File “C:\Users\Shinobi\AppData\Local\temporary projects\pythonapplication1\pythonapplication1.py”, line 36, in
> args = parse_args()

Please, can you help me? thanks

Download Cryptocurrency Data with CCXT

Installing the Library

Other Dependencies

Download Tool Overview

The Code

Usage Examples

Unsupported Timeframe

Unsupported Symbol

Unsupported Exachange

Issues and Improvements

Related

36 Responses

Leave a Reply

Installing the Library

Other Dependencies

Download Tool Overview

The Code

Usage Examples

Unsupported Timeframe

Unsupported Symbol

Unsupported Exachange

Issues and Improvements

Share this:

Related

36 Responses

Leave a Reply