It’s imperative for the crypto-market analysts to have access to all sorts of cryptocurrency historical data. However, because the majority of the crypto-exchanges limit the extraction of the former, one has to wait until the real-time data pumps up inevitably. If lacking time for so, it is perfectly convenient to analyze the information that can be withdrawn from the archives.
Ways to exploit the cryptocurrency historical data
First of all, the information helps with the data analysis—by knowing the market’s dynamics, it’s much easier to predict its behavior in the nearby future. Secondly, the data can assist with training the trade algorithms—it’s possible to train and test the artificial intelligence systems, thus further applying them to the algorithmic trading. Finally, the withdrawn information can be used for checking out the trading strategies. If you have severe plans in regards to the crypto-investments, you surely better test them out with the help of historical evidence.
Background info
Before starting to delve into the crypto-exchanges, it’s vital to comprehend some basic operating principles of the industry. For instance, it’s clear that some of the sites use the DDoS protection systems such as Cloudflare. To this end, it’s better to double-check that the platform isn’t blocking your country’s IP to be sure that there wouldn’t be any inconveniences when accessing the REST API’s of the particular crypto-exchange. Because the CAPTCHA would keep popping up, you would be forced to use VPN or Proxy Servers to secure the flawless working experience.
At the same time, it’s worthwhile knowing that some of the platforms can continuously change their endpoints, which is why there is an apparent need to keep examining the official documentation. Furthermore, one needs to realize that the platforms allow for downloading the transaction history and so-called candles. However, not a single crypto-exchange gives a chance for extracting the orders books as their record is only available in real-time.
Finally, it’s essential to check the names of the currency pairs on the different sites as they could mess up some things in a significant way. In some cases, the names of the currencies can be displayed in various formats when accessing API, only reinforcing the previous claim.
Specifics of Crypto-exchanges
Every single crypto-exchange platform has its features that require a particular approach when working with it as can be seen below (the article published on 01.06.2018 and reflects a market on such a date).
Binance
- Location: Hong Kong
- API: REST, WebSocket
- Docs: Binance official api docs
- Location: $1.5B (01.06.2018)
Currently, Binance, a platform registered in Hong Kong, is one of the most popular and in-demand crypto-exchanges in the world. In fact, such a status can be justified by solely looking at the customer count and the capitalization of the market. Such numbers are undoubtedly impressive, especially if considering that Binance functions less than a single year. Because of such prominence, it is reasonable to believe that Binance would inevitably remain the industry’s leader for a long time.
Pros & Cons:
- It’s feasible to obtain the historical data from the beginning of trades on Binance.
- There are 300 hundred cryptocurrency pairs traded on the platform.
- REST API works pretty stable.
- Binance doesn’t require an additional generation of API Key for getting the historical data.
- All the while, one has to consider the fact that each request has its own size as stated in the Binance’s documentation and API. It’s merely more convenient to use the requests, such as endpoint GET /api/v1/aggTrades as they allow for optimizing the process of obtaining data due to the minimum requests weight.
- Binance allows for the hourly extraction of data, making the extracting process faster.
- There is a need to control a number of requests per seconds, yet the valid queries are more than enough for the simultaneous extraction of a couple of currency pairs.
HitBTC
- Location: Hong Kong
- API: REST, WebSocket
- Docs: HitBTC official api docs
- Location: $300M (01.06.2018)
The crypto-exchange platform HitBTC allows for more than 300 crypto-pairs trade. Created by the Estonian and Israeli entrepreneurs, HitBTC became a major site where Bitcoin(BTC) and Bitcoin Cash(BCH) are traded on the constant basis.
Pros & Cons:
- As for REST API, it works pretty stable with no problems reflected on downloading.
- The historical information can be derived up from the beginning of trades on HitBTC.
- HitBTC allows for the maximum of 1000 transactions per one single request.
- HitBTC doesn’t require the additional authentication and the generation of API Key.
- A number of possible requests is enough for the average extracting of data.
- One should be considerate of the fact that the REST API of HitBTC releases the transaction time with the precise accuracy in the text format, meaning that there could be a need to use the conversion for the timestamp.
Bithumb
- Location: South Korea
- API: REST
- Docs: Bithumb official api docs
- Location: $270M (01.06.2018)
Bithumb is the most popular crypto-exchange platform in South Korea and is one of the largest ones in the world in terms of the trading volume. According to Korea Herald, Bithumb accounts for 75,7% of the overall Bitcoin (BTC) trade in South Korea and 50% of Ethereum (ETH). To this end, it’s no surprise that the platform forms pairs in a matter that follows cryptocurrency - South Korean won. Even though Bithumb is massive in scale, it’s quite difficult to find any information regarding the executives and members of the board as can be seen on Bloomberg. The only thing known of Bithumb is that it belongs to BTC Korea.com Co. LTD and is based in Seoul.
Pros & Cons:
- It’s possible to obtain the historical information since the beginning of trades on Bithumb.
- A number of cryptocurrency pairs is not big (33), yet Bithumb is really popular all across the globe and particularly in South Korea.
- The possible number of request is enough for the simultaneous download of multiple cryptocurrency pairs.
- Bithumb doesn’t require the extra authentication and the generation of API Key.
- In general, REST API works stable, although every now and then one has to handle the 525 or 520 response. If this is the case, there is a need to send a request yet again with the same parameters.
- The downloading process itself is going to go on pretty slowly as Bithumb only grants 100 transactions per one single request.
- The REST API of Bithumb gives the precise transaction time up to seconds accordingly to the South Korean timezone. If your system operates with multiple crypto-exchanges, then it may be helpful to convert the data to a common time zone.
Kraken
- Location: US
- API: REST
- Docs: Kraken official api docs
- Location: $147M (01.06.2018)
The crypto-exchange platform Kraken was opened way back in 2011 in San Francisco. Kraken was the first of its kind and lead the industry: the platform’s trades and volumes started to appear on Bloomberg, and passed the cryptographically verified audit of reserves. Not only does the Kraken trade cryptocurrencies, but also operate with the American and Canadian Dollars, British Pounds, as well as with Japanese Yens. Kraken is the owner of the cryptowatch service, the one that allows for extracting data in the real-time mode. Yet, the platform doesn’t allow for direct obtaining of the cryptocurrency historical data, which is disappointing.
Pros & Cons:
- It’s possible to extract the data up from the beginning of trades on Kraken.
- As the official documentation states, it’s vital to control the number of requests per second: the process goes as fast as one may extract 2000 transactions per one query.
- Close to 50 cryptocurrency pairs are being traded on Kraken.
- Kraken doesn’t require the extra authentication and generation of API Key.
- REST API works really stable, although sometimes a mistake {“error”:[“EService:Unavailable”]} may pop up. If this is the case, then it may be helpful to send the request yet again by using the same parameters.
- When navigating through the historical data, one needs to have the transaction ID. Kraken doesn’t show the ID of every separate transaction but only the ID of the last one, which is received per request. Such a drawback is pretty inconvenient as one has to beforehand known the historical trade ID to move forward with the lookout.
Gate.io
- Location: China
- API: REST
- Docs: Gate.io official api docs
- Location: $134M (01.06.2018)
Gate.io is a cryptocurrency exchange platform that provides trading access to a vast array of digital coins. Available both in Chinese and English, Gate.io started way back in 2017 and seems to be functioning outside of the US. The platform offers pretty competitive fees and is not available for all the investors: Gate.io only supports blockchain transfers in the cryptocurrencies that are currently traded on the platform. The majority of the trades take place in the form of BTC, ETH, EOS, BTM, USDT, ONT and is worth approximately $120M daily.
Pros & Cons:
- The historical data can be extracted from the beginning of trades on the Gate.io platform.
- It’s worthwhile considering that REST API is pretty stable.
- The transaction time is given with the accuracy up to milliseconds.
- There are 300 cryptocurrency pairs traded on the platform.
- Gate.io doesn’t require the extra authentication and the generation of API Key.
- There may be some difficulties in delving through the transaction history because of the nature of each trade ID: it’s unique for the exchange platform as a whole, rather than for every single crypto pair.
bitFlyer
- Location: Japan
- API: REST
- Docs: bitFlyer official api docs
- Location: $68M (01.06.2018)
bitFlyer is the major Japanese trading platform that was created back in 2014. The exchange has its headquarters in Tokyo and is lead by CEO Yuzo Kanzo and CTO Takafumi Komiyama. bitFlyer did undergo some investment rounds while gathering funds for the project: by attracting more than $35 million dollars in total, bitFlyer was able to open the office in San Francisco and obtain a legal allowance to conduct trade in 35 US States. Simultaneously, bitFlyer partnered with Vandle Card, which allowed the platform to release its own Visa Cards: not only did it allow for depositing from various sources but also for having access to various trades. To manage funds, the mobile app called bitFlyer Waller was created, which is pretty convenient to all of the owners.
Pros & Cons:
- It’s possible to get the historical data from up from the beginning of trades.
- bitFlyer doesn’t require the extra authentication and the generation of API Key.
- bitFlyer provides a fairly limited list of trading pairs (BTC, ETH, BCH) but the obtained information may still be useful.
Bitbank
- Location: Japan
- API: REST
- Docs: Bitbank official api docs
- Location: $57M (01.06.2018)
Japan is well known for its fast-growing cryptocurrency market. Bitbank, in turn, is another major player in such a league that aims to capitalize on the country’s existing love of both electronic payments and shopping in the physical stores. Bitbank posits itself as the first online bitcoin payment system in Japan and now includes a wallet service for the local payments. Even though there isn’t a multitude of currencies traded on Bitbank, it still has an influence on the market, particularly in Japan.
Pros & Cons:
- The platform allows for extracting all the historical data since the beginning of trades.
- One has extract 500 transactions per single request.
- Doesn’t need extra authentication or generation of API Key.
- There are only 8 cryptocurrency pairs traded on the platform.
Poloniex
- Location: US
- API: REST
- Docs: Poloniex official api docs
- Location: $40M (01.06.2018)
Poloniex is one of the most massive crypto-exchanges in the world. It is responsible for the biggest volumes of trades for the majority of altcoins that can be exchanged for Bitcoin. Created by Tristan D’Agosta way back in 2014, Poloniex still keeps the anonymity of most of its workers, partially because the digital coins are still in the sector of the informal economy (Poloniex doesn’t have a trading license in the United States). Poloniex at first was focused on the American market, but later on was able to gather the altcoin-traders from all across the globe. The service is fully functioning now and only keeps functioning further with the daily trading volumes being close to $989 million. It goes without saying, Poloniex or Polo is one of the most prominent altcoin-concentrated crypto-exchanges, for which it is valued by the altcoin traders globally.
Pros & Cons:
- The platform allows for extracting all the cryptocurrency historical data since the beginning of trades.
- As for the REST API, it works pretty stable.
- REST API allows to download transaction sorted by days, which assists with extracting data in a fast manner.
- Finally, Poloniex doesn’t require the extra authentication or the generation of API Key.
- There are 100 cryptocurrency pairs traded on the platform.
- The author has had the access issues because of the IP. However, you may be the lucky guy who won’t face such an inconvenience.
CCXT: Extraction of cryptocurrency historical data
In its nurture, CCXT is a JavaScript / Python / PHP library for cryptocurrency trading and e-commerce with support for many bitcoin/ether/altcoin exchange markets and merchant APIs. It is exploited to connect and trade with all sorts of cryptocurrency, both BTC and altcoins, and payment processing services globally. By providing a quick access to market data for storage, a comprehensive analysis, visualization, indicator development, algorithmic trading, strategy backtesting, bot programming, webshop integration and related software engineering, CCXT surely proved to be useful. A library aims to unify the responses from every single crypto-exchange, although one should still double-check the outcome as it’s not always performed in a correct manner.
As a whole, CCXT allows for avoiding the endpoints of crypto-exchanges, but one has to still know and understand the documentation to download the data correctly. Currently, CCXT supports REST API, but the creators plan to include WebSockets API in the nearby future.
Summary
It’s worth noting that the listed crypto-exchanges are stable in their nature and can tolerate a considerably large number of requests. However, such features don’t apply to all of them. The extraction of massive amounts of cryptocurrency historical data from crypto-exchanges is quite a painstaking and lengthy job that includes a lot of complexities that you may encounter. In this article, we shared our experience and did our best to ease that overwhelming process for you.
Important copyright notice © DataRoot Labs and datarootlabs.com, 2019. Unauthorized use and/or duplication of this material without express and written permission from this site’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to DataRoot Labs and datarootlabas.com with appropriate and specific direction to the original content.
Updated Jun 05, 2018 — 00:00 UTC
