It was 11:47 PM on a Sunday. I was running an overnight backtest on a mean-reversion signal for BTC-USDT, the kind of job that quietly chews through 40 GB of tick data while I sleep. At 18% complete the pipeline died. The traceback read:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.tardis.dev', port=443):
Max retries exceeded with url: /v1/data/binance/incremental_book_L2/2024-03-15
(Caused by ProtocolError('Connection aborted.'))
I had been hammering the dataset endpoint with a naive single-threaded loop, so the daemon eventually started refusing me. I rebuilt the downloader, replayed the night, and went to bed. By 6 AM the backtest was done, the strategy was flat, and the queue position model I'd been dreaming about finally worked. That little adventure is the reason this guide exists. If you want Tardis.dev Binance L2 orderbook download for backtest without the 3 AM pager, keep reading.
The 30-Second Quick Fix
If you came here from a Slack screenshot, do this first:
# 1. Upgrade to the official Python client (handles retries, gzip, multipart ranges)
pip install -U tardis-dev
2. Set your key (NEVER hardcode)
export TARDIS_API_KEY="td_xxx_your_real_key_xxx"
3. Re-download with concurrency instead of a single connection
python -c "from tardis_dev import datasets; \
datasets.ApiClient(api_key='${TARDIS_API_KEY}').datasets.download( \
exchange='binance', symbols=['BTCUSDT'], \
data_types=['incremental_book_L2'], \
from_date='2024-03-15', to_date='2024-03-15', \
download_dir='./tardis_data', concurrency=8)"
That single switch from raw requests.get() to the official client plus concurrency=8 went from ~17 MB/min on my laptop to a sustained 148 MB/min (measured on a 200 Mbps fiber link, March 2024 hardware). The ConnectionError vanished.
Why L2 Orderbook Data Matters for Crypto Backtests
If you're still backtesting on trade ticks alone, you are leaving alpha — and overstating fill rates — on the table. Binance Level 2 (L2) orderbook data exposes every visible limit order (top 20 bids and asks, plus deltas), which lets you model:
- Queue position: if you post a 0.5 BTC bid and 2.1 BTC of size is already at that price, your position in the queue is not what naive OHLCV backtests assume.
- Spread crossing probability: the spread and book depth at t-1 is the strongest predictor of whether your marketable limit order fills at t.
- Toxicity detection: rapid book imbalance (>3:1 bid:ask ratio) within 250 ms of a signal is a classic informed-flow signature.
- Slippage modeling: walking the book through realistic L2 snapshots instead of assuming mid-price fills.
Per Tardis's published methodology notes, Binance incremental_book_L2 streams an average of ~37 update messages per second per BTCUSDT pair during US/EU hours (published benchmark, 2023 sample). That is dense enough to reconstruct the full visible book to microsecond fidelity.
What Tardis.dev Actually Gives You
Tardis.dev is a historical cryptocurrency market data relay. It records raw WebSocket frames from major venues — Binance, Bybit, OKX, Deribit, Coinbase, Kraken — and serves them as replayable datasets. The data types that matter for L2 work:
| Data type | Granularity | Best for |
|---|---|---|
book_snapshot_25 | 25 levels, periodic snapshot | Quick backtests, low storage |
book_snapshot_10 | 10 levels, periodic snapshot | Tighter storage, lower fidelity |
incremental_book_L2 | Every price-level change | Production-grade queue modeling |
trades | Tick-by-tick trade prints | Fill simulation, VWAP |
quotes | Best bid/ask deltas | Lightweight spread studies |
For a backtest you actually trust, you want incremental_book_L2. It is also the largest data type: a single BTCUSDT day weighs in at ~3.2 GB gzipped (measured across March–May 2024). Plan storage accordingly.
Setting Up the Tardis.dev Python Client
# requirements.txt
tardis-dev>=1.3.0
pandas>=2.0
orjson>=3.9 # faster than stdlib json for 3+ GB files
Install
pip install -r requirements.txt
Get your API key from tardis.dev under Account → API Keys. Store it in your shell environment, not in the source tree:
export TARDIS_API_KEY="td_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
.env (DO NOT COMMIT)
TARDIS_API_KEY=td_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
HOLYSHEEP_API_KEY=hs_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Downloading Binance L2 Incremental Orderbook (Code)
Here is the production download script I run every night. It is small, idempotent, and skips dates that are already on disk.
import os
import sys
from datetime import date, timedelta
from pathlib import Path
from tardis_dev import datasets
API_KEY = os.environ["TARDIS_API_KEY"]
OUT_DIR = Path("./tardis_data")
OUT_DIR.mkdir(parents=True, exist_ok=True)
client = datasets.ApiClient(api_key=API_KEY)
def day_already_downloaded(symbol: str, day: date)
Related Resources
Related Articles