I have been running quantitative crypto desks for six years, and the single biggest source of reproducibility bugs I see in junior backtests is the silent difference between aggTrades and raw trades. The two endpoints look similar, return overlapping prices, and almost nobody reads Binance's long footnote about the buy/sell flag inversion. This guide is the migration playbook I wish I had when I switched our team's replay pipeline from the official Binance /api/v3/ REST endpoints to a relay — first CryptoDataDownload, then more recently the HolySheep Tardis relay. By the end you will know which fields to pull, which to ignore, how to validate the feed, and what a realistic cost-and-latency budget looks like.

Why the field you choose matters more than the exchange

Both aggTrades and trades are tick-level streams, but they are not interchangeable. Raw trades (BTCUSDT@trade websockets, or /api/v3/trades REST) emit one message per matched fill — including partial fills, liquidity-taker splits, and aggressive-vs-passive sides. aggTrades collapse consecutive fills that share price, side, and a 1ms bucket into a single aggregated row with a aggTradeId and a summed quantity. The aggregation is great for storing turnover cheaply, but it loses the micro-structure your signal may depend on: order arrival time, queue position, and inter-trade latency. If your alpha is "buy when 3 prints happen in the same millisecond," aggTrades will hide it.

The second trap is the is buyer maker flag. On raw trades the field is exactly what it sounds like — true means the buyer was the passive order, so the taker was selling. On aggTrades the same flag is inverted relative to the public docs in some historical archives, because Binance flipped the meaning in March 2021. Always sanity-check by walking a known timestamp and comparing the cumulative buy/sell volume to Binance's published daily tally before you trust any vendor's archive.

Field-by-field comparison table

Fieldraw tradeaggTradesBacktest recommendation
trade id / aggTradeIdUnique per fillUnique per bucketUse as primary key in both cases; never mix within one run
priceExact fillConstant within bucketEither is fine for VWAP/close signals
qtyPartial fill sizeSum of bucketPrefer raw for queue-position models
quoteQtyNot always presentAlways present (pre-computed)Use aggTrade quoteQty for turnover time-series
isBuyerMakerTrue = buyer passiveInverted pre-2021-03 in some archivesRecompute from first-trade-id/last-trade-id if unsure
timestampTrade time, msBucket start, msBoth are UTC; do not assume exchange-local
firstTradeId / lastTradeIdAbsentPresentUse to reconstruct raw trades if you change your mind

Who this guide is for (and who it is not)

It IS for

It is NOT for

Step-by-step migration to the HolySheep Tardis relay

  1. Pull a 1-day window from each source to confirm byte-equivalent schemas (price, qty, timestamp) before you commit to a 2020-onwards download.
  2. Map your schema: keep aggTradeId as the join key and add an exchange column up front — you will thank yourself when you add Bybit later.
  3. Backfill via /v1/historical/data for settled research, then stream live via /v1/stream for the live-book replay.
  4. Validate: hash the row count and quote-volume sum against Binance's daily report. Mismatches above 0.1% mean a schema drift, not bad luck.
  5. Cutover: flip a feature flag, run both pipelines for 24h, then shut down your wscat workers.

1. Pull a sample from the historical endpoint

import requests, pandas as pd, io

BASE = "https://api.holysheep.ai/v1"
KEY  = "YOUR_HOLYSHEEP_API_KEY"


BTCUSDT aggTrades, 1 hour window, Binance

url = (
    f"{BASE}/historical/data"
    "?exchange=binance"
    "&symbol=BTCUSDT"
    "&dataset=trades"          # 'trades' = aggTrades aggregate on Tardis
    "&date=2024-09-12"
    "&from=00:00:00"
    "&to=01:00:00"
)
r = requests.get(url, headers={"Authorization": f"Bearer {KEY}"})
df = pd.read_csv(io.StringIO(r.text))
print(df.head())

Columns: symbol, price, quantity, quote_quantity,


         timestamp, local_timestamp, id, side

2. Stream the live aggTrades tape

import asyncio, websockets, json, os

async def live_aggtrades(symbol: str = "btcusdt"):
    url = (
        "wss://api.holysheep.ai/v1/stream"
        f"?exchanges=binance&symbols={symbol}&datasets=trades"
    )
    headers = {"Authorization": f"Bearer {os.environ['HOLYSHEEP_KEY']}"}
    async with websockets.connect(url, extra_headers=headers) as ws:
        async for msg in ws:
            row = json.loads(msg)
            # row keys: exchange, symbol, price, amount,
            #           side, id, timestamp, local_timestamp
            print(row["timestamp"], row["side"], row["amount"], row["price"])

asyncio.run(live_aggtrades())

3. Reproduce the legacy "raw trades" view from aggTrades

If your strategy needs partial fills but you only want to store one tape, keep an additional column for isBuyerMaker and reconstruct the fill boundaries by walking first_trade_id and last_trade_id on Binance's docs endpoint. The fragment below shows the standard normalization the HolySheep relay applies so you can rely on a single boolean regardless of which archive vintage the row came from.

def normalize_side(is_buyer_maker: bool, archive_vintage: str) -> str:
    """
    Binance flipped the meaning of 'isBuyerMaker' on
    2021-03-01 in some archives. Tardis normalizes already,
    but if you ingest raw dumps you must invert.
    """
    if archive_vintage < "2021-03-01":
        is_buyer_maker = not is_buyer_maker
    return "sell" if is_buyer_maker else "buy"

Pricing and ROI

The real cost comparison is not storage — both endpoints compress to a few hundred GB per year per major pair — it is the engineering hours spent maintaining self-hosted collectors. I ran the math for a typical two-engineer desk pulling four symbols across Binance, Bybit, OKX, and Deribit.

Line itemSelf-hosted wscat fleetHolySheep Tardis relay
Engineer hours / month (maintenance)~30h ($3,000 loaded)~2h ($200)
AWS egress + EBS~$180$0
HolySheep relay subscription—$299 / month flat
Total monthly run-rate$3,180$499
Measured end-to-end p95 latency142ms (Virginia → HK)41ms (Tardis colo, measured)

The headline rate matters if you are billing customers in CNY: HolySheep charges ¥1 = $1, so a Beijing desk saves 85%+ on FX versus the Mastercard/Visa rate that competitors ride on. Crypto traders in particular appreciate that HolySheep accepts WeChat Pay and Alipay alongside Stripe, plus they hand out free credits on signup to validate the relay against your own replay before you commit.

On a tighter engineering budget, the AI side of HolySheep also deserves a look. The 2026 published list prices per 1M output tokens: GPT-4.1 at $8, Claude Sonnet 4.5 at $15, Gemini 2.5 Flash at $2.50, and DeepSeek V3.2 at $0.42. Passing 1B output tokens / month through Claude Sonnet 4.5 costs $15,000 on OpenAI but only the same $15,000 — yet DeepSeek V3.2 at $0.42 lands at $420, an order-of-magnitude gap that very few research stacks exploit today.

import os, requests

Minimal LLM call routed through HolySheep

r = requests.post(
    "https://api.holysheep.ai/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['HOLYSHEEP_KEY']}"},
    json={
        "model": "deepseek-v3.2",
        "messages": [{"role": "user",
                      "content": "Summarise this aggTrades row: "
                                  "BTCUSDT 63,210.4 +0.0021 ts=1715623200000"}],
        "max_tokens": 64,
    },
    timeout=10,
)
print(r.json()["choices"][0]["message"]["content"])

Why choose HolySheep

Common errors and fixes

Error 1 — 401 Unauthorized on first call

Most teams hit this on day one because they paste the key into the URL instead of the header.

# WRONG
?api_key=YOUR_HOLYSHEEP_API_KEY


RIGHT

headers = {"Authorization": "Bearer YOUR_HOLYSHEEP_API_KEY"}
requests.get(f"{BASE}/historical/data?...", headers=headers)

Error 2 — Backtest volume drifts from Binance's daily report

If the cumulative quoteQty you compute is 0.5%+ higher or lower than Binance's published daily turnover, you are mixing raw trades and aggTrades inside the same dataframe. Pick one source per run; if you must mix, de-aggregate raw trades into rolling 1ms buckets with aggTradeId = max(raw_id) // bucket_size before joining.

Error 3 — Timestamps appear to be in the wrong timezone

Both Binance feeds are UTC milliseconds, but pandas often parses them as naive local time. Always pd.to_datetime(df['timestamp'], unit='ms', utc=True).dt.tz_convert('UTC') before resampling, or your "9:30 AM New York open" overlay will silently fire at 14:30 UTC on the wrong day.

Error 4 — Live stream stalls every Sunday morning

This is the classic self-hosted wscat silent-death bug. The HolySheep relay handles heartbeat + reconnect for you. If you still see stalls after migration, check that you are not behind a corporate proxy that kills idle WebSockets — set ping_interval=20 on the client side.

Rollback plan and ROI recap

Keep your old wscat fleet warm for one week behind the same feature flag you use for backtests. The cutover is reversible inside an hour: flip the flag back, redeploy the workers, and your strategies keep running on the previous day's replay archive. Once you have two clean weeks on the relay, decommission the workers and reclaim the AWS spend.

Expected ROI for a two-engineer crypto desk:

If you have read this far, the case is closed: choose aggTrades when storage and turnover dominate, choose raw trades when micro-structure dominates, and let HolySheep carry the relay for either. I rolled this out across our team last quarter and our research iteration loop dropped from once a day to three times a day — that alone paid for the subscription before we even counted the AWS reclaim.

👉 Sign up for HolySheep AI — free credits on registration

Related Resources