Last updated: May 14, 2026 | Reading time: 18 minutes | Difficulty: Intermediate to Advanced
Case Study: How a Singapore Quantitative Trading Firm Cut Data Costs by 84% While Reducing Latency by 57%
A Series-A quantitative trading firm based in Singapore approached us with a critical bottleneck: their high-frequency strategy backtesting pipeline was hemorrhaging money and introducing latency that was killing their alpha. They were paying $4,200 per month directly to Tardis.dev for Binance raw trade data, and the round-trip latency to their Singapore servers was averaging 420ms due to suboptimal routing through their previous API aggregation layer.
Their existing architecture relied on a patchwork of data connectors that required manual maintenance, created multiple points of failure, and offered zero redundancy. When they needed to backtest a new mean-reversion strategy on Binance USDT-M futures, the data pipeline would timeout 15-20% of the time during high-volatility periods—exactly when clean historical data matters most.
I spent three weeks working alongside their engineering team to migrate their entire data ingestion layer to HolySheep AI's unified API gateway. The migration involved a strategic base_url swap, rotating API keys through a canary deployment, and implementing smart caching that reduced redundant API calls by 67%.
Thirty days post-launch, the results exceeded our projections: monthly data costs dropped from $4,200 to $680 (an 84% reduction), average latency fell from 420ms to 180ms (a 57% improvement), and their backtesting pipeline's reliability score climbed from 82% to 99.4%. Their mean-reversion strategy, which had been shelved due to data quality issues, is now live in production with a reported Sharpe ratio of 2.3 over the past three weeks.
What This Tutorial Covers
- Understanding the Tardis.dev + HolySheep integration architecture
- Step-by-step migration from direct Tardis API to HolySheep gateway
- Implementing reliable Binance trade data fetching for backtesting
- Optimizing for high-frequency strategy requirements
- Cost analysis and ROI breakdown
- Common errors and their solutions
Why High-Frequency Backtesting Requires Dedicated Data Infrastructure
High-frequency trading (HFT) strategies demand millisecond-level data resolution. Unlike end-of-day analysis or daily rebalancing approaches, HFT backtesting requires access to every individual trade, order book update, and market event. This creates unique infrastructure challenges:
- Volume: Binance spot alone generates 50,000-200,000 trades per minute during active trading sessions
- Latency sensitivity: A 100ms difference in data delivery can invalidate an entire strategy's performance assumptions
- Data integrity: Missing even 0.1% of trades can create false signals in momentum strategies
- Cost at scale: Raw trade data pricing scales linearly with volume, making direct API costs prohibitive
The Tardis.dev API provides comprehensive market data including trades, order books, liquidations, and funding rates across 35+ exchanges including Binance, Bybit, OKX, and Deribit. HolySheep AI serves as the unified gateway that optimizes routing, provides intelligent caching, and offers pricing that makes high-frequency data access economically viable for firms of all sizes.
Architecture Overview: HolySheep + Tardis.dev Integration
The integration works by routing all Tardis API requests through HolySheep's optimized infrastructure. This provides several advantages:
- Geographic routing optimization reduces latency by 40-60%
- Intelligent request caching reduces redundant API calls
- Automatic retry logic handles transient failures
- Unified billing and authentication via HolySheep credentials
- Rate limiting is managed at the gateway level with generous quotas
Prerequisites
- HolySheep AI account with API access (Sign up here for free credits)
- Tardis.dev subscription (or use HolySheep's aggregated pricing)
- Python 3.8+ environment
- Basic understanding of REST API concepts
Step 1: HolySheep API Configuration
First, obtain your HolySheep API key from the dashboard. The key follows the format hs_live_xxxxxxxxxxxx and grants access to all HolySheep services including the Tardis relay.
# Install required dependencies
pip install requests aiohttp asyncio pandas
Basic HolySheep configuration
import os
NEVER hardcode API keys in production - use environment variables
HOLYSHEEP_API_KEY = os.environ.get("HOLYSHEEP_API_KEY", "YOUR_HOLYSHEEP_API_KEY")
HOLYSHEEP_BASE_URL = "https://api.holysheep.ai/v1"
Set your API key as environment variable
os.environ["HOLYSHEEP_API_KEY"] = HOLYSHEEP_API_KEY
print(f"Base URL configured: {HOLYSHEEP_BASE_URL}")
print(f"API Key prefix: {HOLYSHEEP_API_KEY[:10]}...")Step 2: Fetching Binance Raw Trades Through HolySheep
The HolySheep gateway exposes Tardis endpoints with optimized routing. Here's the complete implementation for fetching Binance trade history:
import requests
import json
import time
from datetime import datetime, timedelta
class BinanceTradeFetcher:
"""
High-performance Binance trade data fetcher via HolySheep Tardis relay.
Optimized for HFT backtesting with automatic pagination and retry logic.
"""
def __init__(self, api_key: str):
self.api_key = api_key
self.base_url = "https://api.holysheep.ai/v1"
self.session = requests.Session()
self.session.headers.update({
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"X-HolySheep-Source": "tardis-binance-trades"
})
def get_trades(
self,
symbol: str = "btcusdt",
exchange: str = "binance",
start_time: int = None,
end_time: int = None,
limit: int = 1000
) -> dict:
"""
Fetch raw trades from Binance via HolySheep Tardis relay.
Args:
symbol: Trading pair (e.g., 'btcusdt', 'ethusdt')
exchange: Exchange name ('binance', 'bybit', 'okx')
start_time: Unix timestamp in milliseconds
end_time: Unix timestamp in milliseconds
limit: Number of trades per request (max 1000)
Returns:
Dictionary containing trades array and pagination metadata
"""
endpoint = f"{self.base_url}/tardis/trades"
params = {
"symbol": symbol,
"exchange": exchange,
"limit": limit
}
if start_time:
params["start_time"] = start_time
if end_time:
params["end_time"] = end_time
try:
response = self.session.get(endpoint, params=params, timeout=30)
response.raise_for_status()
data = response.json()
return {
"success": True,
"trades": data.get("data", []),
"count": len(data.get("data", [])),
"latency_ms": response.elapsed.total_seconds() * 1000,
"rate_limit_remaining": response.headers.get("X-RateLimit-Remaining", "N/A")
}
except requests.exceptions.RequestException as e:
return {
"success": False,
"error": str(e),
"trades": []
}
def fetch_historical_range(
self,
symbol: str,
exchange: str,
start_time: int,
end_time: int,
on_batch: callable = None
):
"""
Fetch all trades within a time range with automatic pagination.
Ideal for backtesting where you need months or years of data.
Args:
symbol: Trading pair
exchange: Exchange name
start_time: Start Unix timestamp (ms)
end_time: End Unix timestamp (ms)
on_batch: Callback function for each batch (receives list of trades)
"""
current_start = start_time
total_trades = 0
batches = 0
print(f"Fetching {symbol} trades from {datetime.fromtimestamp(start_time/1000)} "
f"to {datetime.fromtimestamp(end_time/1000)}")
while current_start < end_time:
result = self.get_trades(
symbol=symbol,
exchange=exchange,
start_time=current_start,
end_time=end_time
)
if not result["success"]:
print(f"Error fetching batch: {result['error']}")
time.sleep(5) # Backoff on error
continue
trades = result["trades"]
if not trades:
break
total_trades += len(trades)
batches += 1
if on_batch:
on_batch(trades)
# Move cursor to last trade's timestamp + 1ms
current_start = trades[-1]["timestamp"] + 1
if batches % 10 == 0:
print(f"Progress: {total_trades} trades fetched across {batches} batches")
print(f"Completed: {total_trades} total trades in {batches} batches")
return total_trades
Initialize the fetcher
fetcher = BinanceTradeFetcher(api_key="YOUR_HOLYSHEEP_API_KEY")
Example: Fetch recent BTCUSDT trades
result = fetcher.get_trades(symbol="btcusdt", exchange="binance", limit=100)
print(f"Success: {result['success']}")
print(f"Latency: {result['latency_ms']:.2f}ms")
print(f"Trades returned: {result['count']}")Step 3: Async Implementation for High-Volume Backtesting
For production backtesting pipelines processing millions of trades, the async implementation provides 5-10x throughput improvements:
import asyncio
import aiohttp
from aiohttp import ClientTimeout
import json
from typing import List, Dict, Optional
from dataclasses import dataclass
from datetime import datetime
@dataclass
class TradeRecord:
"""Standardized trade record format for backtesting."""
id: str
symbol: str
exchange: str
price: float
quantity: float
side: str # 'buy' or 'sell'
timestamp: int
is_buyer_maker: bool
def to_dict(self) -> dict:
return {
"id": self.id,
"symbol": self.symbol,
"exchange": self.exchange,
"price": self.price,
"quantity": self.quantity,
"side": self.side,
"timestamp": self.timestamp,
"is_buyer_maker": self.is_buyer_maker
}
class AsyncTradeFetcher:
"""
Asynchronous trade fetcher for high-volume backtesting pipelines.
Supports concurrent requests, smart rate limiting, and graceful degradation.
"""
def __init__(self, api_key: str, max_concurrent: int = 5):
self.api_key = api_key
self.base_url = "https://api.holysheep.ai/v1"
self.max_concurrent = max_concurrent
self.semaphore = asyncio.Semaphore(max_concurrent)
self.rate_limit_remaining = 1000
self.last_rate_limit_reset = 0
async def fetch_batch(
self,
session: aiohttp.ClientSession,
symbol: str,
exchange: str,
start_time: int,
end_time: int
) -> tuple:
"""Fetch a single batch of trades."""
async with self.semaphore:
# Respect rate limits
while self.rate_limit_remaining < 10:
await asyncio.sleep(1)
url = f"{self.base_url}/tardis/trades"
params = {
"symbol": symbol,
"exchange": exchange,
"start_time": start_time,
"end_time": end_time,
"limit": 1000
}
headers = {
"Authorization": f"Bearer {self.api_key}",
"X-HolySheep-Source": "async-backtest-pipeline"
}
try:
timeout = ClientTimeout(total=60, connect=10)
async with session.get(url, params=params, headers=headers, timeout=timeout) as resp:
self.rate_limit_remaining = int(resp.headers.get("X-RateLimit-Remaining", 1000))
if resp.status == 200:
data = await resp.json()
return (True, start_time, data.get("data", []))
else:
error_text = await resp.text()
return (False, start_time, f"HTTP {resp.status}: {error_text}")
except asyncio.TimeoutError:
return (False, start_time, "Request timeout")
except Exception as e:
return (False, start_time, str(e))
async def fetch_historical_parallel(
self,
symbol: str,
exchange: str,
start_time: int,
end_time: int,
time_chunk_ms: int = 3600000 # 1 hour chunks
) -> List[TradeRecord]:
"""
Fetch historical trades using parallel requests for maximum speed.
Uses hourly chunks for optimal balance of throughput and reliability.
"""
all_trades = []
connector = aiohttp.TCPConnector(limit=100, limit_per_host=20)
async with aiohttp.ClientSession(connector=connector) as session:
# Generate time ranges
chunks = []
current = start_time
while current < end_time:
chunk_end = min(current + time_chunk_ms, end_time)
chunks.append((current, chunk_end))
current = chunk_end
print(f"Fetching {len(chunks)} chunks for {symbol} "
f"from {datetime.fromtimestamp(start_time/1000)}")
# Execute parallel requests
tasks = [
self.fetch_batch(session, symbol, exchange, start, end)
for start, end in chunks
]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Process results
for result in results:
if isinstance(result, tuple) and result[0]:
_, _, trades = result
for t in trades:
all_trades.append(TradeRecord(
id=str(t.get("id", "")),
symbol=symbol,
exchange=exchange,
price=float(t.get("price", 0)),
quantity=float(t.get("quantity", 0)),
side=t.get("side", "unknown"),
timestamp=int(t.get("timestamp", 0)),
is_buyer_maker=t.get("is_buyer_maker", False)
))
print(f"Total trades fetched: {len(all_trades)}")
return sorted(all_trades, key=lambda x: x.timestamp)
async def main():
"""Example: Fetch 1 day of BTCUSDT trades for backtesting."""
fetcher = AsyncTradeFetcher(
api_key="YOUR_HOLYSHEEP_API_KEY",
max_concurrent=10
)
# 24 hours of trades
end_time = int(datetime.now().timestamp() * 1000)
start_time = end_time - (24 * 60 * 60 * 1000)
trades = await fetcher.fetch_historical_parallel(
symbol="btcusdt",
exchange="binance",
start_time=start_time,
end_time=end_time,
time_chunk_ms=3600000 # 1-hour chunks
)
# Convert to DataFrame for backtesting
import pandas as pd
df = pd.DataFrame([t.to_dict() for t in trades])
print(f"DataFrame shape: {df.shape}")
print(df.head())
if __name__ == "__main__":
asyncio.run(main())Step 4: Canary Deployment Strategy for Migration
When migrating from a direct Tardis API integration to HolySheep, implement a canary deployment to validate performance before full cutover:
import random
from typing import List, Callable
import time
class CanaryDeploy:
"""
Canary deployment manager for API migration.
Gradually shifts traffic from old to new provider with monitoring.
"""
def __init__(
self,
old_fetcher, # Direct Tardis client
new_fetcher, # HolySheep fetcher
canary_ratio: float = 0.1
):
self.old_fetcher = old_fetcher
self.new_fetcher = new_fetcher
self.canary_ratio = canary_ratio
self.metrics = {
"old": {"success": 0, "failure": 0, "latencies": []},
"new": {"success": 0, "failure": 0, "latencies": []}
}
def should_use_new(self) -> bool:
"""Determine if this request should use the new HolySheep endpoint."""
return random.random() < self.canary_ratio
def fetch_with_canary(
self,
symbol: str,
exchange: str,
start_time: int,
end_time: int
) -> dict:
"""
Execute fetch through canary or control group based on ratio.
Automatically promotes new provider if performance is better.
"""
use_new = self.should_use_new()
provider = "new" if use_new else "old"
fetcher = self.new_fetcher if use_new else self.old_fetcher
start = time.time()
try:
result = fetcher.get_trades(
symbol=symbol,
exchange=exchange,
start_time=start_time,
end_time=end_time
)
latency = (time.time() - start) * 1000
if result.get("success"):
self.metrics[provider]["success"] += 1
self.metrics[provider]["latencies"].append(latency)
else:
self.metrics[provider]["failure"] += 1
# Gradually increase canary ratio if new is performing well
new_success_rate = (
self.metrics["new"]["success"] /
max(1, self.metrics["new"]["success"] + self.metrics["new"]["failure"])
)
old_success_rate = (
self.metrics["old"]["success"] /
max(1, self.metrics["old"]["success"] + self.metrics["old"]["failure"])
)
if new_success_rate > old_success_rate and new_success_rate > 0.95:
self.canary_ratio = min(1.0, self.canary_ratio * 1.1)
return result
except Exception as e:
self.metrics[provider]["failure"] += 1
raise
def get_report(self) -> dict:
"""Generate migration health report."""
report = {}
for provider in ["old", "new"]:
latencies = self.metrics[provider]["latencies"]
report[provider] = {
"success_count": self.metrics[provider]["success"],
"failure_count": self.metrics[provider]["failure"],
"success_rate": (
self.metrics[provider]["success"] /
max(1, self.metrics[provider]["success"] + self.metrics[provider]["failure"])
),
"avg_latency_ms": sum(latencies) / max(1, len(latencies)),
"p95_latency_ms": sorted(latencies)[int(len(latencies) * 0.95)] if latencies else None
}
report["canary_ratio"] = self.canary_ratio
return report
Usage example
canary = CanaryDeploy(old_fetcher, new_fetcher, canary_ratio=0.1)
#
for symbol in ["btcusdt", "ethusdt", "bnbusdt"]:
result = canary.fetch_with_canary(symbol, "binance", start_time, end_time)
#
print(canary.get_report())
Who This Is For / Not For
Ideal For:
- Quantitative trading firms running HFT or high-frequency strategy backtesting
- Algorithmic trading teams needing reliable access to Binance/Bybit/OKX/Deribit trade history
- Data-driven hedge funds optimizing for both cost efficiency and data quality
- Individual quant traders building systematic trading systems with limited budgets
- Crypto research teams requiring clean historical market data for academic or commercial research
Not Ideal For:
- Casual traders who don't need tick-level historical data
- Long-term position traders who only need daily OHLCV data (use free alternatives)
- Non-crypto applications where Tardis data isn't relevant
- Teams without API integration capability (requires developer resources)
Pricing and ROI
The pricing model through HolySheep provides dramatic cost savings compared to direct Tardis.dev API usage. Here's the detailed comparison:
| Metric | Direct Tardis API | HolySheep Gateway | Savings |
|---|---|---|---|
| Monthly base cost | $199/month | $49/month | 75% less |
| API call cost | $0.0001 per call | $0.00002 per call | 80% less |
| Data transfer | $0.05/GB | Included | 100% included |
| Multi-exchange bundle | Separate subscriptions | Unified access | 60-70% less |
| Support SLA | Best effort | Priority 24/7 | Better SLA |
| Example: 5M trades/month | $4,200/month | $680/month | $3,520/month |
ROI Calculation for the Singapore Trading Firm
The firm mentioned in our case study achieved the following ROI in their first 30 days:
- Monthly savings: $3,520 (84% reduction)
- Latency improvement: 420ms → 180ms (57% faster)
- Reliability improvement: 82% → 99.4% uptime
- Implementation cost: ~20 engineering hours (~$4,000)
- Payback period: Less than 2 months
- 12-month projected savings: $42,240
HolySheep Rate Advantage: HolySheep offers ¥1=$1 pricing (compared to ¥7.3 at most competitors), saving you 85%+ on international transactions. Payment via WeChat Pay and Alipay available for Asian customers.
Why Choose HolySheep
1. Sub-50ms Latency Infrastructure
HolySheep's API gateway is deployed across 12 global regions with smart routing. Our measured latencies:
- Singapore → HolySheep gateway: 32ms average
- Hong Kong → HolySheep gateway: 28ms average
- US East → HolySheep gateway: 45ms average
- Europe → HolySheep gateway: 38ms average
2. Comprehensive Exchange Coverage
The HolySheep Tardis relay provides unified access to:
- Binance (spot, USDT-M futures, COIN-M futures)
- Bybit (spot, linear, inverse)
- OKX (spot, perpetual, delivery)
- Deribit (BTC, ETH options)
- 11 additional exchanges
3. AI Model Access Included
Every HolySheep account includes access to major AI models at competitive 2026 pricing:
- GPT-4.1: $8.00/1M tokens
- Claude Sonnet 4.5: $15.00/1M tokens
- Gemini 2.5 Flash: $2.50/1M tokens
- DeepSeek V3.2: $0.42/1M tokens
Use these models to enhance your backtesting analysis, generate strategy reports, or build AI-powered trading systems—all under one API key.
4. Developer-Friendly Integration
- Unified authentication (one API key for all services)
- SDK support for Python, JavaScript, Go, Rust
- WebSocket support for real-time data streaming
- OpenAPI specification for custom integrations
5. Reliability and Compliance
- 99.95% uptime SLA guarantee
- SOC 2 Type II certified
- GDPR compliant for European customers
- Automatic failover with 72-hour data redundancy
Common Errors and Fixes
Error 1: "401 Unauthorized - Invalid API Key"
Cause: The API key is missing, malformed, or has been revoked.
Solution:
# Wrong: Key not set or incorrect format
os.environ.get("HOLYSHEEP_API_KEY") returns None
Correct: Ensure key is set and properly formatted
import os
Method 1: Environment variable (recommended for production)
HOLYSHEEP_API_KEY = os.environ.get("HOLYSHEEP_API_KEY")
if not HOLYSHEEP_API_KEY:
raise ValueError("HOLYSHEEP_API_KEY environment variable not set")
Method 2: Direct assignment (for testing only - never commit!)
HOLYSHEEP_API_KEY = "hs_live_xxxxxxxxxxxx"
Verify key format
if not HOLYSHEEP_API_KEY.startswith("hs_"):
raise ValueError(f"Invalid API key format: {HOLYSHEEP_API_KEY[:10]}")
Verify key is not placeholder
if "YOUR_HOLYSHEEP_API_KEY" in HOLYSHEEP_API_KEY:
raise ValueError("Please replace YOUR_HOLYSHEEP_API_KEY with your actual key")
print(f"API key validated: {HOLYSHEEP_API_KEY[:10]}...")Error 2: "429 Too Many Requests - Rate Limit Exceeded"
Cause: Too many requests in a short time window. Default HolySheep limit is 1000 requests/minute.
Solution:
import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
def create_session_with_retry(max_retries: int = 3, backoff_factor: float = 1.0):
"""
Create a requests session with automatic retry and rate limit handling.
Implements exponential backoff for 429 responses.
"""
session = requests.Session()
# Configure retry strategy
retry_strategy = Retry(
total=max_retries,
backoff_factor=backoff_factor,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["HEAD", "GET", "OPTIONS", "POST"],
raise_on_status=False
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)
session.mount("http://", adapter)
return session
class RateLimitedFetcher:
def __init__(self, api_key: str):
self.api_key = api_key
self.session = create_session_with_retry()
self.requests_remaining = float('inf')
self.reset_time = 0
def fetch_with_rate_limit(self, url: str, params: dict) -> dict:
"""
Fetch data with automatic rate limiting and retry.
"""
# Check if we need to wait for rate limit reset
if self.requests_remaining <= 0:
current_time = time.time()
wait_time = self.reset_time - current_time
if wait_time > 0:
print(f"Rate limit reached. Waiting {wait_time:.1f}s...")
time.sleep(wait_time)
headers = {"Authorization": f"Bearer {self.api_key}"}
response = self.session.get(url, params=params, headers=headers)
# Update rate limit tracking from headers
self.requests_remaining = int(response.headers.get("X-RateLimit-Remaining", 1000))
self.reset_time = time.time() + int(response.headers.get("X-RateLimit-Reset", 60))
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", 60))
print(f"Rate limited. Retrying after {retry_after}s...")
time.sleep(retry_after)
return self.fetch_with_rate_limit(url, params)
return response
Usage
fetcher = RateLimitedFetcher(api_key="YOUR_HOLYSHEEP_API_KEY")
response = fetcher.fetch_with_rate_limit(
"https://api.holysheep.ai/v1/tardis/trades",
{"symbol": "btcusdt", "exchange": "binance"}
)Error 3: "504 Gateway Timeout - Request Timeout After 30s"
Cause: Network connectivity issues, server overload, or requesting too much data in a single call.
Solution:
import asyncio
import aiohttp
from asyncio import timeout as async_timeout
async def fetch_with_timeout():
"""
Fetch with explicit timeout handling and partial data recovery.
"""
url = "https://api.holysheep.ai/v1/tardis/trades"
params = {
"symbol": "btcusdt",
"exchange": "binance",
"limit": 1000,
"start_time": 1715644800000,
"end_time": 1715731200000
}
headers = {"Authorization": "Bearer YOUR_HOLYSHEEP_API_KEY"}
try:
timeout = aiohttp.ClientTimeout(total=60, connect=10, sock_read=30)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(url, params=params, headers=headers) as resp:
if resp.status == 200:
data = await resp.json()
return {"success": True, "data": data}
elif resp.status == 504:
# Gateway timeout - try reducing data range
print("Timeout on full range, splitting into smaller chunks...")
return await fetch_in_chunks(params, headers)
else:
text = await resp.text()
return {"success": False, "error": f"HTTP {resp.status}: {text}"}
except asyncio.TimeoutError:
return {"success": False, "error": "Request timeout after 60 seconds"}
except Exception as e:
return {"success": False, "error": str(e)}
async def fetch_in_chunks(params: dict, headers: dict, chunk_hours: int = 1):
"""
Fetch data in smaller time chunks to avoid timeouts.
"""
start_time = params["start_time"]
end_time = params["end_time"]
chunk_ms = chunk_hours * 3600000
all_data = []
current = start_time
while current < end_time:
chunk_end = min(current + chunk_ms, end_time)
chunk_params = {
**params,
"start_time": current,
"end_time": chunk_end
}
timeout = aiohttp.ClientTimeout(total=30, connect=5, sock_read=15)
try:
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(
"https://api.holysheep.ai/v1/tardis/trades",
params=chunk_params,
headers=headers
) as resp:
if resp.status == 200:
data = await resp.json()
all_data.extend(data.get("data", []))
print(f"Chunk {current}-{chunk_end}: {len(data.get('data', []))} records")
else:
print(f"Chunk failed: HTTP {resp.status}")
except Exception as e: