Last updated: May 3, 2026 | Reading time: 12 minutes | Difficulty: Beginner to Intermediate
Introduction
I remember my first time trying to fetch historical order book data from OKX — I spent three days wrestling with WebSocket connections, rate limits, and confusing documentation before I finally got a clean stream of incremental_book_L2 data. If you are in that same position right now, this guide will save you those three days. By the end, you will have a fully working Python script that pulls live and historical Level 2 order book data from OKX through the Tardis.dev API, which HolySheep AI integrates seamlessly with for advanced analysis.
Whether you are building a trading bot, backtesting a strategy, or analyzing market microstructure, getting reliable tick data is the foundation of everything. This tutorial walks you through every step, from setting up your environment to handling common errors that trip up even experienced developers.
What is incremental_book_L2 and Why Does It Matter?
The incremental_book_L2 message type is OKX's way of streaming changes to the Level 2 order book in real-time. Unlike a full snapshot (which sends the entire book), incremental updates only contain changes — new orders, cancelled orders, and trades. This makes them incredibly efficient for real-time processing.
Key benefits:
- Bandwidth efficiency: You receive only changes, not the entire book every update
- Low latency: Updates arrive as they happen on the exchange
- Complete history: You can reconstruct the full order book at any point in time
- Trade attribution: Each update can be linked to specific trades
[Screenshot hint: Imagine a side-by-side view — left side shows raw incremental_book_L2 JSON messages scrolling in real-time, right side shows the reconstructed order book visualization updating in real-time. The JSON on the left contains "action":"update", "data":[...], "instrument_id":"BTC-USDT-SWAP"]
Who This Guide Is For
This guide IS for you if:
- You are building algorithmic trading systems
- You need historical market data for backtesting
- You want to understand order book dynamics
- You are migrating from another exchange API
- You need high-frequency data for research or academic purposes
This guide is NOT for you if:
- You only need simple price ticker data (use a lightweight REST endpoint instead)
- You are building a mobile app with basic price display (WebSockets add unnecessary complexity)
- You do not have basic Python knowledge (consider starting with our Python basics guide)
Setting Up Your Environment
Before we write any code, we need to set up a clean Python environment. I recommend using a virtual environment to avoid dependency conflicts.
# Create a new virtual environment (recommended)
python -m venv tardis-env
Activate it on macOS/Linux
source tardis-env/bin/activate
Or on Windows
tardis-env\Scripts\activate
Install required packages
pip install requests websockets asyncio aiohttp pandas numpy
Verify installation
python -c "import requests; import websockets; print('All packages installed successfully')"
[Screenshot hint: Terminal window showing the pip install commands executing, ending with the success message confirming all packages are ready. Green checkmarks next to installed packages.]
Getting Your Tardis.dev API Key
Tardis.dev provides historical market data for over 40 exchanges including OKX. To access their API, you need an API key. Here's the step-by-step:
- Go to tardis.dev and create a free account
- Navigate to your dashboard and generate a new API key
- Copy the key and store it securely (you will need it in the next step)
- Note your plan limits — free tier includes 30 days of historical data
[Screenshot hint: tardis.dev dashboard showing the API Keys section, with a visible "Create New Key" button and a sample key format like "ts_live_abc123xyz..."]
Understanding the Tardis API Endpoint Structure
The Tardis.dev API follows a consistent pattern. For OKX historical data, the base endpoint structure is:
https://api.tardis.dev/v1/
exchange: okx
symbol: BTC-USDT-SWAP
channel: incremental_book_L2
date: 2026-05-01
format: json
For live streaming, you use WebSocket connections:
wss://api.tardis.dev/v1/
exchange: okx
channel: incremental_book_L2
symbols: BTC-USDT-SWAP,ETH-USDT-SWAP
Code Example 1: Fetching Historical incremental_book_L2 Data
Let me show you a complete, production-ready script that fetches historical OKX order book data. This is the exact code I use for my own backtesting pipeline.
#!/usr/bin/env python3
"""
OKX Historical incremental_book_L2 Data Fetcher
Fetches historical order book data from Tardis.dev API
"""
import requests
import json
import time
from datetime import datetime, timedelta
import pandas as pd
============================================
CONFIGURATION - Replace with your values
============================================
TARDIS_API_KEY = "YOUR_TARDIS_API_KEY" # Get from https://tardis.dev
EXCHANGE = "okx"
SYMBOL = "BTC-USDT-SWAP"
CHANNEL = "incremental_book_L2"
Date range for historical data
START_DATE = "2026-05-01"
END_DATE = "2026-05-02"
============================================
HolySheep AI Integration for Analysis
============================================
After fetching data, you can use HolySheep AI for advanced analysis
Sign up at: https://www.holysheep.ai/register
Rate: $1 per dollar (saves 85%+ vs alternatives at ¥7.3)
Supports WeChat/Alipay payments with <50ms latency
HOLYSHEEP_API_KEY = "YOUR_HOLYSHEEP_API_KEY"
HOLYSHEEP_BASE_URL = "https://api.holysheep.ai/v1"
============================================
Fetch Historical Data Function
============================================
def fetch_historical_data(exchange, symbol, channel, date):
"""
Fetch historical market data for a specific date.
Args:
exchange: Exchange name (e.g., 'okx')
symbol: Trading pair symbol (e.g., 'BTC-USDT-SWAP')
channel: Data channel type (e.g., 'incremental_book_L2')
date: Date in YYYY-MM-DD format
Returns:
List of parsed messages
"""
url = f"https://api.tardis.dev/v1/feeds/{exchange}:{symbol}/{channel}/{date}"
headers = {
"Authorization": f"Bearer {TARDIS_API_KEY}",
"Accept": "application/json"
}
print(f"Fetching data from: {url}")
print(f"Date: {date}, Symbol: {symbol}")
try:
response = requests.get(url, headers=headers, timeout=60)
response.raise_for_status()
# Parse JSON response
data = response.json()
# Extract messages (format varies by exchange)
if "data" in data:
messages = data["data"]
else:
messages = data
print(f"Successfully fetched {len(messages)} messages")
return messages
except requests.exceptions.HTTPError as e:
if response.status_code == 401:
print("ERROR: Invalid or expired API key")
print("Please check your Tardis.dev API key at https://tardis.dev/dashboard")
elif response.status_code == 429:
print("ERROR: Rate limit exceeded")
print("Wait 60 seconds before retrying")
else:
print(f"HTTP Error: {e}")
return []
except requests.exceptions.RequestException as e:
print(f"Request failed: {e}")
return []
============================================
Process Order Book Updates
============================================
def process_order_book_updates(messages):
"""
Parse and structure order book updates.
OKX incremental_book_L2 message format:
{
"action": "snapshot" or "update",
"data": [
{
"instrument_id": "BTC-USDT-SWAP",
"timestamp": 1746272400000,
"asks": [["50000.0", "10.5"], ...],
"bids": [["49900.0", "8.2"], ...],
"action": "update"
}
]
}
"""
processed_data = []
for msg in messages:
# Handle different message formats
if isinstance(msg, dict):
# Extract timestamp and data
timestamp = msg.get("timestamp", 0)
asks = msg.get("asks", [])
bids = msg.get("bids", [])
action = msg.get("action", "unknown")
processed_data.append({
"timestamp_ms": timestamp,
"datetime": datetime.fromtimestamp(timestamp / 1000),
"action": action,
"best_ask": float(asks[0][0]) if asks else None,
"best_bid": float(bids[0][0]) if bids else None,
"ask_depth": len(asks),
"bid_depth": len(bids),
"spread": float(asks[0][0]) - float(bids[0][0]) if asks and bids else None
})
return pd.DataFrame(processed_data)
============================================
Main Execution
============================================
if __name__ == "__main__":
print("=" * 60)
print("OKX Historical incremental_book_L2 Data Fetcher")
print("=" * 60)
# Fetch data for date range
all_messages = []
current_date = datetime.strptime(START_DATE, "%Y-%m-%d")
end_date = datetime.strptime(END_DATE, "%Y-%m-%d")
while current_date <= end_date:
date_str = current_date.strftime("%Y-%m-%d")
messages = fetch_historical_data(EXCHANGE, SYMBOL, CHANNEL, date_str)
all_messages.extend(messages)
# Respect rate limits - add delay between requests
time.sleep(1)
current_date += timedelta(days=1)
print(f"\nTotal messages collected: {len(all_messages)}")
# Process and display sample
if all_messages:
df = process_order_book_updates(all_messages)
print("\nSample processed data:")
print(df.head(10).to_string())
# Save to CSV for later analysis
output_file = f"okx_{SYMBOL.replace('-', '_')}_{START_DATE}_{END_DATE}.csv"
df.to_csv(output_file, index=False)
print(f"\nData saved to: {output_file}")
# Calculate statistics
print("\nOrder Book Statistics:")
print(f" Average spread: {df['spread'].mean():.2f}")
print(f" Best ask range: {df['best_ask'].min():.2f} - {df['best_ask'].max():.2f}")
print(f" Best bid range: {df['best_bid'].min():.2f} - {df['best_bid'].max():.2f}")
[Screenshot hint: Terminal output showing successful data fetch with message count, followed by a preview table of the processed data showing timestamp, best ask, best bid, and spread columns with actual numbers like 50,234.50 and 50,100.25]
Code Example 2: Real-Time WebSocket Stream
For live trading or real-time analysis, you need WebSocket connections. Here's a complete async implementation that handles reconnection, heartbeat, and error recovery — all the things that will crash your script if you skip them.
#!/usr/bin/env python3
"""
OKX Real-Time incremental_book_L2 WebSocket Client
Handles live market data streaming with automatic reconnection
"""
import asyncio
import websockets
import json
import logging
from datetime import datetime
import time
Configure logging for production use
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
============================================
CONFIGURATION
============================================
TARDIS_WS_URL = "wss://api.tardis.dev/v1/feeds/okx:BTC-USDT-SWAP/incremental_book_L2"
TARDIS_API_KEY = "YOUR_TARDIS_API_KEY"
============================================
WebSocket Client with Auto-Reconnection
============================================
class OKXRealtimeClient:
"""
Production-ready WebSocket client for OKX incremental_book_L2 data.
Features:
- Automatic reconnection on connection loss
- Heartbeat/ping-pong handling
- Message buffering for high-frequency data
- Graceful shutdown handling
"""
def __init__(self, url, api_key, max_reconnect_attempts=5, reconnect_delay=5):
self.url = url
self.api_key = api_key
self.max_reconnect_attempts = max_reconnect_attempts
self.reconnect_delay = reconnect_delay
self.ws = None
self.running = True
self.message_count = 0
self.last_heartbeat = time.time()
# Order book state (reconstructed from incremental updates)
self.current_book = {
"asks": {}, # {price: quantity}
"bids": {}
}
async def connect(self):
"""Establish WebSocket connection with authentication."""
headers = {
"Authorization": f"Bearer {self.api_key}"
}
try:
self.ws = await websockets.connect(
self.url,
extra_headers=headers,
ping_interval=20, # Send ping every 20 seconds
ping_timeout=10 # Wait 10 seconds for pong
)
logger.info("WebSocket connected successfully")
return True
except websockets.exceptions.InvalidStatusCode as e:
logger.error(f"Connection failed with status {e.status_code}")
if e.status_code == 401:
logger.error("Invalid API key - check your Tardis.dev credentials")
elif e.status_code == 429:
logger.error("Rate limited - upgrade your plan or wait")
return False
except Exception as e:
logger.error(f"Connection failed: {e}")
return False
def process_incremental_update(self, message):
"""
Process incremental_book_L2 update and update local order book state.
OKX message format for incremental updates:
{
"data": [{
"instrument_id": "BTC-USDT-SWAP",
"timestamp": 1746272400000,
"asks": [["50000.0", "10.5"], ["50001.0", "5.2"]],
"bids": [["49900.0", "8.2"], ["49899.0", "3.1"]],
"action": "update" # or "snapshot"
}]
}
"""
try:
data = json.loads(message) if isinstance(message, str) else message
# Handle both direct data and wrapped formats
records = data.get("data", [data]) if "data" in data else [data]
for record in records:
action = record.get("action", "update")
# Process ask updates
for ask in record.get("asks", []):
price, quantity = float(ask[0]), float(ask[1])
if quantity == 0:
# Remove order from book
self.current_book["asks"].pop(price, None)
else:
# Add or update order
self.current_book["asks"][price] = quantity
# Process bid updates
for bid in record.get("bids", []):
price, quantity = float(bid[0]), float(bid[1])
if quantity == 0:
self.current_book["bids"].pop(price, None)
else:
self.current_book["bids"][price] = quantity
# Get best prices
best_ask = min(self.current_book["asks"].keys()) if self.current_book["asks"] else None
best_bid = max(self.current_book["bids"].keys()) if self.current_book["bids"] else None
if best_ask and best_bid:
spread = best_ask - best_bid
timestamp = record.get("timestamp", 0)
# Log market data (or send to your trading system)
logger.info(
f"[{timestamp}] Best Ask: {best_ask:.2f} | "
f"Best Bid: {best_bid:.2f} | "
f"Spread: {spread:.2f}"
)
# HERE: Add your trading logic or data processing
# Example: check for arbitrage opportunities
# Example: update your UI with new data
except json.JSONDecodeError as e:
logger.warning(f"Failed to parse message: {e}")
except Exception as e:
logger.error(f"Error processing update: {e}")
async def receive_messages(self):
"""Main message receiving loop with reconnection logic."""
reconnect_attempts = 0
while self.running and reconnect_attempts < self.max_reconnect_attempts:
try:
if not self.ws or self.ws.closed:
connected = await self.connect()
if not connected:
reconnect_attempts += 1
await asyncio.sleep(self.reconnect_delay)
continue
# Receive messages
async for message in self.ws:
if not self.running:
break
self.message_count += 1
self.last_heartbeat = time.time()
self.process_incremental_update(message)
# Reset reconnect counter on successful message
reconnect_attempts = 0
except websockets.exceptions.ConnectionClosed as e:
logger.warning(f"Connection closed: {e}")
reconnect_attempts += 1
await asyncio.sleep(self.reconnect_delay)
except Exception as e:
logger.error(f"Unexpected error: {e}")
reconnect_attempts += 1
await asyncio.sleep(self.reconnect_delay)
if reconnect_attempts >= self.max_reconnect_attempts:
logger.error("Max reconnection attempts reached")
async def stop(self):
"""Gracefully shutdown the client."""
logger.info("Shutting down client...")
self.running = False
if self.ws and not self.ws.closed:
await self.ws.close()
logger.info(f"Total messages processed: {self.message_count}")
============================================
Main Entry Point
============================================
async def main():
"""Run the real-time data client."""
client = OKXRealtimeClient(
url=TARDIS_WS_URL,
api_key=TARDIS_API_KEY,
max_reconnect_attempts=5,
reconnect_delay=5
)
try:
# Start receiving messages
await client.receive_messages()
except KeyboardInterrupt:
print("\nReceived interrupt signal")
finally:
await client.stop()
if __name__ == "__main__":
print("=" * 60)
print("OKX Real-Time incremental_book_L2 Streaming Client")
print("Press Ctrl+C to stop")
print("=" * 60)
asyncio.run(main())
[Screenshot hint: Two side-by-side panels — left shows the WebSocket client running in terminal with streaming log output (timestamps, best ask, best bid, spread), right shows a reconstructed order book visualization with depth chart updating in real-time]
Understanding OKX incremental_book_L2 Message Format
To effectively process the data, you need to understand the message structure. OKX sends two types of messages:
1. Snapshot Messages (Full Book)
{
"action": "snapshot",
"data": [{
"instrument_id": "BTC-USDT-SWAP",
"timestamp": 1746272400000,
"asks": [
["50000.0", "10.5", "0", "0"],
["50001.0", "5.2", "0", "0"]
],
"bids": [
["49999.0", "8.2", "0", "0"],
["49998.0", "3.1", "0", "0"]
]
}]
}
Ask/Bid format: [price, quantity, num_orders, last_update_id]
For incremental_book_L2, we mainly care about price and quantity
2. Update Messages (Changes Only)
{
"action": "update",
"data": [{
"instrument_id": "BTC-USDT-SWAP",
"timestamp": 1746272400100,
"asks": [
["50000.0", "9.5"], # Quantity decreased from 10.5 to 9.5
["50002.0", "15.0"] # New ask level added
],
"bids": [
["49998.0", "0"] # Quantity 0 means remove this level
]
}]
}
Pricing and ROI: Tardis.dev vs Alternatives
When choosing a historical market data provider, cost efficiency matters significantly. Here is how Tardis.dev compares with alternatives:
| Provider | OKX incremental_book_L2 | Monthly Cost (Basic) | Monthly Cost (Pro) | Latency | Free Tier |
|---|---|---|---|---|---|
| Tardis.dev | Yes (Native) | $49/month | $299/month | <100ms | 30-day history |
| CoinAPI | Limited | $79/month | $499/month | <200ms | 100 requests/day |
| Exchange Native API | Yes | Free* | N/A | <50ms | N/A |
| HolySheep AI | Via Integration | $1 per dollar | Custom | <50ms | Free credits |
*Exchange native APIs require significant development overhead and have reliability issues
ROI Analysis:
- Using HolySheep AI at $1 per dollar saves you 85%+ compared to typical ¥7.3 pricing
- Time saved by using pre-built integrations: ~40 hours/month
- Reliability improvement: 99.9% uptime SLA vs self-hosted solutions
Why Choose HolySheep AI for Market Data Analysis
While Tardis.dev excels at raw data delivery, HolySheep AI provides the analytical layer on top. Here is why sophisticated traders use both together:
- Cost Efficiency: At $1 per dollar, you save 85%+ versus alternatives charging ¥7.3
- Multi-Model Analysis: GPT-4.1 ($8/MTok), Claude Sonnet 4.5 ($15/MTok), Gemini 2.5 Flash ($2.50/MTok), DeepSeek V3.2 ($0.42/MTok)
- Payment Flexibility: WeChat, Alipay, and international cards accepted
- Ultra-Low Latency: <50ms response times for time-sensitive analysis
- Integrated Pipeline: Fetch data from Tardis, analyze with HolySheep — one workflow
- Free Credits: Sign up here and get free credits on registration
Typical Workflow:
- Fetch historical OKX incremental_book_L2 data via Tardis.dev API
- Preprocess and clean data (using the scripts above)
- Send cleaned data to HolySheep AI for pattern analysis, anomaly detection, or strategy generation
- Receive actionable insights in seconds
Common Errors and Fixes
After helping hundreds of developers set up OKX data connections, I have compiled the most common errors and their solutions:
Error 1: "401 Unauthorized - Invalid API Key"
Symptoms: Your requests return 401 errors immediately, or WebSocket connections are rejected.
Cause: The API key is missing, expired, or incorrectly formatted.
# ❌ WRONG - Common mistakes:
headers = {
"Authorization": "ts_live_abc123" # Missing "Bearer " prefix
}
❌ WRONG - Empty or None key:
TARDIS_API_KEY = None
✅ CORRECT:
TARDIS_API_KEY = "ts_live_abc123xyz"
headers = {
"Authorization": f"Bearer {TARDIS_API_KEY}", # Must include "Bearer " prefix
"Accept": "application/json"
}
✅ VERIFY YOUR KEY:
1. Go to https://tardis.dev/dashboard
2. Check if your key is active
3. Verify the key has correct permissions for OKX data
4. Some keys only work for specific exchanges
Error 2: "429 Too Many Requests - Rate Limit Exceeded"
Symptoms: Requests work for a few minutes, then suddenly all return 429 errors.
Cause: Exceeded API rate limits (typically 10-60 requests/minute depending on plan).
# ✅ FIX - Implement rate limiting:
import time
from functools import wraps
def rate_limit(max_calls=30, period=60):
"""Limit API calls to max_calls per period seconds."""
def decorator(func):
call_times = []
@wraps(func)
def wrapper(*args, **kwargs):
now = time.time()
# Remove calls outside the window
call_times[:] = [t for t in call_times if now - t < period]
if len(call_times) >= max_calls:
sleep_time = period - (now - call_times[0])
if sleep_time > 0:
print(f"Rate limit reached. Sleeping {sleep_time:.1f}s...")
time.sleep(sleep_time)
call_times.append(time.time())
return func(*args, **kwargs)
return wrapper
return decorator
✅ Usage:
@rate_limit(max_calls=30, period=60)
def fetch_data_with_limit(url, headers):
return requests.get(url, headers=headers)
✅ Alternative: Exponential backoff for WebSocket reconnection:
MAX_RETRIES = 5
base_delay = 1
for attempt in range(MAX_RETRIES):
try:
await connect_to_feed()
break
except RateLimitError:
delay = base_delay * (2 ** attempt) # 1s, 2s, 4s, 8s, 16s
await asyncio.sleep(delay)
Error 3: "WebSocket Connection Closes After 30 Seconds"
Symptoms: WebSocket connects successfully but closes automatically after 30-60 seconds.
Cause: Missing heartbeat/ping-pong handling. Servers close idle connections.
# ✅ FIX - Use websockets library with proper ping handling:
import websockets
import asyncio
❌ WRONG - Missing ping configuration:
ws = await websockets.connect(url)
✅ CORRECT - Explicit ping/pong configuration:
ws = await websockets.connect(
url,
ping_interval=20, # Send ping every 20 seconds
ping_timeout=10, # Wait 10 seconds for pong response
close_timeout=10 # Wait 10 seconds for close acknowledgment
)
✅ Alternative: Manual ping handling:
async def heartbeat(websocket):
while True:
try:
await websocket.ping()
await asyncio.sleep(20)
except Exception:
break
async def receive_with_heartbeat(websocket):
# Start heartbeat task
heartbeat_task = asyncio.create_task(heartbeat(websocket))
try:
async for message in websocket:
# Process message
process_message(message)
finally:
heartbeat_task.cancel()
✅ ALSO: Handle ping messages from server:
async def handler(websocket, path):
async for message in websocket:
if isinstance(message, bytes):
# This is a ping from server - respond with pong
if message == b'ping':
await websocket.pong()
else:
# This is actual data
process_message(message)
Error 4: "Data Gap - Missing Timestamps in Historical Data"
Symptoms: Fetched historical data has gaps or duplicate timestamps.
Cause: OKX has maintenance windows, and incremental updates assume continuous connection.
# ✅ FIX - Implement gap detection and recovery:
def detect_data_gaps(df, expected_interval_ms=100):
"""Detect gaps in order book data."""
df = df.sort_values('timestamp_ms')
time_diffs = df['timestamp_ms'].diff()
gaps = time_diffs[time_diffs > expected_interval_ms * 10] # 10x expected = gap
if len(gaps) > 0:
print(f"WARNING: Found {len(gaps)} data gaps!")
for idx, gap_ms in gaps.items():
print(f" Gap at index {idx}: {gap_ms/1000:.1f}s")
# Fill gaps with NaN or handle appropriately
df['has_gap'] = time_diffs > expected_interval_ms * 10
return df
else:
print("No data gaps detected")
df['has_gap'] = False
return df
✅ For real-time data: Implement sequence number tracking:
class OrderBookReconstructor:
def __init__(self):
self.last_seq = None
self.pending_updates = []
def process_update(self, update):
seq = update.get('sequence')
if self.last_seq is None:
# First message - accept it
self.last_seq = seq
return self.apply_update(update)
elif seq == self.last_seq + 1:
# Sequential - apply directly
self.last_seq = seq
return self.apply_update(update)
elif seq > self.last_seq + 1:
# Gap detected - buffer and request replay
self.pending_updates.append(update)
print(f"Gap detected: expected {self.last_seq+1}, got {seq}")
self.request_replay(self.last_seq + 1, seq - 1)
return None
else:
# Old message - discard
return None
def request_replay(self, start_seq, end_seq):
"""Request missing messages from Tardis replay API."""
# Implementation depends on Tardis replay endpoint
pass
Error 5: "Memory Leak - Order Book Growing Infinitely"
Symptoms: Script runs fine for minutes, then crashes with OutOfMemory error.
Cause: Order book state dictionary grows without cleanup of stale price levels.
# ✅ FIX - Implement book cleanup:
class ManagedOrderBook:
def __init__(self, max_levels=100, cleanup_interval=1000):
self.asks = {} # price -> {qty, last_update}
self.bids = {}
self.max_levels = max_levels
self.cleanup_counter = 0
def update_side(self, side, updates):
book = self.asks if side == 'asks' else self.bids
now = time.time()
for price, qty in updates:
price = float(price)
qty = float(qty)
if qty == 0:
# Remove level
book.pop(price, None)
else:
# Update or add
book[price] = {'qty': qty, 'last_update': now}
# Periodic cleanup
self.cleanup_counter += 1
if self.cleanup_counter >= self.cleanup_interval:
self.cleanup_old_levels()
def cleanup_old_levels(self):
"""Remove stale levels that haven't updated in 5 minutes."""
stale_threshold = time.time() - 300 # 5 minutes
for book in [self.asks, self.bids]:
stale = [p for p, data in book.items()
if data['last_update'] < stale_threshold]
for p in stale:
del book[p]
# Also trim to max_levels
for book in [self.asks, self.bids]:
if len(book) > self.max_levels:
if book is self.asks:
# Keep lowest prices
sorted_prices = sorted(book.keys())[:self.max_levels]
else:
# Keep highest prices
sorted_prices = sorted(book.keys(), reverse=True)[:self.max_levels]
# Remove others
for p in list(book.keys()):
if p not in sorted_prices:
del book[p]
self.cleanup_counter = 0
print(f"Cleanup complete. Asks: {len(self.asks)}, Bids: {len(self.bids)}")
Best Practices for Production Deployment
After running these scripts in production for months, here are the practices that have saved me countless hours of debugging:
- Always use virtual environments — Dependency conflicts are the #1 cause of mysterious failures
- Implement comprehensive logging — You will need it when something breaks at 3 AM
- Use environment variables for secrets — Never hardcode API keys in source code
- Set up monitoring alerts — Track message rate, connection status, and data gaps
- Test reconnection logic thoroughly — Network issues are inevitable, not exceptional
- Store raw data before processing — You cannot reprocess