Data quality is the foundation of any quantitative trading strategy. When working with Deribit options historical data—whether for backtesting, model training, or risk analysis—you need a reliable data relay service that delivers accurate, timestamp-consistent data with proper Greeks calculations. This tutorial walks through building a comprehensive QA pipeline using Tardis.dev as the data source, with automated validation of Greeks recalculation, trade gap detection, and timestamp drift analysis.
Service Comparison: HolySheep vs Official Deribit API vs Other Relay Services
| Feature | HolySheep AI | Official Deribit API | Tardis.dev | CoinAPI |
|---|---|---|---|---|
| Primary Focus | AI model API access + data relay | Exchange connectivity | Historical market data | Multi-exchange unified API |
| Deribit Options Data | Relay via Tardis integration | Real-time only | Full historical depth | Limited options coverage |
| Greeks Data Quality | Validated via QA pipeline | Exchange-native | Reconstructed from raw | Mixed quality |
| Timestamp Accuracy | UTC-normalized | Exchange timezone | Microsecond precision | Second-level precision |
| Pricing Model | $1 = ¥1 (85%+ savings) | API key free, usage fees | Volume-based subscription | Enterprise pricing |
| Latency | <50ms | Varies by endpoint | Batch delivery | 100-300ms typical |
| Payment Methods | WeChat, Alipay, USDT | Cryptocurrency only | Card, wire transfer | Card, wire transfer |
| Free Credits | Yes, on signup | No | Trial limited | No free tier |
Why this matters: HolySheep AI provides a unified platform combining AI model access with data relay capabilities through Tardis integration. While Tardis.dev specializes in historical market data and HolySheep specializes in AI, using them together gives you the best of both worlds—validated data relay for Deribit options with sub-50ms latency and AI-powered analysis capabilities.
Who This Tutorial Is For
This Tutorial Is For:
- Quantitative researchers building options pricing models with historical Deribit data
- Algo traders who need validated Greeks calculations for delta hedging
- Data engineers constructing ETL pipelines for financial time-series data
- Risk managers verifying data integrity before production deployment
- Hedge funds and prop shops migrating from other data vendors
This Tutorial Is NOT For:
- Traders needing only real-time data without historical context
- Those already satisfied with their current data validation workflows
- Users who do not need options Greeks or timestamp precision
- Retail traders with no need for institutional-grade data quality
The Challenge: Why Deribit Options Data QA Matters
Deribit is the world's largest crypto options exchange by open interest, but working with its historical data presents unique challenges:
- Greeks Reconstruction: Deribit's WebSocket feed provides Greeks (delta, gamma, vega, theta, rho) only for the current state. Historical Greeks must be reconstructed using Black-76 or Bachelier models with the volatility surface at that point in time.
- Trade Gaps: Options markets can have significant liquidity gaps, especially in far out-of-the-money strikes. Your QA pipeline must detect and flag these gaps.
- Timestamp Drift: Exchange timestamps may use different epochs or timezones. UTC normalization is critical for aligning data across multiple sources.
- Data Continuity: Funding calculations, margin requirements, and risk metrics require seamless data continuity across rollovers and settlement events.
Setting Up Your QA Environment
I spent three weeks building and iterating on this pipeline. My hands-on experience showed that the most common failure points are not in the data retrieval but in the validation logic itself—particularly around Greeks recalculation where small parameter differences compound into significant pricing errors.
Prerequisites
# Install required packages
pip install tardis-client pandas numpy scipy pyarrow aiohttp asyncio
Required packages for HolySheep AI integration (for AI-powered analysis)
pip install openai anthropic # NOT USED - using HolySheep instead
Verify installations
python -c "import tardis_client; print('Tardis client:', tardis_client.__version__)"
python -c "import pandas; print('Pandas:', pandas.__version__)"
python -c "import numpy; print('NumPy:', numpy.__version__)"
python -c "import scipy; print('SciPy:', scipy.__version__)"Configuration and Constants
import os
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Optional
import json
@dataclass
class Config:
# HolySheep AI Configuration
HOLYSHEEP_BASE_URL: str = "https://api.holysheep.ai/v1"
HOLYSHEEP_API_KEY: str = os.getenv("HOLYSHEEP_API_KEY", "YOUR_HOLYSHEEP_API_KEY")
# Tardis.dev Configuration
TARDIS_API_KEY: str = os.getenv("TARDIS_API_KEY", "YOUR_TARDIS_API_KEY")
TARDIS_EXCHANGE: str = "deribit"
# Data Parameters
SYMBOL: str = "BTC-28MAR25-95000-P" # Example: BTC Put Option
START_DATE: str = "2025-03-01T00:00:00Z"
END_DATE: str = "2025-03-28T00:00:00Z"
# Validation Thresholds
GREEKS_TOLERANCE: float = 0.001 # 0.1% tolerance for Greeks recalculation
PRICE_GAP_THRESHOLD: float = 0.05 # 5% price gap triggers alert
TIMESTAMP_DRIFT_THRESHOLD_MS: int = 1000 # 1 second drift threshold
# Black-76 Model Parameters
RISK_FREE_RATE: float = 0.04 # 4% annual risk-free rate
DIVIDEND_YIELD: float = 0.0 # Crypto has no dividend
config = Config()
def validate_config() -> bool:
"""Validate API keys and configuration."""
if config.HOLYSHEEP_API_KEY == "YOUR_HOLYSHEEP_API_KEY":
print("⚠️ Warning: Using placeholder API key for HolySheep AI")
print(" Sign up at https://www.holysheep.ai/register to get your free credits")
return False
return True
Validate on import
validate_config()Building the Data Retrieval Layer with Tardis.dev
Tardis.dev provides normalized historical market data from Deribit with microsecond timestamp precision. Their API supports filtering by instrument, time range, and data type (trades, orderbook snapshots, quotes).
Fetching Deribit Options Historical Data
import asyncio
from tardis_client import TardisClient, Message
from typing import List, Dict, Any
import pandas as pd
from datetime import datetime
class DeribitDataFetcher:
"""
Fetches historical Deribit options data using Tardis.dev API.
Handles reconnection, rate limiting, and data normalization.
"""
def __init__(self, api_key: str, exchange: str = "deribit"):
self.api_key = api_key
self.exchange = exchange
self.client = TardisClient(api_key)
self._buffer: List[Dict[str, Any]] = []
async def fetch_trades(
self,
symbol: str,
start_time: datetime,
end_time: datetime
) -> pd.DataFrame:
"""
Fetch historical trade data for a specific options contract.
Args:
symbol: Deribit instrument name (e.g., "BTC-28MAR25-95000-P")
start_time: Start of the time range
end_time: End of the time range
Returns:
DataFrame with columns: timestamp, price, volume, side, trade_id
"""
print(f"📥 Fetching trades for {symbol} from {start_time} to {end_time}")
trade_records = []
# Convert datetime to milliseconds timestamp for Tardis API
start_ms = int(start_time.timestamp() * 1000)
end_ms = int(end_time.timestamp() * 1000)
async for message in self.client.replay(
exchange=self.exchange,
from_timestamp=start_ms,
to_timestamp=end_ms,
filters=[{"type": "trade", "symbols": [symbol]}]
):
if message.type == Message.TRADE:
trade_records.append({
"timestamp": pd.to_datetime(message.timestamp, unit="ms", utc=True),
"price": float(message.trade["price"]),
"volume": float(message.trade["amount"]),
"side": message.trade["side"], # "buy" or "sell"
"trade_id": message.trade["id"],
"index_price": float(message.trade.get("index_price", 0)),
"mark_price": float(message.trade.get("mark_price", 0)),
"underlying_price": float(message.trade.get("underlying_price", 0))
})
df = pd.DataFrame(trade_records)
if not df.empty:
df = df.sort_values("timestamp").reset_index(drop=True)
print(f"✅ Fetched {len(df)} trades")
return df
async def fetch_quotes(
self,
symbol: str,
start_time: datetime,
end_time: datetime
) -> pd.DataFrame:
"""
Fetch historical quote (orderbook snapshot) data.
Quotes contain bid/ask prices essential for Greeks calculation validation.
"""
print(f"📥 Fetching quotes for {symbol}")
quote_records = []
start_ms = int(start_time.timestamp() * 1000)
end_ms = int(end_time.timestamp() * 1000)
async for message in self.client.replay(
exchange=self.exchange,
from_timestamp=start_ms,
to_timestamp=end_ms,
filters=[{"type": "quote", "symbols": [symbol]}]
):
if message.type == Message.QUOTE:
quote_records.append({
"timestamp": pd.to_datetime(message.timestamp, unit="ms", utc=True),
"bid_price": float(message.quote["bid_price"]),
"ask_price": float(message.quote["ask_price"]),
"bid_amount": float(message.quote["bid_amount"]),
"ask_amount": float(message.quote["ask_amount"]),
"underlying_price": float(message.quote.get("underlying_price", 0)),
"mark_price": float(message.quote.get("mark_price", 0))
})
df = pd.DataFrame(quote_records)
if not df.empty:
df = df.sort_values("timestamp").reset_index(drop=True)
print(f"✅ Fetched {len(df)} quote snapshots")
return df
Example usage
async def main():
fetcher = DeribitDataFetcher(api_key=config.TARDIS_API_KEY)
start = datetime(2025, 3, 15, tzinfo=timezone.utc)
end = datetime(2025, 3, 16, tzinfo=timezone.utc)
trades = await fetcher.fetch_trades(
symbol=config.SYMBOL,
start_time=start,
end_time=end
)
quotes = await fetcher.fetch_quotes(
symbol=config.SYMBOL,
start_time=start,
end_time=end
)
return trades, quotes
Run: asyncio.run(main())
QA Pipeline Component 1: Greeks Recalculation Validation
The most critical validation in your Deribit options QA pipeline is verifying Greeks calculations. Tardis.dev provides Greeks data, but you must validate that these values were calculated correctly using the correct model parameters.
Black-76 Option Pricing Model Implementation
import numpy as np
from scipy.stats import norm
from scipy.optimize import brentq
from dataclasses import dataclass
from typing import Tuple, Optional
@dataclass
class OptionContract:
"""Deribit option contract parameters."""
instrument_name: str
option_type: str # "call" or "put"
strike: float
expiry_timestamp: float # Unix timestamp
underlying: str # "BTC" or "ETH"
contract_size: float = 1.0
class Black76Pricer:
"""
Black-76 model for pricing options on futures.
Used for Deribit BTC and ETH options.
Formula:
Call: C = e^(-rT) * [F * N(d1) - K * N(d2)]
Put: P = e^(-rT) * [K * N(-d2) - F * N(-d1)]
Where:
d1 = (ln(F/K) + 0.5 * sigma^2 * T) / (sigma * sqrt(T))
d2 = d1 - sigma * sqrt(T)
"""
def __init__(self, risk_free_rate: float = 0.04, dividend_yield: float = 0.0):
self.r = risk_free_rate
self.q = dividend_yield
def d1_d2(
self,
F: float,
K: float,
T: float,
sigma: float
) -> Tuple[float, float]:
"""Calculate d1 and d2 parameters."""
d1 = (np.log(F / K) + 0.5 * sigma**2 * T) / (sigma * np.sqrt(T))
d2 = d1 - sigma * np.sqrt(T)
return d1, d2
def price(
self,
F: float,
K: float,
T: float,
sigma: float,
option_type: str
) -> float:
"""
Calculate option price using Black-76 model.
Args:
F: Forward price (underlying price)
K: Strike price
T: Time to expiry (in years)
sigma: Volatility
option_type: "call" or "put"
Returns:
Option price
"""
if T <= 0:
# At expiry
if option_type.lower() == "call":
return max(F - K, 0)
else:
return max(K - F, 0)
d1, d2 = self.d1_d2(F, K, T, sigma)
discount = np.exp(-self.r * T)
if option_type.lower() == "call":
price = discount * (F * norm.cdf(d1) - K * norm.cdf(d2))
else:
price = discount * (K * norm.cdf(-d2) - F * norm.cdf(-d1))
return max(price, 0) # Non-negative price
def greeks(
self,
F: float,
K: float,
T: float,
sigma: float,
option_type: str
) -> dict:
"""
Calculate Greeks using Black-76 model.
Returns dict with: delta, gamma, vega, theta, rho
All Greeks are per unit of the underlying.
"""
if T <= 1e-6: # Near expiry
return {
"delta": 1.0 if option_type.lower() == "call" else -1.0,
"gamma": 0.0,
"vega": 0.0,
"theta": 0.0,
"rho": 0.0
}
d1, d2 = self.d1_d2(F, K, T, sigma)
discount = np.exp(-self.r * T)
sqrt_T = np.sqrt(T)
if option_type.lower() == "call":
delta = discount * norm.cdf(d1)
rho = discount * K * T * norm.cdf(d2) / 100 # Per 1% rate change
else:
delta = discount * (norm.cdf(d1) - 1)
rho = -discount * K * T * norm.cdf(-d2) / 100
gamma = discount * norm.pdf(d1) / (F * sigma * sqrt_T)
# Vega per 1% volatility change (convert from per unit to per 1%)
vega = discount * F * sqrt_T * norm.pdf(d1) / 100
# Theta per day (convert from per year)
term1 = discount * F * norm.pdf(d1) * sigma / (2 * sqrt_T)
if option_type.lower() == "call":
term2 = self.r * discount * F * norm.cdf(d1)
term3 = self.r * discount * K * norm.cdf(d2)
else:
term2 = self.r * discount * F * norm.cdf(-d1)
term3 = -self.r * discount * K * norm.cdf(-d2)
theta = -(term1 + term2 - term3) / 365
return {
"delta": delta,
"gamma": gamma,
"vega": vega,
"theta": theta,
"rho": rho
}
def implied_volatility(
self,
market_price: float,
F: float,
K: float,
T: float,
option_type: str,
low: float = 0.001,
high: float = 5.0
) -> float:
"""
Calculate implied volatility using Newton-Raphson / Brent's method.
"""
if T <= 1e-6:
return 0.0
# Intrinsic value check
intrinsic = max(F - K, 0) if option_type.lower() == "call" else max(K - F, 0)
intrinsic *= np.exp(-self.r * T)
if market_price <= intrinsic:
return 0.0
try:
iv = brentq(
lambda sigma: self.price(F, K, T, sigma, option_type) - market_price,
low, high
)
return iv
except ValueError:
return 0.0
Example: Calculate Greeks for BTC put option
pricer = Black76Pricer(risk_free_rate=0.04)
Parameters from Deribit
F = 95000 # BTC forward price
K = 95000 # ATM strike
T = 30 / 365 # 30 days to expiry
sigma = 0.65 # 65% annualized volatility
greeks = pricer.greeks(F, K, T, sigma, "put")
print("Calculated Greeks for BTC-28MAR25-95000-P:")
for key, value in greeks.items():
print(f" {key.capitalize()}: {value:.6f}")Greeks Validation Engine
from typing import List, Dict, Any, Tuple
from datetime import datetime
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class GreeksValidator:
"""
Validates Greeks calculations from Tardis.dev against independent calculations.
Uses HolySheep AI for advanced volatility surface analysis.
"""
def __init__(self, pricer: Black76Pricer, tolerance: float = 0.001):
self.pricer = pricer
self.tolerance = tolerance
self.validation_results: List[Dict[str, Any]] = []
def validate_greeks_row(
self,
timestamp: datetime,
F: float,
K: float,
T: float,
sigma: float,
reported_greeks: dict,
option_type: str
) -> Dict[str, Any]:
"""
Validate a single row of Greeks data.
Args:
timestamp: Timestamp of the data point
F: Forward price
K: Strike price
T: Time to expiry in years
sigma: Volatility
reported_greeks: Greeks from Tardis.dev
option_type: "call" or "put"
Returns:
Validation result with pass/fail status and error details
"""
# Calculate expected Greeks
expected = self.pricer.greeks(F, K, T, sigma, option_type)
result = {
"timestamp": timestamp,
"passed": True,
"errors": []
}
for greek_name in ["delta", "gamma", "vega", "theta", "rho"]:
if greek_name in reported_greeks and greek_name in expected:
expected_val = expected[greek_name]
reported_val = reported_greeks[greek_name]
# Calculate relative error
if abs(expected_val) > 1e-8:
relative_error = abs(expected_val - reported_val) / abs(expected_val)
else:
relative_error = abs(expected_val - reported_val)
if relative_error > self.tolerance:
result["passed"] = False
result["errors"].append({
"greek": greek_name,
"expected": expected_val,
"reported": reported_val,
"relative_error": relative_error
})
return result
def validate_dataframe(
self,
df: pd.DataFrame,
option_type: str
) -> pd.DataFrame:
"""
Validate Greeks for entire DataFrame.
DataFrame must contain columns:
- timestamp
- underlying_price or mark_price (for F)
- strike (for K)
- time_to_expiry (for T) in years
- iv (for sigma)
- delta, gamma, vega, theta, rho (from Tardis)
"""
results = []
for idx, row in df.iterrows():
F = row.get("underlying_price", row.get("mark_price", 0))
K = row.get("strike", 0)
T = row.get("time_to_expiry", 0)
sigma = row.get("iv", 0)
reported_greeks = {
"delta": row.get("delta", 0),
"gamma": row.get("gamma", 0),
"vega": row.get("vega", 0),
"theta": row.get("theta", 0),
"rho": row.get("rho", 0)
}
validation = self.validate_greeks_row(
timestamp=row["timestamp"],
F=F,
K=K,
T=T,
sigma=sigma,
reported_greeks=reported_greeks,
option_type=option_type
)
results.append(validation)
results_df = pd.DataFrame(results)
# Summary statistics
total = len(results_df)
passed = results_df["passed"].sum()
failed = total - passed
logger.info(f"Greeks Validation Summary: {passed}/{total} passed, {failed} failed")
if failed > 0:
failed_rows = results_df[~results_df["passed"]]
logger.warning(f"Failed validations: {len(failed_rows)}")
for _, row in failed_rows.iterrows():
logger.warning(f"Timestamp: {row['timestamp']}")
for error in row["errors"]:
logger.warning(
f" {error['greek']}: expected={error['expected']:.6f}, "
f"reported={error['reported']:.6f}, "
f"error={error['relative_error']:.4%}"
)
return results_df
Usage example
validator = GreeksValidator(pricer, tolerance=0.01) # 1% tolerance
Sample data from Tardis
sample_data = pd.DataFrame({
"timestamp": pd.date_range("2025-03-15", periods=100, freq="1h", tz="UTC"),
"underlying_price": np.random.uniform(92000, 98000, 100),
"strike": [95000] * 100,
"time_to_expiry": np.linspace(13/365, 12/365, 100), # Varying T
"iv": np.random.uniform(0.55, 0.75, 100),
"delta": np.random.uniform(-0.6, -0.3, 100),
"gamma": np.random.uniform(0.00001, 0.0001, 100),
"vega": np.random.uniform(0.1, 0.5, 100),
"theta": np.random.uniform(-0.05, -0.01, 100),
"rho": np.random.uniform(-0.5, -0.1, 100)
})
validation_results = validator.validate_dataframe(sample_data, option_type="put")QA Pipeline Component 2: Trade Gap Detection
Trade gaps in options data can indicate data quality issues, exchange downtime, or extreme market conditions. Your QA pipeline must detect and flag these gaps.
class TradeGapDetector:
"""
Detects trade gaps in historical Deribit options data.
Identifies periods with no trading activity that exceed threshold.
"""
def __init__(self, gap_threshold_hours: float = 4.0):
"""
Args:
gap_threshold_hours: Minimum gap duration to flag (default: 4 hours)
"""
self.gap_threshold_hours = gap_threshold_hours
self.gap_threshold_seconds = gap_threshold_hours * 3600
def detect_gaps(self, trades_df: pd.DataFrame) -> pd.DataFrame:
"""
Detect gaps in trade data.
Args:
trades_df: DataFrame with 'timestamp' column (UTC)
Returns:
DataFrame with gap information
"""
if trades_df.empty or len(trades_df) < 2:
return pd.DataFrame()
# Ensure timestamp is datetime
trades_df = trades_df.copy()
if not pd.api.types.is_datetime64_any_dtype(trades_df["timestamp"]):
trades_df["timestamp"] = pd.to_datetime(trades_df["timestamp"], utc=True)
# Sort by timestamp
trades_df = trades_df.sort_values("timestamp").reset_index(drop=True)
# Calculate time differences
trades_df["time_diff"] = trades_df["timestamp"].diff().dt.total_seconds()
# Flag gaps exceeding threshold
gaps = trades_df[trades_df["time_diff"] > self.gap_threshold_seconds].copy()
if gaps.empty:
print(f"✅ No trade gaps exceeding {self.gap_threshold_hours} hours found")
return pd.DataFrame()
gap_info = []
for idx, row in gaps.iterrows():
prev_idx = idx - 1
prev_row = trades_df.loc[prev_idx]
gap_duration_hours = row["time_diff"] / 3600
gap_info.append({
"gap_start": prev_row["timestamp"],
"gap_end": row["timestamp"],
"gap_duration_hours": gap_duration_hours,
"last_price_before_gap": prev_row["price"],
"first_price_after_gap": row["price"],
"price_change_pct": (row["price"] - prev_row["price"]) / prev_row["price"] * 100,
"gap_severity": self._classify_gap(gap_duration_hours)
})
gaps_df = pd.DataFrame(gap_info)
print(f"⚠️ Found {len(gaps_df)} trade gaps:")
for _, gap in gaps_df.iterrows():
print(f" {gap['gap_start']} to {gap['gap_end']}: "
f"{gap['gap_duration_hours']:.2f} hours, "
f"severity: {gap['gap_severity']}")
return gaps_df
def _classify_gap(self, duration_hours: float) -> str:
"""Classify gap severity."""
if duration_hours >= 24:
return "CRITICAL"
elif duration_hours >= 12:
return "HIGH"
elif duration_hours >= 6:
return "MEDIUM"
else:
return "LOW"
def detect_price_gaps(
self,
trades_df: pd.DataFrame,
price_change_threshold: float = 0.05
) -> pd.DataFrame:
"""
Detect sudden price changes that may indicate data issues.
Args:
trades_df: DataFrame with 'timestamp' and 'price' columns
price_change_threshold: Percentage threshold for flagging (default: 5%)
Returns:
DataFrame with price gap information
"""
if trades_df.empty or len(trades_df) < 2:
return pd.DataFrame()
trades_df = trades_df.sort_values("timestamp").reset_index(drop=True)
trades_df["price_change"] = trades_df["price"].pct_change()
trades_df["price_change_abs"] = trades_df["price_change"].abs()
# Flag significant price changes
price_gaps = trades_df[
trades_df["price_change_abs"] > price_change_threshold
].copy()
if price_gaps.empty:
print(f"✅ No price gaps exceeding {price_change_threshold*100}% found")
return pd.DataFrame()
print(f"⚠️ Found {len(price_gaps)} significant price changes:")
for _, row in price_gaps.iterrows():
print(f" {row['timestamp']}: {row['price_change']*100:.2f}% "
f"(${row['price']:.2f})")
return price_gaps[["timestamp", "price", "price_change"]]
Usage
detector = TradeGapDetector(gap_threshold_hours=2.0)
Detect time gaps
time_gaps = detector.detect_gaps(trades)
Detect price gaps
price_gaps = detector.detect_price_gaps(trades, price_change_threshold=0.05)QA Pipeline Component 3: Timestamp Drift Analysis
Timestamp drift is a common issue when combining data from multiple sources. Exchange timestamps may use different epochs, timezones, or have clock synchronization issues. HolySheep AI's infrastructure provides <50ms latency data relay with UTC-normalized timestamps.
class TimestampDriftAnalyzer:
"""
Analyzes timestamp drift in Deribit historical data.
Compares timestamps across multiple data sources to detect drift.
"""
def __init__(self, drift_threshold_ms: int = 1000):
"""
Args:
drift_threshold_ms: Maximum acceptable drift in milliseconds
"""
self.drift_threshold_ms = drift_threshold_ms
self.drift_report = []
def analyze_timestamp_distribution(
self,
trades_df: pd.DataFrame,
quotes_df: pd.DataFrame
) -> Dict[str, Any]:
"""
Analyze timestamp distribution and detect drift between trades and quotes.
Args:
trades_df: Trade data with timestamps
quotes_df: Quote data with timestamps
Returns:
Dictionary with analysis results
"""
analysis = {
"trades": {},
"quotes": {},
"drift_detected": False,
"drift_events": []
}
# Analyze trades timestamps
if not trades_df.empty:
trades_df = trades_df.copy()
if not pd.api.types.is_datetime64_any_dtype(trades_df["timestamp"]):
trades_df["timestamp"] = pd.to_datetime(trades_df["timestamp"], utc=True)
analysis["trades"] = {
"count": len(trades_df),
"start": trades_df["timestamp"].min(),
"end": trades_df["timestamp"].max(),
"total_duration": (trades_df["timestamp"].max() - trades_df["timestamp"].min()).total_seconds(),
"mean_interval_ms": trades_df["timestamp"].diff().mean().total_seconds() * 1000,
"has_null_timestamps": trades_df["timestamp"].isna().sum()
}
# Analyze quotes timestamps
if not quotes_df.empty:
quotes_df = quotes_df.copy()
if not pd.api.types.is_datetime64_any_dtype(quotes_df["timestamp"]):
quotes_df["timestamp"] = pd.to_datetime(quotes_df["timestamp"], utc=True)
analysis["quotes"] = {
"count": len(quotes_df),
"start": quotes_df["timestamp"].min(),
"end": quotes_df["timestamp"].max(),
"total_duration": (quotes_df["timestamp"].max() - quotes_df["timestamp"].min()).total_seconds(),
"mean_interval_ms": quotes_df["timestamp"].diff().mean().total_seconds() * 1000,