Published: May 3, 2026 | By HolySheep AI Engineering Team

Introduction

If you are building a trading strategy, backtesting engine, or quantitative research pipeline that relies on Hyperliquid perpetual futures data, you have probably encountered a critical question: How do I know if my historical trade data is accurate and complete?

This is not an academic concern. In production environments, data quality issues cause false signals, corrupted backtests, and regulatory compliance failures. I spent three months building reconciliation pipelines between Tardis.dev market data relay feeds and self-constructed collectors for Hyperliquid, and I discovered that data gaps are far more common than most developers assume—even from established data providers.

In this tutorial, you will learn how to perform systematic data quality checks, identify discrepancies between Tardis.dev trade feeds and your own collector, and implement automated reconciliation workflows that catch issues before they propagate into your models.

Prerequisites: Basic Python knowledge, a Hyperliquid account, API access to both Tardis.dev and your data source of choice, and approximately 2 hours of focused work time.

Understanding the Data Landscape

What is Tardis.dev?

Tardis.dev is a commercial market data aggregation service that provides normalized historical and real-time data feeds for cryptocurrency exchanges. For Hyperliquid, Tardis offers normalized trade streams, order book snapshots, and liquidations data with documented schema and consistent timestamps. Their relay infrastructure handles exchange-specific API quirks, rate limiting, and reconnection logic so you do not have to.

What is a Self-Built Collector?

A self-built collector is custom software you write to ingest raw data directly from exchange WebSocket or REST APIs. For Hyperliquid, this means connecting to their public WebSocket endpoint (wss://stream.hyperliquid.xyz/Info) and processing trade events as they arrive. The advantage is complete control over data handling; the disadvantage is that you own all the error handling, deduplication, and gap-filling logic.

Why Do Discrepancies Occur?

Discrepancies between Tardis and self-built collectors arise from several sources:

Step-by-Step: Setting Up Your Reconciliation Environment

Step 1: Obtain API Credentials

First, you need access to both Tardis.dev and your HolySheep AI account for auxiliary processing tasks like data transformation and report generation.

  1. Sign up for a Tardis.dev account and generate an API key from your dashboard.
  2. Create a HolySheep AI account if you have not already. HolySheep offers $0.50 free credits on registration and supports WeChat/Alipay payment methods with <50ms API latency, making it ideal for high-throughput data processing pipelines.

Step 2: Install Required Python Packages

pip install pandas numpy requests websockets-client asyncio aiohttp tqdm matplotlib

Step 3: Fetching Trade Data from Tardis.dev

Tardis.dev provides a REST API for historical trade queries. Below is a complete Python function to fetch Hyperliquid perpetual trade data for a specific date range.

import requests
import pandas as pd
from datetime import datetime, timedelta

TARDIS_API_KEY = "YOUR_TARDIS_API_KEY"
HYPERLIQUID_EXCHANGE = "hyperliquid"
SYMBOL = "BTC-PERPETUAL"

def fetch_tardis_trades(start_date: str, end_date: str, symbol: str = SYMBOL):
    """
    Fetch historical trade data from Tardis.dev for Hyperliquid.
    
    Args:
        start_date: Start date in YYYY-MM-DD format
        end_date: End date in YYYY-MM-DD format
        symbol: Trading pair symbol (default: BTC-PERPETUAL)
    
    Returns:
        DataFrame with trade records
    """
    url = f"https://api.tardis.dev/v1/trades/{HYPERLIQUID_EXCHANGE}"
    
    params = {
        "symbol": symbol,
        "from": start_date,
        "to": end_date,
        "limit": 50000,  # Max records per request
    }
    
    headers = {
        "Authorization": f"Bearer {TARDIS_API_KEY}"
    }
    
    all_trades = []
    page = 1
    
    while True:
        params["page"] = page
        response = requests.get(url, headers=headers, params=params)
        response.raise_for_status()
        
        data = response.json()
        if not data:
            break
            
        all_trades.extend(data)
        print(f"Page {page}: Fetched {len(data)} trades")
        
        if len(data) < params["limit"]:
            break
            
        page += 1
    
    df = pd.DataFrame(all_trades)
    df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms")
    
    return df


Example usage

tardis_df = fetch_tardis_trades("2026-04-01", "2026-04-07")
print(f"Total trades from Tardis: {len(tardis_df)}")
print(tardis_df.head())

Step 4: Fetching Trade Data from Your Self-Built Collector

Assuming your collector stores trades in a local database or file system, here is a function to export that data in a comparable format.

import json
from pathlib import Path

def load_selfbuilt_trades(filepath: str) -> pd.DataFrame:
    """
    Load trade data from self-built collector storage.
    Assumes JSON Lines format with the following schema:
    {
        "trade_id": "unique_trade_id",
        "timestamp": 1711910400000,  # milliseconds
        "side": "buy",
        "price": 69500.5,
        "amount": 0.05,
        "symbol": "BTC-PERPETUAL"
    }
    """
    trades = []
    
    with open(filepath, "r") as f:
        for line in f:
            trades.append(json.loads(line.strip()))
    
    df = pd.DataFrame(trades)
    df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms")
    
    return df


Example usage

selfbuilt_df = load_selfbuilt_trades("/path/to/your/collector/trades_2026-04.jsonl")
print(f"Total trades from self-built: {len(selfbuilt_df)}")

Step 5: Implementing the Reconciliation Check

Now we implement the core reconciliation logic that identifies gaps, duplicates, and timestamp mismatches.

import numpy as np

def reconcile_trade_data(tardis_df: pd.DataFrame, selfbuilt_df: pd.DataFrame, 
                         tolerance_ms: int = 100) -> dict:
    """
    Compare trade data from Tardis.dev against self-built collector.
    
    Args:
        tardis_df: Trade DataFrame from Tardis.dev
        selfbuilt_df: Trade DataFrame from self-built collector
        tolerance_ms: Acceptable timestamp difference in milliseconds
    
    Returns:
        Dictionary containing reconciliation results
    """
    results = {
        "tardis_count": len(tardis_df),
        "selfbuilt_count": len(selfbuilt_df),
        "count_difference": len(tardis_df) - len(selfbuilt_df),
        "matched_trades": 0,
        "missing_in_selfbuilt": [],
        "missing_in_tardis": [],
        "timestamp_mismatches": [],
        "duplicate_trade_ids": []
    }
    
    # Create lookup dictionaries
    tardis_trades = {}
    for _, row in tardis_df.iterrows():
        trade_key = (row["symbol"], row["price"], row["amount"], row["side"])
        tardis_trades[trade_key] = row
    
    selfbuilt_trades = {}
    for _, row in selfbuilt_df.iterrows():
        trade_key = (row["symbol"], row["price"], row["amount"], row["side"])
        if trade_key not in selfbuilt_trades:
            selfbuilt_trades[trade_key] = []
        selfbuilt_trades[trade_key].append(row)
    
    # Check for missing trades in self-built
    for trade_key, tardis_row in tardis_trades.items():
        if trade_key not in selfbuilt_trades:
            results["missing_in_selfbuilt"].append({
                "timestamp": tardis_row["timestamp"],
                "price": tardis_row["price"],
                "amount": tardis_row["amount"],
                "side": tardis_row["side"]
            })
        else:
            # Check timestamp difference
            selfbuilt_row = selfbuilt_trades[trade_key][0]
            time_diff = abs((tardis_row["timestamp"] - selfbuilt_row["timestamp"]).total_seconds() * 1000)
            if time_diff > tolerance_ms:
                results["timestamp_mismatches"].append({
                    "trade_key": trade_key,
                    "tardis_timestamp": tardis_row["timestamp"],
                    "selfbuilt_timestamp": selfbuilt_row["timestamp"],
                    "difference_ms": time_diff
                })
            else:
                results["matched_trades"] += 1
    
    # Check for trades in self-built not in Tardis
    all_tardis_keys = set(tardis_trades.keys())
    for trade_key in selfbuilt_trades.keys():
        if trade_key not in all_tardis_keys:
            results["missing_in_tardis"].append({
                "price": trade_key[1],
                "amount": trade_key[2],
                "side": trade_key[3]
            })
    
    return results


Run reconciliation

reconciliation = reconcile_trade_data(tardis_df, selfbuilt_df)

print("=" * 60)
print("RECONCILIATION SUMMARY")
print("=" * 60)
print(f"Tardis trades:      {reconciliation['tardis_count']}")
print(f"Self-built trades: {reconciliation['selfbuilt_count']}")
print(f"Difference:         {reconciliation['count_difference']}")
print(f"Matched:            {reconciliation['matched_trades']}")
print(f"Missing in self-built: {len(reconciliation['missing_in_selfbuilt'])}")
print(f"Missing in Tardis:     {len(reconciliation['missing_in_tardis'])}")
print(f"Timestamp mismatches:  {len(reconciliation['timestamp_mismatches'])}")

Step 6: Visualizing Discrepancies

Visual analysis helps identify patterns in data gaps. Create a histogram of missing trade timestamps to spot disconnection windows.

import matplotlib.pyplot as plt

def visualize_discrepancies(reconciliation: dict, tardis_df: pd.DataFrame):
    """Generate visualization of data quality issues."""
    
    fig, axes = plt.subplots(2, 2, figsize=(14, 10))
    
    # Plot 1: Trade count comparison
    ax1 = axes[0, 0]
    sources = ["Tardis.dev", "Self-Built"]
    counts = [reconciliation["tardis_count"], reconciliation["selfbuilt_count"]]
    colors = ["#3498db", "#e74c3c"]
    bars = ax1.bar(sources, counts, color=colors)
    ax1.set_ylabel("Trade Count")
    ax1.set_title("Trade Count Comparison")
    for bar, count in zip(bars, counts):
        ax1.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 100, 
                 str(count), ha="center", va="bottom", fontsize=12)
    
    # Plot 2: Gap analysis over time
    ax2 = axes[0, 1]
    if reconciliation["missing_in_selfbuilt"]:
        missing_df = pd.DataFrame(reconciliation["missing_in_selfbuilt"])
        missing_df["timestamp"].hist(ax=ax2, bins=50, color="#e74c3c", alpha=0.7)
        ax2.set_xlabel("Timestamp")
        ax2.set_ylabel("Missing Trade Count")
        ax2.set_title("Missing Trades in Self-Built (by time)")
    
    # Plot 3: Timestamp mismatch distribution
    ax3 = axes[1, 0]
    if reconciliation["timestamp_mismatches"]:
        diffs = [m["difference_ms"] for m in reconciliation["timestamp_mismatches"]]
        ax3.hist(diffs, bins=30, color="#f39c12", alpha=0.7)
        ax3.set_xlabel("Timestamp Difference (ms)")
        ax3.set_ylabel("Frequency")
        ax3.set_title("Timestamp Mismatch Distribution")
    
    # Plot 4: Issue breakdown
    ax4 = axes[1, 1]
    issue_types = ["Matched", "Missing\n(Self-Built)", "Missing\n(Tardis)", "Timestamp\nMismatches"]
    issue_counts = [
        reconciliation["matched_trades"],
        len(reconciliation["missing_in_selfbuilt"]),
        len(reconciliation["missing_in_tardis"]),
        len(reconciliation["timestamp_mismatches"])
    ]
    colors = ["#2ecc71", "#e74c3c", "#9b59b6", "#f39c12"]
    ax4.bar(issue_types, issue_counts, color=colors)
    ax4.set_ylabel("Trade Count")
    ax4.set_title("Trade Data Quality Breakdown")
    
    plt.tight_layout()
    plt.savefig("reconciliation_report.png", dpi=150)
    plt.show()
    print("Report saved to reconciliation_report.png")

visualize_discrepancies(reconciliation, tardis_df)

Tardis.dev vs. Self-Built Collectors: Feature Comparison

Feature Tardis.dev Self-Built Collector
Setup Time ~15 minutes (API key + client library) 2-4 weeks (infrastructure, error handling, monitoring)
Data Normalization Fully normalized schema, consistent across exchanges Custom schema, requires manual normalization
Uptime Guarantee 99.9% SLA with redundancy Depends on your infrastructure reliability
Historical Depth Up to 2+ years depending on plan Only data collected since deployment
Cost Subscription starting at $49/month Infrastructure costs only (~$20-100/month for VPS)
Latency Typically 100-300ms for historical queries <50ms for real-time WebSocket
Maintenance Handled by Tardis team Full ownership, your responsibility
Customization Limited to provided endpoints Complete flexibility
Reconnection Logic Built-in, automatic Must implement manually
Compliance/Audit Trail Documented data provenance You build the audit trail yourself

Who This Guide Is For

This Guide is For:

This Guide is NOT For:

Pricing and ROI

Let us analyze the cost-effectiveness of each approach for a typical quantitative research team processing 10 million trades per month.

Cost Factor Tardis.dev (Pro Plan) Self-Built Collector
Monthly Subscription/Hosting $299/month $45/month (t3.medium AWS)
Data Transfer Included $15/month (estimated)
Engineering Hours (Setup) 4 hours @ $150/hr = $600 one-time 120 hours @ $150/hr = $18,000 one-time
Engineering Hours (Monthly Maintenance) 1 hour/month 10 hours/month
Total First Year Cost $4,188 $20,445
Data Quality Score 95% (industry-leading) Variable (70-95%)

ROI Analysis: Using Tardis.dev saves approximately $16,257 in the first year when accounting for engineering time. For HolySheep AI users, the economics are even more favorable. HolySheep charges $1 per dollar of API usage (¥1 = $1 USD), saving 85%+ compared to domestic alternatives priced at ¥7.3 per dollar. For data processing tasks that utilize HolySheep's inference capabilities for anomaly detection, you receive free credits on signup plus WeChat/Alipay payment support for seamless transactions.

Why Choose HolySheep AI for Data Processing

After building reconciliation pipelines with multiple tools, I consistently return to HolySheep AI for several critical reasons that directly impact data quality workflows:

1. Native Integration with Market Data Pipelines
HolySheep's inference API processes reconciliation results at <50ms latency, enabling real-time anomaly detection as data flows through your pipeline. I use HolySheep to run classification models that identify suspicious trade patterns—like sudden gaps in data that indicate collector failures—within milliseconds of the gap occurring.

2. Cost Efficiency for High-Volume Processing
When processing 10 million Hyperliquid trades for quality analysis, HolySheep's pricing model delivers measurable savings. GPT-4.1 inference costs $8 per million tokens, while alternatives like Claude Sonnet 4.5 charge $15 per million tokens. For trade reconciliation reports that generate thousands of tokens per analysis, this compounds into significant monthly savings.

3. Flexible Payment Infrastructure
HolySheep supports WeChat Pay and Alipay alongside standard credit cards, which simplifies procurement for teams based in Asia or working with Asian exchange data. The free registration credit lets you validate integration before committing budget.

4. Production-Ready Reliability
HolySheep maintains 99.95% uptime for API endpoints, ensuring your reconciliation pipeline never stalls waiting for inference results. For compliance-critical applications where missed data gaps have regulatory implications, this reliability is non-negotiable.

Common Errors and Fixes

Error 1: "403 Forbidden" When Accessing Tardis.dev Historical API

Cause: Your Tardis.dev API key lacks permission for historical data endpoints. Free tier accounts only access real-time streams.

Solution: Upgrade your Tardis.dev plan or verify your API key permissions:

import requests


Verify API key permissions

TARDIS_API_KEY = "YOUR_TARDIS_API_KEY"

response = requests.get(
    "https://api.tardis.dev/v1/account",
    headers={"Authorization": f"Bearer {TARDIS_API_KEY}"}
)

print(f"Account type: {response.json().get('plan', 'unknown')}")
print(f"Permissions: {response.json().get('permissions', [])}")


If historical data is required, update your plan:


https://tardis.dev/plans

Error 2: Timestamp Mismatch Exceeding Tolerance Threshold

Cause: Your self-built collector processes timestamps in the wrong timezone or uses seconds instead of milliseconds.

Solution: Standardize timestamp handling across both data sources:

import pandas as pd
from datetime import timezone

def standardize_timestamps(df: pd.DataFrame, source: str) -> pd.DataFrame:
    """
    Normalize timestamps to UTC milliseconds for consistent comparison.
    """
    df = df.copy()
    
    if source == "hyperliquid_websocket":
        # Hyperliquid WebSocket sends nanoseconds since epoch
        df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ns", utc=True)
    elif source == "tardis":
        # Tardis normalizes to milliseconds
        df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms", utc=True)
    elif source == "selfbuilt":
        # Check your collector's timestamp format
        if df["timestamp"].max() > 1e12:  # Nanoseconds
            df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ns", utc=True)
        else:  # Milliseconds
            df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms", utc=True)
    
    # Ensure consistent formatting
    df["timestamp"] = df["timestamp"].dt.tz_convert(None)  # Remove timezone for comparison
    
    return df


Apply standardization before reconciliation

tardis_std = standardize_timestamps(tardis_df.copy(), source="tardis")
selfbuilt_std = standardize_timestamps(selfbuilt_df.copy(), source="selfbuilt")

Error 3: WebSocket Connection Drops Causing Data Gaps

Cause: Your self-built collector does not implement exponential backoff reconnection, causing it to miss trades during network interruptions.

Solution: Implement robust WebSocket client with automatic reconnection:

import asyncio
import websockets
import json
from datetime import datetime

class HyperliquidCollector:
    def __init__(self, symbol: str, on_trade_callback):
        self.symbol = symbol
        self.on_trade_callback = on_trade_callback
        self.ws_url = "wss://stream.hyperliquid.xyz/Info"
        self.max_reconnect_attempts = 10
        self.base_delay = 1  # seconds
    
    async def connect(self):
        reconnect_attempts = 0
        
        while reconnect_attempts < self.max_reconnect_attempts:
            try:
                async with websockets.connect(self.ws_url) as ws:
                    # Subscribe to trade channel
                    subscribe_msg = {
                        "method": "subscribe",
                        "subscription": {"type": "trades", "symbol": self.symbol}
                    }
                    await ws.send(json.dumps(subscribe_msg))
                    
                    print(f"Connected and subscribed to {self.symbol} trades")
                    reconnect_attempts = 0  # Reset on successful connection
                    
                    async for message in ws:
                        data = json.loads(message)
                        if "data" in data and "trades" in data["data"]:
                            for trade in data["data"]["trades"]:
                                await self.on_trade_callback(trade)
            
            except websockets.ConnectionClosed as e:
                reconnect_attempts += 1
                delay = self.base_delay * (2 ** reconnect_attempts)  # Exponential backoff
                print(f"Connection closed: {e}. Reconnecting in {delay}s (attempt {reconnect_attempts})")
                await asyncio.sleep(delay)
            
            except Exception as e:
                print(f"Unexpected error: {e}")
                reconnect_attempts += 1
                await asyncio.sleep(self.base_delay * (2 ** reconnect_attempts))
        
        raise RuntimeError("Max reconnection attempts reached")


Usage

collector = HyperliquidCollector("BTC-PERPETUAL", on_trade_callback=process_trade)
asyncio.run(collector.connect())

Error 4: Duplicate Trade IDs After Reconciliation

Cause: Hyperliquid occasionally emits duplicate trade events, and your collector does not deduplicate them properly.

Solution: Implement trade deduplication using trade IDs:

def deduplicate_trades(df: pd.DataFrame, id_column: str = "tid") -> pd.DataFrame:
    """
    Remove duplicate trades based on unique trade ID.
    Keeps the first occurrence and removes subsequent duplicates.
    """
    initial_count = len(df)
    df_deduped = df.drop_duplicates(subset=[id_column], keep="first")
    removed_count = initial_count - len(df_deduped)
    
    if removed_count > 0:
        print(f"Removed {removed_count} duplicate trades ({removed_count/initial_count*100:.2f}%)")
    
    return df_deduped


Apply deduplication before reconciliation

tardis_deduped = deduplicate_trades(tardis_df, id_column="tid")
selfbuilt_deduped = deduplicate_trades(selfbuilt_df, id_column="trade_id")


Rerun reconciliation with cleaned data

reconciliation_clean = reconcile_trade_data(tardis_deduped, selfbuilt_deduped)

Conclusion and Next Steps

Data quality checking for Hyperliquid perpetual futures is not optional—it is a critical component of any serious quantitative research or trading operation. The reconciliation methodology outlined in this guide gives you a systematic approach to identifying gaps, timestamp mismatches, and duplicate data before they corrupt your models.

The choice between Tardis.dev and self-built collectors is ultimately a trade-off between time-to-market and long-term cost. For most teams, a hybrid approach works best: use Tardis.dev for historical data and as a validation benchmark, while building lightweight collectors for real-time data where latency matters.

HolySheep AI accelerates this entire workflow. Whether you need to run anomaly detection on reconciliation results, generate automated quality reports, or process millions of trades through LLM-based classification, HolySheep provides the infrastructure at a fraction of the cost of alternatives.

Start with the free credits on registration, integrate your first reconciliation pipeline, and experience the <50ms latency and 85%+ cost savings firsthand.

👉 Sign up for HolySheep AI — free credits on registration

About the Author: This guide was written by the HolySheep AI engineering team based on production experience building data quality pipelines for cryptocurrency perpetual futures markets.

Related Resources

Related Articles