Last validated: May 30, 2026 | Estimated read time: 18 minutes | HolySheep Technical Documentation
Introduction: Why Your AI API SLA Dashboard Matters More Than You Think
I launched an e-commerce AI customer service chatbot last quarter, and for the first two weeks, I thought everything was fine. Response times felt snappy, customers seemed happy, and my error logs showed minimal issues. Then I ran my first P99 latency analysis—and discovered that 1% of my requests were taking over 8 seconds to complete. During peak traffic, that meant 1 in every 100 customers was experiencing a timeout-level delay. My customer satisfaction scores reflected exactly what my monitoring had missed: not the median, but the outliers.
This tutorial walks you through building a comprehensive HolySheep API SLA monitoring dashboard that captures real-time and historical P50/P95/P99 latency metrics, error rates, and uptime statistics. Whether you're running an enterprise RAG system serving thousands of concurrent users or an indie developer project with intermittent traffic, this template gives you production-grade observability for your AI API layer—at a fraction of what enterprise monitoring tools cost.
HolySheep AI provides sub-50ms gateway latency with native support for WeChat and Alipay payments, and their unified API supports GPT-4.1, Claude Sonnet 4.5, Gemini 2.5 Flash, and DeepSeek V3.2 models. Sign up here to access free credits and start building with a platform that charges ¥1=$1 (saving 85%+ compared to domestic alternatives at ¥7.3).
Understanding SLA Metrics: P50, P95, and P99 in Production AI Contexts
Before diving into implementation, let's clarify what these percentile metrics mean for your AI API integration:
- P50 (Median): 50% of requests complete faster than this time. This is your "typical" user experience.
- P95: 95% of requests complete faster than this time. This captures your power users and high-complexity queries.
- P99: 99% of requests complete faster than this time. This is your SLA commitment floor—every user should experience this or better.
For a production AI customer service system, realistic targets look like this:
| Metric | Target | Acceptable | Critical |
|---|---|---|---|
| P50 Latency | <200ms | 200-500ms | >500ms |
| P95 Latency | <800ms | 800-2000ms | >2000ms |
| P99 Latency | <3000ms | 3000-8000ms | >8000ms |
| Error Rate | <0.1% | 0.1-1% | >1% |
| Uptime | >99.9% | 99-99.9% | <99% |
Architecture Overview
Our monitoring solution consists of four layers:
┌─────────────────────────────────────────────────────────────┐
│ MONITORING LAYER │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ Prometheus │ │ Grafana │ │ Alert Manager │ │
│ │ (Metrics) │ │ (Dashboard) │ │ (Paging) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ COLLECTION LAYER │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Python Collector + FastAPI Wrapper │ │
│ │ - Request timing interceptor │ │
│ │ - Error classification (4xx/5xx/timeouts) │ │
│ │ - Histogram bucketing for percentile calculation │ │
│ └─────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ API LAYER (HolySheep) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ base_url: https://api.holysheep.ai/v1 │ │
│ │ Supported: GPT-4.1, Claude Sonnet 4.5, Gemini 2.5 │ │
│ │ Gateway Latency: <50ms │ │
│ └─────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ DATA STORAGE │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ TimescaleDB │ │ InfluxDB │ │ Redis (hot cache) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘Step 1: Installing Dependencies and Setting Up the Environment
Create a new Python project and install the required monitoring stack. We recommend Python 3.10+ for optimal async performance:
mkdir holysheep-sla-monitor
cd holysheep-sla-monitor
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install httpx prometheus-client fastapi uvicorn
pip install pandas numpy structlog timescalesDB-driver
Optional: for visualization
pip install streamlit plotly dash
Verify installation
python -c "import httpx, prometheus_client; print('Dependencies OK')"Create a .env file with your HolySheep API credentials:
HOLYSHEEP_API_KEY=YOUR_HOLYSHEEP_API_KEY
HOLYSHEEP_BASE_URL=https://api.holysheep.ai/v1
LOG_LEVEL=INFO
PROMETHEUS_PORT=9090
METRICS_WINDOW_SECONDS=3600Step 2: Building the Metrics Collection Engine
The core of your SLA dashboard is the metrics collector that intercepts every API call, measures latency with microsecond precision, and classifies errors. Here's a production-ready implementation:
import httpx
import time
import structlog
from prometheus_client import Counter, Histogram, Gauge, start_http_server
from datetime import datetime
from typing import Optional, Dict, Any
import asyncio
import numpy as np
from collections import deque
import threading
logger = structlog.get_logger()
Prometheus metrics definitions
REQUEST_COUNT = Counter(
'holysheep_requests_total',
'Total API requests to HolySheep',
['model', 'endpoint', 'status_code']
)
REQUEST_LATENCY = Histogram(
'holysheep_request_latency_seconds',
'Request latency in seconds',
['model', 'endpoint'],
buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
)
ERROR_RATE = Counter(
'holysheep_errors_total',
'Total errors by type',
['model', 'error_type', 'status_code']
)
ACTIVE_REQUESTS = Gauge(
'holysheep_active_requests',
'Currently processing requests',
['model']
)
class SLAMetricsCollector:
"""
HolySheep API SLA Metrics Collector
Captures P50/P95/P99 latency, error rates, and uptime metrics
with thread-safe sliding window calculations.
"""
def __init__(self, window_size: int = 3600):
self.window_size = window_size # 1-hour sliding window
self._lock = threading.Lock()
self._latencies: Dict[str, deque] = {} # model -> list of latencies
self._request_counts: Dict[str, Dict[str, int]] = {} # model -> {status: count}
def record_request(
self,
model: str,
latency_ms: float,
status_code: int,
endpoint: str = "/chat/completions"
):
"""Thread-safe request recording"""
with self._lock:
key = f"{model}:{endpoint}"
# Initialize data structures if needed
if key not in self._latencies:
self._latencies[key] = deque(maxlen=self.window_size)
self._request_counts[key] = {}
# Record latency (convert to seconds for histogram)
self._latencies[key].append(latency_ms / 1000.0)
# Increment counters
REQUEST_LATENCY.labels(model=model, endpoint=endpoint).observe(latency_ms / 1000.0)
REQUEST_COUNT.labels(model=model, endpoint=endpoint, status_code=str(status_code)).inc()
if status_code >= 400:
error_type = self._classify_error(status_code)
ERROR_RATE.labels(model=model, error_type=error_type, status_code=str(status_code)).inc()
# Track request counts for uptime calculation
self._request_counts[key][str(status_code)] = \
self._request_counts[key].get(str(status_code), 0) + 1
def _classify_error(self, status_code: int) -> str:
"""Classify error type for alerting"""
if status_code == 400:
return "bad_request"
elif status_code == 401:
return "auth_failure"
elif status_code == 429:
return "rate_limited"
elif status_code == 500:
return "server_error"
elif status_code == 503:
return "service_unavailable"
return "unknown_error"
def calculate_percentiles(self, model: str) -> Dict[str, float]:
"""Calculate P50, P95, P99 for a given model"""
with self._lock:
all_latencies = []
for key, latencies in self._latencies.items():
if key.startswith(model):
all_latencies.extend(latencies)
if not all_latencies:
return {"p50": 0, "p95": 0, "p99": 0}
sorted_latencies = np.sort(all_latencies)
n = len(sorted_latencies)
return {
"p50": float(np.percentile(sorted_latencies, 50)) * 1000, # Back to ms
"p95": float(np.percentile(sorted_latencies, 95)) * 1000,
"p99": float(np.percentile(sorted_latencies, 99)) * 1000,
"count": n
}
def get_error_rate(self, model: str) -> float:
"""Calculate error rate percentage"""
with self._lock:
total = 0
errors = 0
for key, counts in self._request_counts.items():
if key.startswith(model):
for status, count in counts.items():
total += count
if int(status) >= 400:
errors += count
return (errors / total * 100) if total > 0 else 0.0
Global metrics collector instance
metrics = SLAMetricsCollector(window_size=3600)
class HolySheepMonitoredClient:
"""
Wrapped HolySheep API client with automatic SLA metrics collection.
Usage:
client = HolySheepMonitoredClient(api_key="YOUR_HOLYSHEEP_API_KEY")
response = await client.chat_completions(model="gpt-4.1", messages=[...])
"""
BASE_URL = "https://api.holysheep.ai/v1"
def __init__(self, api_key: str):
self.api_key = api_key
self._client = httpx.AsyncClient(
base_url=self.BASE_URL,
timeout=httpx.Timeout(60.0, connect=10.0),
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
)
logger.info("HolySheepMonitoredClient initialized", base_url=self.BASE_URL)
async def chat_completions(
self,
model: str,
messages: list,
temperature: float = 0.7,
max_tokens: Optional[int] = None,
**kwargs
) -> Dict[str, Any]:
"""Send chat completion request with metrics collection"""
ACTIVE_REQUESTS.labels(model=model).inc()
start_time = time.perf_counter()
error_status = 200
try:
payload = {
"model": model,
"messages": messages,
"temperature": temperature
}
if max_tokens:
payload["max_tokens"] = max_tokens
payload.update(kwargs)
response = await self._client.post(
"/chat/completions",
json=payload
)
error_status = response.status_code
response.raise_for_status()
result = response.json()
elapsed_ms = (time.perf_counter() - start_time) * 1000
# Record successful request
metrics.record_request(
model=model,
latency_ms=elapsed_ms,
status_code=error_status
)
logger.info(
"HolySheep request completed",
model=model,
latency_ms=round(elapsed_ms, 2),
status_code=error_status
)
return result
except httpx.TimeoutException:
elapsed_ms = (time.perf_counter() - start_time) * 1000
metrics.record_request(model=model, latency_ms=elapsed_ms, status_code=504)
logger.error("Request timeout", model=model, latency_ms=round(elapsed_ms, 2))
raise
except httpx.HTTPStatusError as e:
elapsed_ms = (time.perf_counter() - start_time) * 1000
metrics.record_request(
model=model,
latency_ms=elapsed_ms,
status_code=e.response.status_code
)
logger.error(
"HTTP error",
model=model,
status_code=e.response.status_code,
response=e.response.text[:500]
)
raise
finally:
ACTIVE_REQUESTS.labels(model=model).dec()
async def close(self):
await self._client.aclose()Step 3: Running a Multi-Hour Load Test with Real-Time Percentile Calculations
Now let's create a load test script that simulates production traffic patterns and measures your actual SLA metrics. This script generates realistic request patterns with varying complexity:
#!/usr/bin/env python3
"""
HolySheep SLA Load Test & Monitoring Script
Simulates production traffic patterns and continuously reports:
- P50/P95/P99 latency by model
- Error rates and error type distribution
- Request throughput (RPM/RPS)
- Cost estimation based on actual token usage
Usage:
python sla_load_test.py --duration 3600 --models gpt-4.1 claude-sonnet-4.5
"""
import asyncio
import argparse
import random
import time
import json
from datetime import datetime, timedelta
from typing import List
import structlog
logger = structlog.get_logger()
Sample conversation templates for realistic load testing
CONVERSATION_TEMPLATES = [
# E-commerce customer service
[
{"role": "user", "content": "Hi, I placed an order #12345 yesterday but it hasn't shipped yet. Can you check the status?"},
{"role": "assistant", "content": "I'd be happy to help you check on order #12345. Let me look that up for you."},
{"role": "user", "content": "Also, can I change the shipping address to a different city?"}
],
# Technical support
[
{"role": "user", "content": "Our integration with your API is returning 401 errors randomly. We've verified the API key is correct."},
{"role": "assistant", "content": "I understand how frustrating intermittent authentication errors can be. Let me help troubleshoot."},
{"role": "user", "content": "It happens about 5% of the time. We're calling the API from a Kubernetes cluster with multiple replicas."}
],
# RAG system query
[
{"role": "system", "content": "You are a technical documentation assistant. Use the provided context to answer questions accurately."},
{"role": "user", "content": "Based on the API documentation, how do I implement streaming responses with proper error handling?"}
]
]
COMPLEXITY_LEVELS = [
{"name": "simple", "tokens": (50, 150), "weight": 0.4},
{"name": "moderate", "tokens": (150, 500), "weight": 0.4},
{"name": "complex", "tokens": (500, 2000), "weight": 0.2}
]
async def generate_request_payload(include_history: bool = True):
"""Generate realistic request payload"""
template = random.choice(CONVERSATION_TEMPLATES)
if include_history and random.random() > 0.3:
# Multi-turn conversation
messages = template.copy()
# Add a new question
messages.append({
"role": "assistant",
"content": "I understand. Let me provide more details about that."
})
messages.append({
"role": "user",
"content": "Can you elaborate on the second point?"
})
else:
# Single turn
messages = [{"role": "user", "content": template[0]["content"]}]
return messages
async def run_sla_test(
api_key: str,
models: List[str],
duration_seconds: int,
target_rps: float = 10.0
):
"""Execute SLA load test with continuous monitoring"""
from main import HolySheepMonitoredClient, metrics
client = HolySheepMonitoredClient(api_key)
# Start Prometheus metrics server on port 9090
start_http_server(9090)
logger.info("Prometheus metrics server started on :9090")
start_time = time.time()
end_time = start_time + duration_seconds
request_interval = 1.0 / target_rps
print("\n" + "="*80)
print("HOLYSHEEP API SLA MONITORING DASHBOARD")
print("="*80)
print(f"Start Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
print(f"Duration: {duration_seconds}s | Target RPS: {target_rps}")
print(f"Models: {', '.join(models)}")
print("="*80 + "\n")
total_requests = 0
successful_requests = 0
failed_requests = 0
total_cost = 0.0
# Model pricing (2026 rates in $/1M tokens input/output)
MODEL_PRICING = {
"gpt-4.1": {"input": 2.00, "output": 8.00},
"claude-sonnet-4.5": {"input": 3.00, "output": 15.00},
"gemini-2.5-flash": {"input": 0.35, "output": 2.50},
"deepseek-v3.2": {"input": 0.14, "output": 0.42}
}
async def single_request(model: str):
nonlocal total_requests, successful_requests, failed_requests, total_cost
try:
messages = await generate_request_payload()
# Estimate tokens (rough approximation)
complexity = random.choices(
COMPLEXITY_LEVELS,
weights=[c["weight"] for c in COMPLEXITY_LEVELS]
)[0]
estimated_input_tokens = random.randint(*complexity["tokens"])
estimated_output_tokens = random.randint(
complexity["tokens"][0] // 2,
complexity["tokens"][1]
)
response = await client.chat_completions(
model=model,
messages=messages,
max_tokens=estimated_output_tokens
)
# Calculate actual cost if usage info available
if "usage" in response:
usage = response["usage"]
input_tokens = usage.get("prompt_tokens", estimated_input_tokens)
output_tokens = usage.get("completion_tokens", estimated_output_tokens)
if model in MODEL_PRICING:
pricing = MODEL_PRICING[model]
input_cost = (input_tokens / 1_000_000) * pricing["input"]
output_cost = (output_tokens / 1_000_000) * pricing["output"]
total_cost += input_cost + output_cost
successful_requests += 1
except Exception as e:
failed_requests += 1
logger.error("Request failed", model=model, error=str(e))
total_requests += 1
# Reporting task
async def print_dashboard():
while time.time() < end_time:
await asyncio.sleep(30) # Update every 30 seconds
elapsed = time.time() - start_time
uptime = (successful_requests / total_requests * 100) if total_requests > 0 else 100
print(f"\n[{datetime.now().strftime('%H:%M:%S')}] --- 30-Second SLA Snapshot ---")
print(f"Requests: {total_requests} | Success: {successful_requests} | Failed: {failed_requests}")
print(f"Uptime: {uptime:.2f}% | Elapsed: {elapsed:.0f}s | Cost: ${total_cost:.4f}")
print("-" * 60)
for model in models:
percentiles = metrics.calculate_percentiles(model)
error_rate = metrics.get_error_rate(model)
print(f"\n {model.upper()}:")
print(f" P50 Latency: {percentiles['p50']:.1f}ms")
print(f" P95 Latency: {percentiles['p95']:.1f}ms")
print(f" P99 Latency: {percentiles['p99']:.1f}ms")
print(f" Error Rate: {error_rate:.3f}%")
print("\n" + "="*80)
# Start dashboard task
dashboard_task = asyncio.create_task(print_dashboard())
# Main request loop
request_tasks = []
while time.time() < end_time:
model = random.choice(models)
request_tasks.append(asyncio.create_task(single_request(model)))
# Batch requests to maintain target RPS
if len(request_tasks) >= 10:
await asyncio.gather(*request_tasks, return_exceptions=True)
request_tasks = []
await asyncio.sleep(request_interval)
# Wait for remaining tasks
if request_tasks:
await asyncio.gather(*request_tasks, return_exceptions=True)
# Wait for dashboard to finish
await asyncio.sleep(2)
dashboard_task.cancel()
# Final report
print("\n" + "="*80)
print("FINAL SLA REPORT")
print("="*80)
print(f"Total Requests: {total_requests}")
print(f"Successful: {successful_requests}")
print(f"Failed: {failed_requests}")
print(f"Overall Success Rate: {successful_requests/total_requests*100:.2f}%")
print(f"Total Cost: ${total_cost:.4f}")
print("-" * 60)
for model in models:
percentiles = metrics.calculate_percentiles(model)
error_rate = metrics.get_error_rate(model)
print(f"\n {model.upper()} Metrics:")
print(f" P50 (Median): {percentiles['p50']:.1f}ms")
print(f" P95 (95th): {percentiles['p95']:.1f}ms")
print(f" P99 (99th): {percentiles['p99']:.1f}ms")
print(f" Sample Size: {percentiles['count']}")
print(f" Error Rate: {error_rate:.3f}%")
await client.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="HolySheep API SLA Load Test")
parser.add_argument("--api-key", default="YOUR_HOLYSHEEP_API_KEY")
parser.add_argument("--models", nargs="+",
default=["gpt-4.1", "deepseek-v3.2"],
help="Models to test")
parser.add_argument("--duration", type=int, default=300,
help="Test duration in seconds")
parser.add_argument("--rps", type=float, default=5.0,
help="Target requests per second")
args = parser.parse_args()
asyncio.run(run_sla_test(
api_key=args.api_key,
models=args.models,
duration_seconds=args.duration,
target_rps=args.rps
))Step 4: Building the Grafana Dashboard JSON Template
Import this JSON template into Grafana to visualize your SLA metrics in real-time. Navigate to Grafana → Dashboards → Import and paste this configuration:
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"title": "P50/P95/P99 Latency by Model",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
"targets": [
{
"expr": "histogram_quantile(0.50, sum(rate(holysheep_request_latency_seconds_bucket[5m])) by (le, model)) * 1000",
"legendFormat": "{{model}} P50",
"refId": "A"
},
{
"expr": "histogram_quantile(0.95, sum(rate(holysheep_request_latency_seconds_bucket[5m])) by (le, model)) * 1000",
"legendFormat": "{{model}} P95",
"refId": "B"
},
{
"expr": "histogram_quantile(0.99, sum(rate(holysheep_request_latency_seconds_bucket[5m])) by (le, model)) * 1000",
"legendFormat": "{{model}} P99",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "ms",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 500},
{"color": "orange", "value": 1000},
{"color": "red", "value": 3000}
]
}
}
}
},
{
"title": "Error Rate %",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
"targets": [
{
"expr": "sum(rate(holysheep_errors_total[5m])) by (model, error_type) / sum(rate(holysheep_requests_total[5m])) by (model) * 100",
"legendFormat": "{{model}} - {{error_type}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 0.1},
{"color": "red", "value": 1}
]
}
}
}
},
{
"title": "Request Throughput (RPM)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 8, "x": 0, "y": 8},
"targets": [
{
"expr": "sum(rate(holysheep_requests_total[1m])) by (model) * 60",
"legendFormat": "{{model}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqpm",
"thresholds": {
"mode": "absolute",
"steps": [{"color": "green", "value": null}]
}
}
}
},
{
"title": "Active Requests",
"type": "gauge",
"gridPos": {"h": 8, "w": 8, "x": 8, "y": 8},
"targets": [
{
"expr": "sum(holysheep_active_requests)",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 50},
{"color": "red", "value": 100}
]
}
}
}
},
{
"title": "Cost per Hour ($)",
"type": "stat",
"gridPos": {"h": 8, "w": 8, "x": 16, "y": 8},
"targets": [
{
"expr": "holysheep_estimated_cost_per_hour",
"refId": "A"
}
],
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": ["lastNotNull"],
"fields": ""
}
}
}
],
"schemaVersion": 27,
"style": "dark",
"tags": ["holySheep", "API", "SLA", "monitoring"],
"templating": {
"list": [
{
"name": "model",
"type": "query",
"query": "label_values(holysheep_requests_total, model)"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"title": "HolySheep API SLA Dashboard",
"uid": "holysheep-sla-001",
"version": 1
}Step 5: Integrating with AlertManager for SLA Violations
Configure Prometheus AlertManager to page your team when SLA thresholds are breached. Add this to your Prometheus rules configuration:
groups:
- name: holySheep-sla-alerts
interval: 30s
rules:
# P99 Latency Alert - Critical
- alert: HolySheepP99LatencyCritical
expr: histogram_quantile(0.99, sum(rate(holysheep_request_latency_seconds_bucket[5m])) by (le, model)) > 3
for: 5m
labels:
severity: critical
service: holySheep-api
annotations:
summary: "HolySheep P99 latency exceeds 3 seconds"
description: "Model {{ $labels.model }} P99 latency is {{ $value | humanizeDuration }} (threshold: 3s)"
# Error Rate Alert - Warning
- alert: HolySheepErrorRateHigh
expr: sum(rate(holysheep_errors_total[5m])) by (model) / sum(rate(holysheep_requests_total[5m])) by (model) > 0.01
for: 2m
labels:
severity: warning
service: holySheep-api
annotations:
summary: "HolySheep error rate exceeds 1%"
description: "Model {{ $labels.model }} error rate is {{ $value | humanizePercentage }}"
# Service Down Alert
- alert: HolySheepServiceDown
expr: sum(rate(holysheep_requests_total[5m])) by () < 1
for: 1m
labels:
severity: critical
service: holySheep-api
annotations:
summary: "HolySheep API appears to be down or unreachable"
description: "No successful requests in the last 5 minutes"
# Rate Limit Alert
- alert: HolySheepRateLimited
expr: sum(rate(holysheep_errors_total{error_type="rate_limited"}[5m])) by () > 0
for: 30s
labels:
severity: warning
service: holySheep-api
annotations:
summary: "Rate limiting detected from HolySheep API"
description: "Your application is being rate limited. Consider implementing exponential backoff."
AlertManager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
route:
group_by: ['alertname', 'severity']
group_wait: 10s
group_interval: 10s
repeat_interval: 12h
receiver: 'team-notifications'
routes:
- match:
severity: critical
receiver: 'pagerduty-critical'
group_wait: 0s
- match:
severity: warning
receiver: 'slack-notifications'
continue: trueLong-Term Data Retention and Trend Analysis
For 30-day and 90-day SLA trend analysis, configure TimescaleDB to store your metrics with automatic data compression. This allows you to track SLA improvements over time and generate monthly compliance reports:
-- Create hypertable for long-term metric storage
CREATE TABLE holysheep_sla_metrics (