Tháng 3/2026, tôi triển khai hệ thống RAG cho một doanh nghiệp thương mại điện tử với 50K người dùng hàng ngày. Đỉnh điểm Black Friday, API response time tăng từ 200ms lên 8 giây — khách hàng chờ đợi, đội ngũ hoảng loạn, và tôi nhận ra: không có monitoring = không có kiểm soát. Bài viết này chia sẻ cách tôi xây dựng production monitoring hoàn chỉnh cho HolySheep AI multi-model gateway với Datadog và Grafana, giúp bạn phát hiện vấn đề trước khi người dùng phàn nàn.
Vì sao cần giám sát API Gateway?
Khi làm việc với multi-model AI gateway, bạn đối mặt với những thách thức đặc thù:
- Latency biến động: Mỗi model (GPT-4.1, Claude Sonnet 4.5, Gemini 2.5 Flash, DeepSeek V3.2) có thời gian response khác nhau, từ 50ms đến 30 giây
- 5xx errors không thể đoán trước: Model provider downtime, rate limiting, quota exceeded
- Cost spike: Token usage tăng đột biến có thể khiến chi phí vượt tầm kiểm soát
- Model routing thất bại: Fallback không hoạt động đúng cách
Trong dự án thương mại điện tử đó, tôi mất 4 tiếng để identify nguyên nhân root cause vì thiếu metrics. Sau đó, với monitoring setup chuẩn, tôi phát hiện và xử lý vấn đề trong vòng 15 phút.
Kiến trúc Monitoring System
Trước khi vào code, hãy hiểu kiến trúc tổng thể:
┌─────────────────────────────────────────────────────────────────┐
│ HOLYSHEEP AI GATEWAY │
│ base_url: https://api.holysheep.ai/v1 │
├─────────────────────────────────────────────────────────────────┤
│ Request Flow: │
│ Client → API Gateway → Model Router → [Model Pool] │
│ ↓ │
│ Metrics Collector ← Prometheus/StatsD │
│ ↓ │
│ Datadog / Grafana ← Alert Manager │
└─────────────────────────────────────────────────────────────────┘
Model Pool:
├── GPT-4.1 (OpenAI compatible) P95: 800ms, Cost: $8/MTok
├── Claude Sonnet 4.5 (Anthropic) P95: 1200ms, Cost: $15/MTok
├── Gemini 2.5 Flash (Google) P95: 200ms, Cost: $2.50/MTok
└── DeepSeek V3.2 (DeepSeek) P95: 150ms, Cost: $0.42/MTok
1. Cài đặt Prometheus Metrics Collector
Đầu tiên, tôi tạo một middleware để expose Prometheus metrics từ API requests. Đây là phần quan trọng nhất — nếu không có metrics, thì không có gì để giám sát.
# metrics_server.py - Prometheus metrics collector
Install: pip install prometheus-client fastapi uvicorn httpx
from prometheus_client import Counter, Histogram, Gauge, start_http_server
from fastapi import FastAPI, Request, Response
from fastapi.responses import JSONResponse
import httpx
import time
import os
from typing import Optional
============================================
HOLYSHEEP API CONFIGURATION
============================================
HOLYSHEEP_BASE_URL = "https://api.holysheep.ai/v1"
HOLYSHEEP_API_KEY = os.getenv("YOUR_HOLYSHEEP_API_KEY", "sk-holysheep-xxxxx")
============================================
METRICS DEFINITIONS
============================================
Request counter by model, status code
request_counter = Counter(
'holysheep_requests_total',
'Total API requests',
['model', 'status_code', 'endpoint']
)
Latency histogram in milliseconds
latency_histogram = Histogram(
'holysheep_request_latency_ms',
'Request latency in milliseconds',
['model', 'endpoint'],
buckets=[50, 100, 200, 500, 1000, 2000, 5000, 10000, 30000]
)
Token usage gauge
token_usage = Counter(
'holysheep_tokens_total',
'Total tokens used',
['model', 'token_type'] # token_type: prompt, completion
)
5xx error counter
error_5xx_counter = Counter(
'holysheep_5xx_errors_total',
'Total 5xx errors',
['model', 'error_type']
)
Active requests gauge
active_requests = Gauge(
'holysheep_active_requests',
'Number of active requests',
['model']
)
Cost tracking
cost_accumulator = Counter(
'holysheep_cost_usd',
'Accumulated cost in USD',
['model']
)
============================================
MODEL PRICING (2026 rates)
============================================
MODEL_PRICING = {
'gpt-4.1': {'prompt': 0.000002, 'completion': 0.000006}, # $8/MTok
'claude-sonnet-4.5': {'prompt': 0.000003, 'completion': 0.000012}, # $15/MTok
'gemini-2.5-flash': {'prompt': 0.000000125, 'completion': 0.0000005}, # $2.50/MTok
'deepseek-v3.2': {'prompt': 0.000000021, 'completion': 0.00000007}, # $0.42/MTok
}
============================================
API GATEWAY APPLICATION
============================================
app = FastAPI(title="HolySheep AI Gateway Monitored")
@app.api_route("/{full_path:path}", methods=["GET", "POST", "PUT", "DELETE"])
async def proxy_to_holysheep(request: Request, full_path: str):
"""
Proxy requests to HolySheep AI with full metrics collection
"""
start_time = time.perf_counter()
model = request.headers.get('X-Model', 'gpt-4.1')
active_requests.labels(model=model).inc()
try:
# Build target URL
target_url = f"{HOLYSHEEP_BASE_URL}/{full_path}"
# Prepare headers
headers = dict(request.headers)
headers['Authorization'] = f'Bearer {HOLYSHEEP_API_KEY}'
headers.pop('host', None)
# Get request body
body = await request.body()
# Forward request to HolySheep
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.request(
method=request.method,
url=target_url,
headers=headers,
content=body if body else None
)
# Calculate latency in milliseconds
latency_ms = (time.perf_counter() - start_time) * 1000
# Record metrics
status_code = str(response.status_code)
request_counter.labels(model=model, status_code=status_code, endpoint=full_path).inc()
latency_histogram.labels(model=model, endpoint=full_path).observe(latency_ms)
# Track 5xx errors
if response.status_code >= 500:
error_type = f"http_{response.status_code}"
error_5xx_counter.labels(model=model, error_type=error_type).inc()
# Extract and record token usage from response
try:
response_data = response.json()
usage = response_data.get('usage', {})
prompt_tokens = usage.get('prompt_tokens', 0)
completion_tokens = usage.get('completion_tokens', 0)
if prompt_tokens > 0:
token_usage.labels(model=model, token_type='prompt').inc(prompt_tokens)
cost = prompt_tokens * MODEL_PRICING.get(model, MODEL_PRICING['gpt-4.1'])['prompt']
cost_accumulator.labels(model=model).inc(cost)
if completion_tokens > 0:
token_usage.labels(model=model, token_type='completion').inc(completion_tokens)
cost = completion_tokens * MODEL_PRICING.get(model, MODEL_PRICING['gpt-4.1'])['completion']
cost_accumulator.labels(model=model).inc(cost)
except:
pass # Response might not be JSON or have usage field
return Response(
content=response.content,
status_code=response.status_code,
headers=dict(response.headers)
)
except httpx.TimeoutException:
latency_ms = (time.perf_counter() - start_time) * 1000
error_5xx_counter.labels(model=model, error_type='timeout').inc()
request_counter.labels(model=model, status_code='504', endpoint=full_path).inc()
latency_histogram.labels(model=model, endpoint=full_path).observe(latency_ms)
return JSONResponse(status_code=504, content={"error": "Gateway timeout"})
except Exception as e:
latency_ms = (time.perf_counter() - start_time) * 1000
error_5xx_counter.labels(model=model, error_type='internal_error').inc()
request_counter.labels(model=model, status_code='500', endpoint=full_path).inc()
latency_histogram.labels(model=model, endpoint=full_path).observe(latency_ms)
return JSONResponse(status_code=500, content={"error": str(e)})
finally:
active_requests.labels(model=model).dec()
@app.get("/metrics")
async def prometheus_metrics():
"""Prometheus metrics endpoint - automatically formatted by prometheus_client"""
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST
return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)
@app.get("/health")
async def health_check():
"""Health check endpoint"""
return {"status": "healthy", "service": "holysheep-monitored-gateway"}
============================================
STARTUP
============================================
if __name__ == "__main__":
import uvicorn
# Start Prometheus metrics server on port 9090
start_http_server(9090)
print("Prometheus metrics exposed on http://localhost:9090/metrics")
# Start FastAPI on port 8000
uvicorn.run(app, host="0.0.0.0", port=8000)
Run: python metrics_server.py
Test: curl http://localhost:8000/health
Metrics: curl http://localhost:9090/metrics
2. Cấu hình Datadog Dashboard
Datadog là lựa chọn tuyệt vời cho production monitoring với alerting thông minh. Tôi sử dụng Datadog APM + Custom Metrics để track P95 latency và 5xx error rates.
# datadog_monitor.py - Datadog integration
Install: pip install datadog
from datadog import initialize, statsd
import time
import os
============================================
DATADOG CONFIGURATION
============================================
options = {
'api_key': os.getenv('DD_API_KEY', 'your-datadog-api-key'),
'app_key': os.getenv('DD_APP_KEY', 'your-datadog-app-key'),
'statsd_host': os.getenv('DD_AGENT_HOST', 'localhost'),
'statsd_port': 8125,
}
initialize(**options)
============================================
CUSTOM METRICS FUNCTIONS
============================================
def record_request_metrics(model: str, latency_ms: float, status_code: int,
tokens_used: int = 0, cost_usd: float = 0.0):
"""
Record comprehensive request metrics to Datadog
"""
tags = [
f'model:{model}',
f'status_code:{status_code}',
'service:holysheep-gateway'
]
# Request count
statsd.increment('holysheep.requests', tags=tags)
# Latency histogram (Datadog will calculate P95, P99 automatically)
statsd.histogram('holysheep.latency', latency_ms, tags=tags)
# Token usage
if tokens_used > 0:
statsd.increment('holysheep.tokens', tokens_used, tags=tags + [f'type:{"prompt" if tokens_used < 1000 else "completion"}'])
# Cost tracking
if cost_usd > 0:
statsd.gauge('holysheep.cost', cost_usd, tags=tags)
# Error tracking
if status_code >= 500:
statsd.increment('holysheep.errors.5xx', tags=tags + [f'error_type:http_{status_code}'])
elif status_code >= 400:
statsd.increment('holysheep.errors.4xx', tags=tags)
def record_gateway_health(model: str, is_healthy: bool, response_time_ms: float):
"""
Record gateway health status
"""
tags = [f'model:{model}', 'service:holysheep-gateway']
statsd.gauge('holysheep.gateway.health', 1 if is_healthy else 0, tags=tags)
statsd.gauge('holysheep.gateway.response_time', response_time_ms, tags=tags)
============================================
ALERT THRESHOLDS CONFIGURATION
============================================
ALERT_CONFIG = {
'p95_latency_threshold_ms': 2000, # Alert if P95 > 2 seconds
'error_rate_threshold_percent': 5, # Alert if 5xx > 5%
'cost_hourly_threshold_usd': 100, # Alert if hourly cost > $100
'active_requests_threshold': 1000, # Alert if concurrent requests > 1000
}
============================================
MONITOR CLASS
============================================
class HolySheepMonitor:
"""
HolySheep Gateway Monitor with Datadog integration
"""
def __init__(self):
self.metrics_buffer = []
self.buffer_size = 100
def check_gateway_health(self, model: str) -> dict:
"""Health check endpoint for HolySheep API"""
import httpx
start = time.perf_counter()
try:
response = httpx.get(
f"https://api.holysheep.ai/v1/models",
headers={"Authorization": f"Bearer {os.getenv('YOUR_HOLYSHEEP_API_KEY')}"},
timeout=10.0
)
response_time = (time.perf_counter() - start) * 1000
is_healthy = response.status_code == 200
record_gateway_health(model, is_healthy, response_time)
return {'healthy': is_healthy, 'response_time_ms': response_time}
except Exception as e:
response_time = (time.perf_counter() - start) * 1000
record_gateway_health(model, False, response_time)
return {'healthy': False, 'error': str(e), 'response_time_ms': response_time}
def calculate_p95(self, latencies: list) -> float:
"""Calculate P95 from latency list"""
if not latencies:
return 0.0
sorted_latencies = sorted(latencies)
index = int(len(sorted_latencies) * 0.95)
return sorted_latencies[min(index, len(sorted_latencies) - 1)]
def check_alerts(self, metrics: list) -> list:
"""Check alert conditions"""
alerts = []
if not metrics:
return alerts
# Calculate P95
latencies = [m['latency_ms'] for m in metrics if 'latency_ms' in m]
p95 = self.calculate_p95(latencies)
if p95 > ALERT_CONFIG['p95_latency_threshold_ms']:
alerts.append({
'type': 'high_latency',
'severity': 'warning',
'message': f'P95 latency {p95:.0f}ms exceeds threshold {ALERT_CONFIG["p95_latency_threshold_ms"]}ms',
'p95_value': p95
})
# Error rate check
error_count = sum(1 for m in metrics if m.get('status_code', 0) >= 500)
total_count = len(metrics)
error_rate = (error_count / total_count * 100) if total_count > 0 else 0
if error_rate > ALERT_CONFIG['error_rate_threshold_percent']:
alerts.append({
'type': 'high_error_rate',
'severity': 'critical',
'message': f'5xx error rate {error_rate:.1f}% exceeds threshold {ALERT_CONFIG["error_rate_threshold_percent"]}%',
'error_rate': error_rate
})
return alerts
============================================
USAGE EXAMPLE
============================================
if __name__ == "__main__":
monitor = HolySheepMonitor()
# Check gateway health
health = monitor.check_gateway_health('gpt-4.1')
print(f"Gateway Health: {health}")
# Record sample metrics
record_request_metrics(
model='gpt-4.1',
latency_ms=150.5,
status_code=200,
tokens_used=500,
cost_usd=0.004
)
print("Metrics sent to Datadog successfully!")
Run: python datadog_monitor.py
Environment variables needed:
DD_API_KEY, DD_APP_KEY, DD_AGENT_HOST, YOUR_HOLYSHEEP_API_KEY
3. Grafana Dashboard Configuration
Với Grafana, tôi tạo dashboard trực quan để track real-time performance. Kết hợp Prometheus làm data source.
# grafana_dashboard.json - Grafana Dashboard Configuration
Import this JSON to Grafana
{
"dashboard": {
"title": "HolySheep AI Gateway Monitor",
"uid": "holysheep-gateway-monitor",
"timezone": "browser",
"panels": [
{
"title": "P95 Latency by Model",
"type": "timeseries",
"gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate(holysheep_request_latency_ms_bucket[5m])) by (le, model))",
"legendFormat": "P95 - {{model}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ms",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 1000},
{"color": "red", "value": 2000}
]
}
}
},
"alert": {
"name": "High P95 Latency Alert",
"conditions": [
{
"evaluator": {"params": [2000], "type": "gt"},
"operator": {"type": "and"},
"query": {"params": ["A", "5m", "now"]},
"reducer": {"type": "avg"}
}
],
"frequency": "1m",
"handler": 1,
"message": "HolySheep Gateway P95 latency exceeded 2000ms threshold"
}
},
{
"title": "5xx Error Rate %",
"type": "timeseries",
"gridPos": {"x": 12, "y": 0, "w": 12, "h": 8},
"targets": [
{
"expr": "sum(rate(holysheep_5xx_errors_total[5m])) by (model) / sum(rate(holysheep_requests_total[5m])) by (model) * 100",
"legendFormat": "{{model}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 2},
{"color": "red", "value": 5}
]
}
}
},
"alert": {
"name": "High 5xx Error Rate Alert",
"conditions": [
{
"evaluator": {"params": [5], "type": "gt"},
"operator": {"type": "and"}
}
],
"frequency": "30s",
"handler": 1,
"message": "5xx error rate exceeded 5% threshold - Immediate action required"
}
},
{
"title": "Request Volume by Model",
"type": "bargauge",
"gridPos": {"x": 0, "y": 8, "w": 8, "h": 6},
"targets": [
{
"expr": "sum(rate(holysheep_requests_total[5m])) by (model)",
"legendFormat": "{{model}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": {"mode": "palette-classic"}
}
}
},
{
"title": "Token Usage (Thousands)",
"type": "timeseries",
"gridPos": {"x": 8, "y": 8, "w": 8, "h": 6},
"targets": [
{
"expr": "sum(rate(holysheep_tokens_total[1h])) by (model, token_type) / 1000",
"legendFormat": "{{model}} - {{token_type}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
}
}
},
{
"title": "Hourly Cost (USD)",
"type": "stat",
"gridPos": {"x": 16, "y": 8, "w": 8, "h": 6},
"targets": [
{
"expr": "sum(increase(holysheep_cost_usd[1h]))",
"legendFormat": "Hourly Cost"
}
],
"fieldConfig": {
"defaults": {
"unit": "currencyUSD",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 50},
{"color": "red", "value": 100}
]
}
}
}
},
{
"title": "Active Requests",
"type": "gauge",
"gridPos": {"x": 0, "y": 14, "w": 6, "h": 6},
"targets": [
{
"expr": "sum(holysheep_active_requests)",
"legendFormat": "Active"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"max": 1000,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 500},
{"color": "red", "value": 800}
]
}
}
}
},
{
"title": "Error Breakdown",
"type": "piechart",
"gridPos": {"x": 6, "y": 14, "w": 6, "h": 6},
"targets": [
{
"expr": "sum(holysheep_5xx_errors_total) by (error_type)",
"legendFormat": "{{error_type}}"
}
]
}
],
"time": {"from": "now-1h", "to": "now"},
"refresh": "10s"
}
}
============================================
PROMETHEUS ALERT RULES (prometheus_rules.yml)
============================================
groups:
- name: holysheep-alerts
rules:
- alert: HolySheepHighP95Latency
expr: histogram_quantile(0.95, sum(rate(holysheep_request_latency_ms_bucket[5m])) by (le)) > 2000
for: 5m
labels:
severity: warning
annotations:
summary: "High P95 latency detected on HolySheep Gateway"
description: "P95 latency is {{ $value }}ms (threshold: 2000ms)"
- alert: HolySheepHighErrorRate
expr: sum(rate(holysheep_5xx_errors_total[5m])) / sum(rate(holysheep_requests_total[5m])) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: "High 5xx error rate on HolySheep Gateway"
description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"
- alert: HolySheepCostSpike
expr: sum(increase(holysheep_cost_usd[1h])) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Hourly cost exceeded $100"
description: "Current hourly cost: ${{ $value }}"
4. Test Script - End-to-End Verification
Trước khi deploy lên production, tôi luôn chạy test script này để verify mọi thứ hoạt động đúng.
# test_monitoring.py - Comprehensive monitoring test
Install: pip install httpx prometheus_client pytest pytest-asyncio
import asyncio
import httpx
import time
from prometheus_client import REGISTRY
HOLYSHEEP_BASE_URL = "https://api.holysheep.ai/v1"
API_KEY = "YOUR_HOLYSHEEP_API_KEY" # Replace with your key
============================================
TEST CONFIGURATION
============================================
TEST_MODELS = ['gpt-4.1', 'gemini-2.5-flash', 'deepseek-v3.2']
CONCURRENT_REQUESTS = 10
TOTAL_REQUESTS = 50
async def test_single_request(client: httpx.AsyncClient, model: str) -> dict:
"""Test single request to HolySheep"""
start_time = time.perf_counter()
try:
response = await client.post(
f"{HOLYSHEEP_BASE_URL}/chat/completions",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
"X-Model": model
},
json={
"model": model,
"messages": [{"role": "user", "content": "Count to 3"}],
"max_tokens": 50,
"temperature": 0.7
},
timeout=30.0
)
latency_ms = (time.perf_counter() - start_time) * 1000
return {
'success': response.status_code == 200,
'status_code': response.status_code,
'latency_ms': latency_ms,
'model': model,
'response': response.json() if response.status_code == 200 else None,
'error': None
}
except Exception as e:
latency_ms = (time.perf_counter() - start_time) * 1000
return {
'success': False,
'status_code': 0,
'latency_ms': latency_ms,
'model': model,
'response': None,
'error': str(e)
}
async def run_load_test():
"""Run concurrent load test"""
print("=" * 60)
print("HOLYSHEEP MONITORING TEST SUITE")
print("=" * 60)
async with httpx.AsyncClient() as client:
# Test each model
results = {'gpt-4.1': [], 'gemini-2.5-flash': [], 'deepseek-v3.2': []}
for model in TEST_MODELS:
print(f"\n🔄 Testing {model}...")
tasks = [test_single_request(client, model) for _ in range(TOTAL_REQUESTS)]
model_results = await asyncio.gather(*tasks)
results[model] = model_results
# Calculate statistics
successful = [r for r in model_results if r['success']]
failed = [r for r in model_results if not r['success']]
latencies = [r['latency_ms'] for r in successful]
latencies.sort()
print(f" ✅ Success: {len(successful)}/{TOTAL_REQUESTS}")
print(f" ❌ Failed: {len(failed)}/{TOTAL_REQUESTS}")
if latencies:
p50 = latencies[int(len(latencies) * 0.5)]
p95 = latencies[int(len(latencies) * 0.95)]
p99 = latencies[int(len(latencies) * 0.99)]
avg = sum(latencies) / len(latencies)
print(f" 📊 Latency Stats:")
print(f" P50: {p50:.2f}ms")
print(f" P95: {p95:.2f}ms")
print(f" P99: {p99:.2f}ms")
print(f" Avg: {avg:.2f}ms")
# Check P95 threshold
if p95 > 2000:
print(f" ⚠️ WARNING: P95 ({p95:.0f}ms) exceeds 2000ms threshold!")
# Check error rate
error_rate = len(failed) / TOTAL_REQUESTS * 100
if error_rate > 5:
print(f" 🚨 ERROR: Error rate ({error_rate:.1f}%) exceeds 5% threshold!")
# Summary
print("\n" + "=" * 60)
print("SUMMARY")
print("=" * 60)
total_success = sum(len([r for r in results[m] if r['success']]) for m in TEST_MODELS)
total_requests = TOTAL_REQUESTS * len(TEST_MODELS)
overall_success_rate = total_success / total_requests * 100
print(f"Total Requests: {total_requests}")
print(f"Success Rate: {overall_success_rate:.1f}%")
# Check Prometheus metrics
print("\n📈 Prometheus Metrics Check:")
for metric in REGISTRY.collect():
print(f" - {metric.name}: {len(metric.samples)} samples")
return results
if __name__ == "__main__":
# Run test
results = asyncio.run(run_load_test())
print("\n✅ Test completed! Metrics should be available at http://localhost:9090/metrics")
Run: python test_monitoring.py
Verify: curl http://localhost:9090/metrics | grep holysheep
5. Alerting Rules - P95 Latency & 5xx Error Rate
Đây là phần quan trọng nhất - alerting rules giúp bạn biết vấn đề trước khi người dùng phàn nàn.
# ============================================
ALERT RULES CONFIGURATION
============================================
DATADOG MONITOR CONFIGURATION (datadog_monitors.yml)
monitors:
- name: "HolySheep P95 Latency Alert"
type: "metric alert"
query: |
p95(holysheep.latency{service:holysheep-gateway}.as_count(), 5m) > 2000
message: |
🚨 ALERT: HolySheep Gateway High Latency
P95 Latency: {{ value }}ms
Threshold: 2000ms
Models affected: {{ tags.model }}
Action Required:
1. Check model provider status
2. Verify network connectivity
3. Check for rate limiting
4. Consider scaling gateway instances
@slack-holysheep-alerts @pagerduty-holysheep
tags:
- "service:holysheep-gateway"
- "severity:warning"
- name: "HolySheep 5xx Error Rate Alert"
type: "metric alert"
query: |
(sum(holysheep.errors.5xx{service:holysheep-gateway}.as_count()) / sum(holysheep.requests{service:holysheep-gateway}.as_count())) * 100 > 5
message: |
🚨🚨 CRITICAL: HolySheep Gateway High Error Rate
5xx Error Rate: {{ value }}%
Threshold: 5%
Error Types:
{{#each groups}}
- {{ error_type }}: {{ this }}
{{/each}}
IMMEDIATE ACTION REQUIRED!
1. Check HolySheep API status at https://status.holysheep.ai
2. Check model provider dashboards
3. Enable fallback routing
4. Scale up gateway if needed
@pagerduty-holysheep-critical @slack-holysheep-alerts
tags:
- "service:holysheep-gateway"
- "severity:critical"