Prologue : Le Mardi où tout a basculé
C'était 14h23 un mardi après-midi. Mon téléphone a vibré avec une alerte que je n'oublierai jamais : ConnectionError: timeout after 30000ms. Notre application de traitement de documents, entièrement dépendante de l'API AI pour l'analyse sémantique, tombait en cascade. Les utilisateurs voyaient des timeouts à répétition, et notre taux d'erreur bondissait à 23% — bien au-delà du seuil acceptable de 1% que nous nous étions fixés.
Cette expérience m'a coûté 3 heures de debug intensif, 47 tickets de support, et surtout, la confiance d'un client enterprise qui menaçait de rompre son contrat de 12 000$/mois. C'est ce jour-là que j'ai compris l'importance critique d'une infrastructure SLO robuste pour tout middleware API AI. Aujourd'hui, je vais vous montrer comment implémenter une surveillance professionnelle qui aurait détecté ce problème en 30 secondes plutôt qu'en 3 heures.
Comprendre les SLO pour votre API Relay
Un SLO (Service Level Objective) est un engagement mesurable que vous prenez envers vos utilisateurs concernant la disponibilité et les performances de votre service. Pour un relayeur API AI comme HolySheep, les métriques essentielles sont :
- Taux de succès des requêtes : Objectif 99,9% (max 0,1% d'erreurs)
- Latence P99 : Objectif < 500ms pour les appels simples
- Disponibilité globale : Objectif 99,95% (max 4h38min d'indisponibilité/an)
- Temps de récupération (MTTR) : Objectif < 15 minutes
Chez HolySheep AI, notre infrastructure multi-régions garantit une latence moyenne de 42ms vers les endpoints asiatiques, avec un taux de disponibilité vérifiable de 99,97% sur les 6 derniers mois.
Architecture de Surveillance Recommandée
Stack Technique
Notre architecture de monitoring s'appuie sur Prometheus pour la collecte, Grafana pour la visualisation, et AlertManager pour les notifications. Cette stack open-source constitue le standard industriel pour la surveillance SLO.
Implémentation : Le Client Python avec Monitoring Intégré
Voici le code complet du client Python que j'utilise en production depuis 18 mois. Il intègre nativement la surveillance SLO et les alertes automatiques :
# holy_sheep_monitored_client.py
Client Python avec monitoring SLO intégré pour HolySheep AI API
Compatible Python 3.8+, requires prometheus-client, requests
import time
import requests
import logging
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List
from prometheus_client import Counter, Histogram, Gauge, CollectorRegistry, push_to_gateway
import threading
import queue
Configuration du registre Prometheus
REGISTRY = CollectorRegistry()
Métriques Prometheus pour le monitoring SLO
REQUEST_COUNT = Counter(
'holysheep_api_requests_total',
'Total des requêtes API HolySheep',
['method', 'endpoint', 'status_code'],
registry=REGISTRY
)
REQUEST_LATENCY = Histogram(
'holysheep_api_request_duration_seconds',
'Latence des requêtes API HolySheep',
['method', 'endpoint'],
buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
registry=REGISTRY
)
ACTIVE_REQUESTS = Gauge(
'holysheep_api_active_requests',
'Requêtes actives en cours',
registry=REGISTRY
)
ERROR_RATE = Gauge(
'holysheep_api_error_rate',
'Taux d\'erreur sur la fenêtre glissante',
registry=REGISTRY
)
BUDGET_CONSUMPTION = Gauge(
'holysheep_api_budget_consumption_dollars',
'Consommation budgétaire en USD',
registry=REGISTRY
)
class SLOThreshold:
"""Configuration des seuils SLO"""
SUCCESS_RATE_MIN = 0.999 # 99.9% minimum
LATENCY_P99_MAX = 0.5 # 500ms maximum
LATENCY_P95_MAX = 0.2 # 200ms maximum
ERROR_BUDGET_WINDOW = timedelta(hours=24)
ERROR_BUDGET_MAX_RATE = 0.001 # 0.1% d'erreurs sur 24h
class AlertManager:
"""Gestionnaire d'alertes intelligent"""
def __init__(self, webhook_url: str, slack_webhook: Optional[str] = None):
self.webhook_url = webhook_url
self.slack_webhook = slack_webhook
self.alert_history: List[Dict] = []
self.alert_cooldown = timedelta(minutes=5)
self.last_alert_time: Dict[str, datetime] = {}
def should_alert(self, alert_type: str) -> bool:
"""Évite les alertes spam avec un cooldown"""
now = datetime.now()
if alert_type in self.last_alert_time:
if now - self.last_alert_time[alert_type] < self.alert_cooldown:
return False
self.last_alert_time[alert_type] = now
return True
def send_alert(self, severity: str, title: str, message: str, metrics: Dict = None):
"""Envoie une alerte via plusieurs canaux"""
if not self.should_alert(title):
logging.debug(f"Alerte {title} en cooldown, ignorée")
return
alert_payload = {
"timestamp": datetime.now().isoformat(),
"severity": severity,
"title": title,
"message": message,
"metrics": metrics or {},
"source": "holysheep-slo-monitor"
}
# Push vers Prometheus AlertManager
try:
requests.post(self.webhook_url, json=alert_payload, timeout=5)
except Exception as e:
logging.error(f"Échec envoi alerte: {e}")
# Notification Slack optionnelle
if self.slack_webhook:
self._send_slack(severity, title, message, metrics)
self.alert_history.append(alert_payload)
logging.warning(f"🚨 ALERT [{severity.upper()}] {title}: {message}")
@dataclass
class SLOClient:
"""Client HolySheep avec monitoring SLO complet"""
api_key: str
base_url: str = "https://api.holysheep.ai/v1"
timeout: float = 30.0
max_retries: int = 3
retry_delay: float = 1.0
budget_limit: float = 1000.0 # Limite budgétaire en USD
# Composants internes
_alert_manager: AlertManager = field(default=None, init=False)
_error_window: queue.Queue = field(default_factory=queue.Queue, init=False)
_last_cost: float = field(default=0.0, init=False)
_total_tokens: int = field(default=0, init=False)
def __post_init__(self):
self.session = requests.Session()
self.session.headers.update({
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
"X-Holysheep-Client": "slo-monitor-v2.1"
})
self._alert_manager = AlertManager(
webhook_url="http://alertmanager:9093/api/v1/alerts"
)
logging.basicConfig(level=logging.INFO)
self.logger = logging.getLogger(__name__)
def _track_request(self, method: str, endpoint: str, duration: float,
status_code: int, tokens: int = 0, cost: float = 0.0):
"""Enregistre les métriques de la requête"""
REQUEST_COUNT.labels(method=method, endpoint=endpoint,
status_code=str(status_code)).inc()
REQUEST_LATENCY.labels(method=method, endpoint=endpoint).observe(duration)
# Mise à jour du coût total
self._last_cost += cost
self._total_tokens += tokens
BUDGET_CONSUMPTION.set(self._last_cost)
# Tracking du taux d'erreur sur fenêtre glissante
self._error_window.put((datetime.now(), status_code >= 400))
self._clean_error_window()
# Calcul du taux d'erreur actuel
error_count = sum(1 for _, is_error in list(self._error_window.queue) if is_error)
total_count = self._error_window.qsize()
if total_count > 0:
current_error_rate = error_count / total_count
ERROR_RATE.set(current_error_rate)
# Alertes basées sur les SLO
self._check_slo_violations(current_error_rate, duration)
def _clean_error_window(self, window_seconds: int = 3600):
"""Nettoie les données hors fenêtre temporelle"""
cutoff = datetime.now() - timedelta(seconds=window_seconds)
while not self._error_window.empty():
timestamp, _ = self._error_window.queue[0]
if timestamp < cutoff:
self._error_window.get()
else:
break
def _check_slo_violations(self, error_rate: float, latency_p99: float):
"""Vérifie les violations SLO et déclenche les alertes"""
# Violation du taux de succès
if error_rate > (1 - SLOThreshold.SUCCESS_RATE_MIN):
self._alert_manager.send_alert(
severity="critical",
title="SLO_VIOLATION_ERROR_RATE",
message=f"Taux d'erreur {error_rate*100:.2f}% dépasse le SLO de 0.1%",
metrics={
"current_error_rate": error_rate,
"slo_threshold": 0.001,
"breach_severity": "critical"
}
)
# Warning pour budget临近
if self._last_cost > self.budget_limit * 0.8:
self._alert_manager.send_alert(
severity="warning",
title="BUDGET_THRESHOLD_WARNING",
message=f"80% du budget utilisé: ${self._last_cost:.2f} / ${self.budget_limit:.2f}",
metrics={
"budget_used_percent": (self._last_cost / self.budget_limit) * 100
}
)
# Alerte budget épuisé
if self._last_cost >= self.budget_limit:
self._alert_manager.send_alert(
severity="critical",
title="BUDGET_EXHAUSTED",
message=f"Budget limite atteint: ${self._last_cost:.2f}",
metrics={"action_required": "service_degradation"}
)
def _estimate_cost(self, model: str, tokens: int, cached_tokens: int = 0) -> float:
"""Estime le coût selon le modèle utilisé (tarifs HolySheep 2026)"""
pricing = {
"gpt-4.1": {"input": 2.0, "output": 8.0, "cache_write": 0.50, "cache_read": 0.0},
"gpt-4.1-mini": {"input": 0.50, "output": 2.0, "cache_write": 0.125, "cache_read": 0.0},
"claude-sonnet-4.5": {"input": 3.0, "output": 15.0, "cache_write": 3.75, "cache_read": 0.30},
"gemini-2.5-flash": {"input": 0.125, "output": 0.50, "cache_write": 0.10, "cache_read": 0.0},
"gemini-2.5-pro": {"input": 1.25, "output": 10.0, "cache_write": 10.0, "cache_read": 0.0},
"deepseek-v3.2": {"input": 0.07, "output": 0.28, "cache_write": 0.14, "cache_read": 0.0},
}
if model not in pricing:
self.logger.warning(f"Modèle {model} non reconnu, utilisation tarif par défaut")
return tokens * 0.0001 # Approximation conservative
rates = pricing[model]
input_cost = (tokens - cached_tokens) * rates["input"] / 1_000_000
output_cost = tokens * rates["output"] / 1_000_000
cache_cost = cached_tokens * rates["cache_read"] / 1_000_000
return input_cost + output_cost + cache_cost
def chat_completions(self, model: str, messages: List[Dict],
**kwargs) -> Dict[str, Any]:
"""Appel completions avec monitoring"""
endpoint = f"{self.base_url}/chat/completions"
ACTIVE_REQUESTS.inc()
start_time = time.time()
attempt = 0
while attempt < self.max_retries:
try:
response = self.session.post(
endpoint,
json={
"model": model,
"messages": messages,
**{k: v for k, v in kwargs.items() if k not in ['stream']}
},
timeout=self.timeout
)
duration = time.time() - start_time
status_code = response.status_code
# Parsing de la réponse
if response.status_code == 200:
data = response.json()
usage = data.get("usage", {})
prompt_tokens = usage.get("prompt_tokens", 0)
completion_tokens = usage.get("completion_tokens", 0)
total_tokens = prompt_tokens + completion_tokens
cached_tokens = usage.get("prompt_tokens_details", {}).get("cached_tokens", 0)
cost = self._estimate_cost(model, total_tokens, cached_tokens)
self._track_request("POST", endpoint, duration, status_code,
total_tokens, cost)
ACTIVE_REQUESTS.dec()
return {"success": True, "data": data, "cost_usd": cost}
elif response.status_code == 429:
# Rate limit - retry with exponential backoff
attempt += 1
wait_time = self.retry_delay * (2 ** attempt)
self.logger.warning(f"Rate limited, retry dans {wait_time}s")
time.sleep(wait_time)
continue
else:
error_msg = response.json().get("error", {}).get("message", "Unknown error")
self.logger.error(f"Erreur API {status_code}: {error_msg}")
self._track_request("POST", endpoint, duration, status_code, 0, 0)
ACTIVE_REQUESTS.dec()
return {"success": False, "error": error_msg, "status_code": status_code}
except requests.exceptions.Timeout:
self.logger.error("Timeout de requête")
self._track_request("POST", endpoint, self.timeout, 408, 0, 0)
if attempt == self.max_retries - 1:
ACTIVE_REQUESTS.dec()
return {"success": False, "error": "Request timeout", "status_code": 408}
attempt += 1
except requests.exceptions.ConnectionError as e:
self.logger.error(f"Connection error: {e}")
self._track_request("POST", endpoint, time.time() - start_time, 503, 0, 0)
ACTIVE_REQUESTS.dec()
return {"success": False, "error": "Connection failed", "status_code": 503}
except Exception as e:
self.logger.exception(f"Erreur inattendue: {e}")
duration = time.time() - start_time
self._track_request("POST", endpoint, duration, 500, 0, 0)
ACTIVE_REQUESTS.dec()
return {"success": False, "error": str(e), "status_code": 500}
ACTIVE_REQUESTS.dec()
return {"success": False, "error": "Max retries exceeded", "status_code": 503}
def get_slo_report(self) -> Dict[str, Any]:
"""Génère un rapport SLO complet"""
return {
"timestamp": datetime.now().isoformat(),
"budget": {
"spent_usd": round(self._last_cost, 4),
"limit_usd": self.budget_limit,
"remaining_usd": round(self.budget_limit - self._last_cost, 4),
"utilization_percent": round((self._last_cost / self.budget_limit) * 100, 2)
},
"tokens": {
"total": self._total_tokens,
"estimated_models": ["gpt-4.1", "claude-sonnet-4.5", "deepseek-v3.2"]
},
"slo_status": {
"success_rate_target": f"{SLOThreshold.SUCCESS_RATE_MIN * 100}%",
"latency_p99_target_ms": SLOThreshold.LATENCY_P99_MAX * 1000,
"monitoring_window_seconds": 3600
}
}
Exemple d'utilisation
if __name__ == "__main__":
# IMPORTANT: Remplacez par votre vraie clé API HolySheep
API_KEY = "YOUR_HOLYSHEEP_API_KEY"
client = SLOClient(
api_key=API_KEY,
base_url="https://api.holysheep.ai/v1",
timeout=30.0,
budget_limit=500.0 # Limite de 500$ par période
)
# Test avec un appel simple
messages = [
{"role": "system", "content": "Tu es un assistant technique expert."},
{"role": "user", "content": "Explique la différence entre SLO et SLA en 2 phrases."}
]
result = client.chat_completions(
model="deepseek-v3.2",
messages=messages,
temperature=0.7,
max_tokens=200
)
if result["success"]:
print(f"✅ Réponse reçue")
print(f" Coût: ${result['cost_usd']:.6f}")
print(f" Contenu: {result['data']['choices'][0]['message']['content'][:100]}...")
else:
print(f"❌ Erreur: {result['error']}")
# Affichage du rapport SLO
print("\n📊 Rapport SLO:")
report = client.get_slo_report()
for key, value in report.items():
print(f" {key}: {value}")
Dashboard Grafana pour la Visualisation SLO
Maintenant, créons le dashboard Grafana qui vous donnera une visibilité temps réel sur vos SLO. Ce dashboard intègre les métriques Prometheus collectées par notre client :
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 99.5 },
{ "color": "red", "value": 99.9 }
]
},
"unit": "percentunit",
"max": 1
},
"overrides": []
},
"gridPos": { "h": 6, "w": 8, "x": 0, "y": 0 },
"id": 1,
"options": {
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": ["lastNotNull"],
"fields": ""
},
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"pluginVersion": "8.0.0",
"targets": [
{
"expr": "1 - (sum(rate(holysheep_api_requests_total{status_code=~'5..'}[5m])) / sum(rate(holysheep_api_requests_total[5m])))",
"legendFormat": "Taux de succès",
"refId": "A"
}
],
"title": "📈 SLO: Taux de Succès (Objectif: 99.9%)",
"type": "gauge"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisLabel": "Latence (ms)",
"axisWidth": 0,
"barAlignment": 0,
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"scaleDistribution": { "type": "linear" },
"showPoints": "never"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 200 },
{ "color": "red", "value": 500 }
]
},
"unit": "ms"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 10, "x": 8, "y": 0 },
"id": 2,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" },
"tooltip": { "mode": "multi" }
},
"pluginVersion": "8.0.0",
"targets": [
{
"expr": "histogram_quantile(0.50, sum(rate(holysheep_api_request_duration_seconds_bucket[5m])) by (le)) * 1000",
"legendFormat": "P50",
"refId": "A"
},
{
"expr": "histogram_quantile(0.95, sum(rate(holysheep_api_request_duration_seconds_bucket[5m])) by (le)) * 1000",
"legendFormat": "P95",
"refId": "B"
},
{
"expr": "histogram_quantile(0.99, sum(rate(holysheep_api_request_duration_seconds_bucket[5m])) by (le)) * 1000",
"legendFormat": "P99 ⚠️",
"refId": "C"
}
],
"title": "⏱️ Latence des Requêtes API (en ms)",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisLabel": "Dollars ($)",
"lineWidth": 2,
"pointSize": 5
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 800 },
{ "color": "red", "value": 1000 }
]
},
"unit": "currencyUSD"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 6, "x": 18, "y": 0 },
"id": 3,
"targets": [
{
"expr": "holysheep_api_budget_consumption_dollars",
"legendFormat": "Budget utilisé",
"refId": "A"
}
],
"title": "💰 Consommation Budgétaire",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"gridPos": { "h": 4, "w": 24, "x": 0, "y": 6 },
"id": 4,
"targets": [
{
"expr": "sum(holysheep_api_error_rate > 0.001)",
"legendFormat": "❌ Violations SLO actives",
"refId": "A"
}
],
"title": "🚨 Alertes SLO Actives",
"type": "stat"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "lineWidth": 1 }
},
"overrides": [
{
"matcher": { "id": "byName", "options": "Requêtes" },
"properties": [{ "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" }}]
},
{
"matcher": { "id": "byName", "options": "Erreurs" },
"properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" }}]
}
]
},
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 10 },
"id": 5,
"targets": [
{
"expr": "sum(rate(holysheep_api_requests_total[5m])) * 60",
"legendFormat": "Requêtes/min",
"refId": "A"
},
{
"expr": "sum(rate(holysheep_api_requests_total{status_code=~'5..'}[5m])) * 60",
"legendFormat": "Erreurs/min",
"refId": "B"
}
],
"title": "📊 Volume de Requêtes vs Erreurs",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"max": 100,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "yellow", "value": 50 },
{ "color": "green", "value": 80 }
]
},
"unit": "percent"
}
},
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 10 },
"id": 6,
"targets": [
{
"expr": "(1 - holysheep_api_error_rate) * 100",
"legendFormat": "Error Budget Remaining",
"refId": "A"
}
],
"title": "🎯 Error Budget Restant (Objectif: 100%)",
"type": "gauge"
}
],
"schemaVersion": 27,
"style": "dark",
"tags": ["holy-sheap", "slo", "monitoring", "api"],
"templating": { "list": [] },
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "HolySheep AI - Dashboard SLO Monitoring",
"uid": "holysheep-slo-001",
"version": 1
}Configuration AlertManager : Règles d'Alerte
Le fichier de configuration AlertManager ci-dessous définit les règles d'alerte qui se déclenchent automatiquement lors des violations SLO :
# alertmanager-rules.yml
Règles d'alerte SLO pour HolySheep AI Relay
groups:
- name: holysheep_slo_alerts
rules:
# 🚨 Alerte critique: Taux d'erreur dépasse le SLO de 99.9%
- alert: HolySheepSLOErrorRateViolation
expr: |
(
sum(rate(holysheep_api_requests_total{status_code=~"5.."}[5m]))
/
sum(rate(holysheep_api_requests_total[5m]))
) > 0.001
for: 2m
labels:
severity: critical
service: holysheep-api
slo: error-rate
annotations:
summary: "🚨 Violation SLO: Taux d'erreur критический"
description: |
Le taux d'erreur API HolySheep dépasse le seuil SLO de 0.1%
**Métriques actuelles:**
- Taux d'erreur: {{ $value | printf "%.3f" }}%
- Seuil SLO: 0.1%
- Excès: {{ printf "%.2f" (mul (sub (mul $value 100) 0.1) 1) }}x
**Action requise:**
- Vérifier la santé des services upstream
- Examiner les logs d'erreur récents
- Envisager un failover vers région backup
runbook_url: "https://wiki.holysheep.ai/runbooks/slo-error-rate"
# ⚠️ Alerte warning: Latence P99 supérieure à 500ms
- alert: HolySheepSLOLatencyViolation
expr: |
histogram_quantile(0.99,
sum(rate(holysheep_api_request_duration_seconds_bucket[5m])) by (le)
) > 0.5
for: 5m
labels:
severity: warning
service: holysheep-api
slo: latency-p99
annotations:
summary: "⚠️ Latence P99 élevée: {{ $value | humanizeDuration }}"
description: |
La latence P99 dépasse l'objectif de 500ms
**Métriques:**
- P50: {{ printf "%.0f" (mul (histogram_quantile(0.50, sum(rate(holysheep_api_request_duration_seconds_bucket[5m])) by (le))) 1000) }}ms
- P95: {{ printf "%.0f" (mul (histogram_quantile(0.95, sum(rate(holysheep_api_request_duration_seconds_bucket[5m])) by (le))) 1000) }}ms
- P99: {{ $value | humanizeDuration }}
runbook_url: "https://wiki.holysheep.ai/runbooks/slo-latency"
# 💰 Alerte critique: Budget接近 épuisé
- alert: HolySheepBudgetExhausted
expr: |
holysheep_api_budget_consumption_dollars >= 1000
for: 1m
labels:
severity: critical
service: holysheep-api
type: budget
annotations:
summary: "💸 Budget API Kritique: ${{ $value | printf "%.2f" }}"
description: |
Le budget mensuel API a atteint la limite configurée.
**Consommation:**
- Actuel: ${{ $value | printf "%.2f" }}
- Limite: $1000.00
- Taux d'utilisation: {{ printf "%.1f" (mul (div $value 1000) 100) }}%
**Actions possibles:**
1. Contacter le support HolySheep pour augmentation de limite
2. Réviser les règles de cache pour réduire les coûts
3. Activer le mode dégradé automatiquement
# 🔴 Alerte emergency: Service complètement indisponible
- alert: HolySheepServiceDown
expr: |
sum(rate(holysheep_api_requests_total[5m])) < 1
for: 1m
labels:
severity: emergency
service: holysheep-api
type: availability
annotations:
summary: "🔴 HolySheep API Indisponible!"
description: |
Aucune requête réussie depuis plus d'une minute.
Le service est potentiellement down.
**Diagnostics:**
- Requêtes/min actuelle: {{ printf "%.1f" (mul $value 60) }}
- Vérifier: https://status.holysheep.ai
# 📊 Alerte information: pic de traffic anormal
- alert: HolySheepTrafficAnomaly
expr: |
(
sum(rate(holysheep_api_requests_total[5m]))
/
avg_over_time(sum(rate(holysheep_api_requests_total[1h]))[7d:1h])
) > 3
for: 10m
labels:
severity: info
service: holysheep-api
type: traffic
annotations:
summary: "📈 Pic de traffic détecté: {{ printf "%.1f" $value }}x normal"
description: |
Le volume de requêtes est significativement supérieur à la normale.
- Traffic actuel: {{ printf "%.1f" (mul $value 100) }}% de la moyenne
- Possible cause: Campaign marketing, viral content, ou attaque DDoS
**Recommandation:** Vérifier la source du traffic
- name: holysheep_error_breakdown
interval: 30s
rules:
# Détail des erreurs par type
- alert: HolySheepErrorRateByCode
expr: |
sum by (status_code) (
rate(holysheep_api_requests_total{status_code=~"4.."}[5m])
) > 0.1
labels:
severity: warning
category: error-breakdown
annotations: