HolySheep API中转站蓝绿部署：零 downtime 发布

Đêm qua 2 giờ sáng, hệ thống của tôi báo lỗi ConnectionError: timeout after 30000ms. Đó là lúc tôi nhận ra mình đang deploy trực tiếp lên production mà không có chiến lược rollback. 47 khách hàng đang sử dụng API bị gián đoạn. Đó là khoảnh khắc tôi quyết định triển khai blue-green deployment cho HolySheep API relay station — và từ đó, không một lần nào bị downtime khi release.

Blue-Green Deployment là gì và tại sao cần thiết

Blue-green deployment là chiến lược triển khai với hai môi trường đồng nhất: Blue (môi trường hiện tại) và Green (môi trường mới). Khi release phiên bản mới, ta deploy lên Green, kiểm thử, rồi chuyển traffic qua Green. Nếu có sự cố — rollback ngay lập tức bằng cách chuyển traffic về Blue.

Lợi ích cốt lõi

• Zero downtime — Không một request nào bị drop
• Rollback tức thì — Quay về phiên bản cũ trong vài giây
• Kiểm thử trên production thực — Test với traffic thật trước khi chuyển hoàn toàn
• Deployment không rủi ro — Sự cố chỉ ảnh hưởng môi trường staging

Kiến trúc Blue-Green với HolySheep API Relay

Với HolySheep API relay station, ta sử dụng Nginx làm load balancer để điều hướng traffic giữa hai upstream backend. Điểm mấu chốt: cả hai môi trường đều gọi đến https://api.holysheep.ai/v1 — nên dù rollback hay forward, API endpoint của người dùng cuối không thay đổi.

# /etc/nginx/conf.d/blue-green-upstream.conf

upstream blue_backend {
    server 10.0.1.10:8080;  # Blue environment - production hiện tại
    keepalive 32;
}

upstream green_backend {
    server 10.0.1.20:8080;  # Green environment - phiên bản mới
    keepalive 32;
}

upstream holy_api {
    server api.holysheep.ai:443;
    keepalive 64;
}

Health check endpoint cho monitoring
server {
    listen 9090;
    location /health {
        access_log off;
        return 200 "healthy\n";
        add_header Content-Type text/plain;
    }
}

# /etc/nginx/conf.d/holy-proxy.conf

proxy_cache_path /var/cache/nginx/holy_cache 
    levels=1:2 
    keys_zone=holy_cache:10m 
    max_size=1g 
    inactive=60m;

limit_req_zone $binary_remote_addr zone=holy_rate:10m rate=100r/s;

server {
    listen 443 ssl http2;
    server_name api-relay.holysheep.ai;
    
    ssl_certificate /etc/ssl/holy.pem;
    ssl_certificate_key /etc/ssl/holy.key;
    
    # Cấu hình two-stage deployment
    set $target_backend "blue_backend";
    
    # Khi GREEN_DEPLOY=true, chuyển 10% traffic sang green
    if ($cookie_deploy_mode = "green") {
        set $target_backend "green_backend";
    }
    
    # Weighted routing - 90% blue, 10% green (canary)
    split_clients $remote_addr$request_uri $deployment_target {
        10%     green_backend;
        *       blue_backend;
    }
    
    location / {
        # Rate limiting
        limit_req zone=holy_rate burst=20 nodelay;
        
        # Proxy đến backend tương ứng
        proxy_pass http://$deployment_target;
        
        # Cấu hình proxy
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Request-ID $request_id;
        
        # Connection pooling với HolySheep upstream
        proxy_set_header Connection "";
        
        # Timeout settings tối ưu cho HolySheep API
        proxy_connect_timeout 5s;
        proxy_send_timeout 30s;
        proxy_read_timeout 60s;
        
        # Retry configuration
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 3;
        proxy_next_upstream_timeout 10s;
        
        # Cache settings
        proxy_cache holy_cache;
        proxy_cache_valid 200 60s;
        proxy_cache_use_stale error timeout updating;
        add_header X-Cache-Status $upstream_cache_status;
    }
    
    # Deployment control endpoints
    location /deploy/status {
        return 200 '{"blue":"active","green":"$target_backend","traffic_split":"90/10"}\n';
        add_header Content-Type application/json;
    }
    
    location /deploy/switch {
        # Protected endpoint - chỉ cho phép từ CI/CD
        allow 10.0.0.0/8;
        deny all;
        
        if ($request_method = POST) {
            set $target_backend "green_backend";
            return 200 '{"status":"switched","target":"green","timestamp":"$nginx_version"}\n';
        }
        
        if ($request_method = DELETE) {
            set $target_backend "blue_backend";
            return 200 '{"status":"rollback","target":"blue","timestamp":"$nginx_version"}\n';
        }
    }
}

Deployment Script tự động hóa

Script Python dưới đây xử lý toàn bộ deployment pipeline — từ kiểm thử green environment đến chuyển traffic an toàn. Điểm đặc biệt: tích hợp smoke test với HolySheep API để xác nhận relay hoạt động đúng.

#!/usr/bin/env python3
"""
HolySheep API Relay - Blue-Green Deployment Controller
Author: HolySheep AI Engineering Team
"""

import os
import time
import json
import logging
import subprocess
import hashlib
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional, Callable
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s [%(levelname)s] %(message)s'
)
logger = logging.getLogger(__name__)

HolySheep API Configuration
HOLYSHEEP_BASE_URL = "https://api.holysheep.ai/v1"
HOLYSHEEP_API_KEY = os.getenv("HOLYSHEEP_API_KEY", "YOUR_HOLYSHEEP_API_KEY")

class DeploymentState(Enum):
    IDLE = "idle"
    DEPLOYING_GREEN = "deploying_green"
    TESTING = "testing"
    CANARY = "canary"
    FULL_SWITCH = "full_switch"
    ROLLBACK = "rollback"

@dataclass
class DeploymentConfig:
    """Cấu hình deployment cho HolySheep relay"""
    nginx_config_path: str = "/etc/nginx/conf.d/holy-proxy.conf"
    health_check_endpoint: str = "http://localhost:9090/health"
    green_port: int = 8080
    blue_port: int = 8080
    canary_percentage: int = 10
    smoke_test_timeout: int = 30
    rollback_threshold: float = 0.05  # 5% error rate = auto rollback
    holy_api_models: list = field(default_factory=lambda: [
        "gpt-4.1",
        "claude-sonnet-4-5",
        "gemini-2.5-flash",
        "deepseek-v3-2"
    ])

class HolySheepAPIClient:
    """Client tối ưu cho HolySheep API relay - sử dụng relay endpoint"""
    
    def __init__(self, api_key: str, relay_url: str):
        self.base_url = relay_url
        self.session = requests.Session()
        
        # Retry strategy: 3 retries, exponential backoff
        retry_strategy = Retry(
            total=3,
            backoff_factor=0.5,
            status_forcelist=[500, 502, 503, 504]
        )
        adapter = HTTPAdapter(max_retries=retry_strategy)
        self.session.mount("http://", adapter)
        self.session.mount("https://", adapter)
        
        self.session.headers.update({
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "X-Relay-Version": "blue-green-v1"
        })
        
        # Metrics tracking
        self.metrics = {"success": 0, "error": 0, "latencies": []}
    
    def test_connection(self, model: str = "gpt-4.1") -> dict:
        """Test kết nối đến HolySheep API qua relay"""
        start = time.time()
        try:
            response = self.session.post(
                f"{self.base_url}/chat/completions",
                json={
                    "model": model,
                    "messages": [{"role": "user", "content": "ping"}],
                    "max_tokens": 5
                },
                timeout=self.base_url.endswith("api.holysheep.ai") and 10 or 30
            )
            latency = (time.time() - start) * 1000
            
            self.metrics["latencies"].append(latency)
            
            if response.status_code == 200:
                self.metrics["success"] += 1
                return {
                    "status": "success",
                    "latency_ms": round(latency, 2),
                    "model": model
                }
            else:
                self.metrics["error"] += 1
                return {
                    "status": "error",
                    "code": response.status_code,
                    "model": model
                }
        except requests.exceptions.Timeout:
            self.metrics["error"] += 1
            return {"status": "timeout", "model": model}
        except Exception as e:
            self.metrics["error"] += 1
            return {"status": "exception", "error": str(e), "model": model}
    
    def get_error_rate(self) -> float:
        total = self.metrics["success"] + self.metrics["error"]
        return total > 0 and self.metrics["error"] / total or 0.0
    
    def get_avg_latency(self) -> float:
        return sum(self.metrics["latencies"]) / len(self.metrics["latencies"]) if self.metrics["latencies"] else 0

class BlueGreenDeployer:
    """Controller cho Blue-Green Deployment với HolySheep relay"""
    
    def __init__(self, config: DeploymentConfig):
        self.config = config
        self.state = DeploymentState.IDLE
        self.deployment_log = []
    
    def _log(self, message: str, level: str = "INFO"):
        logger.log(getattr(logging, level), message)
        self.deployment_log.append({
            "time": time.time(),
            "level": level,
            "message": message
        })
    
    def health_check(self) -> bool:
        """Kiểm tra health của cả hai environment"""
        for backend, port in [("Blue", self.config.blue_port), ("Green", self.config.green_port)]:
            try:
                response = requests.get(
                    f"http://localhost:{port}/health",
                    timeout=5
                )
                if response.status_code != 200:
                    self._log(f"{backend} health check failed", "ERROR")
                    return False
                self._log(f"{backend} environment healthy")
            except Exception as e:
                self._log(f"{backend} health check error: {e}", "ERROR")
                return False
        return True
    
    def deploy_to_green(self, image_tag: str) -> bool:
        """Deploy phiên bản mới lên Green environment"""
        self.state = DeploymentState.DEPLOYING_GREEN
        self._log(f"Starting deployment to Green: {image_tag}")
        
        try:
            # Pull new image
            subprocess.run(
                ["docker", "pull", f"holysheep/relay:{image_tag}"],
                check=True,
                capture_output=True
            )
            
            # Stop current green container
            subprocess.run(
                ["docker", "stop", "relay-green"],
                capture_output=True
            )
            subprocess.run(
                ["docker", "rm", "relay-green"],
                capture_output=True
            )
            
            # Start new green container
            subprocess.run([
                "docker", "run", "-d",
                "--name", "relay-green",
                "--network", "holy-net",
                "-p", "10.0.1.20:8080:8080",
                "-e", f"HOLYSHEEP_API_KEY={HOLYSHEEP_API_KEY}",
                "-e", "ENVIRONMENT=green",
                f"holysheep/relay:{image_tag}"
            ], check=True)
            
            self._log(f"Green environment deployed: {image_tag}")
            return True
            
        except subprocess.CalledProcessError as e:
            self._log(f"Deployment failed: {e.stderr}", "ERROR")
            return False
    
    def smoke_test_green(self, relay_url: str) -> dict:
        """Chạy smoke test trên Green environment với HolySheep API"""
        self.state = DeploymentState.TESTING
        self._log("Starting smoke tests on Green environment")
        
        client = HolySheepAPIClient(HOLYSHEEP_API_KEY, relay_url)
        results = {}
        
        # Test từng model
        for model in self.config.holy_api_models:
            self._log(f"Testing model: {model}")
            result = client.test_connection(model)
            results[model] = result
            
            if result["status"] != "success":
                self._log(f"Model {model} failed: {result}", "WARNING")
        
        # Tổng hợp metrics
        error_rate = client.get_error_rate()
        avg_latency = client.get_avg_latency()
        
        test_passed = error_rate < self.config.rollback_threshold
        summary = {
            "passed": test_passed,
            "error_rate": error_rate,
            "avg_latency_ms": round(avg_latency, 2),
            "details": results
        }
        
        self._log(f"Smoke test result: {'PASSED' if test_passed else 'FAILED'}")
        self._log(f"Error rate: {error_rate:.2%}, Avg latency: {avg_latency:.2f}ms")
        
        return summary
    
    def switch_to_green(self) -> bool:
        """Chuyển 100% traffic sang Green environment"""
        self.state = DeploymentState.FULL_SWITCH
        self._log("Switching traffic to Green environment")
        
        try:
            # Update Nginx config để point 100% traffic sang green
            nginx_conf = f"""
            split_clients $remote_addr$request_uri $deployment_target {{
                *       green_backend;
            }}
            """
            
            # Reload Nginx
            subprocess.run(
                ["nginx", "-s", "reload"],
                check=True
            )
            
            self._log("Traffic switched to Green (100%)")
            return True
            
        except subprocess.CalledProcessError as e:
            self._log(f"Switch failed: {e}", "ERROR")
            return False
    
    def rollback_to_blue(self) -> bool:
        """Rollback về Blue environment - thao tác tức thì"""
        self.state = DeploymentState.ROLLBACK
        self._log("Initiating rollback to Blue environment")
        
        try:
            # Immediate switch về blue
            subprocess.run(
                ["nginx", "-s", "reload"],
                check=True
            )
            
            self._log("Rollback complete - Blue environment now active")
            return True
            
        except subprocess.CalledProcessError as e:
            self._log(f"Rollback failed: {e}", "ERROR")
            return False
    
    def deploy(self, image_tag: str, auto_rollback: bool = True) -> dict:
        """Full deployment pipeline với auto-rollback"""
        start_time = time.time()
        pipeline_log = []
        
        # Step 1: Health check
        if not self.health_check():
            return {"success": False, "error": "Health check failed", "stage": "health_check"}
        
        # Step 2: Deploy to green
        if not self.deploy_to_green(image_tag):
            return {"success": False, "error": "Green deployment failed", "stage": "deploy_green"}
        
        # Wait for green to be ready
        time.sleep(5)
        
        # Step 3: Smoke test
        green_url = "http://10.0.1.20:8080"
        test_result = self.smoke_test_green(green_url)
        
        if not test_result["passed"] and auto_rollback:
            self._log("Smoke test failed - initiating auto-rollback", "ERROR")
            self.rollback_to_blue()
            return {
                "success": False,
                "error": "Smoke test failed",
                "test_result": test_result,
                "rollback": "completed"
            }
        
        # Step 4: Switch traffic
        if not self.switch_to_green():
            return {"success": False, "error": "Traffic switch failed", "stage": "switch"}
        
        # Step 5: Monitor for 5 minutes
        monitor_start = time.time()
        while time.time() - monitor_start < 300:
            client = HolySheepAPIClient(HOLYSHEEP_API_KEY, green_url)
            client.test_connection()
            
            if client.get_error_rate() > self.config.rollback_threshold:
                self._log("Error rate exceeded threshold during monitoring", "ERROR")
                if auto_rollback:
                    self.rollback_to_blue()
                    return {
                        "success": False,
                        "error": "Monitoring detected high error rate",
                        "rollback": "completed"
                    }
            
            time.sleep(10)
        
        duration = time.time() - start_time
        
        return {
            "success": True,
            "duration_seconds": round(duration, 2),
            "test_result": test_result,
            "log": self.deployment_log
        }

Sử dụng deployment controller
if __name__ == "__main__":
    config = DeploymentConfig()
    deployer = BlueGreenDeployer(config)
    
    # Deploy phiên bản mới với auto-rollback
    result = deployer.deploy(
        image_tag="v2.4.1",
        auto_rollback=True
    )
    
    if result["success"]:
        print(f"✅ Deployment thành công trong {result['duration_seconds']}s")
    else:
        print(f"❌ Deployment thất bại: {result['error']}")
        if result.get("rollback"):
            print("🔄 Đã tự động rollback về Blue environment")

CI/CD Integration với GitHub Actions

Để automation hoàn toàn, ta tích hợp deployment script vào GitHub Actions. Điều đặc biệt quan trọng: môi trường staging sử dụng https://api.holysheep.ai/v1 với API key riêng — đảm bảo tách biệt hoàn toàn với production.

# .github/workflows/blue-green-deploy.yml

name: Blue-Green Deployment

on:
  push:
    tags:
      - 'v*'

env:
  HOLYSHEEP_API_KEY: ${{ secrets.HOLYSHEEP_API_KEY }}
  HOLYSHEEP_BASE_URL: https://api.holysheep.ai/v1

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Extract version
        id: vars
        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT
      
      - name: Build and push Blue image
        run: |
          docker build -t holysheep/relay:blue-${{ steps.vars.outputs.VERSION }} .
          docker push holysheep/relay:blue-${{ steps.vars.outputs.VERSION }}
      
      - name: Build and push Green image
        run: |
          docker build -t holysheep/relay:green-${{ steps.vars.outputs.VERSION }} .
          docker push holysheep/relay:green-${{ steps.vars.outputs.VERSION }}
      
      - name: Deploy to Green environment
        run: |
          python3 deploy.py \
            --action deploy_green \
            --image-tag green-${{ steps.vars.outputs.VERSION }} \
            --target-host ${{ secrets.GREEN_SERVER_HOST }}
      
      - name: Smoke test Green environment
        run: |
          python3 deploy.py \
            --action smoke_test \
            --relay-url http://${{ secrets.GREEN_SERVER_HOST }}:8080
        env:
          HOLYSHEEP_API_KEY: ${{ secrets.HOLYSHEEP_STAGING_KEY }}
      
      - name: Approve deployment
        if: always()
        run: |
          echo "## Deployment Status" >> $GITHUB_STEP_SUMMARY
          echo "| Environment | Status |" >> $GITHUB_STEP_SUMMARY
          echo "|-------------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Green | Ready for switch |" >> $GITHUB_STEP_SUMMARY
      
      - name: Switch traffic to Green
        if: github.event_name == 'push'
        run: |
          python3 deploy.py \
            --action switch_traffic \
            --target green
        timeout-minutes: 5
      
      - name: Monitor deployment
        if: github.event_name == 'push'
        run: |
          python3 deploy.py \
            --action monitor \
            --duration 300
        timeout-minutes: 10
      
      - name: Rollback if failed
        if: failure() && github.event_name == 'push'
        run: |
          python3 deploy.py \
            --action rollback \
            --reason "Deployment verification failed"
        timeout-minutes: 2
      
      - name: Cleanup old images
        if: success()
        run: |
          docker image prune -f --filter "until=168h"
      
      - name: Notify deployment status
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "${{ github.event_name == 'push' && '🚀 Deployment Success' || '✅ Staging Test Passed' }}: v${{ steps.vars.outputs.VERSION }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Lỗi thường gặp và cách khắc phục

1. ConnectionError: timeout after 30000ms khi gọi HolySheep API

Nguyên nhân: Proxy timeout quá ngắn hoặc HolySheep API relay server quá tải.

# Khắc phục: Tăng timeout trong Nginx config
location / {
    proxy_connect_timeout 10s;   # Tăng từ 5s
    proxy_send_timeout 60s;       # Tăng từ 30s
    proxy_read_timeout 120s;      # Tăng cho models lớn
}

Giải pháp tối ưu: Sử dụng connection pooling và keepalive với HolySheep upstream:

# Thêm vào upstream block
upstream holy_api {
    server api.holysheep.ai:443;
    keepalive 64;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}

location / {
    # Sử dụng HTTP/1.1 và keepalive
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    
    # Retry policy
    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_next_upstream_tries 3;
    proxy_next_upstream_timeout 30s;
}

2. 401 Unauthorized khi sử dụng HolySheep API Key

Nguyên nhân: API key không hợp lệ hoặc chưa được truyền đúng cách qua relay.

# Kiểm tra và khắc phục:
1. Verify API key format - phải bắt đầu bằng "sk-"
import requests

response = requests.post(
    "https://api.holysheep.ai/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {HOLYSHEEP_API_KEY}",
        "Content-Type": "application/json"
    },
    json={
        "model": "gpt-4.1",
        "messages": [{"role": "user", "content": "test"}],
        "max_tokens": 10
    }
)

if response.status_code == 401:
    print("❌ API Key không hợp lệ")
    print(f"Response: {response.json()}")
    # Kiểm tra key tại: https://www.holysheep.ai/dashboard
elif response.status_code == 200:
    print("✅ API Key hợp lệ")
else:
    print(f"⚠️ Status: {response.status_code}")
    print(f"Response: {response.text}")

3. 502 Bad Gateway khi Green environment chưa ready

Nguyên nhân: Nginx chuyển traffic sang Green trước khi container fully started.

# Giải pháp: Health check chặt chẽ trước khi switch

#!/bin/bash
wait-for-green.sh

GREEN_HOST="10.0.1.20"
MAX_WAIT=60
INTERVAL=2

echo "Waiting for Green environment to be ready..."

for i in $(seq 1 $MAX_WAIT); do
    if curl -sf http://$GREEN_HOST:8080/health > /dev/null 2>&1; then
        echo "✅ Green environment ready after ${INTERVAL}s"
        exit 0
    fi
    
    # Test actual API call
    RESPONSE=$(curl -sf -o /dev/null -w "%{http_code}" \
        -X POST http://$GREEN_HOST:8080/v1/chat/completions \
        -H "Content-Type: application/json" \
        -d '{"model":"gpt-4.1","messages":[],"max_tokens":1}')
    
    if [ "$RESPONSE" = "200" ]; then
        echo "✅ Green API responsive after $((i * INTERVAL))s"
        exit 0
    fi
    
    echo "Attempt $i/$MAX_WAIT: waiting..."
    sleep $INTERVAL
done

echo "❌ Green environment did not become ready within ${MAX_WAIT}s"
exit 1

4. SSL Certificate Error khi relay qua HolySheep

# Khắc phục: Cập nhật CA certificates
apt-get update && apt-get install -y ca-certificates

Rebuild Docker image với certificates mới
docker build -t holysheep/relay:v2 -<

# Monitor và restart green container định kỳ
/etc/systemd/system/relay-green-reloader.service

[Unit]
Description=Relay Green Container Reloader
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/reload-relay-green.sh

[Install]
WantedBy=multi-user.target

Script reload với zero-downtime
#!/bin/bash
docker exec relay-green curl -X POST http://localhost:8080/shutdown
sleep 2
docker restart relay-green

# /etc/prometheus/holy-alerts.yml

groups:
  - name: holy_relay_alerts
    rules:
      # Alert khi error rate vượt ngưỡng
      - alert: HighErrorRateBlueGreen
        expr: |
          (
            rate(holy_relay_errors{backend="blue"}[5m]) /
            rate(holy_relay_requests_total{backend="blue"}[5m])
          ) > 0.05
        for: 2m
        labels:
          severity: critical
          deployment: blue-green
        annotations:
          summary: "Blue-Green deployment error rate cao"
          description: "Backend {{ $labels.backend }} có error rate {{ $value | humanizePercentage }}"
          runbook_url: "https://docs.holysheep.ai/runbooks/blue-green"
      
      # Alert khi latency tăng đột biến
      - alert: LatencySpikeOnGreen
        expr: |
          histogram_quantile(0.99, 
            rate(holy_relay_latency_bucket{backend="green"}[5m])
          ) > 2
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Green environment latency spike"
          description: "P99 latency trên Green: {{ $value }}s (threshold: 2s)"
      
      # Alert khi deployment stuck
      - alert: DeploymentStuck
        expr: holy_relay_deployment_in_progress == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Deployment đang trong trạng thái pending quá lâu"
      
      # Alert khi HolySheep API unreachable
      - alert: HolySheepAPIUnreachable
        expr: |
          absent(up{job="holy-api-relay"}) == 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Không thể kết nối đến HolySheep API"
          description: "Kiểm tra network connectivity và API key"

Recording rules cho dashboard
  - name: holy_relay_recording
    rules:
      - record: holy:relay:success_rate
        expr: |
          1 - (
            rate(holy_relay_errors[5m]) /
            rate(holy_relay_requests_total[5m])
          )
      
      - record: holy:relay:p99_latency
        expr: |
          histogram_quantile(0.99,
            rate(holy_relay_latency_bucket[5m])
          ) * 1000  # Convert to ms