Deploying AI models for real-time inference at scale is one of the most challenging problems in modern infrastructure engineering. When I first encountered GPU scheduling in Kubernetes three years ago, I spent three weeks debugging why my PyTorch models kept getting OOM killed despite having plenty of GPU memory available. This guide walks you through everything I wish someone had explained to me on day one—no prior Kubernetes or cloud infrastructure experience required.

Understanding GPU Scheduling: Why It Matters for AI Inference

Before we write any code, let's understand what GPU scheduling actually does. When you deploy an AI model for inference (making predictions), your model needs access to a Graphics Processing Unit (GPU) to compute results quickly. Kubernetes, the container orchestration system, manages which pods (your application instances) get access to which physical GPUs on your cluster.

The challenge? Kubernetes doesn't automatically know that your inference service needs a GPU. You must explicitly tell it. Without proper configuration, your pods will sit in "Pending" state forever, waiting for a resource that Kubernetes doesn't know how to allocate.

Real-world context: A misconfigured GPU scheduler can add 200-500ms of latency to every inference request. At scale, this means your users experience sluggish responses even when your model is fast. Proper scheduling eliminates this bottleneck entirely.

Prerequisites: What You Need Before Starting

Step 1: Installing the NVIDIA Device Plugin for Kubernetes

Kubernetes cannot interact with GPUs without the NVIDIA Device Plugin. This component runs as a DaemonSet on every node and tells Kubernetes's scheduler which nodes have GPUs and how much GPU memory each has available.

# Add the NVIDIA Helm repository
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update


Install the NVIDIA Device Plugin


This enables Kubernetes to see and schedule GPU resources

helm install nvdp/nvidia-device-plugin \
    --namespace nvidia-device-plugins \
    --create-namespace \
    --generate-name


Verify the installation

kubectl get pods -n nvidia-device-plugins


Expected output shows the device plugin daemon running


NAME                                  READY   STATUS    RESTARTS   AGE


nvidia-device-plugin-daemonset-xxxxx  1/1     Running   0          30s

What just happened: We installed the NVIDIA Device Plugin via Helm, Kubernetes's package manager. The plugin automatically detects all GPUs on your nodes and advertises them to the Kubernetes scheduler as "nvidia.com/gpu" resources. Without this, Kubernetes treats GPU nodes identically to CPU-only nodes.

Step 2: Verifying GPU Availability

Before deploying your inference service, confirm that Kubernetes can see your GPUs. This step saves hours of debugging later.

# Check if any nodes have GPU resources available
kubectl describe nodes | grep -A 10 "Allocated resources"


More specific check for nvidia.com/gpu

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.capacity.nvidia.com/gpu"


If GPU is available, you'll see:


NAME                    GPU


gpu-node-1              1


gpu-node-2              2



Test GPU scheduling with a simple pod

cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-test
      image: nvidia/cuda:11.8.0-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
EOF


Check if pod scheduled successfully

kubectl get pods gpu-test


View the nvidia-smi output to confirm GPU access

kubectl logs gpu-test

Important: The resource limit nvidia.com/gpu: 1 is mandatory. Omitting this limit means Kubernetes will schedule your pod on a CPU-only node, and your CUDA code will fail with "CUDA not available" errors.

Step 3: Building Your Inference Service Container

Now we build a production-ready inference service. I'll use a simple Flask API that wraps a model, but the principles apply to any framework (FastAPI, TensorFlow Serving, Triton).

# Dockerfile for your inference service
FROM nvidia/cuda:11.8.0-base-ubuntu22.04

WORKDIR /app


Install Python and dependencies

RUN apt-get update && apt-get install -y \
    python3.10 \
    python3-pip \
    && rm -rf /var/lib/apt/lists/*


Install ML framework (example with PyTorch)

RUN pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu118


Install Flask for API

RUN pip3 install flask flask-cors


Copy application files

COPY inference_service.py .
COPY requirements.txt .


Expose port for HTTP API

EXPOSE 5000


Run the service

CMD ["python3", "inference_service.py"]
# inference_service.py - Simple inference API
from flask import Flask, request, jsonify
import torch
import time

app = Flask(__name__)


Load model on startup (once per container)


Using CPU for demo; in production, use .cuda()

model = None

def load_model():
    global model
    # Load your trained model here
    # model = torch.load('model.pth').eval()
    print("Model loaded successfully on", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU")
    return True

@app.before_request
def initialize():
    global model
    if model is None:
        load_model()

@app.route('/predict', methods=['POST'])
def predict():
    start_time = time.time()
    
    data = request.json
    input_data = data.get('input')
    
    # Run inference
    # In production: tensor = torch.tensor(input_data).cuda()
    # For demo, we simulate processing
    result = {"prediction": "sample_output", "model": "demo-v1"}
    
    latency_ms = (time.time() - start_time) * 1000
    
    return jsonify({
        "success": True,
        "result": result,
        "latency_ms": round(latency_ms, 2)
    })

@app.route('/health', methods=['GET'])
def health():
    return jsonify({
        "status": "healthy",
        "gpu_available": torch.cuda.is_available(),
        "gpu_device": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None
    })

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Step 4: Deploying to Kubernetes with GPU Resources

Create a Kubernetes Deployment that explicitly requests GPU resources. This is the configuration that makes or breaks your inference service.

# gpu-inference-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-inference-service
  labels:
    app: ai-inference
    version: v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ai-inference
  template:
    metadata:
      labels:
        app: ai-inference
        version: v1
    spec:
      containers:
      - name: inference-container
        image: your-registry/ai-inference:v1
        ports:
        - containerPort: 5000
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "4Gi"
            cpu: "2"
          requests:
            memory: "2Gi"
            cpu: "1"
        env:
        - name: MODEL_PATH
          value: "/models/latest"
        - name: HOLYSHEEP_API_KEY
          valueFrom:
            secretKeyRef:
              name: holysheep-api-key
              key: api-key
        readinessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 30
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: ai-inference-service
spec:
  selector:
    app: ai-inference
  ports:
  - port: 80
    targetPort: 5000
  type: LoadBalancer
# Apply the deployment
kubectl apply -f gpu-inference-deployment.yaml


Watch the deployment progress

kubectl rollout status deployment/ai-inference-service


Check pod status (look for "Running" with GPU allocated)

kubectl get pods -l app=ai-inference -o wide


Verify GPU allocation

kubectl describe pods -l app=ai-inference | grep -A 5 "Allocated resources"


Test the service (get external IP first)

kubectl get service ai-inference-service


Expected response from health endpoint

curl http://<EXTERNAL-IP>/health

Step 5: Integrating HolySheep AI for Cost-Effective AI Inference

While running your own GPU cluster gives you control, many production scenarios benefit from managed inference APIs. HolySheep AI offers GPU-accelerated inference at dramatically lower costs: their rate of ¥1=$1 saves you 85%+ compared to ¥7.3 pricing from competitors. With support for WeChat and Alipay payments, sub-50ms latency, and free credits on signup, it's an excellent complement to self-managed deployments.

# Example: Integrating HolySheep AI API for model inference

This replaces expensive local inference for certain workloads


import requests
import json

def call_holysheep_inference(prompt, model="deepseek-v3.2"):
    """
    Call HolySheep AI API for inference.
    Pricing as of 2026:
    - GPT-4.1: $8.00/1M tokens
    - Claude Sonnet 4.5: $15.00/1M tokens
    - Gemini 2.5 Flash: $2.50/1M tokens
    - DeepSeek V3.2: $0.42/1M tokens (best value)
    """
    base_url = "https://api.holysheep.ai/v1"
    
    headers = {
        "Authorization": f"Bearer {YOUR_HOLYSHEEP_API_KEY}",
        "Content-Type": "application/json"
    }
    
    payload = {
        "model": model,
        "messages": [
            {"role": "user", "content": prompt}
        ],
        "temperature": 0.7,
        "max_tokens": 1000
    }
    
    try:
        response = requests.post(
            f"{base_url}/chat/completions",
            headers=headers,
            json=payload,
            timeout=30
        )
        response.raise_for_status()
        
        result = response.json()
        
        return {
            "success": True,
            "content": result["choices"][0]["message"]["content"],
            "model": result["model"],
            "usage": result.get("usage", {}),
            "latency_ms": response.elapsed.total_seconds() * 1000
        }
        
    except requests.exceptions.RequestException as e:
        return {
            "success": False,
            "error": str(e),
            "suggestion": "Check API key and network connectivity"
        }


Usage example

result = call_holysheep_inference(
    prompt="Explain Kubernetes GPU scheduling in simple terms",
    model="deepseek-v3.2"
)

if result["success"]:
    print(f"Response: {result['content']}")
    print(f"Latency: {result['latency_ms']:.2f}ms")
    print(f"Model used: {result['model']}")
else:
    print(f"Error: {result['error']}")

Optimizing GPU Scheduling for Production

Basic scheduling works, but production systems need advanced strategies. Here are three optimizations I implemented after my first deployment failed under load:

GPU Bin-Packing with nodeSelector

By default, Kubernetes spreads pods across nodes. For GPU workloads, this can cause fragmentation where no single node has enough free GPUs for larger models.

# deployment-gpu-pinned.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: large-model-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: large-model
  template:
    metadata:
      labels:
        app: large-model
    spec:
      # Request 2 GPUs for larger models
      containers:
      - name: inference
        image: your-registry/large-model:v1
        resources:
          limits:
            nvidia.com/gpu: "2"  # Request 2 GPUs
            memory: "16Gi"
            cpu: "4"
        # Pin to specific node type (A100 preferred)
        nodeSelector:
          gpu-type: nvidia-tesla-a100
        tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"

Horizontal Pod Autoscaling with GPU Metrics

Standard HPA uses CPU/memory, but GPU workloads need custom metrics. Install the NVIDIA DCGM exporter for GPU-aware scaling.

# Install DCGM exporter for GPU metrics
helm install dcgm-exporter nvidia/dcgm-exporter \
    --namespace monitoring \
    --create-namespace


GPU-aware HPA configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-inference-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: "dcgm-exporter_gpu_utilization"
        selector:
          matchLabels:
            gpu: "0"
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: "nvidia.com/gpu"
      target:
        type: Utilization
        averageUtilization: 80

Monitoring GPU Utilization

You cannot optimize what you cannot measure. Set up comprehensive GPU monitoring to identify bottlenecks.

# Get detailed GPU info for all pods
kubectl get pods --all-namespaces -o json | \
  jq '.items[] | {name: .metadata.name, gpu: .spec.containers[].resources.limits.nvidia.com/gpu}'


Real-time GPU usage on nodes

kubectl debug node/gpu-node-1 -it --image=nvidia/cuda:11.8.0-base \
  -- nvidia-smi


View GPU metrics in Prometheus (if metrics-server installed)

kubectl port-forward -n monitoring prometheus-0 9090:9090


Then query: sum(rate(container_gpu_utilization_seconds_total[5m])) by (pod)

Common Errors and Fixes

Error 1: " pods have Unschedulable status - insufficient nvidia.com/gpu"

Symptom: Pods stuck in Pending state with event "insufficient nvidia.com/gpu".

Root Cause: Either no GPUs available in the cluster, or the NVIDIA Device Plugin is not running.

# Diagnose and fix
kubectl get nodes -o wide


Check if device plugin is running

kubectl get pods -n nvidia-device-plugins


If not running, reinstall

kubectl delete -n nvidia-device-plugins daemonset nvdp-nvidia-device-plugin
helm repo update
helm install nvdp/nvidia-device-plugin -n nvidia-device-plugins


If node doesn't have GPU labels, add them

kubectl label node <node-name> nvidia.com/gpu=true


Verify node has GPU capacity

kubectl get node <node-name> -o json | jq '.status.capacity'

Error 2: "CUDA out of memory" despite GPU having enough memory

Symptom: Container logs show "CUDA out of memory" but nvidia-smi shows available memory.

Root Cause: Multiple pods sharing GPU without proper memory limits, or model not releasing memory between inference calls.

# Check actual GPU memory allocation
kubectl exec -it <pod-name> -- nvidia-smi


Fix by limiting GPU memory in container


Add to container spec:

resources:
  limits:
    nvidia.com/gpu: 1
    memory: "8Gi"


If using PyTorch, set memory growth limit in code

import torch
torch.cuda.set_per_process_memory_fraction(0.8)  # Use max 80% of GPU


Force memory cleanup between requests

import gc
gc.collect()
torch.cuda.empty_cache()

Error 3: "Failed to initialize CUDA" in container logs

Symptom: Application starts but CUDA operations fail with initialization errors.

Root Cause: Container image not built with CUDA support, or NVIDIA libraries not accessible.

# Verify container has CUDA toolkit
kubectl exec -it <pod-name> -- nvcc --version


If nvcc missing, rebuild container with CUDA base image

FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04


Verify LD_LIBRARY_PATH includes CUDA libs

kubectl exec -it <pod-name> -- sh -c 'echo $LD_LIBRARY_PATH'


Should include: /usr/local/cuda/lib64



If missing, add to deployment

env:
- name: LD_LIBRARY_PATH
  value: /usr/local/cuda/lib64:$(LD_LIBRARY_PATH)

Error 4: API 401 Unauthorized from HolySheep AI

Symptom: Python client receives 401 status code when calling HolySheep API.

Root Cause: Invalid or missing API key in request headers.

# Correct authentication format
import os

HOLYSHEEP_API_KEY = os.environ.get("HOLYSHEEP_API_KEY", "YOUR_HOLYSHEEP_API_KEY")

headers = {
    "Authorization": f"Bearer {HOLYSHEEP_API_KEY}",
    "Content-Type": "application/json"
}


Verify key format: should start with "hs-" prefix

if not HOLYSHEEP_API_KEY.startswith("hs-"):
    print("Warning: API key should start with 'hs-'")


Get your key from: https://www.holysheep.ai/register


Free credits ($5 value) available on signup



Test connection

import requests
response = requests.get(
    "https://api.holysheep.ai/v1/models",
    headers={"Authorization": f"Bearer {HOLYSHEEP_API_KEY}"}
)
print(f"Status: {response.status_code}")
print(f"Available models: {response.json()}")

Performance Benchmarks and Real-World Numbers

Based on my testing across multiple deployments, here are realistic performance expectations:

Cost comparison: A production inference workload processing 10M tokens/day costs approximately $4.20 with HolySheep AI (DeepSeek V3.2) versus $73 with Claude Sonnet 4.5. The savings compound significantly at scale.

Conclusion

GPU scheduling in Kubernetes is initially intimidating, but becomes straightforward once you understand the core concepts: install the device plugin, request GPU resources explicitly, and monitor utilization. For workloads that don't require real-time local inference, managed services like HolySheep AI eliminate operational complexity while delivering sub-50ms latency at fraction of traditional costs.

The hybrid approach—self-managed Kubernetes for latency-critical paths, HolySheep API for bulk processing—gives you the best of both worlds. Start with the basics, measure everything, and iterate based on real traffic patterns.

👉 Sign up for HolySheep AI — free credits on registration

Related Resources

Related Articles