Cost-Efficient AI Infrastructure Setup: From GPU Selection to Production

GPU Instance Selection: Matching Hardware to Workload

The single biggest cost mistake in AI infrastructure is defaulting to the most powerful GPU available. An NVIDIA H100 is extraordinary hardware, but using it for fine-tuning a 7B-parameter model is like renting a cargo ship to deliver a single package.

Understanding GPU Tiers

Cloud GPU instances fall into three tiers, each optimized for different workload profiles. Choosing the right tier is the foundation of cost-efficient AI infrastructure.

The Decision Framework

Use this hierarchy to select the right GPU for each workload stage:

1. Calculate VRAM requirement: Model parameters × bytes per parameter. A 7B model at FP16 needs ~14 GB; at FP32, ~28 GB. Add 20-30% overhead for optimizer states and activation memory during training.

2. Assess bandwidth needs: Distributed training across nodes requires high NVLink/InfiniBand bandwidth. Single-node fine-tuning does not. If you only need one GPU, bandwidth is irrelevant—choose the cheapest option that fits.

3. Match to cloud instance: AWS g5 (A10G) for mid-tier, p4d (A100 40GB) or p5 (H100) for high-tier. GCP a2-highgpu (A100 40GB), a3 (H100). Azure NC-series (A100), ND-series (H100).

4. Start one tier lower: Benchmark on the cheaper tier first. If training throughput is acceptable, you just saved 40-70% without any other optimization.

Multi-Cloud GPU Arbitrage

GPU pricing varies significantly across cloud providers. AWS, GCP, and Azure each have different supply-demand dynamics for GPU instances. Organizations running large-scale training can exploit these differences:

• AWS: Deepest Spot pool for A10G/g5 instances, best for training workloads tolerant of interruptions. Highest on-demand prices for H100.

• GCP: Preemptible VMs offer fixed 60-91% discounts with guaranteed 24-hour runtime. Better for shorter training jobs that complete within a day.

• Azure: Often has H100 spot availability when AWS and GCP are constrained. Committed Use Discounts for predictable baseline workloads.

• Specialized providers: Lambda Labs, CoreWeave, and RunPod offer bare-metal GPU rentals at 30-50% below hyperscaler on-demand pricing for long-running training.

Spot Instances for Training: 60-90% Savings with Checkpointing

Training workloads are inherently fault-tolerant: the computation is deterministic, progress can be saved, and interrupted jobs can resume. This makes them ideal candidates for Spot/Preemptible instances, where cloud providers offer unused capacity at steep discounts in exchange for the ability to reclaim it with short notice.

Building a Spot-First Training Pipeline

A robust Spot training pipeline requires three components working together:

1. Automatic checkpointing: Save model weights, optimizer state, and training progress to durable storage (S3/GCS) every 15-30 minutes. Modern frameworks like PyTorch Lightning and Hugging Face Transformers support this natively.

2. Interruption handling: Listen for the 2-minute Spot termination notice (AWS) or 30-second preemption signal (GCP). Trigger an immediate checkpoint save and graceful shutdown.

3. Automatic resume: When a replacement instance launches, detect the latest checkpoint and resume training from that point. Total wasted compute per interruption: 15-30 minutes maximum.

Spot Diversification Strategy

The key to reliable Spot availability is diversification across multiple instance types and availability zones. Instead of requesting a specific instance like p3.2xlarge, define your requirements abstractly:

• GPU requirement: 1x GPU with at least 16 GB VRAM

• Acceptable instance types: g5.xlarge, g5.2xlarge, g4dn.xlarge, g4dn.2xlarge, p3.2xlarge

• AZ preference: Any of us-east-1a, us-east-1b, us-east-1c, us-east-1d

With 5 instance types across 4 AZs, you have 20 Spot pools to draw from. The probability of all 20 being simultaneously interrupted is negligible.

Kubernetes GPU Scheduling with Karpenter

Running AI workloads on Kubernetes provides orchestration benefits—declarative scheduling, health checks, automatic restarts—but introduces GPU-specific challenges. Standard Kubernetes scheduling treats GPUs as opaque resources, leading to fragmentation and waste.

The GPU Scheduling Problem

Consider a cluster with three GPU node groups:

• Pool A: g5.xlarge (1× A10G, 24 GB VRAM) — for inference

• Pool B: g5.12xlarge (4× A10G, 96 GB total) — for fine-tuning

• Pool C: p4d.24xlarge (8× A100, 320 GB total) — for pre-training

The Cluster Autoscaler scales each pool independently. If a fine-tuning job requests 2 GPUs, Pool B adds a 4-GPU node—leaving 2 GPUs idle at $3/hr each. Over a month, this fragmentation wastes $4,300 per node.

Karpenter: Just-in-Time GPU Provisioning

Karpenter eliminates node group rigidity. Instead of predefined pools, you declare GPU requirements as constraints, and Karpenter selects the optimal instance type at scheduling time:

When a training pod requesting 2× A10G GPUs enters pending state, Karpenter evaluates available instance types in real time:

• g5.2xlarge (1× A10G): insufficient—needs 2 GPUs

• g5.8xlarge (2× A10G, Spot $2.40/hr): matches exactly

• g5.12xlarge (4× A10G, Spot $4.80/hr): over-provisioned

Karpenter selects the g5.8xlarge—the cheapest option that satisfies the requirements. No wasted GPUs, no manual node group management.

Bin-Packing and Consolidation for GPU Workloads

Karpenter's consolidation loop runs continuously. When inference pods scale down at night (traffic drops 80%), Karpenter migrates remaining pods to fewer nodes and terminates the empty ones. In the morning, as traffic ramps up, new nodes launch within 60-90 seconds.

For mixed workloads (training + inference on the same cluster), Karpenter co-locates small inference pods on partially utilized training nodes, filling GPU fragments that would otherwise sit idle.

Model Serving Optimization: Reducing Inference Costs

Training is a burst activity—expensive but finite. Serving (inference) runs 24/7, making it the dominant cost driver for production AI systems. A model serving $0.50/hr in GPU costs accumulates to $4,380/year per replica. Optimize serving, and you reduce your largest recurring expense.

Strategy 1: Model Quantization

Quantization reduces the numerical precision of model weights, trading minimal accuracy loss for dramatic resource savings:

Moving from FP32 to INT8 reduces serving cost from $4.10/hr to $0.50/hr—a 88% reduction. Over a year per replica, this saves $31,536. For a deployment with 4 replicas, annual savings exceed $125,000.

Strategy 2: Dynamic Batching

GPUs are massively parallel processors. Serving one request at a time wastes 90%+ of available compute. Dynamic batching groups incoming requests and processes them simultaneously:

• Without batching: 1 request per GPU cycle, ~50 requests/second on A10G

• With dynamic batching (batch=32): 32 requests per GPU cycle, ~800 requests/second on A10G

• Result: 16x throughput improvement. Serve the same traffic with 1 GPU instead of 16, or handle 16x growth at the same cost.

Serving frameworks like vLLM, TensorRT-LLM, and Triton Inference Server implement continuous batching, which dynamically adjusts batch size based on available requests and latency SLOs.

Strategy 3: Autoscaling Inference Replicas

Production inference traffic follows daily patterns—high during business hours, low at night. Scale GPU replicas to match:

Combined with Karpenter, this creates a fully elastic GPU infrastructure: KEDA scales inference pods based on queue depth, Karpenter provisions GPU nodes for those pods, and both scale back down during off-peak—paying only for active inference.

Storage Optimization for AI Workloads

Storage is the hidden cost multiplier in AI infrastructure. A single training run generates GBs of checkpoints. Datasets accumulate over time. Model artifacts persist across versions. Without lifecycle management, storage costs grow monotonically even as you optimize compute.

The Tiered Storage Strategy

Automated Lifecycle Policies

Manual storage management doesn't scale. Implement automated policies:

• Checkpoints: Keep last 3 checkpoints per training run on hot storage. Promote best checkpoint to warm tier on training completion. Archive all others after 30 days.

• Datasets: Active training datasets on hot tier. Move to warm after 14 days of no access. Transition to cold after 90 days.

• Model artifacts: Production model on warm tier (fast retrieval for serving). Previous versions to cold tier. Archive after deprecation.

• Logs and metrics: 7 days on hot, 30 days on warm, archive after 90 days.

Case Study: Israeli Computer Vision Startup

A Tel Aviv-based Series B startup building real-time object detection models for autonomous logistics. Their AI infrastructure costs had grown to $89,000/month—threatening runway—while their models still needed 2x more training compute to reach production accuracy targets.

Before Optimization

• 8× p4d.24xlarge instances running 24/7 on-demand ($62,800/month compute)

• 30 TB on gp3 EBS volumes ($7,200/month storage)

• Static Cluster Autoscaler with rigid node groups

• All models served at FP32 precision

• Average GPU utilization: 28%

Optimization Steps

1. GPU right-sizing: Discovered that 60% of fine-tuning jobs fit on A10G instances. Migrated those workloads from p4d to g5, cutting per-job costs by 65%.

2. Spot for training: Implemented checkpointing every 20 minutes. Moved all training to Spot with on-demand fallback. Achieved 72% average Spot usage.

3. Karpenter deployment: Replaced Cluster Autoscaler. GPU utilization jumped from 28% to 74%. Night-time consolidation reduced active nodes by 60% during off-peak.

4. Model quantization: Production inference models quantized to INT8 using GPTQ. Serving moved from A100 to T4 instances with 1.2% accuracy drop (within acceptable threshold).

5. Storage tiering: Implemented S3 Intelligent-Tiering for checkpoints and lifecycle policies for datasets. Reduced storage footprint from 30 TB EBS to 3 TB hot + 27 TB tiered.

After Optimization

The savings extended their runway by 8 months and enabled them to double their training compute budget while spending less. The additional experiments accelerated their model accuracy past the production threshold 3 months ahead of schedule.

Implementation Roadmap: 30-Day Plan

Implement these optimizations incrementally to minimize risk and demonstrate value early:

• Week 1 — Visibility: Tag all GPU instances by workload type (training/inference/dev). Instrument GPU utilization metrics with DCGM Exporter and Prometheus. Identify underutilized instances.

• Week 2 — Quick wins: Right-size GPU instances based on actual VRAM usage. Quantize inference models to FP16/INT8. Implement dynamic batching in serving layer.

• Week 3 — Spot migration: Add checkpointing to training pipelines. Start Spot instances for dev/test workloads. Gradually promote to production training with on-demand fallback.

• Week 4 — Karpenter + Storage: Deploy Karpenter alongside existing autoscaler. Configure GPU NodePools with Spot priority. Implement S3 lifecycle policies for checkpoints and datasets.